Just Downlink
基于 scrapy + elasticsearch + django 搭建的分布式电影搜索
- 利用 scrapy 爬取知名电影网站的下载链接
- 利用 elasticsearch 存储数据
- 利用 django 搭建电影搜索界面
数据采集
- 支持同步、异步数据存储至 Mysql 数据库
- 支持 elasticsearch 数据存储
- 支持 json 数据导出
- 保存日志至 logs 目录
搜索界面
- 搜索建议
- 简化电影下载页面
GFigure / JustDownlink
Licence: other
基于Scrapy+Elasticsearch+Django搭建的分布式电影搜索
Stars: ✭ 28
Programming Languages
javascript
184084 projects - #8 most used programming language
python
139335 projects - #7 most used programming language
CSS
56736 projects
HTML
75241 projects
shell
77523 projects
Projects that are alternatives of or similar to JustDownlink
scrapy-cookies
A middleware of cookies persistence for Scrapy
Stars: ✭ 19 (-32.14%)
Mutual labels: scrapy
OpenScraper
An open source webapp for scraping: towards a public service for webscraping
Stars: ✭ 80 (+185.71%)
Mutual labels: scrapy
proxi
Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.
Stars: ✭ 32 (+14.29%)
Mutual labels: scrapy
photo-spider-scrapy
10 photo website spiders, 10 个国外图库的 scrapy 爬虫代码
Stars: ✭ 17 (-39.29%)
Mutual labels: scrapy
scrapy-distributed
A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.
Stars: ✭ 38 (+35.71%)
Mutual labels: scrapy
scraping-ebay
Scraping Ebay's products using Scrapy Web Crawling Framework
Stars: ✭ 79 (+182.14%)
Mutual labels: scrapy
OLX Scraper
📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.
Stars: ✭ 15 (-46.43%)
Mutual labels: scrapy
elves
🎊 Design and implement of lightweight crawler framework.
Stars: ✭ 322 (+1050%)
Mutual labels: scrapy
Data-Engineering-Projects
Personal Data Engineering Projects
Stars: ✭ 167 (+496.43%)
Mutual labels: scrapy
IMDB-Scraper
Scrapy project for scraping data from IMDB with Movie Dataset including 58,623 movies' data.
Stars: ✭ 37 (+32.14%)
Mutual labels: scrapy
python-spider
python爬虫小项目【持续更新】【笔趣阁小说下载、Tweet数据抓取、天气查询、网易云音乐逆向、天天基金网查询、微博数据抓取（生成cookie）、有道翻译逆向、企查查免登陆爬虫、大众点评svg加密破解、B站用户爬虫、拉钩免登录爬虫、自如租房字体加密、知乎问答
Stars: ✭ 45 (+60.71%)
Mutual labels: scrapy
logparser
A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.
Stars: ✭ 70 (+150%)
Mutual labels: scrapy
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].