追书神器（一）—Scrapy入门
- 运行脚本可以抓取指定的url上的最新小说
- 过滤掉无用标签和信息
- 存成本地HTML文件
- 按照链接抓取
- 按章节命名
todo
二期
- html文件模板 -(jinjia2)
- 调动scrapy框架
- 从指定站点抓取任意的小说
- 每天定时抓取
三期
- 多站点抓取
- 阿里云部署
- 微信公众平台 or 小程序
YanyeZhang / NovelCrawler
Licence: other
基于Scrapy的爬虫demo
Stars: ✭ 15
Labels
Projects that are alternatives of or similar to NovelCrawler
elves
🎊 Design and implement of lightweight crawler framework.
Stars: ✭ 322 (+2046.67%)
Mutual labels: scrapy
IMDB-Scraper
Scrapy project for scraping data from IMDB with Movie Dataset including 58,623 movies' data.
Stars: ✭ 37 (+146.67%)
Mutual labels: scrapy
Data-Engineering-Projects
Personal Data Engineering Projects
Stars: ✭ 167 (+1013.33%)
Mutual labels: scrapy
OLX Scraper
📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.
Stars: ✭ 15 (+0%)
Mutual labels: scrapy
OpenScraper
An open source webapp for scraping: towards a public service for webscraping
Stars: ✭ 80 (+433.33%)
Mutual labels: scrapy
proxi
Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.
Stars: ✭ 32 (+113.33%)
Mutual labels: scrapy
scrapy.dart
Scrapy, a fast high-level web crawling & scraping framework for dart and Flutter
Stars: ✭ 50 (+233.33%)
Mutual labels: scrapy
photo-spider-scrapy
10 photo website spiders, 10 个国外图库的 scrapy 爬虫代码
Stars: ✭ 17 (+13.33%)
Mutual labels: scrapy
logparser
A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.
Stars: ✭ 70 (+366.67%)
Mutual labels: scrapy
python-spider
python爬虫小项目【持续更新】【笔趣阁小说下载、Tweet数据抓取、天气查询、网易云音乐逆向、天天基金网查询、微博数据抓取（生成cookie）、有道翻译逆向、企查查免登陆爬虫、大众点评svg加密破解、B站用户爬虫、拉钩免登录爬虫、自如租房字体加密、知乎问答
Stars: ✭ 45 (+200%)
Mutual labels: scrapy
scrapy-cookies
A middleware of cookies persistence for Scrapy
Stars: ✭ 19 (+26.67%)
Mutual labels: scrapy
scrapy-distributed
A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.
Stars: ✭ 38 (+153.33%)
Mutual labels: scrapy
scraping-ebay
Scraping Ebay's products using Scrapy Web Crawling Framework
Stars: ✭ 79 (+426.67%)
Mutual labels: scrapy
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].