今日头条爬虫
- 1.代码基于python的scrapy爬虫框架。爬取url保存在Redis,爬取数据主要保存在MongoDB
- 2.依赖模块有pymongo，scrapy-redis，scrapy，redis，通过以下命令安装：
$ pip install pymongo scrapy scrapy-redis redis
- 使用方法:下载项目以后，进入项目根文件夹，运行：
$ scrapy crawl toutiao
Harhao / toutiao
Licence: other
今日头条科技新闻接口爬虫
Stars: ✭ 17
Programming Languages
python
139335 projects - #7 most used programming language
Projects that are alternatives of or similar to toutiao
elves
🎊 Design and implement of lightweight crawler framework.
Stars: ✭ 322 (+1794.12%)
Mutual labels: spider, scrapy
NScrapy
NScrapy is a .net core corss platform Distributed Spider Framework which provide an easy way to write your own Spider
Stars: ✭ 88 (+417.65%)
Mutual labels: spider, scrapy
devsearch
A web search engine built with Python which uses TF-IDF and PageRank to sort search results.
Stars: ✭ 52 (+205.88%)
Mutual labels: spider, scrapy
OpenScraper
An open source webapp for scraping: towards a public service for webscraping
Stars: ✭ 80 (+370.59%)
Mutual labels: spider, scrapy
python-spider
python爬虫小项目【持续更新】【笔趣阁小说下载、Tweet数据抓取、天气查询、网易云音乐逆向、天天基金网查询、微博数据抓取（生成cookie）、有道翻译逆向、企查查免登陆爬虫、大众点评svg加密破解、B站用户爬虫、拉钩免登录爬虫、自如租房字体加密、知乎问答
Stars: ✭ 45 (+164.71%)
Mutual labels: spider, scrapy
scrapy facebooker
Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.
Stars: ✭ 22 (+29.41%)
Mutual labels: spider, scrapy
Web-Iota
Iota is a web scraper which can find all of the images and links/suburls on a webpage
Stars: ✭ 60 (+252.94%)
Mutual labels: spider, scrapy
scrapy helper
Dynamic configurable crawl (动态可配置化爬虫)
Stars: ✭ 84 (+394.12%)
Mutual labels: spider, scrapy
photo-spider-scrapy
10 photo website spiders, 10 个国外图库的 scrapy 爬虫代码
Stars: ✭ 17 (+0%)
Mutual labels: spider, scrapy
Gerapy
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
Stars: ✭ 2,601 (+15200%)
Mutual labels: spider, scrapy
Spiderkeeper
admin ui for scrapy/open source scrapinghub
Stars: ✭ 2,562 (+14970.59%)
Mutual labels: spider, scrapy
scrapy-distributed
A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.
Stars: ✭ 38 (+123.53%)
Mutual labels: spider, scrapy
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].