DoubanPyspider
使用Pyspider框架的豆瓣爬虫
依赖:
python第三方库：pyspider, pymongo, requests
数据库：mongodb
代理池：IPProxy
Usage
安装mongodb， 使用pip安装pyspider和pymongo，在pyspider新建一个爬虫项目，将movie_spider.py文件的内容拷贝到项目中运行即可。
xrlin / DoubanPyspider
Licence: other
使用Pyspider框架的豆瓣爬虫
Stars: ✭ 26
Programming Languages
python
139335 projects - #7 most used programming language
Projects that are alternatives of or similar to DoubanPyspider
robots.txt
🤖 robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API
Stars: ✭ 13 (-50%)
Mutual labels: spiders
Free proxy pool
对免费代理IP网站进行爬取，收集汇总为自己的代理池。关键是验证代理的有效性、匿名性、去重复
Stars: ✭ 66 (+153.85%)
Mutual labels: spiders
BaiduSpider
项目已经移动至：https://github.com/BaiduSpider/BaiduSpider ！！ 一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。
Stars: ✭ 29 (+11.54%)
Mutual labels: spiders
Awesome Python Login Model
模拟登陆基本采用的是直接登录或者使用selenium+webdriver的方式，有的网站直接登录难度很大，比如qq空间，bilibili等如果采用selenium就相对轻松一些。
Stars: ✭ 13,953 (+53565.38%)
Mutual labels: spiders
Abot
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Stars: ✭ 1,961 (+7442.31%)
Mutual labels: spiders
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].