Haipproxy💖 High available distributed ip proxy pool, powerd by Scrapy and Redis
Stars: ✭ 4,993 (+693.8%)
Crawlab LiteLite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
Stars: ✭ 122 (-80.6%)
Marmot💐Marmot | Web Crawler/HTTP protocol Download Package 🐭
Stars: ✭ 186 (-70.43%)
CrawlabDistributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
Stars: ✭ 8,392 (+1234.18%)
FbcrawlA Facebook crawler
Stars: ✭ 536 (-14.79%)
Python3 SpiderPython爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️
Stars: ✭ 2,129 (+238.47%)
Goribot[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。
Stars: ✭ 190 (-69.79%)
NewcrawlerFree Web Scraping Tool with Java
Stars: ✭ 589 (-6.36%)
toutiao今日头条科技新闻接口爬虫
Stars: ✭ 17 (-97.3%)
Wechatsogou基于搜狗微信搜索的微信公众号爬虫接口
Stars: ✭ 5,220 (+729.89%)
arachnodHigh performance crawler for Nodejs
Stars: ✭ 17 (-97.3%)
ScrappleA framework for creating semi-automatic web content extractors
Stars: ✭ 464 (-26.23%)
Happy Spiders🔧 🔩 🔨 收集整理了爬虫相关的工具、模拟登陆技术、代理IP、scrapy模板代码等内容。
Stars: ✭ 261 (-58.51%)
CrawlyCrawly, a high-level web crawling & scraping framework for Elixir.
Stars: ✭ 440 (-30.05%)
PttImageSpiderPTT 圖片下載器 (抓取整個看板的圖片，並用文章標題作為資料夾的名稱 ) (使用Scrapy)
Stars: ✭ 16 (-97.46%)
Bt Btt磁力網站U3C3介紹以及域名更新
Stars: ✭ 261 (-58.51%)
NetdiscoveryNetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。
Stars: ✭ 573 (-8.9%)
Scrapy RedisRedis-based components for Scrapy.
Stars: ✭ 4,998 (+694.59%)
XsrfprobeThe Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
Stars: ✭ 532 (-15.42%)
Gospidergolang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。
Stars: ✭ 285 (-54.69%)
ToapiEvery web site provides APIs.
Stars: ✭ 3,209 (+410.17%)
flink-crawlerContinuous scalable web crawler built on top of Flink and crawler-commons
Stars: ✭ 48 (-92.37%)
crawlerA simple and flexible web crawler framework for java.
Stars: ✭ 20 (-96.82%)
scrapy-adminA django admin site for scrapy
Stars: ✭ 44 (-93%)
WebCrawler一个轻量级、快速、多线程、多管道、灵活配置的网络爬虫。
Stars: ✭ 39 (-93.8%)
slime🍰 一个可视化的爬虫平台
Stars: ✭ 27 (-95.71%)
scrapy facebookerCollection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.
Stars: ✭ 22 (-96.5%)
galerA fast tool to fetch URLs from HTML attributes by crawl-in.
Stars: ✭ 138 (-78.06%)
Douban CrawlerUno Crawler por https://douban.com
Stars: ✭ 13 (-97.93%)
DouyinAPI of DouYin for Humans used to Crawl Popular Videos and Musics
Stars: ✭ 580 (-7.79%)
ip proxy poolGenerating spiders dynamically to crawl and check those free proxy ip on the internet with scrapy.
Stars: ✭ 39 (-93.8%)
Go jobs带你了解一下Golang的市场行情
Stars: ✭ 526 (-16.38%)
Python Spider豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章
Stars: ✭ 615 (-2.23%)
Crawlertutorial爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例
Stars: ✭ 282 (-55.17%)
Weixin Spider微信公众号爬虫，公众号历史文章，文章评论，文章阅读及在看数据，可视化web页面，可部署于Windows服务器。基于Python3之flask/mysql/redis/mitmproxy/pywin32等实现，高效微信爬虫，微信公众号爬虫，历史文章，文章评论，数据更新。
Stars: ✭ 287 (-54.37%)
Hacker News Digest📰 A responsive interface of Hacker News with summaries and thumbnails.
Stars: ✭ 278 (-55.8%)
91porn Api🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新） 及 在线web预览
Stars: ✭ 341 (-45.79%)
Zhihu Login知乎模拟登录，支持提取验证码和保存 Cookies
Stars: ✭ 340 (-45.95%)
AlltheplacesA set of spiders and scrapers to extract location information from places that post their location on the internet.
Stars: ✭ 277 (-55.96%)
Elves🎊 Design and implement of lightweight crawler framework.
Stars: ✭ 315 (-49.92%)
Ttbot今日头条机器人，支持用户登陆、关注、取消关注、获取关注粉丝、发文、发悟空问答、点赞、评论、采集各种类型新闻讯息等，使用今日头条网页版API实现
Stars: ✭ 338 (-46.26%)
Xcrawler快速、简洁且强大的PHP爬虫框架
Stars: ✭ 344 (-45.31%)
Webstera reliable high-level web crawling & scraping framework for Node.js.
Stars: ✭ 364 (-42.13%)
Fictiondown小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对
Stars: ✭ 362 (-42.45%)
Spider Flow新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
Stars: ✭ 365 (-41.97%)
Bilili🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器
Stars: ✭ 379 (-39.75%)
Freshonions TorscraperFresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion
Stars: ✭ 348 (-44.67%)
Signature algorithm各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)
Stars: ✭ 380 (-39.59%)
GosintOSINT Swiss Army Knife
Stars: ✭ 401 (-36.25%)
Xxl CrawlerA distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）
Stars: ✭ 561 (-10.81%)
