Java Spider一个基于webmagic框架二次开发的java爬虫框架实战,已实现能爬取腾讯,搜狐,今日头条(单独集成功能)等资讯内容,配合elasticsearch框架用法,实现了自动爬虫,已投入线上生产使用。
Stars: ✭ 276 (+1352.63%)
WswpCode for the second edition Web Scraping with Python book by Packt Publications
Stars: ✭ 112 (+489.47%)
Douban CrawlerUno Crawler por https://douban.com
Stars: ✭ 13 (-31.58%)
Email ExtractorThe main functionality is to extract all the emails from one or several URLs - La funcionalidad principal es extraer todos los correos electrónicos de una o varias Url
Stars: ✭ 81 (+326.32%)
Warta ScrapIndonesia Index News Crawler, including 10 online media
Stars: ✭ 57 (+200%)
Haipproxy💖 High available distributed ip proxy pool, powerd by Scrapy and Redis
Stars: ✭ 4,993 (+26178.95%)
JspiderJSpider会每周更新至少一个网站的JS解密方式,欢迎 Star,交流微信:13298307816
Stars: ✭ 914 (+4710.53%)
ScrapitScraping scripts for various websites.
Stars: ✭ 25 (+31.58%)
AvbookAV 电影管理系统, avmoo , javbus , javlibrary 爬虫,线上 AV 影片图书馆,AV 磁力链接数据库,Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
Stars: ✭ 8,133 (+42705.26%)
PoliteBe nice on the web
Stars: ✭ 253 (+1231.58%)
Alipayspider ScrapyAlipaySpider on Scrapy(use chrome driver); 支付宝爬虫(基于Scrapy)
Stars: ✭ 70 (+268.42%)
Reptile🏀 Python3 网络爬虫实战(部分含详细教程)猫眼 腾讯视频 豆瓣 研招网 微博 笔趣阁小说 百度热点 B站 CSDN 网易云阅读 阿里文学 百度股票 今日头条 微信公众号 网易云音乐 拉勾 有道 unsplash 实习僧 汽车之家 英雄联盟盒子 大众点评 链家 LPL赛程 台风 梦幻西游、阴阳师藏宝阁 天气 牛客网 百度文库 睡前故事 知乎 Wish
Stars: ✭ 1,048 (+5415.79%)
allitebooks.comDownload all the ebooks with indexed csv of "allitebooks.com"
Stars: ✭ 24 (+26.32%)
Crawlab LiteLite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
Stars: ✭ 122 (+542.11%)
Copybook用爬虫爬取小说网站上所有小说,存储到数据库中,并用爬到的数据构建自己的小说网站
Stars: ✭ 117 (+515.79%)
Scrapy demoall kinds of scrapy demo
Stars: ✭ 128 (+573.68%)
ScralaUnmaintained 🐳 ☕️ 🕷 Scala crawler(spider) framework, inspired by scrapy, created by @gaocegege
Stars: ✭ 113 (+494.74%)
Python3 SpiderPython爬虫实战 - 模拟登陆各大网站 包含但不限于:滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝,如果喜欢请start ❤️
Stars: ✭ 2,129 (+11105.26%)
Linkedin Profile Scraper🕵️♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.
Stars: ✭ 171 (+800%)
Hivelots of spider (很多爬虫)
Stars: ✭ 110 (+478.95%)
Python Spider豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章
Stars: ✭ 615 (+3136.84%)
CollyElegant Scraper and Crawler Framework for Golang
Stars: ✭ 15,535 (+81663.16%)
ScrapydwebWeb app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO 👉
Stars: ✭ 2,385 (+12452.63%)
Spiderkeeperadmin ui for scrapy/open source scrapinghub
Stars: ✭ 2,562 (+13384.21%)
PttImageSpiderPTT 圖片下載器 (抓取整個看板的圖片,並用文章標題作為資料夾的名稱 ) (使用Scrapy)
Stars: ✭ 16 (-15.79%)
scraper图片爬取下载工具,极速爬取下载 站酷https://www.zcool.com.cn/, CNU 视觉 http://www.cnu.cc/ 设计师/用户 上传的 图片/照片/插画。
Stars: ✭ 64 (+236.84%)
AutoscraperA Smart, Automatic, Fast and Lightweight Web Scraper for Python
Stars: ✭ 4,077 (+21357.89%)
GosintOSINT Swiss Army Knife
Stars: ✭ 401 (+2010.53%)
wget-luaWget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
Stars: ✭ 52 (+173.68%)
OLX Scraper📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.
Stars: ✭ 15 (-21.05%)
antA web crawler for Go
Stars: ✭ 264 (+1289.47%)
SpydanA web spider for shodan.io without using the Developer API.
Stars: ✭ 30 (+57.89%)
scrapy-distributedA series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.
Stars: ✭ 38 (+100%)
newspaperjsNews extraction and scraping. Article Parsing
Stars: ✭ 59 (+210.53%)
aliexscrapeGet Aliexpress product details in JSON
Stars: ✭ 80 (+321.05%)
metacritic apiPHP Metacritic API - Mirrored by my GitLab
Stars: ✭ 31 (+63.16%)
Elves🎊 Design and implement of lightweight crawler framework.
Stars: ✭ 315 (+1557.89%)
XidelCommand line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
Stars: ✭ 335 (+1663.16%)
arachnodHigh performance crawler for Nodejs
Stars: ✭ 17 (-10.53%)
scrapy-LBCAraignée LeBonCoin avec Scrapy et ElasticSearch
Stars: ✭ 14 (-26.32%)
scrapy-adminA django admin site for scrapy
Stars: ✭ 44 (+131.58%)
LinkedinLinkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy
Stars: ✭ 309 (+1526.32%)
Xcrawler快速、简洁且强大的PHP爬虫框架
Stars: ✭ 344 (+1710.53%)
Freshonions TorscraperFresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion
Stars: ✭ 348 (+1731.58%)
Scrapy RedisRedis-based components for Scrapy.
Stars: ✭ 4,998 (+26205.26%)
XsrfprobeThe Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
Stars: ✭ 532 (+2700%)
JikanUnofficial MyAnimeList PHP+REST API which provides functions other than the official API
Stars: ✭ 531 (+2694.74%)
SurgeonDeclarative DOM extraction expression evaluator. 👨⚕️
Stars: ✭ 653 (+3336.84%)
GoogledictionaryapiGoogle does not provide Google Dictionary API so I created one.
Stars: ✭ 528 (+2678.95%)
Go jobs带你了解一下Golang的市场行情
Stars: ✭ 526 (+2668.42%)