WeReadScan扫描“微信读书”已购图书并下载本地PDF的爬虫
Stars: ✭ 273 (+992%)
antA web crawler for Go
Stars: ✭ 264 (+956%)
doc crawler.pyExplore a website recursively and download all the wanted documents (PDF, ODT…)
Stars: ✭ 22 (-12%)
Market-Trend-PredictionThis is a project of build knowledge graph course. The project leverages historical stock price, and integrates social media listening from customers to predict market Trend On Dow Jones Industrial Average (DJIA).
Stars: ✭ 57 (+128%)
Strong Web Crawler基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码、触发各类事件、操纵页面Dom结构。
Stars: ✭ 238 (+852%)
KochatOpensource Korean chatbot framework
Stars: ✭ 204 (+716%)
AntchAntch, a fast, powerful and extensible web crawling & scraping framework for Go
Stars: ✭ 198 (+692%)
NutchApache Nutch is an extensible and scalable web crawler
Stars: ✭ 2,277 (+9008%)
Crawler CommonsA set of reusable Java components that implement functionality common to any web crawler
Stars: ✭ 173 (+592%)
AbotCross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Stars: ✭ 1,961 (+7744%)
Collector HttpNorconex HTTP Collector is a flexible web crawler for collecting, parsing, and manipulating data from the Internet (or Intranet) to various data repositories such as search engines.
Stars: ✭ 130 (+420%)
ProxyA simple tool for fetching usable proxies from several websites.
Stars: ✭ 124 (+396%)
Crawlab LiteLite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
Stars: ✭ 122 (+388%)
Pspider简单易用的Python爬虫框架,QQ交流群:597510560
Stars: ✭ 1,611 (+6344%)
PulsarTurn large Web sites into tables and charts using simple SQLs.
Stars: ✭ 100 (+300%)
InfinitycrawlerA simple but powerful web crawler library for .NET
Stars: ✭ 97 (+288%)
Ospider开源矢量地理数据获取与预处理工具(POI/AOI/行政区/路网/土地利用)
Stars: ✭ 74 (+196%)
Cvpr2019Displays all the 2019 CVPR Accepted Papers in a way that they are easy to parse.
Stars: ✭ 65 (+160%)
AbotxCross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.
Stars: ✭ 63 (+152%)
CrawlabDistributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
Stars: ✭ 8,392 (+33468%)
MamanRust Web Crawler saving pages on Redis
Stars: ✭ 39 (+56%)
Dutsso快速登录大连理工大学统一身份认证系统(SSO)的Python模块,可轻松实现成绩提醒、抢课、玉兰卡信息、个人信息查询等功能。
Stars: ✭ 32 (+28%)
Storm CrawlerA scalable, mature and versatile web crawler based on Apache Storm
Stars: ✭ 703 (+2712%)
SpidrA versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Stars: ✭ 656 (+2524%)
Awesome CrawlerA collection of awesome web crawler,spider in different languages
Stars: ✭ 4,793 (+19072%)
Spider Flow新一代爬虫平台,以图形化方式定义爬虫流程,不写代码即可完成爬虫。
Stars: ✭ 365 (+1360%)
SparklerSpark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
Stars: ✭ 362 (+1348%)
AcheACHE is a web crawler for domain-specific search.
Stars: ✭ 320 (+1180%)
SupercrawlerA web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
Stars: ✭ 306 (+1124%)
Gopa[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
Stars: ✭ 277 (+1008%)
SpidyThe simple, easy to use command line web crawler.
Stars: ✭ 257 (+928%)
LagoujobJob data mining repo for lagou.com
Stars: ✭ 256 (+924%)
UnChainA tool to find redirection chains in multiple URLs
Stars: ✭ 77 (+208%)
ComicBookMakerScript to fetch webcomics and use them to create ebooks.
Stars: ✭ 27 (+8%)
CrawlBoxEasy way to brute-force web directory.
Stars: ✭ 118 (+372%)
flink-crawlerContinuous scalable web crawler built on top of Flink and crawler-commons
Stars: ✭ 48 (+92%)
SchweizerMesser🎯Python 3 网络爬虫实战、数据分析合集 | 当当 | 网易云音乐 | unsplash | 必胜客 | 猫眼 |
Stars: ✭ 89 (+256%)
evineInteractive CLI Web Crawler
Stars: ✭ 140 (+460%)
pyCreeper一个用来快速提取网页内容的信息采集(爬虫)框架, 实现了对网页的动态加载与控制。
Stars: ✭ 25 (+0%)
proxiProxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.
Stars: ✭ 32 (+28%)
Mimo-CrawlerA web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.
Stars: ✭ 22 (-12%)
siteshooter📷 Automate full website screenshots and PDF generation with multiple viewport support.
Stars: ✭ 63 (+152%)
OLX Scraper📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.
Stars: ✭ 15 (-40%)
WebCrawlerJust a simple web crawler which return crawled links as IObservable using reactive extension and async await.
Stars: ✭ 55 (+120%)
learncpp-downloadScrape bot, to get you an offline copy of tutorials
Stars: ✭ 23 (-8%)
bolsaBiblioteca feita em Python com o objetivo de facilitar o acesso a dados de seus investimentos na bolsa de valores(B3/CEI) através do Portal CEI.
Stars: ✭ 46 (+84%)
leekDistributed task redisqueue(最简单python分布式函数调度框架)
Stars: ✭ 60 (+140%)
json-web-crawlerUse JSON to list all elements (with css 3 and jquery selector) that you want to crawl.
Stars: ✭ 17 (-32%)
soCareers-DataData and data processing scripts of StackOverflow Careers pages
Stars: ✭ 18 (-28%)