Ospider开源矢量地理数据获取与预处理工具(POI/AOI/行政区/路网/土地利用)
Stars: ✭ 74 (-6.33%)
Cvpr2019Displays all the 2019 CVPR Accepted Papers in a way that they are easy to parse.
Stars: ✭ 65 (-17.72%)
AbotxCross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.
Stars: ✭ 63 (-20.25%)
CrawlabDistributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
Stars: ✭ 8,392 (+10522.78%)
MamanRust Web Crawler saving pages on Redis
Stars: ✭ 39 (-50.63%)
Dutsso快速登录大连理工大学统一身份认证系统(SSO)的Python模块,可轻松实现成绩提醒、抢课、玉兰卡信息、个人信息查询等功能。
Stars: ✭ 32 (-59.49%)
Storm CrawlerA scalable, mature and versatile web crawler based on Apache Storm
Stars: ✭ 703 (+789.87%)
SpidrA versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Stars: ✭ 656 (+730.38%)
Awesome CrawlerA collection of awesome web crawler,spider in different languages
Stars: ✭ 4,793 (+5967.09%)
Spider Flow新一代爬虫平台,以图形化方式定义爬虫流程,不写代码即可完成爬虫。
Stars: ✭ 365 (+362.03%)
SparklerSpark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
Stars: ✭ 362 (+358.23%)
AcheACHE is a web crawler for domain-specific search.
Stars: ✭ 320 (+305.06%)
SupercrawlerA web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
Stars: ✭ 306 (+287.34%)
Gopa[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
Stars: ✭ 277 (+250.63%)
SpidyThe simple, easy to use command line web crawler.
Stars: ✭ 257 (+225.32%)
LagoujobJob data mining repo for lagou.com
Stars: ✭ 256 (+224.05%)
UnChainA tool to find redirection chains in multiple URLs
Stars: ✭ 77 (-2.53%)
ComicBookMakerScript to fetch webcomics and use them to create ebooks.
Stars: ✭ 27 (-65.82%)
CrawlBoxEasy way to brute-force web directory.
Stars: ✭ 118 (+49.37%)
flink-crawlerContinuous scalable web crawler built on top of Flink and crawler-commons
Stars: ✭ 48 (-39.24%)
SchweizerMesser🎯Python 3 网络爬虫实战、数据分析合集 | 当当 | 网易云音乐 | unsplash | 必胜客 | 猫眼 |
Stars: ✭ 89 (+12.66%)
evineInteractive CLI Web Crawler
Stars: ✭ 140 (+77.22%)
pyCreeper一个用来快速提取网页内容的信息采集(爬虫)框架, 实现了对网页的动态加载与控制。
Stars: ✭ 25 (-68.35%)
proxiProxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.
Stars: ✭ 32 (-59.49%)
Mimo-CrawlerA web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.
Stars: ✭ 22 (-72.15%)
siteshooter📷 Automate full website screenshots and PDF generation with multiple viewport support.
Stars: ✭ 63 (-20.25%)
OLX Scraper📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.
Stars: ✭ 15 (-81.01%)
WebCrawlerJust a simple web crawler which return crawled links as IObservable using reactive extension and async await.
Stars: ✭ 55 (-30.38%)
learncpp-downloadScrape bot, to get you an offline copy of tutorials
Stars: ✭ 23 (-70.89%)
bolsaBiblioteca feita em Python com o objetivo de facilitar o acesso a dados de seus investimentos na bolsa de valores(B3/CEI) através do Portal CEI.
Stars: ✭ 46 (-41.77%)
leekDistributed task redisqueue(最简单python分布式函数调度框架)
Stars: ✭ 60 (-24.05%)
json-web-crawlerUse JSON to list all elements (with css 3 and jquery selector) that you want to crawl.
Stars: ✭ 17 (-78.48%)
StackOverflow-CrawlerIt is a web crawler which crawls the stackoverfolw website (http://stackoverflow.com/) and finds the most popular technologies at current point of time by getting the tags info of the newest questions asked on the website.
Stars: ✭ 25 (-68.35%)
WeReadScan扫描“微信读书”已购图书并下载本地PDF的爬虫
Stars: ✭ 273 (+245.57%)
antA web crawler for Go
Stars: ✭ 264 (+234.18%)
doc crawler.pyExplore a website recursively and download all the wanted documents (PDF, ODT…)
Stars: ✭ 22 (-72.15%)
Market-Trend-PredictionThis is a project of build knowledge graph course. The project leverages historical stock price, and integrates social media listening from customers to predict market Trend On Dow Jones Industrial Average (DJIA).
Stars: ✭ 57 (-27.85%)
Strong Web Crawler基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码、触发各类事件、操纵页面Dom结构。
Stars: ✭ 238 (+201.27%)
KochatOpensource Korean chatbot framework
Stars: ✭ 204 (+158.23%)
AntchAntch, a fast, powerful and extensible web crawling & scraping framework for Go
Stars: ✭ 198 (+150.63%)
NutchApache Nutch is an extensible and scalable web crawler
Stars: ✭ 2,277 (+2782.28%)
Crawler CommonsA set of reusable Java components that implement functionality common to any web crawler
Stars: ✭ 173 (+118.99%)
AbotCross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Stars: ✭ 1,961 (+2382.28%)
Collector HttpNorconex HTTP Collector is a flexible web crawler for collecting, parsing, and manipulating data from the Internet (or Intranet) to various data repositories such as search engines.
Stars: ✭ 130 (+64.56%)
ProxyA simple tool for fetching usable proxies from several websites.
Stars: ✭ 124 (+56.96%)
Crawlab LiteLite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
Stars: ✭ 122 (+54.43%)
Pspider简单易用的Python爬虫框架,QQ交流群:597510560
Stars: ✭ 1,611 (+1939.24%)
PulsarTurn large Web sites into tables and charts using simple SQLs.
Stars: ✭ 100 (+26.58%)
InfinitycrawlerA simple but powerful web crawler library for .NET
Stars: ✭ 97 (+22.78%)