StackOverflow-CrawlerIt is a web crawler which crawls the stackoverfolw website (http://stackoverflow.com/) and finds the most popular technologies at current point of time by getting the tags info of the newest questions asked on the website.
Stars: ✭ 25 (+47.06%)
WeReadScan扫描“微信读书”已购图书并下载本地PDF的爬虫
Stars: ✭ 273 (+1505.88%)
antA web crawler for Go
Stars: ✭ 264 (+1452.94%)
doc crawler.pyExplore a website recursively and download all the wanted documents (PDF, ODT…)
Stars: ✭ 22 (+29.41%)
Market-Trend-PredictionThis is a project of build knowledge graph course. The project leverages historical stock price, and integrates social media listening from customers to predict market Trend On Dow Jones Industrial Average (DJIA).
Stars: ✭ 57 (+235.29%)
Strong Web Crawler基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码、触发各类事件、操纵页面Dom结构。
Stars: ✭ 238 (+1300%)
KochatOpensource Korean chatbot framework
Stars: ✭ 204 (+1100%)
AntchAntch, a fast, powerful and extensible web crawling & scraping framework for Go
Stars: ✭ 198 (+1064.71%)
NutchApache Nutch is an extensible and scalable web crawler
Stars: ✭ 2,277 (+13294.12%)
Crawler CommonsA set of reusable Java components that implement functionality common to any web crawler
Stars: ✭ 173 (+917.65%)
AbotCross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Stars: ✭ 1,961 (+11435.29%)
Collector HttpNorconex HTTP Collector is a flexible web crawler for collecting, parsing, and manipulating data from the Internet (or Intranet) to various data repositories such as search engines.
Stars: ✭ 130 (+664.71%)
ProxyA simple tool for fetching usable proxies from several websites.
Stars: ✭ 124 (+629.41%)
Crawlab LiteLite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
Stars: ✭ 122 (+617.65%)
Pspider简单易用的Python爬虫框架,QQ交流群:597510560
Stars: ✭ 1,611 (+9376.47%)
PulsarTurn large Web sites into tables and charts using simple SQLs.
Stars: ✭ 100 (+488.24%)
InfinitycrawlerA simple but powerful web crawler library for .NET
Stars: ✭ 97 (+470.59%)
Ospider开源矢量地理数据获取与预处理工具(POI/AOI/行政区/路网/土地利用)
Stars: ✭ 74 (+335.29%)
Cvpr2019Displays all the 2019 CVPR Accepted Papers in a way that they are easy to parse.
Stars: ✭ 65 (+282.35%)
AbotxCross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.
Stars: ✭ 63 (+270.59%)
CrawlabDistributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
Stars: ✭ 8,392 (+49264.71%)
MamanRust Web Crawler saving pages on Redis
Stars: ✭ 39 (+129.41%)
Dutsso快速登录大连理工大学统一身份认证系统(SSO)的Python模块,可轻松实现成绩提醒、抢课、玉兰卡信息、个人信息查询等功能。
Stars: ✭ 32 (+88.24%)
Storm CrawlerA scalable, mature and versatile web crawler based on Apache Storm
Stars: ✭ 703 (+4035.29%)
SpidrA versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Stars: ✭ 656 (+3758.82%)
Awesome CrawlerA collection of awesome web crawler,spider in different languages
Stars: ✭ 4,793 (+28094.12%)
Spider Flow新一代爬虫平台,以图形化方式定义爬虫流程,不写代码即可完成爬虫。
Stars: ✭ 365 (+2047.06%)
SparklerSpark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
Stars: ✭ 362 (+2029.41%)
AcheACHE is a web crawler for domain-specific search.
Stars: ✭ 320 (+1782.35%)
SupercrawlerA web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
Stars: ✭ 306 (+1700%)
Gopa[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
Stars: ✭ 277 (+1529.41%)
SpidyThe simple, easy to use command line web crawler.
Stars: ✭ 257 (+1411.76%)
LagoujobJob data mining repo for lagou.com
Stars: ✭ 256 (+1405.88%)
UnChainA tool to find redirection chains in multiple URLs
Stars: ✭ 77 (+352.94%)
ComicBookMakerScript to fetch webcomics and use them to create ebooks.
Stars: ✭ 27 (+58.82%)
CrawlBoxEasy way to brute-force web directory.
Stars: ✭ 118 (+594.12%)
flink-crawlerContinuous scalable web crawler built on top of Flink and crawler-commons
Stars: ✭ 48 (+182.35%)
SchweizerMesser🎯Python 3 网络爬虫实战、数据分析合集 | 当当 | 网易云音乐 | unsplash | 必胜客 | 猫眼 |
Stars: ✭ 89 (+423.53%)
evineInteractive CLI Web Crawler
Stars: ✭ 140 (+723.53%)
pyCreeper一个用来快速提取网页内容的信息采集(爬虫)框架, 实现了对网页的动态加载与控制。
Stars: ✭ 25 (+47.06%)
proxiProxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.
Stars: ✭ 32 (+88.24%)
Mimo-CrawlerA web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.
Stars: ✭ 22 (+29.41%)
siteshooter📷 Automate full website screenshots and PDF generation with multiple viewport support.
Stars: ✭ 63 (+270.59%)
OLX Scraper📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.
Stars: ✭ 15 (-11.76%)
WebCrawlerJust a simple web crawler which return crawled links as IObservable using reactive extension and async await.
Stars: ✭ 55 (+223.53%)
learncpp-downloadScrape bot, to get you an offline copy of tutorials
Stars: ✭ 23 (+35.29%)
bolsaBiblioteca feita em Python com o objetivo de facilitar o acesso a dados de seus investimentos na bolsa de valores(B3/CEI) através do Portal CEI.
Stars: ✭ 46 (+170.59%)
leekDistributed task redisqueue(最简单python分布式函数调度框架)
Stars: ✭ 60 (+252.94%)