Top 54 web-crawler open source projects

基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码、触发各类事件、操纵页面Dom结构。

Opensource Korean chatbot framework

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Apache Nutch is an extensible and scalable web crawler

A simple distributed crawler for zhihu && data analysis

A set of reusable Java components that implement functionality common to any web crawler

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

A collection of awesome web scaper, crawler.

Norconex HTTP Collector is a flexible web crawler for collecting, parsing, and manipulating data from the Internet (or Intranet) to various data repositories such as search engines.

A simple tool for fetching usable proxies from several websites.

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

简单易用的Python爬虫框架，QQ交流群：597510560

Turn large Web sites into tables and charts using simple SQLs.

A simple but powerful web crawler library for .NET

Web Crawler

开源矢量地理数据获取与预处理工具(POI/AOI/行政区/路网/土地利用)

Displays all the 2019 CVPR Accepted Papers in a way that they are easy to parse.

Cross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.

Parser and database to index the terpene profile of different strains of Cannabis from online databases

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Rust Web Crawler saving pages on Redis

快速登录大连理工大学统一身份认证系统（SSO）的Python模块，可轻松实现成绩提醒、抢课、玉兰卡信息、个人信息查询等功能。

A scalable, mature and versatile web crawler based on Apache Storm

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

A collection of awesome web crawler,spider in different languages

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.

ACHE is a web crawler for domain-specific search.

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

The simple, easy to use command line web crawler.

Job data mining repo for lagou.com

A tool to find redirection chains in multiple URLs

Script to fetch webcomics and use them to create ebooks.

Easy way to brute-force web directory.

Continuous scalable web crawler built on top of Flink and crawler-commons

🎯Python 3 网络爬虫实战、数据分析合集 | 当当 | 网易云音乐 | unsplash | 必胜客 | 猫眼 |

Interactive CLI Web Crawler

一个用来快速提取网页内容的信息采集（爬虫）框架， 实现了对网页的动态加载与控制。

Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.

A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

📷 Automate full website screenshots and PDF generation with multiple viewport support.

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

Just a simple web crawler which return crawled links as IObservable using reactive extension and async await.

Scrape bot, to get you an offline copy of tutorials

Biblioteca feita em Python com o objetivo de facilitar o acesso a dados de seus investimentos na bolsa de valores(B3/CEI) através do Portal CEI.

Distributed task redisqueue(最简单python分布式函数调度框架)

Use JSON to list all elements (with css 3 and jquery selector) that you want to crawl.

It is a web crawler which crawls the stackoverfolw website (http://stackoverflow.com/) and finds the most popular technologies at current point of time by getting the tags info of the newest questions asked on the website.

扫描“微信读书”已购图书并下载本地PDF的爬虫

Raspagem de dados para iniciante usando Scrapy e outras libs básicas

A web crawler for Go

Explore a website recursively and download all the wanted documents (PDF, ODT…)

This is a project of build knowledge graph course. The project leverages historical stock price, and integrates social media listening from customers to predict market Trend On Dow Jones Industrial Average (DJIA).