Top 18 webcrawler open source projects

新闻网页正文通用抽取器 Beta 版.

蓝天采集器是一款免费的数据采集发布爬虫软件，采用php+mysql开发，可部署在云服务器，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Save content you enjoy!

Java web crawling library

단 몇줄의 코드로 다양한 쇼핑 플랫폼에서 결제 내역을 긁어오자!

A Unix-style personal search engine and web crawler for your digital footprint.

A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

新闻网页正文通用抽取器 Beta 版.

Document Search Engine Tool

Multithreaded Konachan / Yandere (moebooru based site) Image Bulk Downloader | 多线程K站Y站下载器

Yummy Recipe Crawler and Search

Bot para monitoramento de promoções no fórum do Hardmob http://www.hardmob.com.br/promocoes/

This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.

Because Chegg only gave few days to access their solutions for free when I rent their books so I decided to create a web crawler to save all the needed solutions offline.

一个致力于用Python提高部门工作自动化水平的程序库！（包括数据采集、办公自动化、辅助研究、图网络、复杂系统、3D可视化等）

O maior livro de receitas culinárias em língua portuguesa

Python Web Crawler with Selenium and PhantomJS