Top 615 crawler open source projects

Distributed Image Search Engine Crawler

爬取amazon/bestbuy/costco/6pm 的商品详情

获取最新可备案域名列表爬虫

Implements the crawler package into Symfony

[ Closed ] 🎨 image collector, which supports custom acquisition source configuration and is compatible with MacOS and Windows operating systems.

massive SQL injection vulnerability scanner

Android应用市场网络爬虫

Scraping scripts for various websites.

基于appium的app自动遍历工具

A Tumblr Blog Backup Application

👧 美女写真套图爬虫（二）

Elasticsearch File System Crawler (FS Crawler)

The Last Web Recon Tool You'll Need

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

Crawls a website, gets PageSpeed Insights data for each page, and exports an HTML report.

Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机

Dark Web OSINT Tool

Life is short, *****.

📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.

[Unmaintained] A simple and clean video/music/image downloader 👾

Gospider - Fast web spider written in Go

A high performance web crawler in Elixir.

A tool for pixiv.net. 人人可用的P站爬虫

🐾 Creeper - The Next Generation Crawler Framework (Go)

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

✌️ Python3 BitTorrent DHT crawler

基金投资管理回测引擎

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

HTTP API for Scrapy spiders

京东商品价格监控：监控用户设定商品价格，降价邮件/微信提醒。技术：Python爬虫/IP代理池/JS接口爬取/Selenium页面爬取

A multi-thread crawler framework with many builtin image crawlers provided.

🎓 中国大学MOOC、学堂在线、网易云课堂、好大学在线、爱课程 MOOC 课程下载。

一个超级轻量的百度图片爬虫

Free Web Scraping Tool with Java

Simple but useful Python web scraping tutorial code.

API of DouYin for Humans used to Crawl Popular Videos and Musics

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

A search application to explore, discover and share online files

Fess is very powerful and easily deployable Enterprise Search Server.

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

基于搜狗微信搜索的微信公众号爬虫接口

Distributed crawler powered by Headless Chrome

A Facebook crawler

Redis-based components for Scrapy.

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

支援 PTT 還有 PTT2 的 PTT API

带你了解一下Golang的市场行情

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

Doujinshi downloader 绅士漫画下载

a new crawler based on python with more function including Network fingerprint search

A collection of awesome web crawler,spider in different languages

🐨实时监控1000家中国企业的新闻动态

Declarative web scraping

A framework for creating semi-automatic web content extractors

Crawler for Nintendo Switch eShop

LinkedIn Scraper (currently working 2020)

Python的基础练习代码与各种爬虫代码

Crawl BookCorpus

Crawly, a high-level web crawling & scraping framework for Elixir.