891 Open source projects that are alternatives of or similar to Pspider

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

A simple distributed crawler for zhihu && data analysis

🥄 A package for building specific Proxy Pool for different Sites.

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

稳健高效的评分制-针对性- IP代理池 + API服务，可以自己插入采集器进行代理IP的爬取，针对你的爬虫的一个或多个目标网站分别生成有效的IP代理数据库，支持MongoDB 4.0 使用 Python3.7（Scored IP proxy pool ,customise proxy data crawler can be added anytime）

A collection of awesome web crawler,spider in different languages

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

Rust Web Crawler saving pages on Redis

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

多线程知乎用户爬虫，基于python3

Continuous scalable web crawler built on top of Flink and crawler-commons

爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例

Html网页正文提取

Crawly, a high-level web crawling & scraping framework for Elixir.

Python的基础练习代码与各种爬虫代码

B站用户爬虫 好耶~是爬虫

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

A Facebook crawler

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

A multi-thread crawler framework with many builtin image crawlers provided.

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

APIs for loginning some websites by using requests.

Golang爬虫 爬取豆瓣电影Top250

API of DouYin for Humans used to Crawl Popular Videos and Musics

A high performance web crawler in Elixir.

Dark Web OSINT Tool

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。

OSINT Swiss Army Knife

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

带你了解一下Golang的市场行情

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)

一个超级轻量的百度图片爬虫

🐾 Creeper - The Next Generation Crawler Framework (Go)

Free Web Scraping Tool with Java

A web crawler (for bug hunting) that gathers more than you can imagine.

Gospider - Fast web spider written in Go

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

💐 Full Amazon Automatic Download

蓝天采集器是一款免费的数据采集发布爬虫软件，采用php+mysql开发，可部署在云服务器，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

[DEPRECATED] Simple, flexible, delightful web crawler/spider package

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

Golang爬虫 爬取汽车之家 二手车产品库

自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。

Sample of using proxies to crawl baidu search results.

massive SQL injection vulnerability scanner

Incredibly fast crawler designed for OSINT.

获取免费socks/https/http代理的网站集合

爬取北大法宝网http://www.pkulaw.cn/Case/

python crawler spider

Powerful web scraping framework for Crystal

Some classic web crawler projects.一些经典的爬虫

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

Cross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.