2403 Open source projects that are alternatives of or similar to Ppspider

A collection of awesome web crawler,spider in different languages

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

一个超级轻量的百度图片爬虫

Example Storefront is Reaction Commerce’s headless ecommerce storefront - Next.js, GraphQL, React. Built using Apollo Client and the commerce-focused React UI components provided in the Storefront Component Library (reactioncommerce/reaction-component-library). It connects with Reaction backend with the GraphQL API.

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

A high performance web crawler in Elixir.

A multi-thread crawler framework with many builtin image crawlers provided.

Web page PDF/PNG rendering done right. Self-hosted service for rendering receipts, invoices, or any content.

Gospider - Fast web spider written in Go

Web crawler.

大数据生态解决方案数据平台：基于大数据、数据平台、微服务、机器学习、商城、自动化运维、DevOps、容器部署平台、数据平台采集、数据平台存储、数据平台计算、数据平台开发、数据平台应用搭建的大数据解决方案。

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

💐 Full Amazon Automatic Download

网站签到脚本

Rust Web Crawler saving pages on Redis

A powerful dynamic crawler for web vulnerability scanners

Golang爬虫 爬取汽车之家 二手车产品库

Cross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.

[DEPRECATED] Simple, flexible, delightful web crawler/spider package

疫情数据爬虫，2019新型冠状病毒数据仓库，轨迹数据，同乘数据，报道

A Lightweight High Performance ESMTP email server

Some classic web crawler projects.一些经典的爬虫

Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!

python crawler spider

蓝天采集器是一款免费的数据采集发布爬虫软件，采用php+mysql开发，可部署在云服务器，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

Async Python 3.6+ web scraping micro-framework based on asyncio

BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。

Python的基础练习代码与各种爬虫代码

APIs for loginning some websites by using requests.

A proper monster for front-end automated testing

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

yspider -- 轻量级爬虫系统

[https://www.sofineday.com], golang项目开发脚手架,集成最佳实践(gin+gorm+go-redis+mongo+cors+jwt+json日志库zap(支持日志收集到kafka或mongo)+消息队列kafka+微信支付宝支付gopay+api加密+api反向代理+go modules依赖管理+headless爬虫chromedp+makefile+二进制压缩+livereload热加载)

all kinds of scrapy demo

Sample of using proxies to crawl baidu search results.

😱Full Automatic Amazon Distributed Spider | 亚马逊分布式四国际站采集选款产品|账号admin,密码adminadmin

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

MM131网站图片爬取 🚨

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

A lite distributed Java spider framework :-)

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

B站用户爬虫 好耶~是爬虫

A simple distributed crawler for zhihu && data analysis

book.zi5.me全站kindle电子书籍爬取，按照作者书籍名分类，每本书有mobi和equb两种格式，采用分布式进行全站爬取

Headless Chrome For Java （Java 爬虫）

Crawl some picture for fun

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 ，点星支持，仅供学习参考，请勿用于商业用途，后果自负。

ScrapingOutsourcing专注分享爬虫代码 尽量每周更新一个

多线程知乎用户爬虫，基于python3

JavaScript + BeautifulSoup = JSSoup

Html网页正文提取

Crawly, a high-level web crawling & scraping framework for Elixir.

Golang爬虫 爬取豆瓣电影Top250

JS逆向研究

Elegant Scraper and Crawler Framework for Golang

Getting started with Puppeteer and Chrome Headless for Web Scraping