1790 Open source projects that are alternatives of or similar to Jlitespider

A simple and flexible web crawler framework for java.

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

🍰 一个可视化的爬虫平台

Microservice abstract class

Observability Signaling for Distributed Computation

NScrapy is a .net core corss platform Distributed Spider Framework which provide an easy way to write your own Spider

A distributed and coördination-free log management system

📰 A responsive interface of Hacker News with summaries and thumbnails.

golang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

👊SpringBoot+Zookeeper+Dubbo打造分布式高并发商品秒杀系统

Every web site provides APIs.

知乎模拟登录，支持提取验证码和保存 Cookies

Celery Tasks Monitoring Tool

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

A HTTP Ruby API for Consul

各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

OSINT Swiss Army Knife

libvineyard: an in-memory immutable data manager.

MM131网站图片爬取 🚨

快速、简洁且强大的PHP爬虫框架

A collection of awesome web crawler,spider in different languages

Open-source distributed computation and storage platform

Python的基础练习代码与各种爬虫代码

A Facebook crawler

Redis-based components for Scrapy.

Moqui Framework tool component for Hazelcast, used for distributed async services, entity distributed cache invalidation, web session replication, and distributed cache (javax.cache)

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

A high performance web crawler in Elixir.

A multi-thread crawler framework with many builtin image crawlers provided.

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

Dark Web OSINT Tool

Distributed Image Search Engine Crawler

一个超级轻量的百度图片爬虫

B站用户爬虫 好耶~是爬虫

APIs for loginning some websites by using requests.

Rust Web Crawler saving pages on Redis

Incredibly fast crawler designed for OSINT.

[DEPRECATED] Simple, flexible, delightful web crawler/spider package

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。

Golang爬虫 爬取汽车之家 二手车产品库

Some classic web crawler projects.一些经典的爬虫

Free Web Scraping Tool with Java

GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）

Pluggable foundation blocks for building distributed apps.

Ongoing Storj v3 development. Decentralized cloud object storage that is affordable, easy to use, private, and secure.

A web crawler (for bug hunting) that gathers more than you can imagine.

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

Sandglass is a distributed, horizontally scalable, persistent, time sorted message queue.

Google search results crawler, get google search results that you need

A distributed, fast open-source graph database featuring horizontal scalability and high availability

API of DouYin for Humans used to Crawl Popular Videos and Musics

a puppeteer walker 🕷 🕸

Golang爬虫 爬取豆瓣电影Top250

获取免费socks/https/http代理的网站集合