1134 Open source projects that are alternatives of or similar to Ruia

稳健高效的评分制-针对性- IP代理池 + API服务，可以自己插入采集器进行代理IP的爬取，针对你的爬虫的一个或多个目标网站分别生成有效的IP代理数据库，支持MongoDB 4.0 使用 Python3.7（Scored IP proxy pool ,customise proxy data crawler can be added anytime）

Web crawling framework based on asyncio.

🍿爬虫代理IP池(proxy pool) python🍟一个还ok的IP代理池

owllook-小说搜索引擎

🧊 一个可爱且任性的 B 站视频下载器（bilili V2）

Python的基础练习代码与各种爬虫代码

带你了解一下Golang的市场行情

抖音 SDK，数据采集，爬虫抓取不是梦

一个超级轻量的百度图片爬虫

A multi-thread crawler framework with many builtin image crawlers provided.

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

Dependency injection framework for Python

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

Html网页正文提取

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

Jobs scheduler for managing background task (asyncio)

An aiohttp transport for raven-python

Simple but useful Python web scraping tutorial code.

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

A Facebook crawler

Simple, clear and fast Web Crawler framework build on python3.6+, powered by asyncio.

Mixcloud API wrapper for Python and Async IO

Scraping scripts for various websites.

[DEPRECATED] Simple, flexible, delightful web crawler/spider package

Popular Pentesting scanner in Python3.6 for SQLi/XSS/LFI/RFI and other Vulns

Modern and powerful asynchronous telegram bot framework.

💐 Full Amazon Automatic Download

OSINT Swiss Army Knife

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

Crawly, a high-level web crawling & scraping framework for Elixir.

各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)

Demos for aiohttp project

A collection of awesome web crawler,spider in different languages

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

API of DouYin for Humans used to Crawl Popular Videos and Musics

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

Free Web Scraping Tool with Java

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

a puppeteer walker 🕷 🕸

asyncio support for botocore library using aiohttp

GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）

a reliable high-level web crawling & scraping framework for Node.js.

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

Dark Web OSINT Tool

slacker wrapper for asyncio

Gospider - Fast web spider written in Go

Some classic web crawler projects.一些经典的爬虫

A project starter template for deploying an aiohttp app to Heroku

Rust Web Crawler saving pages on Redis

A high performance web crawler in Elixir.

自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向

Incredibly fast crawler designed for OSINT.

Golang爬虫 爬取汽车之家 二手车产品库

hproxy - Asynchronous IP proxy pool, aims to make getting proxy as convenient as possible.(异步爬虫代理池)

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。

Implement aiohttp.web OpenAPI 3 server applications with schema first approach.

Simple failure handling. Failsafe implementation in Python

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion