1134 Open source projects that are alternatives of or similar to Gain

golang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。

爬取 www.mzitu.com 全站图片，截至目前共5162个图集，16.5万多张美女图片，使用 asyncio 和 aiohttp 实现的异步版本只需要不到2小时就能爬取完成。按日期创建图集目录，保存更合理。控制台只显示下载的进度条，详细信息保存在日志文件中。支持异常处理，不会终止爬虫程序。失败的请求，下次再执行爬虫程序时会自动下载

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

Slack Developer Kit for Python

Learn how modern async web frameworks work, by writing simple clone from scratch

磁力網站U3C3介紹以及域名更新

今日头条机器人，支持用户登陆、关注、取消关注、获取关注粉丝、发文、发悟空问答、点赞、评论、采集各种类型新闻讯息等，使用今日头条网页版API实现

🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新） 及 在线web预览

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

知乎模拟登录，支持提取验证码和保存 Cookies

各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

mxget的Python实现

🎉 Ultimate Emoji Generator

Asynchronous HTTP client/server framework for asyncio and Python

Python的基础练习代码与各种爬虫代码

Demos for aiohttp project

Crawly, a high-level web crawling & scraping framework for Elixir.

MM131网站图片爬取 🚨

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

Digger is a powerful and flexible web crawler implemented by pure golang

Html网页正文提取

微博超级话题爬虫，微博词频统计+情感分析+简单分类，新增肺炎超话爬取数据

Simple but useful Python web scraping tutorial code.

A multi-thread crawler framework with many builtin image crawlers provided.

API of DouYin for Humans used to Crawl Popular Videos and Musics

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

A high performance web crawler in Elixir.

A fast tool to fetch URLs from HTML attributes by crawl-in.

Scraping scripts for various websites.

slacker wrapper for asyncio

[DEPRECATED] Simple, flexible, delightful web crawler/spider package

Mixcloud API wrapper for Python and Async IO

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

💐 Full Amazon Automatic Download

自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

Simple failure handling. Failsafe implementation in Python

Powerful web scraping framework for Crystal

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

hproxy - Asynchronous IP proxy pool, aims to make getting proxy as convenient as possible.(异步爬虫代理池)

Simple, clear and fast Web Crawler framework build on python3.6+, powered by asyncio.

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

Simple REST client for python 3.6+

Dark Web OSINT Tool

Asynchronous OAuth 2.0 framework and provider for Python 3

蓝天采集器是一款免费的数据采集发布爬虫软件，采用php+mysql开发，可部署在云服务器，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

抖音 SDK，数据采集，爬虫抓取不是梦

爬取北大法宝网http://www.pkulaw.cn/Case/

BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。

获取免费socks/https/http代理的网站集合

GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）

APIs for loginning some websites by using requests.

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

Crawl some picture for fun

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

🐛一只登录正方教务管理系统，爬取数据的小爬虫

Binance chain SDK in Python

Gospider - Fast web spider written in Go