2337 Open source projects that are alternatives of or similar to Haipproxy

Redis-based components for Scrapy.

🥄 A package for building specific Proxy Pool for different Sites.

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

A multi-thread crawler framework with many builtin image crawlers provided.

A lite distributed Java spider framework :-)

Python爬虫代理IP池(proxy pool)

NScrapy is a .net core corss platform Distributed Spider Framework which provide an easy way to write your own Spider

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

Redisson - Redis Java client with features of In-Memory Data Grid. Over 50 Redis based Java objects and services: Set, Multimap, SortedSet, Map, List, Queue, Deque, Semaphore, Lock, AtomicLong, Map Reduce, Publish / Subscribe, Bloom filter, Spring Cache, Tomcat, Scheduler, JCache API, Hibernate, MyBatis, RPC, local cache ...

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

💐 Full Amazon Automatic Download

ScrapingOutsourcing专注分享爬虫代码 尽量每周更新一个

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

book.zi5.me全站kindle电子书籍爬取，按照作者书籍名分类，每本书有mobi和equb两种格式，采用分布式进行全站爬取

💐Marmot | Web Crawler/HTTP protocol Download Package 🐭

A Facebook crawler

Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索

豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

免费 IP 代理池。Scrapy 爬虫框架插件

A collection of awesome web crawler,spider in different languages

V2EX爬虫

A django admin site for scrapy

一个基于Scrapy的数据采集爬虫代码库

收集各种免费的 Python 爬虫项目

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

python爬虫小项目【持续更新】【笔趣阁小说下载、Tweet数据抓取、天气查询、网易云音乐逆向、天天基金网查询、微博数据抓取（生成cookie）、有道翻译逆向、企查查免登陆爬虫、大众点评svg加密破解、B站用户爬虫、拉钩免登录爬虫、自如租房字体加密、知乎问答

A simple and flexible web crawler framework for java.

Continuous scalable web crawler built on top of Flink and crawler-commons

High performance crawler for Nodejs

基于Scrapy框架的豆瓣电影爬虫

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

LEMP stack in a Kubernetes cluster

PTT 網路版爬蟲

🍰 一个可视化的爬虫平台

A framework for creating semi-automatic web content extractors

🐛一只登录正方教务管理系统，爬取数据的小爬虫

PTT 圖片下載器 (抓取整個看板的圖片，並用文章標題作為資料夾的名稱 ) (使用Scrapy)

Uno Crawler por https://douban.com

今日头条科技新闻接口爬虫

Generating spiders dynamically to crawl and check those free proxy ip on the internet with scrapy.

A fast tool to fetch URLs from HTML attributes by crawl-in.

磁力網站U3C3介紹以及域名更新

🔧 🔩 🔨 收集整理了爬虫相关的工具、模拟登陆技术、代理IP、scrapy模板代码等内容。

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

📰 A responsive interface of Hacker News with summaries and thumbnails.

An open source webapp for scraping: towards a public service for webscraping

一个轻量级、快速、多线程、多管道、灵活配置的网络爬虫。

百度贴吧爬虫(基于scrapy和mysql)

A set of spiders and scrapers to extract location information from places that post their location on the internet.

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

微信公众号爬虫，公众号历史文章，文章评论，文章阅读及在看数据，可视化web页面，可部署于Windows服务器。基于Python3之flask/mysql/redis/mitmproxy/pywin32等实现，高效微信爬虫，微信公众号爬虫，历史文章，文章评论，数据更新。

golang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。

*DEPRECATED* Please use https://gopkg.in/redsync.v1 (https://github.com/go-redsync/redsync)

🎊 Design and implement of lightweight crawler framework.

Every web site provides APIs.

A distributed task scheduling framework.（分布式任务调度平台XXL-JOB）