1824 Open source projects that are alternatives of or similar to Scrapy Redis

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

🥄 A package for building specific Proxy Pool for different Sites.

HTTP API for Scrapy spiders

腾讯新闻、知乎话题、微博粉丝，Tumblr爬虫、斗鱼弹幕、妹子图爬虫、分布式设计等

Simple but useful Python web scraping tutorial code.

Parser and database to index the terpene profile of different strains of Cannabis from online databases

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!

a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine

ScrapingOutsourcing专注分享爬虫代码 尽量每周更新一个

基于微博的数据挖掘与社交舆情分析

This is a sina weibo spider built by scrapy [微博爬虫/持续维护]

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

A Facebook crawler

用严肃的数据来回答“什么样的企业会到什么样的大学招聘”？

💐 Full Amazon Automatic Download

Scrapy-based Crawlers for news of Taiwan

Life is short, *****.

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

A lite distributed Java spider framework :-)

Python爬虫代理IP池(proxy pool)

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

Github 仓库及用户分析爬虫

Dynamic file detection tool based on crawler 基于爬虫的动态敏感文件探测工具

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

慕课网python分布式爬虫源码-长期更新维护

Crawlera middleware for Scrapy

*DEPRECATED* Please use https://gopkg.in/redsync.v1 (https://github.com/go-redsync/redsync)

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

基于搜狗微信搜索的微信公众号爬虫接口

A multi-thread crawler framework with many builtin image crawlers provided.

豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章

Scrapy as a Web Crawler for Azure Search Samples

Distributed Image Search Engine Crawler

Android应用市场网络爬虫

POOPAK - TOR Hidden Service Crawler

Some scrapy and web.py exmaples

A framework for creating semi-automatic web content extractors

Spring Boot 脚手架 Mybatis Spring Security JWT 权限 Spring Cache + Redis

基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论等，并且分享了QQ音乐中排名前6400名的内地和港台歌手的49万+的音乐语料

《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题 ；HTTP协议介绍； Requests使用 ；解析器Xpath介绍； MongoDB与MySQL； 多线程爬虫； Scrapy介绍 ；Scrapy-redis介绍； 使用docker部署； 使用nomad管理docker集群； 使用EFK查询docker日志

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

scrapy专利爬虫（停止维护）

💐Marmot | Web Crawler/HTTP protocol Download Package 🐭

swiss army knife for hackers

crawler framework, distributed crawler extractor

爬虫, http代理, 模拟登陆!

PTT 網路版爬蟲

NScrapy is a .net core corss platform Distributed Spider Framework which provide an easy way to write your own Spider

Redisson - Redis Java client with features of In-Memory Data Grid. Over 50 Redis based Java objects and services: Set, Multimap, SortedSet, Map, List, Queue, Deque, Semaphore, Lock, AtomicLong, Map Reduce, Publish / Subscribe, Bloom filter, Spring Cache, Tomcat, Scheduler, JCache API, Hibernate, MyBatis, RPC, local cache ...

Distributed crawling/scraping, Kafka And Redis based components for Scrapy

Distributed WebSocket broker

Redis for humans. 🌎🌍🌏

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

码云仓库链接:AJay13/ECommerceCrawlers Github 仓库链接:DropsDevopsOrg/ECommerceCrawlers 项目展示平台链接:http://wechat.doonsec.com

这是一个支持分布式和集群的java游戏服务器框架，可用于开发棋牌、回合制等游戏。基于netty实现高性能通讯，支持tcp、http、websocket等协议。支持消息加解密、攻击拦截、黑白名单机制。封装了redis缓存、mysql数据库的连接与使用。轻量级，便于上手。

Simplified distributed locking implementation using Redis