2016 Open source projects that are alternatives of or similar to Scrapy Cluster

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

分布式架构开发套件。包括缓存(一二级缓存、自动缓存管理)、队列、分布式定时任务、文件服务(七牛、阿里云OSS、fastDFS)、日志、搜索、分布式锁、分布式事务、集成dubbo、spring boot支持以及常用的工具包等。

Redis-based components for Scrapy.

Example end to end data engineering project.

Spin up Tor containers and then proxy HTTP requests via these Tor instances

深圳地铁大数据客流分析系统🚇🚄🌟

Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.

*DEPRECATED* Please use https://gopkg.in/redsync.v1 (https://github.com/go-redsync/redsync)

Surging is a micro-service engine that provides a lightweight, high-performance, modular RPC request pipeline. The service engine supports http, TCP, WS,Grpc, Thrift,Mqtt, UDP, and DNS protocols. It uses ZooKeeper and Consul as a registry, and integrates it. Hash, random, polling, Fair Polling as a load balancing algorithm, built-in service gove…

Zenko is the open source multi-cloud data controller: own and keep control of your data on any cloud.

Build and deploy a machine learning app from scratch 🚀

spring cloud 版分布式电商项目，全力打造顶级多模块，高可用，高扩展电商项目

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

Simple and friendly Bot for Instagram, using Selenium and Scrapy with Python.

Distributed crawling/scraping, Kafka And Redis based components for Scrapy

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

Crawlera middleware for Scrapy

基于Gradle构建，使用SpringBoot在各个场景的应用，包括集成消息中间件、前后端分离、数据库、缓存、分布式锁、分布式事务等

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

🎉🎉🎉JAVA高级架构师技术栈==任何技能通过 “刻意练习” 都可以达到融会贯通的境界，就像烹饪一样，这里有一份JAVA开发技术手册，只需要增加自己练习的次数。🏃🏃🏃

spring、spring-boot、spring-cloud 常用整合用例

SpringBoot框架源码实战（已更新到springboot2版本实现）~基本用法，Rest，Controller，事件监听，连接数据库MySQL，jpa，redis集成，mybatis集成（声明式与xml两种方式~对应的添删查改功能），日志处理，devtools配置，拦截器用法，资源配置读取，测试集成，Web层实现请求映射，security安全验证，rabbitMq集成，kafka集成，分布式id生成器等。项目实战：https://github.com/hemin1003/yfax-parent 已投入生产线上使用

✍️ Java 工程师必备架构体系知识总结：涵盖分布式、微服务、RPC等互联网公司常用架构，以及数据存储、缓存、搜索等必备技能

Container auto-configurations for spring-boot based integration tests

dnc 去中心化 开源社区 轻联盟 dncto.com QQ群 779699538

java-study 是本人学习Java过程中记录的一些代码！从Java基础的数据类型、jdk1.8的Lambda、Stream和日期的使用、 IO流、数据集合、多线程使用、并发编程、23种设计模式示例代码、常用的工具类， 以及一些常用框架，netty、mina、springboot、kafka、storm、zookeeper、redis、elasticsearch、hbase、hive等等。

syncClient，数据实时同步中间件（同步mysql到kafka、redis、elasticsearch、httpmq）！

以教学为目的的电商系统。包含ToB复杂业务、互联网高并发业务、缓存应用；DDD、微服务指导。模型驱动、数据驱动。了解大型服务进化路线，编码技巧、学习Linux，性能调优。Docker/k8s助力、监控、日志收集、中间件学习。前端技术、后端实践等。主要技术：SpringBoot+JPA+Mybatis-plus+Antd+Vue3。

With Selenium headless browsing and CAPTCHA solving

⚡️ Nepxion Thunder is a distribution RPC framework based on Netty + Hessian + Kafka + ActiveMQ + Tibco + Zookeeper + Redis + Spring Web MVC + Spring Boot + Docker 多协议、多组件、多序列化的分布式RPC调用框架

NScrapy is a .net core corss platform Distributed Spider Framework which provide an easy way to write your own Spider

A Scrapy extension to log items coverage when the spider shuts down

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

我的个人技术博客（Python、Django、Docker、Go、Redis、ElasticSearch、Kafka、Linux）

Building a model to recognize incentives for landscape restoration in environmental policies from Latin America, the US and India. Bringing NLP to the world of policy analysis through an extensible framework that includes scraping, preprocessing, active learning and text analysis pipelines.

API for scrapping common meme sites

Redisson - Redis Java client with features of In-Memory Data Grid. Over 50 Redis based Java objects and services: Set, Multimap, SortedSet, Map, List, Queue, Deque, Semaphore, Lock, AtomicLong, Map Reduce, Publish / Subscribe, Bloom filter, Spring Cache, Tomcat, Scheduler, JCache API, Hibernate, MyBatis, RPC, local cache ...

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

这是一个支持分布式和集群的java游戏服务器框架，可用于开发棋牌、回合制等游戏。基于netty实现高性能通讯，支持tcp、http、websocket等协议。支持消息加解密、攻击拦截、黑白名单机制。封装了redis缓存、mysql数据库的连接与使用。轻量级，便于上手。

Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy

全栈工程师手册

♒︎ [WIP] An experimental ~distributed~ commit-log

Test your code without writing mocks with ephemeral Docker containers 📦 Setup popular services with just a couple lines of code ⏱️ No bash, no yaml, only code 💻

基于 scrapy-redis 的通用分布式爬虫框架

Spring Boot 学习教程

Web tool for Kafka Connect |

A framework for creating semi-automatic web content extractors

A lightweight stream processing library for Go

C++ Parallel Computing and Asynchronous Networking Engine

豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章

A cohesive & pragmatic framework of FP centric Scala libraries

🐂基于springcloud +dubbo构建的模拟秒杀项目，模块化设计，集成了分库分表、elasticsearch🔍、gateway、mybatis-plus、spring-session等常用开源组件

Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索

Simple but useful Python web scraping tutorial code.

Fast job queuing and RPC in python with asyncio and redis.

istio 微服务示例代码 grpc+protobuf+echo+websocket+mysql+redis+kafka+docker-compose

Serverless Platform for the stateful services

Simplified distributed locking implementation using Redis

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

Redis distributed locks in PHP