582 Open source projects that are alternatives of or similar to scrapy-kafka-redis

An open source framework that provides a simple, universal API for building distributed applications. Ray is packaged with RLlib, a scalable reinforcement learning library, and Tune, a scalable hyperparameter tuning library.

Distributed tensors and Machine Learning framework with GPU and MPI acceleration in Python

Blockchain-based marketplace for file storage. Project has moved to GitLab: https://gitlab.com/NebulousLabs/Sia

使用asyncio和aiohttp开发的轻量级异步协程web爬虫框架

Spring Boot 脚手架 Mybatis Spring Security JWT 权限 Spring Cache + Redis

Classificador de poemas do Fernando Pessoa de acordo com os seus heterônimos

Your Gateway to the Distributed Web

Xmutca-rpc是一个基于netty开发的分布式服务框架，提供稳定高性能的RPC远程服务调用功能，支持注册中心，服务治理，负载均衡等特性，开箱即用。

Brain Imaging Analysis Kit

Crawling zhihu, jobbole, lagou by Scrapy, and using Elasticsearch+Django to build a Search Engine website --- README_zh.md (including: implementation roadmap, distributed-crawler and coping with anti-crawling strategies).

☁️ Nextcloud server, a safe home for all your data

Your next actor system

A distributed MQTT message broker based on Erlang/OTP. Built for high quality & Industrial use cases.

Credits(CRDS) - An Evolving Currency For An Evolving Society

Dynamic configurable crawl (动态可配置化爬虫)

The modular distributed fingerprinting engine

Java基于Netty,Protostuff和Zookeeper实现分布式RPC框架

Simple Pose: Rethinking and Improving a Bottom-up Approach for Multi-Person Pose Estimation

The Federated Social Audio Platform

Distributed web chat application base websocket built on akka.

Galaxy is an asynchronous parallel visualization ray tracer for performant rendering in distributed computing environments. Galaxy builds upon Intel OSPRay and Intel Embree, including ray queueing and sending logic inspired by TACC GraviT.

Distributed WebSocket broker

A scrapy middleware to use rotated proxy ip list.

Distributed compute platform implemented in Rust, and powered by Apache Arrow.

A developing game server framework,based on Entity Component System(ECS).

A JVM-embeddable Distributed Database

Ape-X DQN & DDPG with pytorch & tensorboard

book.zi5.me全站kindle电子书籍爬取，按照作者书籍名分类，每本书有mobi和equb两种格式，采用分布式进行全站爬取

Inventus is a spider designed to find subdomains of a specific domain by crawling it and any subdomains it discovers.

🚀构建分布式即时聊天、消息推送系统。 Building distributed instant messaging, push notification systems.

A toolkit for rate limite,support memory and redis

A privacy-aware, distributed, open source social network.

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

Distributed Keras Engine, Make Keras faster with only one line of code.

Using scrapy to scrape tripadvisor in order to get users' reviews.

🥄 A package for building specific Proxy Pool for different Sites.

Rust Implementation of Erlang Distribution Protocol

A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.

A Distributed ResNet on multi-machines each with one GPU card.

JavaScript implementation for Distributed Object Protocol

基于区块链的符合W3C DID和Verifiable Credential规范的分布式身份解决方案

A modern, distributed monitoring system written in Go

python+scrapy+splash 爬取拉勾全站职位信息

Scrapy downloader middleware that stores response HTMLs to disk.

Ansible Role - Git

Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

🌊 .NET Standard 2.1 framework which makes easy to scaffold distributed systems and dispatch incoming load into units of work in a deterministic way.

TiKV based Pub/Sub server

Deploy Dask on job schedulers like PBS, SLURM, and SGE

Personal data silos with secure sharing

Web-based and pixels-based collaborative synthesizer

Hazelcast .NET Client

Server

FastNN provides distributed training examples that use EPL.

The multi-node setup of TimescaleDB 🐯🐯🐯 🐘 🐯🐯🐯

Scraping the real estate agencies for up-to-date house listings as soon as they arrive!

WebHunger is an extensible, full-scale crawler framework that supports distributed crawling, aiming at getting users focused on web page parsing without concerning for the crawling process.

With Selenium headless browsing and CAPTCHA solving

pip install funboost，python全功能分布式函数调度框架,。支持python所有类型的并发模式和全球一切知名消息队列中间件，python函数加速器，框架包罗万象，一统编程思维，兼容50% python编程业务场景，适用范围广。只需要一行代码即可分布式执行python一切函数。旧名字是function_scheduling_distributed_framework