582 Open source projects that are alternatives of or similar to scrapy-kafka-redis

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Redis-based components for Scrapy.

NScrapy is a .net core corss platform Distributed Spider Framework which provide an easy way to write your own Spider

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

A Go implementation of CITA. https://docs.nervos.org/cita

Task queue, Long lived workers for work based parallelization, with processes and Redis as back-end. For distributed computing.

Distributed, Column-oriented storage, Realtime analysis, High performance Database

Browser computing unleashed!

Common interface for data container classes

高可用，高性能的分布式发号服务

A prototype for distributed training/validation/evaluation/extraction with PyTorch.

GraviT is a distributed ray tracing framework that enables applications to leverage hardware-optimized ray tracers within a single environment across many nodes for large-scale rendering tasks.

Alpha implementation of the Sprawl distributed marketplace protocol.

Iota is a web scraper which can find all of the images and links/suburls on a webpage

An electronic health record (EHR) system built on Hyperledger Composer blockchain

A peer-to-peer networking framework to work across languages

Distributed Honeypot

CLI for orbit-db

scrapy mysql pipeline

C++ development framework for building reliable cache-friendly distributed and concurrent multicore software

My code for scraping financial data in Vietnam

Analyze scraped data

A peer-to-peer decentralized database

Distributed SQL Engine in Python using Dask

Araignée LeBonCoin avec Scrapy et ElasticSearch

Distributed Operations and Data Organizer built on Apache BookKeeper

FedScale is a scalable and extensible open-source federated learning (FL) platform.

Distributed tensors and Machine Learning framework with GPU and MPI acceleration in Python

使用asyncio和aiohttp开发的轻量级异步协程web爬虫框架

Classificador de poemas do Fernando Pessoa de acordo com os seus heterônimos

Xmutca-rpc是一个基于netty开发的分布式服务框架，提供稳定高性能的RPC远程服务调用功能，支持注册中心，服务治理，负载均衡等特性，开箱即用。

Crawling zhihu, jobbole, lagou by Scrapy, and using Elasticsearch+Django to build a Search Engine website --- README_zh.md (including: implementation roadmap, distributed-crawler and coping with anti-crawling strategies).

Your next actor system

Credits(CRDS) - An Evolving Currency For An Evolving Society

Dynamic configurable crawl (动态可配置化爬虫)

Java基于Netty,Protostuff和Zookeeper实现分布式RPC框架

The Federated Social Audio Platform

Galaxy is an asynchronous parallel visualization ray tracer for performant rendering in distributed computing environments. Galaxy builds upon Intel OSPRay and Intel Embree, including ray queueing and sending logic inspired by TACC GraviT.

A scrapy middleware to use rotated proxy ip list.

A developing game server framework,based on Entity Component System(ECS).

Ape-X DQN & DDPG with pytorch & tensorboard

Inventus is a spider designed to find subdomains of a specific domain by crawling it and any subdomains it discovers.

A toolkit for rate limite,support memory and redis

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

Using scrapy to scrape tripadvisor in order to get users' reviews.

Rust Implementation of Erlang Distribution Protocol

A Distributed ResNet on multi-machines each with one GPU card.

基于区块链的符合W3C DID和Verifiable Credential规范的分布式身份解决方案

python+scrapy+splash 爬取拉勾全站职位信息

Scrapy downloader middleware that stores response HTMLs to disk.

Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

TiKV based Pub/Sub server

World’s single largest Internet domains dataset

Personal data silos with secure sharing

Hazelcast .NET Client

Crawling the data from lazada, websosanh, compare.vn, cdiscount and cungmua with flexible configs

Open Source, Google Zanzibar-inspired fine-grained permissions database

WebHunger is an extensible, full-scale crawler framework that supports distributed crawling, aiming at getting users focused on web page parsing without concerning for the crawling process.

FastNN provides distributed training examples that use EPL.