tonywangcn / Scaleable Crawler With Docker Cluster
a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine
Stars: ✭ 96
Programming Languages
python
139335 projects - #7 most used programming language
Projects that are alternatives of or similar to Scaleable Crawler With Docker Cluster
Broccoli
Broccoli - distributed task queues for ESP32 cluster
Stars: ✭ 280 (+191.67%)
Mutual labels: celery, distributed, cluster
Jlitespider
A lite distributed Java spider framework :-)
Stars: ✭ 151 (+57.29%)
Mutual labels: crawler, rabbitmq, distributed
Docker Cluster With Celery And Rabbitmq
Build Docker clusters with Celery and RabbitMQ in 10 minutes
Stars: ✭ 72 (-25%)
Mutual labels: rabbitmq, celery, cluster
Haipproxy
💖 High available distributed ip proxy pool, powerd by Scrapy and Redis
Stars: ✭ 4,993 (+5101.04%)
Mutual labels: crawler, distributed
Flower
Real-time monitor and web admin for Celery distributed task queue
Stars: ✭ 5,036 (+5145.83%)
Mutual labels: rabbitmq, celery
Remit
RabbitMQ-backed microservices supporting RPC, pubsub, automatic service discovery and scaling with no code changes.
Stars: ✭ 24 (-75%)
Mutual labels: rabbitmq, distributed
Crate
CrateDB is a distributed SQL database that makes it simple to store and analyze
massive amounts of data in real-time.
Stars: ✭ 3,254 (+3289.58%)
Mutual labels: distributed, cluster
Xxl Crawler
A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）
Stars: ✭ 561 (+484.38%)
Mutual labels: crawler, distributed
Disec
Distributed Image Search Engine Crawler
Stars: ✭ 11 (-88.54%)
Mutual labels: crawler, distributed
Nebula
Nebula is a powerful framwork for building highly concurrent, distributed, and resilient message-driven applications for C++.
Stars: ✭ 385 (+301.04%)
Mutual labels: distributed, cluster
Scrapy Redis
Redis-based components for Scrapy.
Stars: ✭ 4,998 (+5106.25%)
Mutual labels: crawler, distributed
Letsmapyournetwork
Lets Map Your Network enables you to visualise your physical network in form of graph with zero manual error
Stars: ✭ 305 (+217.71%)
Mutual labels: rabbitmq, celery
Tossit
Library for distributed job/worker logic.
Stars: ✭ 56 (-41.67%)
Mutual labels: rabbitmq, distributed
Python Devops
gathers Python stack for DevOps, these are usually my basic templates use for my implementations, so, feel free to use it and evolve it! Everything is Docker!
Stars: ✭ 61 (-36.46%)
Mutual labels: rabbitmq, celery
Clearly
Clearly see and debug your celery cluster in real time!
Stars: ✭ 287 (+198.96%)
Mutual labels: celery, distributed
scaleable-crawler-with-docker-cluster
a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine Check out here for full document
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].