All Projects → long2ice → rearq

long2ice / rearq

Licence: Apache-2.0 license
A distributed task queue built with asyncio and redis, with built-in web interface

Programming Languages

python
139335 projects - #7 most used programming language
HTML
75241 projects
CSS
56736 projects

Projects that are alternatives of or similar to rearq

Node Rethinkdb Job Queue
A persistent job or task queue backed by RethinkDB.
Stars: ✭ 158 (+95.06%)
Mutual labels:  task, queue, distributed
Fennel
A task queue library for Python and Redis
Stars: ✭ 24 (-70.37%)
Mutual labels:  task, queue, asyncio
Arq
Fast job queuing and RPC in python with asyncio and redis.
Stars: ✭ 695 (+758.02%)
Mutual labels:  queue, distributed, asyncio
Clearly
Clearly see and debug your celery cluster in real time!
Stars: ✭ 287 (+254.32%)
Mutual labels:  task, queue, distributed
leek
Celery Tasks Monitoring Tool
Stars: ✭ 77 (-4.94%)
Mutual labels:  task, queue, distributed
Node Celery
Celery client for Node.js
Stars: ✭ 648 (+700%)
Mutual labels:  task, queue
Inc.runtime.queue
An runtime queue use Asynchronous program
Stars: ✭ 19 (-76.54%)
Mutual labels:  task, queue
Je
A distributed job execution engine for the execution of batch jobs, workflows, remediations and more.
Stars: ✭ 30 (-62.96%)
Mutual labels:  task, distributed
Ytask
YTask is an asynchronous task queue for handling distributed jobs in golang(go异步任务框架)
Stars: ✭ 121 (+49.38%)
Mutual labels:  task, queue
linda
Linda is a simple dispatcher library.
Stars: ✭ 12 (-85.19%)
Mutual labels:  task, queue
Rq
Simple job queues for Python
Stars: ✭ 8,065 (+9856.79%)
Mutual labels:  task, rq
Task Easy
A simple, customizable, and lightweight priority queue for promises.
Stars: ✭ 244 (+201.23%)
Mutual labels:  task, queue
Machinery
Machinery is an asynchronous task queue/job queue based on distributed message passing.
Stars: ✭ 5,821 (+7086.42%)
Mutual labels:  task, queue
Xxl Job
A distributed task scheduling framework.(分布式任务调度平台XXL-JOB)
Stars: ✭ 20,197 (+24834.57%)
Mutual labels:  task, distributed
horse-messaging
Open Source Messaging Framework. Queues, Channels, Events, Transactions, Distributed Cache
Stars: ✭ 65 (-19.75%)
Mutual labels:  queue, distributed
qless-php
PHP Bindings for qless
Stars: ✭ 25 (-69.14%)
Mutual labels:  task, queue
aioScrapy
基于asyncio与aiohttp的异步协程爬虫框架 欢迎Star
Stars: ✭ 34 (-58.02%)
Mutual labels:  queue, asyncio
Loafer
Asynchronous message dispatcher - Currently using asyncio and amazon SQS
Stars: ✭ 104 (+28.4%)
Mutual labels:  queue, asyncio
coyotes
非常简单的异步命令执行队列
Stars: ✭ 24 (-70.37%)
Mutual labels:  task, queue
flask-redis-docker
A minimal template for dockerized flask app with redis task queue
Stars: ✭ 49 (-39.51%)
Mutual labels:  queue, rq

ReArq

image image image image

Introduction

ReArq is a distributed task queue with asyncio and redis, which rewrite from arq to make improvement and include web interface.

You can try Demo Online here.

Screenshots

dashboard worker task job result

Requirements

  • Redis >= 5.0

Install

Use MySQL backend:

> pip install rearq[mysql]

Use PostgreSQL backend:

> pip install rearq[postgres]

Quick Start

Task Definition

# main.py
from rearq import ReArq

rearq = ReArq(db_url='mysql://root:[email protected]:3306/rearq')


@rearq.on_shutdown
async def on_shutdown():
    # you can do some clean work here like close db and so on...
    print("shutdown")


@rearq.on_startup
async def on_startup():
    # you should do some initialization work here
    print("startup")


@rearq.task(queue="q1")
async def add(self, a, b):
    return a + b


@rearq.task(cron="*/5 * * * * * *")  # run task per 5 seconds
async def timer(self):
    return "timer"

Run rearq worker

> rearq main:rearq worker -q q1 -q q2 # consume tasks from q1 and q2 as the same time
2021-03-29 09:54:50.464 | INFO     | rearq.worker:_main:95 - Start worker success with queue: rearq:queue:default
2021-03-29 09:54:50.465 | INFO     | rearq.worker:_main:96 - Registered tasks: add, sleep, timer_add
2021-03-29 09:54:50.465 | INFO     | rearq.worker:log_redis_info:86 - redis_version=6.2.1 mem_usage=1.43M clients_connected=5 db_keys=6

Run rearq timer

If you have timing task or delay task, you should run another command also:

> rearq main:rearq timer
2021-03-29 09:54:43.878 | INFO     | rearq.worker:_main:275 - Start timer success
2021-03-29 09:54:43.887 | INFO     | rearq.worker:_main:277 - Registered timer tasks: timer_add
2021-03-29 09:54:43.894 | INFO     | rearq.worker:log_redis_info:86 - redis_version=6.2.1 mem_usage=1.25M clients_connected=2 db_keys=6

Integration in FastAPI

from fastapi import FastAPI

app = FastAPI()


@app.on_event("startup")
async def startup() -> None:
    await rearq.init()


@app.on_event("shutdown")
async def shutdown() -> None:
    await rearq.close()


# then run task in view
@app.get("/test")
async def test():
    job = await add.delay(args=(1, 2))
    # or
    job = await add.delay(kwargs={"a": 1, "b": 2})
    # or
    job = await add.delay(1, 2)
    # or
    job = await add.delay(a=1, b=2)
    result = await job.result(timeout=5)  # wait result for 5 seconds
    print(result.result)
    return result

Start web interface

> rearq main:rearq server
Usage: rearq server [OPTIONS]

  Start rest api server.

Options:
  --host TEXT         Listen host.  [default: 0.0.0.0]
  -p, --port INTEGER  Listen port.  [default: 8000]
  -h, --help          Show this message and exit..

After server run, you can visit https://127.0.0.1:8000/docs to see all apis and https://127.0.0.1:8000 to see web interface.

Other options will pass into uvicorn directly, such as --root-path etc.

rearq main:rearq server --host 0.0.0.0 --root-path /rearq

ThanksTo

  • arq, Fast job queuing and RPC in python with asyncio and redis.

License

This project is licensed under the Apache-2.0 License.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].