All Projects → actumn → celery.node

actumn / celery.node

Licence: MIT License
Celery task queue client/worker for nodejs

Programming Languages

typescript
32286 projects

Projects that are alternatives of or similar to celery.node

Celery
Distributed Task Queue (development branch)
Stars: ✭ 18,378 (+11106.1%)
Mutual labels:  amqp, task-runner, task-manager, queued-jobs, queue-workers
Qutee
PHP Background Jobs (Tasks) Manager
Stars: ✭ 63 (-61.59%)
Mutual labels:  queue, workers, background-jobs, task-runner, job-queue
Node Celery
Celery client for Node.js
Stars: ✭ 648 (+295.12%)
Mutual labels:  queue, amqp, background-jobs, celery
Rq
Simple job queues for Python
Stars: ✭ 8,065 (+4817.68%)
Mutual labels:  workers, background-jobs, job-queue, task-queue
orkid-node
Reliable and modern Redis Streams based task queue for Node.js 🤖
Stars: ✭ 61 (-62.8%)
Mutual labels:  queue, job-queue, task-queue, worker-queue
Honeydew
Job Queue for Elixir. Clustered or Local. Straight BEAM. Optional Ecto. 💪🍈
Stars: ✭ 670 (+308.54%)
Mutual labels:  queue, workers, background, job-queue
bikeshed
Lock free hierarchical work scheduler
Stars: ✭ 78 (-52.44%)
Mutual labels:  task-runner, task-manager, queued-jobs, queue-workers
Resque
Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.
Stars: ✭ 9,031 (+5406.71%)
Mutual labels:  queue, background-jobs, job-queue
Sidekiq Job Php
Push and schedule jobs to Sidekiq from PHP
Stars: ✭ 34 (-79.27%)
Mutual labels:  queue, worker, workers
leek
Celery Tasks Monitoring Tool
Stars: ✭ 77 (-53.05%)
Mutual labels:  queue, worker, celery
rqmonitor
Flask based more dynamic and actionable frontend dashboard for monitoring Redis Queue 👩🏿‍💻 http://python-rq.org
Stars: ✭ 152 (-7.32%)
Mutual labels:  workers, job-queue, task-queue
Toro
Multithreaded message processing on Postgres
Stars: ✭ 39 (-76.22%)
Mutual labels:  queue, workers, background-jobs
Workq
Job server in Go
Stars: ✭ 1,546 (+842.68%)
Mutual labels:  queue, worker, workers
dispatcher
Dispatcher is an asynchronous task queue/job queue based on distributed message passing.
Stars: ✭ 60 (-63.41%)
Mutual labels:  queue, amqp, celery
Simpleue
PHP queue worker and consumer - Ready for AWS SQS, Redis, Beanstalkd and others.
Stars: ✭ 124 (-24.39%)
Mutual labels:  queue, worker, workers
fastlane
Fastlane is a redis and docker based queueing service.
Stars: ✭ 48 (-70.73%)
Mutual labels:  queue, worker, task-manager
Django Rq
A simple app that provides django integration for RQ (Redis Queue)
Stars: ✭ 1,361 (+729.88%)
Mutual labels:  background-jobs, job-queue, task-queue
Flower
Real-time monitor and web admin for Celery distributed task queue
Stars: ✭ 5,036 (+2970.73%)
Mutual labels:  workers, celery, task-queue
Ytask
YTask is an asynchronous task queue for handling distributed jobs in golang(go异步任务框架)
Stars: ✭ 121 (-26.22%)
Mutual labels:  queue, worker, celery
filequeue
light weight, high performance, simple, reliable and persistent queue for Java applications
Stars: ✭ 35 (-78.66%)
Mutual labels:  queue, queued-jobs, queue-workers

Celery client / worker for in node.js
This project focuses on implementing task queue using celery protocol in node.js influenced by node-celery

What is a Task queue and Celery?

Task Queue

Task queue is a mechanism to distribute or dispatch "tasks" or "jobs" across "workers" or "machines" for executing them asynchronously.

Common use cases of task queue:

  • Video Encoding & Decoding
  • Resizing Pictures
  • Processing Bulk Updates
  • Any task which can be executed asynchronously
    image

Applications, also called as "Producers", "Publishers" register logical blocks of code as "tasks".
Workers, also called "Consumers" consume these "task" and optionally store any results to a "message backend".
The broker (task queue) receives tasks encapsulated as messages from "producers" and routes them to "consumers".

But managing messages is not as simple as storing them in a data sotre as aqueue.
Suppose that a number of messages sent and dispatched by large number of producers and workers.
We have to consider below.

  • Detecting poison messages
  • Ensuring reliability of the messaging sysstem
  • Scaling the messaging system

Celery

image

Celery is a one of most famous task queue open source software. A Celery system can consist of multiple workers and brokers, giving way to high availability and horizontal scaling.
The features of celery is

  • simple
  • highly available
  • fast
  • flexible

Celery is written in Python, but the protocol can be implemneted in any languages. There's gocelery for Go and like gocelery, here's celery.node.

Why celery.node?

image

We usually make programs using different languages because of the specific features of each language and sometimes the programs should be communicated with each others by task-queueing, such as python web application with go worker or nodejs worker for better performance.

We can make the programs distribute the tasks to processes written in different languages super easily by using celery, gocelery, and celery.node.

Also, you can use celery.node as pure nodejs task queue.

Protocol

celery protocol reference
Celery.node now supports Celery Message Protocol Version 1 and Version 2.

client.conf.TASK_PROTOCOL = 1; // 1 or 2. default is 2.

Install

$ npm install celery-node

Getting started

Client

celery.node

const celery = require('celery-node');

const client = celery.createClient(
  "amqp://",
  "amqp://"
);

const task = client.createTask("tasks.add");
const result = task.applyAsync([1, 2]);
result.get().then(data => {
  console.log(data);
  client.disconnect();
});

python

from celery import Celery

app = Celery('tasks',
    broker='amqp://',
    backend='amqp://'
)

@app.task
def add(x, y):
    return x + y

if __name__ == '__main__':
    result = add.apply_async((1, 2), serializer='json')
    print(result.get())

Worker

celery.node

const celery = require('celery-node');

const worker = celery.createWorker(
  "amqp://",
  "amqp://"
);
worker.register("tasks.add", (a, b) => a + b);
worker.start();

python

from celery import Celery

app = Celery('tasks',
    broker='amqp://',
    backend='amqp://'
)

@app.task
def add(x, y):
    return x + y
$ celery worker -A tasks --loglevel=INFO

Run Example

Ensure you already installed nodejs, npm, and docker-compose

# Generate dist/ directory, tutorial files depend on it
$ npm run dist

# start a docker container rabbitmq in the background
$ docker-compose -f examples/docker-compose.yml up -d rabbit

# run celery.node client with rabbitmq
$ node examples/tutorial/client.js

# run celery.node worker with rabbitmq
# when you run worker, you can see the result printed out from client
$ node examples/tutorial/worker.js

# stop and remove containers
$ docker-compose -f examples/docker-compose.yml down

Contributing

Before contributing, please read contributing.md
Let's make this project more awesome!

License

  • MIT
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].