orkid

Reliable and modern Redis-Streams based task queue for Node.js.

Screenshot

Table of Contents

• Features
• Requirements
• Install
• Examples
• API Documentation
• Monitoring and Management UI/Admin Panel
• FAQ

Features

• Orkid lets Redis do the heavy lifting with Redis-Streams.
• Adjustable concurrency per consumer instance for scaling task processing. See example code. See example code.
• Job timeouts and retries. All configurable per consumer. See example code.
• Task Deduplication. If a task is already waiting in the queue, it can be configured to avoid queueing the same task again. (Useful for operations like posting database record updates to elasticsearch for re-indexing. Deduplication is a common pattern here to avoid unnecessary updates). See example code.
• Add tasks in bulk in one call. The producer will handle chunking and optimize round-trips to redis using pipelining.
• Monitoring and management UI for better visibility.
• Rate-limiting workers. (work in progress)

Requirements

• Node.js >= 10
• Redis >= 5

👏 Important: Redis-Streams feature is not available before Redis version 5.

Install

npm install orkid --save

Examples

Basic example of producing and consuming tasks:

Producing tasks:

const { Producer } = require('orkid');

// `basic` is the queue name here
//  We'll use the same name in the consumer to process this task
const producer = new Producer('basic');

async function addTasks() {
  for (let i = 0; i < 10; i++) {
    console.log('Adding task ', i);
    await producer.addTask(i);
  }
}

addTasks()
  .then(() => {
    console.log('Done');
    process.exit(); // To disconnect from redis
  })
  .catch(e => console.error(e));

Consuming tasks:

const { Consumer } = require('orkid');

// Worker Function
async function workerFn(data, metadata) {
  let result;
  /*
    Do operation on `data` here
    and store the result in `result` variable

    Anything you return from this function will
    be saved in redis and can be viewed in the Orkid UI.

    Returning nothing is fine too.

    Throwing error will mark the job as failed,
    which can be retried too.
  */

  console.log('Task done!');
  return result;
}

// Consume task from the `basic` queue
const consumer = new Consumer('basic', workerFn);

// Start processing tasks!
// Important: Until you call this method, orkid consumer will do nothing.
consumer.start();

👏 More examples are available in the ./examples directory, including how to do task de-duplication, retry on failure, timeout etc. 👏

API Documentation

API Documentation is available here.

Monitoring and Management UI/Admin Panel

You need to run orkid-ui separately for the dashboard. Detail instructions on how to run orkid-ui locally or in production using docker/docker-compose can be found here: https://github.com/mugli/orkid-ui#running-locally

Task/job life-cycle

[TODO: Add a flowchart here]

FAQ

Maintainer(s)

• Mehdi Hasan Khan (@MehdiHK)

License

MIT

Related Projects

• orkid-ui: Dashboard to monitor and manage Orkid task queue.
• orkid-api: GraphQL API to monitor and manage Orkid task queue (used internally by orkid-ui).