Lightflow - a lightweight, distributed workflow system

Lightflow is a Python 3.5+ library and command-line tool for executing workflows, composed of individual tasks, in a distributed fashion. It is based on Celery and provides task dependencies, data exchange between tasks and an intuitive description of workflows.

Dependencies

Python

Lightflow is written in Python 3 and requires Python 3.5 or higher.

Operating system

Lightflow is being developed and tested on Linux, with Debian and RedHat being the main platforms.

redis

The redis database is required by Lightflow as a communication broker between tasks. It is also used as the default broker for the Celery queuing system, but could be replaced with any other supported Celery broker.

MongoDB

Lightflow makes use of MongoDB for storing persistent data during a workflow run that can be accessed by all tasks.

Getting started

The following getting started guide assumes a redis database running on localhost and port 6379 as well as a MongoDB database running on localhost and port 27017.

Install Lightflow from PyPi:

pip install lightflow

Create a default configuration file and copy the provided example workflows to a local directory of your choice:

lightflow config default .
lightflow config examples .

If you like, list all available example workflows:

lightflow workflow list

In order to execute a workflow, start a worker that consumes jobs from the workflow, dag and task queues. Then start a workflow from the list of available examples. The following example starts the workflow simple:

lightflow worker start
lightflow workflow start simple