fastlane-queue / fastlane

fastlane service

fastlane is a redis-based queueing service that outsmarts everyone else by using containers.

More seriously, though, fastlane allows you to easily implement new workers in the form of containers.

Instead of the tedious, repetitive work of yesteryear where you had to implement a worker in language X or Y, you just spin a new container with all the dependencies you require already previously installed, and instruct fastlane to run a command in that container. Bang! Instant Super-Powered Workers!

Features

• Ad-Hoc execution of jobs (run job right now)
• Scheduled execution of jobs (run job next sunday at 6am, or run in 10 minutes from now);
• Crontab execution of jobs (run job at */10 * * * * - every ten minutes);
• Allows job details to be updated;
• API to retrieve tasks, /tasks;
• API to retrieve task details, /tasks/my-task (<taskUrl>);
• API to retrieve job details, <taskUrl>/jobs/<jobId> (<jobUrl>);
• API to stop running job (<jobUrl>/stop);
• API to retry job (<jobUrl>/retry);
• API to get logs(<jobUrl>/logs), stdout (<jobUrl>/stdout) and stderr (<jobUrl>/stderr) for last execution in jobs;
• Job log output streaming using WebSockets (ws://<jobUrl>/ws) and <jobUrl>/stream;
• API to retrieve execution details, <jobUrl>/executions/<executionId> (<executionUrl>);
• API to stop execution, <executionUrl>/stop;
• API to get logs(<executionUrl>/stdout), stdout (<executionUrl>/stdout) and stderr (<executionUrl>/stderr) for execution;
• Job execution log output streaming using WebSockets (ws://<executionUrl>/ws) and <executionUrl>/stream;
• Additional Job Metadata (useful for webhooks);
• Configurable retries per job;
• Configurable exponential back-off for retries and failures in monitoring of jobs;
• Configurable hard timeout for each execution;
• E-mail subscription to tasks;
• Healthcheck of the MongoDB and Redis Connections;
• Web hooks on job start;
• Web hooks on job completion;
• Redact any env that contains blacklisted keywords;
• Exponential back-off parameters per job;
• Self-healing handling of interrupted jobs;
• Self-healing handling of unscheduled periodical jobs;
• Workers should handle SIGTERM and exit gracefully;
• Docker Container Runner (with docker host pool);
• Docker Pool per task name (Regular Expressions);
• Rename docker containers after processing their details;
• Command to prune processed containers;
• Routes to remove/put back docker host in job balancing;
• Docker SSL connections;
• Circuit breaking when docker host is unavailable;
• Container Environment Variables per Job;
• Configurable global limit for number of running jobs per task name (Regular Expressions);
• Limit of concurrent job executions per task;
• Kubernetes Container Runner;
• MongoDB Task and Job Storage;
• Structured Logging;
• Monitoring of job completion;
• Job Expiration;
• Status Page with details on the farm status (executors, scheduled tasks and queue sizes);
• Error handling mechanism (Sentry built-in, extensible)
• Per-job Error handling mechanism (Sentry built-in, extensible)
• Usage metrics (extensible);
• Support Redis and Redis Sentinel;
• Support Redis Cluster;
• Comprehensive test coverage;
• CORS headers in every API request (configurable);
• Basic Auth support;
• gzip all JSON responses for the API (for requests that accept gzip);
• Store IP address of enqueued job for auditing (X-Real-IP, then X-Forwarded-For, then request.addr);
• Admin to inspect tasks and jobs.

Getting started

Getting fastlane up and running is very simple if you have both docker and docker-compose installed.

We'll use a sample docker compose that gets all our requirements up (ps: this docker-compose file runs Docker In Docker and requires privileged mode to run):

$ curl https://raw.githubusercontent.com/fastlane-queue/fastlane/master/docker-compose-sample.yml | docker-compose -f - up -d

Starting fastlane...
Creating fastlane_mongo_1       ... done
Creating fastlane_docker-host_1 ... done
Creating fastlane_redis_1       ... done
Creating fastlane_fastlane_1    ... done
fastlane started successfully.

After that you can start using fastlane. For more details on getting started, read the following page.

Documentation

Read more about fastlane at read the docs.

Contributing

Logo was created by Artur Sousa.

License

Fastlane is licensed under the MIT license.