All Projects → kodless → leek

kodless / leek

Licence: Apache-2.0 license
Celery Tasks Monitoring Tool

Programming Languages

typescript
32286 projects
python
139335 projects - #7 most used programming language
HCL
1544 projects
javascript
184084 projects - #8 most used programming language
groovy
2714 projects
Dockerfile
14818 projects

Projects that are alternatives of or similar to leek

Clearly
Clearly see and debug your celery cluster in real time!
Stars: ✭ 287 (+272.73%)
Mutual labels:  task, monitor, queue, distributed, celery, monitoring-tool
Ytask
YTask is an asynchronous task queue for handling distributed jobs in golang(go异步任务框架)
Stars: ✭ 121 (+57.14%)
Mutual labels:  task, queue, worker, celery
Node Celery
Celery client for Node.js
Stars: ✭ 648 (+741.56%)
Mutual labels:  task, queue, rabbitmq, celery
celery-monitor
The celery monitor app was written by Django.
Stars: ✭ 92 (+19.48%)
Mutual labels:  monitor, distributed, celery
Node Rethinkdb Job Queue
A persistent job or task queue backed by RethinkDB.
Stars: ✭ 158 (+105.19%)
Mutual labels:  task, queue, distributed
Tossit
Library for distributed job/worker logic.
Stars: ✭ 56 (-27.27%)
Mutual labels:  rabbitmq, worker, distributed
Scaleable Crawler With Docker Cluster
a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine
Stars: ✭ 96 (+24.68%)
Mutual labels:  rabbitmq, distributed, celery
qless-php
PHP Bindings for qless
Stars: ✭ 25 (-67.53%)
Mutual labels:  task, queue, worker
celery.node
Celery task queue client/worker for nodejs
Stars: ✭ 164 (+112.99%)
Mutual labels:  queue, worker, celery
Arq
Fast job queuing and RPC in python with asyncio and redis.
Stars: ✭ 695 (+802.6%)
Mutual labels:  queue, worker, distributed
rearq
A distributed task queue built with asyncio and redis, with built-in web interface
Stars: ✭ 81 (+5.19%)
Mutual labels:  task, queue, distributed
Machinery
Machinery is an asynchronous task queue/job queue based on distributed message passing.
Stars: ✭ 5,821 (+7459.74%)
Mutual labels:  task, queue, rabbitmq
python-asynchronous-tasks
😎Asynchronous tasks in Python with Celery + RabbitMQ + Redis
Stars: ✭ 37 (-51.95%)
Mutual labels:  queue, rabbitmq, celery
Opensa
资产管理、资产采集、灰度发布、反向代理、批量任务、任务编排、计划任务、日志审计、权限管理、角色管理、部门管理、运维自动化
Stars: ✭ 220 (+185.71%)
Mutual labels:  task, celery
Selinon
An advanced distributed task flow management on top of Celery
Stars: ✭ 237 (+207.79%)
Mutual labels:  task, celery
Task Easy
A simple, customizable, and lightweight priority queue for promises.
Stars: ✭ 244 (+216.88%)
Mutual labels:  task, queue
dispatcher
Dispatcher is an asynchronous task queue/job queue based on distributed message passing.
Stars: ✭ 60 (-22.08%)
Mutual labels:  queue, celery
ExDeMon
A general purpose metrics monitor implemented with Apache Spark. Kafka source, Elastic sink, aggregate metrics, different analysis, notifications, actions, live configuration update, missing metrics, ...
Stars: ✭ 19 (-75.32%)
Mutual labels:  monitor, monitoring-tool
Je
A distributed job execution engine for the execution of batch jobs, workflows, remediations and more.
Stars: ✭ 30 (-61.04%)
Mutual labels:  task, distributed
FastAPI Tortoise template
FastAPI - Tortoise ORM - Celery - Docker template
Stars: ✭ 144 (+87.01%)
Mutual labels:  celery, flower

Leek


Leek Celery Monitoring Tool
Celery Tasks Monitoring Tool
Documentation: https://tryleek.com

Docs Build Status App Build Status Python 3.8 pulls

License PRs Welcome Code of Conduct All Contributors

Watch on GitHub Star on GitHub Tweet

What is Leek?

Leek is a celery tasks monitoring tool, the main difference between Leek and other monitoring tools is that Leek can connect to and monitor many brokers with a single container whereas other tools can monitor only a single broker at a time.

Also leek supports environments branching, multiple applications, Google SSO, charts, issues monitoring, advanced filtering and search, indexation and persistence, slack notifications and provides an awesome UI for a better user experience.

Leek came to remediate the issues found with other celery monitoring tools and provide a reliable results and cool features to ease the process of monitoring your celery cluster, finding and respond to issues quickly.

What Leek is not?

Leek is not a celery tasks/workers control tool and you cannot use leek to revoke/terminate/start tasks, restart your workers fleet, or manage your brokers. however control features could be supported with future releases.

Leek is not a package that can be installed/imported but a full stack application published as a docker image.

Features

As opposed to many other alternatives, leek came to fix the issues existing in other tools and offer many other cool features that does not exist in other tools:

  • Google SSO - you can connect to leek using GSuite accounts for organizations and standard GMail accounts for individuals.

  • Multi brokers support - other monitoring tools can connect to only one broker at a time, which enforces you to deploy many instances to monitor them all. however, Leek with its Agent, it can monitor tasks from multiple brokers with only a single instance of leek.

  • Multi ENVs support - when connecting Leek agent to brokers, you can specify an environment tag for that broker and each event sent from that broker will be tagged with that environment name, allowing you to split celery events into qa, stg, prod subsets so later you can filter task by environment name.

  • Enhanced storage - unlike other alternatives that stores the events in volatile RAM, celery events are indexed to elasticsearch to offer persistence and fast retrieval/search.

  • Beatiful UI - unlike other alternatives which are either a command line tool or have an ugly UI, Leek offers a great user experience thanks to its beautiful well designed UI.

  • Notification - you can define notification rules that will trigger a slack notification to inform you about critical events, the notification triggers rules can match against task state, task name exclusion/exclusions, environment name, and runtime upper bound.

  • Monitor Issues - Leek can also monitor issues by aggregating the failed tasks by exception name, and for each exception it will calculate occurrences, recovered, pending, failed and critical exceptions.

  • Charts - Leek generate multiple charts giving you an idea about the application state, these chart includes: tasks states distribution, tasks queues distribution, top 5 executed tasks, top 5 slow tasks, tasks execution over time, tasks queue over time, tasks failure over time ...

  • Filter by anything - unlike other alternatives that doesn't provide a good support for filters, leek provides a wide range of filters.

  • Tasks control - for now leek can retry tasks only, more tasks/workers control features may be introduced in the future.

Running a local demo

To experiment with leek, you can run one of these demo docker-compose files:

curl -sSL https://raw.githubusercontent.com/kodless/leek/master/demo/docker-compose-rmq-no-auth.yml > docker-compose.yml
docker-compose up
curl -sSL https://raw.githubusercontent.com/kodless/leek/master/demo/docker-compose-redis-no-auth.yml > docker-compose.yml
docker-compose up

This is an example of a demo, that includes 4 services:

  • Leek main application

  • A RabbitMQ or Redis broker

  • An elasticsearch node

  • Demo celery client (publisher)

  • Demo celery workers (consumer)

  • After running the services with docker-compose up, wait for the services to start and navigate to http://0.0.0.0:8000.

  • Create an application with the same name as in LEEK_AGENT_SUBSCRIPTIONS, which is leek.

  • Enjoy the demo

version: "2.4"
services:
  # Main app
  app:
    image: kodhive/leek
    environment:
      # General
      - LEEK_API_LOG_LEVEL=WARNING
      - LEEK_AGENT_LOG_LEVEL=INFO
      # Components
      - LEEK_ENABLE_API=true
      - LEEK_ENABLE_AGENT=true
      - LEEK_ENABLE_WEB=true
      # URLs
      - LEEK_API_URL=http://0.0.0.0:5000
      - LEEK_WEB_URL=http://0.0.0.0:8000
      - LEEK_ES_URL=http://es01:9200
      # Authentication
      - LEEK_API_ENABLE_AUTH=false
      # Subscriptions
      - |
        LEEK_AGENT_SUBSCRIPTIONS=
        [
          {
            "broker": "amqp://admin:admin@mq//",
            "backend": null,
            "exchange": "celeryev",
            "queue": "leek.fanout",
            "routing_key": "#",
            "org_name": "mono",
            "app_name": "leek",
            "app_env": "prod",
            "prefetch_count": 1000,
            "concurrency_pool_size": 2,
            "batch_max_size_in_mb": 1,
            "batch_max_number_of_messages": 1000,
            "batch_max_window_in_seconds": 5
          }
        ]
      - LEEK_AGENT_API_SECRET=not-secret
    ports:
      - 5000:5000
      - 8000:8000
    depends_on:
      mq:
        condition: service_healthy

  # Just for local demo!! (Test worker)
  worker:
    image: kodhive/leek-demo
    environment:
      - BROKER_URL=pyamqp://admin:admin@mq:5672
    depends_on:
      mq:
        condition: service_healthy

  # Just for local demo!! (Test client)
  publisher:
    image: kodhive/leek-demo
    environment:
      - BROKER_URL=pyamqp://admin:admin@mq:5672
    command: >
      bash -c "python3 publisher.py"
    depends_on:
      mq:
        condition: service_healthy

  # Just for local demo!! (Test broker)
  mq:
    image: rabbitmq:3.8.9-management-alpine
    environment:
      - RABBITMQ_DEFAULT_USER=admin
      - RABBITMQ_DEFAULT_PASS=admin
      - "RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS=-rabbit log [{console,[{level,error}]}]"
    ports:
      - 15672:15672
      - 5672:5672
    healthcheck:
      test: [ "CMD", "nc", "-z", "localhost", "5672" ]
      interval: 2s
      timeout: 4s
      retries: 20

  # Just for local development!! (Test index db)
  es01:
    image: elasticsearch:7.10.1
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - cluster.initial_master_nodes=es01
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    command: ["elasticsearch", "-Elogger.level=ERROR"]
    healthcheck:
      test: ["CMD-SHELL", "curl --silent --fail localhost:9200/_cluster/health || exit 1"]
      interval: 30s
      timeout: 30s
      retries: 3
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65535
        hard: 65535
    ports:
      - 9200:9200
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].