camunda-community-hub / zeeqs

Licence: Apache-2.0 License
Query API for aggregated Zeebe data

Programming Languages

kotlin
9241 projects
shell
77523 projects

Projects that are alternatives of or similar to zeeqs

zeebe-simple-monitor
A monitoring application to show insides of Zeebe for developers
Stars: ✭ 110 (+197.3%)
Mutual labels:  zeebe, zeebe-tool
bpmn-spec
A tool to run tests for BPMN processes on Zeebe
Stars: ✭ 28 (-24.32%)
Mutual labels:  zeebe, zeebe-tool
camunda-cloud-helm
Camunda Platform 8 Self-Managed Helm charts
Stars: ✭ 41 (+10.81%)
Mutual labels:  zeebe
Data-Engineering-Projects
Personal Data Engineering Projects
Stars: ✭ 167 (+351.35%)
Mutual labels:  data-lake
zeebe-kafka-exporter
Export events from Zeebe to Kafka
Stars: ✭ 21 (-43.24%)
Mutual labels:  zeebe
analyzing-reddit-sentiment-with-aws
Learn how to use Kinesis Firehose, AWS Glue, S3, and Amazon Athena by streaming and analyzing reddit comments in realtime. 100-200 level tutorial.
Stars: ✭ 40 (+8.11%)
Mutual labels:  data-lake
spring-zeebe
Easily use the Zeebe Java Client in your Spring or Spring Boot projects
Stars: ✭ 103 (+178.38%)
Mutual labels:  zeebe
zeebe-simple-tasklist
Zeebe worker to manage manual/user tasks
Stars: ✭ 50 (+35.14%)
Mutual labels:  zeebe
zeebe-client-node-js
Node.js client library for Zeebe Microservices Orchestration Engine
Stars: ✭ 109 (+194.59%)
Mutual labels:  zeebe
terraform-module-azure-datalake
Terraform module for an Azure Data Lake
Stars: ✭ 28 (-24.32%)
Mutual labels:  data-lake
jobAnalytics and search
JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.
Stars: ✭ 25 (-32.43%)
Mutual labels:  data-lake
zeebe-helm
Public Zeebe K8s HELM Charts
Stars: ✭ 13 (-64.86%)
Mutual labels:  zeebe
Azure-Certification-DP-200
Road to Azure Data Engineer Part-I: DP-200 - Implementing an Azure Data Solution
Stars: ✭ 54 (+45.95%)
Mutual labels:  data-lake
nestjs-zeebe
Zeebe transport and client for nestjs framework
Stars: ✭ 40 (+8.11%)
Mutual labels:  zeebe
smart-data-lake
Smart Automation Tool for building modern Data Lakes and Data Pipelines
Stars: ✭ 79 (+113.51%)
Mutual labels:  data-lake
SparkProgrammingInScala
Apache Spark Course Material
Stars: ✭ 57 (+54.05%)
Mutual labels:  data-lake
zeebe-client-csharp
Contains an Zeebe C# client implementation.
Stars: ✭ 71 (+91.89%)
Mutual labels:  zeebe
herd-mdl
Herd-MDL, a turnkey managed data lake in the cloud. See https://finraos.github.io/herd-mdl/ for more information.
Stars: ✭ 11 (-70.27%)
Mutual labels:  data-lake
awesome-camunda-cloud
Awesome Camunda Cloud Projects
Stars: ✭ 81 (+118.92%)
Mutual labels:  zeebe
zeebe-script-worker
Zeebe worker for script evaluation
Stars: ✭ 17 (-54.05%)
Mutual labels:  zeebe

ZeeQS - Zeebe Query Service

License

A Zeebe community extension that provides a GraphQL query API over Zeebe's data. The data is imported from the broker using an exporter (e.g. Hazelcast, Elasticsearch) and aggregated in the service.

architecture view

Usage

The application provides an endpoint /graphql for GraphQL queries.

A query can be send via HTTP POST request and a JSON body containing the query. For example:

curl \
  -X POST \
  -H "Content-Type: application/json" \
  --data '{ "query": "{ processes { nodes { key } } }" }' \
  http://localhost:9000/graphql

While development, the graph can be explored using the integrated GraphiQL: http://localhost:9000/graphiql

Queries

The API provide the following queries:

  • processes
  • processInstances
  • jobs
  • messages
  • incidents
  • errors

And additional queries to request a single object by its key (e.g. process(key: "2251799813685249")).

Pagination

In order to limit the response size and the query processing, each list query uses pagination to return only a subset of the data. By default, it returns the first 10 items of the list.

In addition to the items, the query result contains also the total count of the items.

{
  processes(perPage: 10, page: 0) {
    totalCount
    nodes {
      key
    }
  }
}

Filters

Some queries allow to filter the result by passing arguments in the query.

{
  processInstances(stateIn: [ACTIVATED]) {
    nodes {
      key
    }
  }
}

Examples

Get processes including their process instance count:

{
  processes {
    totalCount
    nodes {
      key
      bpmnProcessId
      version
      processInstances {
        totalCount
      }
    }
  }
}
Query Response

{
  "data": {
    "processes": {
      "totalCount": 3,
      "nodes": [
        {
          "key": "2251799813685249",
          "bpmnProcessId": "demo-process",
          "version": 1,
          "processInstances": {
            "totalCount": 3
          }
        },
        {
          "key": "2251799813685251",
          "bpmnProcessId": "demo-2",
          "version": 1,
          "processInstances": {
            "totalCount": 2
          }
        },
        {
          "key": "2251799813685269",
          "bpmnProcessId": "demo-3",
          "version": 1,
          "processInstances": {
            "totalCount": 1
          }
        }
      ]
    }
  }
}

Get process instances that are active including their active element instances:

{
  processInstances(stateIn: [ACTIVATED]) {
    totalCount
    nodes {
      key
      state
      process {
        bpmnProcessId
      }
      elementInstances(stateIn: [ACTIVATED]) {
        elementId
        elementName
        bpmnElementType
        state
      }
    }
  }
}
Query Response

{
  "data": {
    "processInstances": {
      "totalCount": 6,
      "nodes": [
        {
          "key": "2251799813685261",
          "state": "ACTIVATED",
          "process": {
            "bpmnProcessId": "demo-process"
          },
          "elementInstances": [
            {
              "elementId": "demo-process",
              "elementName": null,
              "bpmnElementType": "PROCESS",
              "state": "ACTIVATED"
            },
            {
              "elementId": "ServiceTask_0f8l2yf",
              "elementName": "task 3",
              "bpmnElementType": "SERVICE_TASK",
              "state": "ACTIVATED"
            }
          ]
        },
        {
          "key": "2251799813685271",
          "state": "ACTIVATED",
          "process": {
            "bpmnProcessId": "demo-2"
          },
          "elementInstances": [
            {
              "elementId": "demo-2",
              "elementName": null,
              "bpmnElementType": "PROCESS",
              "state": "ACTIVATED"
            },
            {
              "elementId": "ServiceTask_0mhbd5b",
              "elementName": "task 3",
              "bpmnElementType": "SERVICE_TASK",
              "state": "ACTIVATED"
            }
          ]
        },
        {
          "key": "2251799813685277",
          "state": "ACTIVATED",
          "process": {
            "bpmnProcessId": "demo-2"
          },
          "elementInstances": [
            {
              "elementId": "demo-2",
              "elementName": null,
              "bpmnElementType": "PROCESS",
              "state": "ACTIVATED"
            },
            {
              "elementId": "ServiceTask_19bf066",
              "elementName": "task 1",
              "bpmnElementType": "SERVICE_TASK",
              "state": "ACTIVATED"
            }
          ]
        },
        {
          "key": "2251799813685287",
          "state": "ACTIVATED",
          "process": {
            "bpmnProcessId": "demo-3"
          },
          "elementInstances": [
            {
              "elementId": "demo-3",
              "elementName": null,
              "bpmnElementType": "PROCESS",
              "state": "ACTIVATED"
            },
            {
              "elementId": "ServiceTask_0t5azxx",
              "elementName": "task 2",
              "bpmnElementType": "SERVICE_TASK",
              "state": "ACTIVATED"
            }
          ]
        },
        {
          "key": "2251799813685304",
          "state": "ACTIVATED",
          "process": {
            "bpmnProcessId": "demo-process"
          },
          "elementInstances": [
            {
              "elementId": "demo-process",
              "elementName": null,
              "bpmnElementType": "PROCESS",
              "state": "ACTIVATED"
            },
            {
              "elementId": "ServiceTask_01flslu",
              "elementName": "task 2",
              "bpmnElementType": "SERVICE_TASK",
              "state": "ACTIVATED"
            }
          ]
        },
        {
          "key": "2251799813685310",
          "state": "ACTIVATED",
          "process": {
            "bpmnProcessId": "demo-process"
          },
          "elementInstances": [
            {
              "elementId": "demo-process",
              "elementName": null,
              "bpmnElementType": "PROCESS",
              "state": "ACTIVATED"
            },
            {
              "elementId": "ServiceTask_0m7fzva",
              "elementName": "task 1",
              "bpmnElementType": "SERVICE_TASK",
              "state": "ACTIVATED"
            }
          ]
        }
      ]
    }
  }
}

Get jobs that are activate (i.e. not completed or canceled) and have one of the given job types:

{
  jobs(stateIn: [ACTIVATABLE, ACTIVATED], jobTypeIn: ["task-1", "task-2", "task-3"]) {
    totalCount
    nodes {
      key
      jobType
      state
      elementInstance {
        elementId
        elementName
        processInstance {
          key
          process {
            key
            bpmnProcessId
          }
        }
      }
    }
  }
}
Query Response

{
  "data": {
    "jobs": {
      "totalCount": 5,
      "nodes": [
        {
          "key": "2251799813685282",
          "jobType": "task-1",
          "state": "ACTIVATABLE",
          "elementInstance": {
            "elementId": "ServiceTask_19bf066",
            "elementName": "task 1",
            "processInstance": {
              "key": "2251799813685277",
              "process": {
                "key": "2251799813685251",
                "bpmnProcessId": "demo-2"
              }
            }
          }
        },
        {
          "key": "2251799813685303",
          "jobType": "task-2",
          "state": "ACTIVATABLE",
          "elementInstance": {
            "elementId": "ServiceTask_0t5azxx",
            "elementName": "task 2",
            "processInstance": {
              "key": "2251799813685287",
              "process": {
                "key": "2251799813685269",
                "bpmnProcessId": "demo-3"
              }
            }
          }
        },
        {
          "key": "2251799813685315",
          "jobType": "task-1",
          "state": "ACTIVATABLE",
          "elementInstance": {
            "elementId": "ServiceTask_0m7fzva",
            "elementName": "task 1",
            "processInstance": {
              "key": "2251799813685310",
              "process": {
                "key": "2251799813685249",
                "bpmnProcessId": "demo-process"
              }
            }
          }
        },
        {
          "key": "2251799813685321",
          "jobType": "task-3",
          "state": "ACTIVATABLE",
          "elementInstance": {
            "elementId": "ServiceTask_0f8l2yf",
            "elementName": "task 3",
            "processInstance": {
              "key": "2251799813685261",
              "process": {
                "key": "2251799813685249",
                "bpmnProcessId": "demo-process"
              }
            }
          }
        },
        {
          "key": "2251799813685324",
          "jobType": "task-2",
          "state": "ACTIVATABLE",
          "elementInstance": {
            "elementId": "ServiceTask_01flslu",
            "elementName": "task 2",
            "processInstance": {
              "key": "2251799813685304",
              "process": {
                "key": "2251799813685249",
                "bpmnProcessId": "demo-process"
              }
            }
          }
        }
      ]
    }
  }
}

Get jobs from a specific process instance

{ 
  processInstance(key: "2251799814310503") { 
    state 
    jobs { 
      key 
      jobType 
      elementInstance { 
        elementId 
      } 
    } 
  } 
}
Query Response

{ 
  "data": {
    "processInstance": {
      "state": "COMPLETED",
      "jobs": [
        {
          "key": "2251799814310522",
          "jobType": "DMN",
          "elementInstance": {
            "elementId": "demo-1"
          }
        },
        {
          "key": "2251799814310525",
          "jobType": "DMN",
          "elementInstance": {
            "elementId": "demo-2"
          }
        },
        {
          "key": "2251799814310527",
          "jobType": "demo-3",
          "elementInstance": {
            "elementId": "demo-3"
          }
        }
      ]
    }
  }
}

Install

Docker

The docker image for the ZeeQS application is published to GitHub Packages.

docker pull ghcr.io/camunda-community-hub/zeeqs:2.1.0
  • ensure that a Zeebe broker is running with a Hazelcast exporter (>= 1.0.0)
  • forward the Hazelcast port to the docker container (default: 5701)
  • configure the connection to Hazelcast by setting zeebe.client.worker.hazelcast.connection (default: localhost:5701)

If the Zeebe broker runs on your local machine with the default configs then start the container with the following command:

docker run --network="host" ghcr.io/camunda-community-hub/zeeqs:2.1.0

For a local setup, the repository contains a docker-compose file. It starts a Zeebe broker with the Hazelcast exporter and the ZeeQS application.

mvn clean install -DskipTests
cd docker
docker-compose up
  • the GraphQL endpoint is available under the port 9000

Manual

  1. Download the latest application JAR (zeeqs-%{VERSION}.jar )

  2. Start the application java -jar zeeqs-{VERSION}.jar

Configuration

The application is a Spring Boot application. The configuration can be changed via environment variables or an application.yaml file. See also the following resources:

By default, the port is set to 9000 and the database is only in-memory (i.e. not persistent).

zeebe:
  client:
    worker:
      hazelcast:
        connection: localhost:5701
        connectionTimeout: PT1M
        ringbuffer: zeebe
        connectionInitialBackoff: PT15S
        connectionBackoffMultiplier: 2.0
        connectionMaxBackoff: PT30S

spring:

  datasource:
    url: jdbc:h2:mem:zeeqs;DB_CLOSE_DELAY=-1
    username: sa
    password:
    driverClassName: org.h2.Driver

  jpa:
    database-platform: org.hibernate.dialect.H2Dialect
    hibernate:
      ddl-auto: create

server:
  port: 9000

Using a different database, for example, PostgreSQL:

  • change the following database configuration settings
- spring.datasource.url=jdbc:postgresql://db:5432/postgres
- spring.datasource.username=postgres
- spring.datasource.password=zeebe
- spring.datasource.driverClassName=org.postgresql.Driver
- spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.PostgreSQLDialect
Full docker-compose.yml with PostgreSQL

version: "2"

networks:
  zeebe_network:
    driver: bridge

services:
  zeebe:
    container_name: zeebe_broker
    image: camunda/zeebe:1.0.0
    environment:
      - ZEEBE_LOG_LEVEL=debug
    ports:
      - "26500:26500"
      - "9600:9600"
      - "5701:5701"
    volumes:
      - ../target/exporter/zeebe-hazelcast-exporter.jar:/usr/local/zeebe/exporters/zeebe-hazelcast-exporter.jar
      - ./application.yaml:/usr/local/zeebe/config/application.yaml
    networks:
      - zeebe_network
      
  zeeqs:
    container_name: zeeqs
    image: ghcr.io/camunda-community-hub/zeeqs:2.0.0
    environment:
      - zeebe.client.worker.hazelcast.connection=zeebe:5701
      - spring.datasource.url=jdbc:postgresql://db:5432/postgres
      - spring.datasource.username=postgres
      - spring.datasource.password=zeebe
      - spring.datasource.driverClassName=org.postgresql.Driver
      - spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.PostgreSQLDialect
    ports:
      - "9000:9000"
    depends_on:
      - db
    networks:
      - zeebe_network

  db:
    image: postgres:12.2
    restart: always
    environment:
      POSTGRES_PASSWORD: zeebe
    volumes:
      - database-data:/var/lib/postgresql/data/
    networks:
      - zeebe_network

volumes:
  database-data:

Build from Source

Build with Maven

mvn clean install

Code of Conduct

This project adheres to the Contributor Covenant Code of Conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to [email protected].

License

Apache License, Version 2.0

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].