Promscale

Promscale is an open-source long-term store for Prometheus data designed for analytics. It is a horizontally scalable, highly-compressed, and operationally mature platform for Prometheus data that offers the combined power of PromQL and SQL, enabling developers to ask any question, create any dashboard, and achieve greater visibility into their systems. Promscale is built on top of TimescaleDB, the leading relational database for time-series.

By allowing a user to use SQL, in addition to PromQL, this platform empowers the user to ask complex analytical queries from their metrics data, and thus extract more meaningful insights.

✅ Promscale has consistently been one of the only long-term stores for Prometheus data that continue to maintain its top-performance, receiving 100% compliance test score each time (with no cross-cutting concerns) from PromLab's PromQL Compliance Test Suite.

For a detailed description of this architecture, please see our design doc.

If you have any questions, please join the #promscale channel on TimescaleDB Slack, or the Promscale Users Google Group.

This repository also contains the source code for prom-migrator. Prom-migrator is an open-source, community-driven and free-to-use, universal prometheus data migration tool, that migrates data from one storage system to another, leveraging Prometheus's remote storage endpoints. For more information about prom-migrator, visit prom-migrator's README.

Documentation

• About TimescaleDB
• Features
• Installation
  • The Observability Suite for Kubernetes
  • Docker
  • Binaries
  • Source
  • Helm
  • Tutorial: Bare Metal Setup for a Monitoring Suite
• Analyzing Data Using SQL
  • Data Model
  • Filtering Series
  • Data Retention
• Writing Data to Promscale
• Alerting & Recording Rules
• Deleting Data
• Quick Tips
• High Availability
• Multi-Node TimescaleDB
• FAQ
• Contributing

🐯 About TimescaleDB

TimescaleDB is a distributed time-series database built on PostgreSQL that scales to over 10 million of metrics per second, supports native compression, handles high cardinality, and offers native time-series capabilities, such as data retention policies, continuous aggregate views, downsampling, data gap-filling and interpolation.

TimescaleDB also supports full SQL, a variety of data types (numerics, text, arrays, JSON, booleans), and ACID semantics. Operationally mature capabilities include high-availability, streaming backups, upgrades over time, roles and permissions, and security.

TimescaleDB has a large and active user community (tens of millions of downloads, hundreds of thousands of active deployments, Slack channel with 4,000+ members). Users include Comcast, Fujitsu, Schneider Electric, Siemens, Walmart, Warner Music, and thousands of others.

Developers and organizations around the world trust TimescaleDB with their time-series data. AppDynamics (now part of Cisco Systems and one of the largest application performance monitoring providers) relies on TimescaleDB as its main metrics database. TimescaleDB is also the preferred (recommended) backend datasource for Zabbix users and is natively supported in Grafana.

🌟 Features

Click the video below for an overview of Promscale:

• Analysis in PromQL and SQL get the ability to analyze data in both query languages. Use PromQL for monitoring and alerting and SQL for deeper analytics and compatibility with a huge ecosystem of data visualization, analysis, and AI/ML tools.
• Rock-solid stability due to being built on top of PostgreSQL, with 30+ years of development work.
• Support for backfilling to ingest data from the past.
• High-Availability support for Promethus HA deployments as well as high-availability deployments of TimescaleDB itself.
• Simple architecture. Unlike some other long-term stores, our architecture consists of only three components: Prometheus, Promscale, and TimescaleDB.
• ACID compliance to ensure consistency of your data.
• Horizontal scalability using multinode support with TimescaleDB version 2.0.

🔧 Choose your own (installation) adventure

We have four main ways to set up Promscale:

tobs (recommended for Kubernetes environments)

The Observability Suite for Kubernetes is a CLI tool and Helm chart that makes installing a full observability suite into your Kubernetes cluster really easy. Tobs includes Prometheus, TimescaleDB, Promscale Connector, and Grafana.

To get started, run the following in your terminal, then follow the on-screen instructions.

curl --proto '=https' --tlsv1.2 -sSLf  https://tsdb.co/install-tobs-sh |sh

Or visit the tobs GitHub repo for more information and instructions on advanced configuration.

🐳 Docker

We provide docker images with every release.

Instructions on how to use our docker images are available here.

🔟 Binaries

We have pre-packaged binaries available for MacOS and Linux on both the x86_64 and i386 architectures. Instructions on how to use our prepackaged binaries are available here.

You can also build binaries from source.

⎈ Helm (sub)chart for Promscale Connector only

A Helm chart for only the Promscale Connector is available in the helm-chart directory of this repository.

This is used as a Helm dependency from the tobs Helm chart and can be used as a dependency in your own custom Helm chart as well.

🔍 Analyzing the data using SQL

We describe how to use our pre-defined views and functions to work with the prometheus data in the SQL schema doc.

A Reference for our SQL API is available here.

💡 Quick tips

Configuring Data Retention

By default, data is stored for 90 days and then deleted. This default can be changed globally or overridden for individual metrics.

If using tobs, these settings can be changed using the tobs metrics retention command (use tobs metrics retention -h to see all options).

These setting can also be changed using the appropriate SQL commands.

🌐 Prometheus HTTP API

The Promscale Connector can be used directly as a Prometheus Data Source in Grafana, or other software.

The connector implements some endpoints of the currently stable (V1) Prometheus HTTP API. The API is accessible at http://<promscale_connector_address>:9201/api/v1 and can be used to execute instant or range PromQL queries against the data in TimescaleDB, as well as retrieve the metadata for series, label names and label values.

A Reference for the implemented endpoints of the Prometheus HTTP API is available here

Configuring Prometheus to filter which metrics are sent (optional)

You can limit the metrics being sent to the adapter (and thus being stored in your long-term storage) by setting up write_relabel_configs in Prometheus, via the prometheus.yml file. Doing this can reduce the amount of space used by your database and thus increase query performance.

The example below drops all metrics starting with the prefix go_, which matches Golang process information exposed by exporters like node_exporter:

remote_write:
 - url: "http://promscale_connector:9201/write"
   write_relabel_configs:
      - source_labels: [__name__]
        regex: 'go_.*'
        action: drop

Additional information about setting up relabel configs, the source_labels field, and the possible actions can be found in the Prometheus Docs.

✏️ Contributing

We welcome contributions to the Promscale Connector, which is licensed and released under the open-source Apache License, Version 2. The same Contributor's Agreement applies as in TimescaleDB; please sign the Contributor License Agreement (CLA) if you're a new contributor.