All Projects → last9 → timescaledb-metrics

last9 / timescaledb-metrics

Licence: MIT license
Send TimescaleDB policy stats (and other things) as metrics.

Programming Languages

go
31211 projects - #10 most used programming language
shell
77523 projects

Projects that are alternatives of or similar to timescaledb-metrics

aws-lambda-http-check
Lambda function to check http endpoint wrapped in serverless project
Stars: ✭ 30 (+42.86%)
Mutual labels:  cloudwatch
2021-nolto
부담없이 자랑하는 작고 소중한 내 프로젝트 🧸✨
Stars: ✭ 34 (+61.9%)
Mutual labels:  cloudwatch
aws-o11y-recipes
recipes for observability solutions at AWS
Stars: ✭ 110 (+423.81%)
Mutual labels:  cloudwatch
sensu-plugins-aws
This plugin provides native AWS instrumentation for monitoring and metrics collection, including: health and metrics for various AWS services, such as EC2, RDS, ELB, and more, as well as handlers for EC2, SES, and SNS.
Stars: ✭ 79 (+276.19%)
Mutual labels:  cloudwatch
fluent-plugin-cloudwatch-ingest
Alternative to ryotarai/fluent-plugin-cloudwatch-logs for ingesting AWS Cloudwatch logs via fluentd
Stars: ✭ 11 (-47.62%)
Mutual labels:  cloudwatch
Awesome Aws
A curated list of awesome Amazon Web Services (AWS) libraries, open source repos, guides, blogs, and other resources. Featuring the Fiery Meter of AWSome.
Stars: ✭ 9,895 (+47019.05%)
Mutual labels:  cloudwatch
gozeit
GoZeit
Stars: ✭ 19 (-9.52%)
Mutual labels:  cloudwatch
openapi-tf-example
Example of how you can use OpenAPI with AWS API Gateway, Also includes integrations with AWSLambda, AWS Cognito, AWS SNS and CloudWatch logs
Stars: ✭ 38 (+80.95%)
Mutual labels:  cloudwatch
awsctl
Control AWS infrastructure easily from a single command line written in Go
Stars: ✭ 97 (+361.9%)
Mutual labels:  cloudwatch
amazon-cloudwatch-auto-alarms
Automatically create and configure Amazon CloudWatch alarms for EC2 instances, RDS, and AWS Lambda using tags for standard and custom CloudWatch Metrics.
Stars: ✭ 52 (+147.62%)
Mutual labels:  cloudwatch
iis
开源微博平台 Open source cloud powered microblog
Stars: ✭ 42 (+100%)
Mutual labels:  cloudwatch
terraform-aws-ecs-cloudwatch-sns-alarms
Terraform module to create CloudWatch Alarms on ECS Service level metrics.
Stars: ✭ 23 (+9.52%)
Mutual labels:  cloudwatch
terraform-aws-cloudwatch-logs
Terraform Module to Provide a CloudWatch Logs Endpoint
Stars: ✭ 59 (+180.95%)
Mutual labels:  cloudwatch
terraform-aws-ecs-web-app
Terraform module that implements a web app on ECS and supports autoscaling, CI/CD, monitoring, ALB integration, and much more.
Stars: ✭ 175 (+733.33%)
Mutual labels:  cloudwatch
cloudwatch-dashboards-cloudformation-sample
A sample project to demonstrate using Cloudformation, how to create and configure CloudWatch metric filters, alarms and a dashboard to monitor an AWS Lambda function.
Stars: ✭ 61 (+190.48%)
Mutual labels:  cloudwatch
monitoring-jump-start
Monitor AWS resources with ease
Stars: ✭ 67 (+219.05%)
Mutual labels:  cloudwatch
lambda-watchtower
Serverless HTTP(S) Endpoint Monitoring With AWS Lambda & CloudWatch
Stars: ✭ 68 (+223.81%)
Mutual labels:  cloudwatch
cwlog
🐾 cli AWS Cloudwatch Logs Downloader
Stars: ✭ 26 (+23.81%)
Mutual labels:  cloudwatch
lawsg
The AWS Cloudwatch Log Viewer
Stars: ✭ 31 (+47.62%)
Mutual labels:  cloudwatch
Multi-Node-TimescaleDB
The multi-node setup of TimescaleDB 🐯🐯🐯 🐘 🐯🐯🐯
Stars: ✭ 42 (+100%)
Mutual labels:  timescaledb

timescaledb-metrics

Send TimescaleDB policy stats (and other things) as metrics.

Usage

We advise not to pass credentials in configurations, but via Environment or Instance Profiles

env AWS_DEFAULT_REGION="ap-south-1" CLOUDWATCH_NAMESPACE="timescaledb-dev" DATABASE_URL="postgres://dbuser:[email protected]:5432/db" ./timescaledb-metrics

Output

Cloudwatch

Motivation

While running timescaleDB, there are a lot of background jobs and policies which need constant observation.

Example of such Jobs are:

  • drop_chunks policy
  • compress_chunks policy
  • reorder policy
  • continuous_aggregate policy

etc.

Example of the stats that can be pulled from TimescaleDB tables.

| entity_name            | job_type             | last_start_on | last_success_on | total_failures |
|------------------------|----------------------|---------------|-----------------|----------------|
| sli_300                | drop_chunks          | 1600262129    | 1600262129      | 0              |
| custom_sli_archive     | reorder              | 1600195293    | 1600195293      | 6              |
| custom_sli_archive     | compress_chunks      | 1600248496    | 1600248496      | 0              |
| custom_sli_archive     | drop_chunks          | 1600259297    | 1600259297      | 0              |
| sli_archive            | reorder              | 1600195323    | 1600195323      | 10             |
| sli_300                | continuous_aggregate | 1600262169    | 1600262170      | 0              |
| sli_archive            | drop_chunks          | 1600259297    | 1600259297      | 0              |
| sli_archive            | compress_chunks      | 1600257782    | 1600257782      | 8              |
| sli_violations_archive | drop_chunks          | 1600248501    | 1600248501      | 0              |

Keeping a manual eye on these is painful and tedious, and by emitting these as metrics we can be alerted when things go wrong.

While some may argue that the interval of these jobs is not as periodic where they qualify to be a metric system, but as the usage increases these policies end up running closer to every 5 minutes. Alerts on such slopes can catch staleness pretty effectively.

Another problem with the default policy_stats is that the job_id is hardly human readable. It's tough to keep a track of an IntegerID since it changes, on any recreation. But that's a minor win.

Components

Only timescaleDB.

Considerations

  • As of now we output to Cloudwatch but it can surely be extended to other metric sources. If you want more metric sources, send a Pull Request or just open a new Issue.

  • The reason to not use go-metrics, is the overall lack of precision of timestamps which Cloudwatch is excellent at.

Prometheus Exporter

You can export timescale db metrics to Prometheus by setting the path of this file as an environment variable which the postgres-exporter recognizes by default.

export PG_EXPORTER_EXTEND_QUERY_PATH=<path_to_queries.yaml>

Or you can just add this to your existing custom queries.yaml

References: https://github.com/wrouesnel/postgres_exporter#adding-new-metrics-via-a-config-file

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].