Elastic Stack on Docker with Preconfigured Security, Tools, Self-Monitoring, and Prometheus Metrics Exporters

With tools like Curator, Rubban, ElastAlert for Alerting.

Introduction

Elastic Stack (ELK) Docker Composition, preconfigured with Security, Monitoring, and Tools; Up with a Single Command.

Based on Official Elastic Docker Images

Stack Version: 7.11.1

You can change Elastic Stack version by setting ELK_VERSION in .env file and rebuild your images. Any version >= 7.0.0 is compatible with this template.

Main Features 📜

• Configured as Production Single Node Cluster. (With a multi-node cluster option for experimenting).
• Deployed on a Single Docker Host or a Docker Swarm Cluster.
• Security Enabled (under basic license).
• SSL Enabled for Transport Layer and Kibana.
• Use Docker-Compose and .env to configure your entire stack parameters.
• Persist Elasticsearch's Keystore and SSL Certifications.
• Self-Monitoring Metrics Enabled.
• Prometheus Exporters for Stack Metrics.
• Embedded Container Healthchecks for Stack Images.
• ElastAlert preconfigured for Alerting.
• Curator with Crond preconfigured for Automated Scheduled tasks (e.g Snapshots to S3).
• Rubban for Kibana curating tasks.

More points

And comparing Elastdocker and the popular deviantony/docker-elk

Requirements

• Docker 17.05 or higher
• Docker-Compose 3 or higher
• 4GB RAM (For Windows and MacOS make sure Docker's VM has more than 4GB+ memory.)

Setup

1. Clone the Repository

   git clone https://github.com/sherifabdlnaby/elastdocker.git
   

2. Initialize Elasticsearch Keystore and TLS Self-Signed Certificates

   $ make setup
   

   For Linux's docker hosts only. By default virtual memory is not enough so run the next command as root sysctl -w vm.max_map_count=262144

3. Start Elastic Stack

   $ make elk           <OR>         $ docker-compose up -d
   

4. Visit Kibana at https://localhost:5601 or https://<your_public_ip>:5601

   Default Username: elastic, Password: changeme

   • Notice that Kibana is configured to use HTTPS, so you'll need to write https:// before localhost:5601 in the browser.
   • Modify .env file for your needs, most importantly ELASTIC_PASSWORD that setup your superuser elastic's password, ELASTICSEARCH_HEAP & LOGSTASH_HEAP for Elasticsearch & Logstash Heap Size.

Whatever your Host (e.g AWS EC2, Azure, DigitalOcean, or on-premise server), once you expose your host to the network, ELK component will be accessible on their respective ports.

Docker Swarm Support

Elastdocker can be deployed to Docker Swarm using make swarm-deploy

Additional Commands

$ make monitoring

$ make tools

$ make all

$ make nodes

$ make build

$ make down

$ make prune

Configuration

• Some Configuration are parameterized in the .env file.
  • ELASTIC_PASSWORD, user elastic's password (default: changeme pls).
  • ELK_VERSION Elastic Stack Version (default: 7.11.1)
  • ELASTICSEARCH_HEAP, how much Elasticsearch allocate from memory (default: 1GB -good for development only-)
  • LOGSTASH_HEAP, how much Logstash allocate from memory.
  • Other configurations which their such as cluster name, and node name, etc.
• Elasticsearch Configuration in elasticsearch.yml at ./elasticsearch/config.
• Logstash Configuration in logstash.yml at ./elasticsearch/config/logstash.yml.
• Logstash Pipeline in main.conf at ./elasticsearch/pipeline/main.conf.
• Kibana Configuration in kibana.yml at ./kibana/config.
• ElastAlert Configuration in ./tools/elastalert/config.
• ElastAlert Alert rules in ./tools/elastalert/rules, head to ElastAlert docs to lookup how to create alerts.
• Curator Actions at ./tools/curator/actions and ./tools/curator/crontab.
• Rubban Configuration using Docker-Compose passed Environment Variables.

Setting Up Keystore

You can extend the Keystore generation script by adding keys to ./setup/keystore.sh script. (e.g Add S3 Snapshot Repository Credentials)

To Re-generate Keystore:

make keystore

Enable SSL on HTTP

By default, only Transport Layer has SSL Enabled, to enable SSL on HTTP layer, add the following lines to elasticsearch.yml

## - http
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certs/elasticsearch.key
xpack.security.http.ssl.certificate: certs/elasticsearch.crt
xpack.security.http.ssl.certificate_authorities: certs/ca.crt
xpack.security.http.ssl.client_authentication: optional

⚠️ Enabling SSL on HTTP layer will require all clients that connect to Elasticsearch to configure SSL connection for HTTP, this includes all the current configured parts of the stack (e.g Logstash, Kibana, Curator, etc) plus any library/binding that connects to Elasticsearch from your application code.

Notes

• Adding Two Extra Nodes to the cluster will make the cluster depending on them and won't start without them again.

• Makefile is a wrapper around Docker-Compose commands, use make help to know every command.

• Elasticsearch will save its data to a volume named elasticsearch-data

• Elasticsearch Keystore (that contains passwords and credentials) and SSL Certificate are generated in the ./secrets directory by the setup command.

• Make sure to run make setup if you changed ELASTIC_PASSWORD and to restart the stack afterwards.

• For Linux Users it's recommended to set the following configuration (run as root)

  sysctl -w vm.max_map_count=262144
  

  By default, Virtual Memory is not enough.

Monitoring The Cluster

Via Prometheus Exporters

If you started Prometheus Exporters using make monitoring command. Prometheus Exporters will expose metrics at the following ports.

Via Self-Monitoring

Head to Stack Monitoring tab in Kibana to see cluster metrics for all stack components.

In Production, cluster metrics should be shipped to another dedicated monitoring cluster.

License

MIT License Copyright (c) 2020 Sherif Abdel-Naby

Contribution

PR(s) are Open and Welcomed.