GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → ypereirareis → docker-metricbeat-example

ypereirareis / docker-metricbeat-example

Licence: MIT license
Elasticsearch Metricbeat example configuration to monitor Host and Services with docker

Programming Languages

Makefile
30231 projects
shell
77523 projects

Labels

dockerelasticsearchkibanamonitoringbeatsmetricbeat

Projects that are alternatives of or similar to docker-metricbeat-example

EnterpriseApplicationLog
Enterprise Application Log with RabbitMQ, LogStash, ElasticSearch and Kibana
Stars: ✭ 88 (+10%)
Mutual labels:  kibana, beats
cloudwatchmetricbeat
A beat for AWS CloudWatch Metrics. Located in NYC? We are hiring: https://www.narmi.com/careers/
Stars: ✭ 17 (-78.75%)
Mutual labels:  beats, metricbeat
icingabeat
Elastic Beat fetching events & status from Icinga 2
Stars: ✭ 36 (-55%)
Mutual labels:  kibana, beats
elastic-stax
elastic-stax Docker Compose projects => for learning/training purposes only!
Stars: ✭ 13 (-83.75%)
Mutual labels:  kibana, beats
elastic-stack-testing
Elastic Stack Testing Framework (ESTF) 🤖
Stars: ✭ 47 (-41.25%)
Mutual labels:  kibana, beats
awesome-elastic-stack
Awesome Elastic Stack
Stars: ✭ 29 (-63.75%)
Mutual labels:  kibana, beats
dockelk
ELK log transport and aggregation at scale
Stars: ✭ 31 (-61.25%)
Mutual labels:  kibana
twitter-sentiment-analysis
Streaming tweets with spark, language detection & sentiment analysis, dashboard with Kibana
Stars: ✭ 100 (+25%)
Mutual labels:  kibana
elk-tls-docker
This repository contains code to create a ELK stack with certificates & security enabled using docker-compose
Stars: ✭ 152 (+90%)
Mutual labels:  kibana
elastic-stack
A complete documentation on how to install Elastic Stack on Ubuntu 16.04 Server ASAP 😎
Stars: ✭ 12 (-85%)
Mutual labels:  metricbeat
elastic-query-export
🚚 Export Data from ElasticSearch to CSV/JSON using a Lucene Query (e.g. from Kibana) or a raw JSON Query string
Stars: ✭ 56 (-30%)
Mutual labels:  kibana
docker-elk-stack
The ELK stack Docker containerization (Elasticsearch, Logstash and Kibana)
Stars: ✭ 20 (-75%)
Mutual labels:  kibana
graylog-beats-plugin
Graylog input plugin for Elastic Beats
Stars: ✭ 19 (-76.25%)
Mutual labels:  beats
logrus-redis-hook
Hook for Logrus which enables logging to RELK stack (Redis, Elasticsearch, Logstash and Kibana)
Stars: ✭ 33 (-58.75%)
Mutual labels:  kibana
kibana-multitenant-proxy
A proxy behind nginx while before kibana (4.x, 5.x) to provide data isolation for different users
Stars: ✭ 25 (-68.75%)
Mutual labels:  kibana
factbeat
Ship Facter facts to Elasticsearch or Logstash.
Stars: ✭ 13 (-83.75%)
Mutual labels:  beats
area3d vis
3D Graph Visualization for Kibana 5
Stars: ✭ 28 (-65%)
Mutual labels:  kibana
kibana diagram
Experimental Flow Diagram Vis for Kibana 6.x
Stars: ✭ 15 (-81.25%)
Mutual labels:  kibana
kibana-xlsx-import
Kibana plugin for import XLSX/CSV file to ElasticSearch
Stars: ✭ 58 (-27.5%)
Mutual labels:  kibana
aws-elb-logs-to-logstash
Consumes ELB & ALB logs and sends them to logstash for ingestion. Kibana dashboard included.
Stars: ✭ 26 (-67.5%)
Mutual labels:  kibana
View All Similar Projects ➔

docker-metricbeat-example

Docker and Metricbeat configurations examples. Learn how to use Docker and Metricbeat to monitor you host system, all your nodes, your docker containers and all your (distributed) services.

Warning

This project is built for example and testing.
DO NOT use any part of it in production unless you know exactly what you are doing.

Software versions

Name Version
Docker 17.10.0-ce (f4ffd25) (Tue Oct 17 19:05:05 2017)
Docker-compose 1.17.0, build ac53b73
Elasticsearch 6.0.0
Metricbeat 6.0.0
Kibana 6.0.0
Apache 2.4.29-alpine
MongoDB 3.5.13-jessie
MySQL 8.0
Nginx 1.13.7-alpine
RabbitMQ 3.6.14-management-alpine
Redis 3.2.11-alpine

Install

Clone the repository

git clone [email protected]:ypereirareis/docker-metricbeat-example.git && cd docker-metricbeat-example

Setup your host (sudo/root needed). You shoud have something like this.

$ make setup
=> ACLs on /var/run/docker.sock OK
vm.max_map_count = 262144
=> vm.max_map_count=262144 OK
2105e36d174f6ad3d0d23cac5769454d60a51fa8998fc0fbe946c4e713f1493c
Building metricbeat-host
Step 1/6 : FROM docker.elastic.co/beats/metricbeat:6.0.0
 ---> bc8ff400feac
Step 2/6 : ARG METRICBEAT_FILE=metricbeat.yml
 ---> Using cache
 ---> ebd68e798d93
Step 3/6 : COPY ${METRICBEAT_FILE} /usr/share/metricbeat/metricbeat.yml
 ---> Using cache
 ---> e24c78b4b7e8
Step 4/6 : USER root
 ---> Using cache
 ---> 84b29d7e9635
Step 5/6 : RUN mkdir /var/log/metricbeat     && chown metricbeat /usr/share/metricbeat/metricbeat.yml     && chmod go-w /usr/share/metricbeat/metricbeat.yml     && chown metricbeat /var/log/metricbeat
 ---> Using cache
 ---> 0d1485e2e4c1
Step 6/6 : USER metricbeat
 ---> Using cache
 ---> 7c5172b2912c
Successfully built 7c5172b2912c
Successfully tagged dockermetricbeatexample_metricbeat-host:latest
elasticsearch uses an image, skipping
Building metricbeat
Step 1/6 : FROM docker.elastic.co/beats/metricbeat:6.0.0
 ---> bc8ff400feac
Step 2/6 : ARG METRICBEAT_FILE=metricbeat.yml
 ---> Using cache
 ---> ebd68e798d93
Step 3/6 : COPY ${METRICBEAT_FILE} /usr/share/metricbeat/metricbeat.yml
 ---> Using cache
 ---> 19c48873ed8e
Step 4/6 : USER root
 ---> Using cache
 ---> fa6d8c6138d6
Step 5/6 : RUN mkdir /var/log/metricbeat     && chown metricbeat /usr/share/metricbeat/metricbeat.yml     && chmod go-w /usr/share/metricbeat/metricbeat.yml     && chown metricbeat /var/log/metricbeat
 ---> Using cache
 ---> b691cc7eb5bf
Step 6/6 : USER metricbeat
 ---> Using cache
 ---> 5e02571e6fb2
Successfully built 5e02571e6fb2
Successfully tagged dockermetricbeatexample_metricbeat:latest
kibana uses an image, skipping

Host Monitoring

We are assuming your docker0 interface IP is: inet 172.17.0.1 netmask 255.255.0.0 broadcast 0.0.0.0 If not the case, please adjust the configuration in docker-compose.yml file for service metricbeat-host.

metricbeat-host:
  ...
  environment:
    - HOST_ELASTICSEARCH=elasticsearch:9222
    - HOST_KIBANA=kibana:5666
  extra_hosts:
    - "elasticsearch:172.17.0.1" # The IP of docker0 interface to access host from container
    - "kibana:172.17.0.1" # The IP of docker0 interface to access host from container
  network_mode: host # Mandatory to monitor host filesystem, memory, processes,...

Start monitoring your host.

$ make start-monitoring-host 
Creating metricbeat-elasticsearch ... 
Creating metricbeat-elasticsearch ... done
metricbeat-elasticsearch is up-to-date
Creating metricbeat-kibana ... 
Creating metricbeat-kibana ... done
Waiting for elasticsearch...
Creating metricbeat-metricbeat-host ... 
Creating metricbeat-metricbeat-host ... done
  • You can check everything is OK, and you should have 3 containers running...
  • Be careful Elasticsearch and Kibana ports are exposed on 0.0.0.0 network (every IP address).
  • Default Metricbeat dashboard are automatically loaded into Kibana (setup.dashboards.enabled: true)
$ docker ps                 
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                              NAMES
e25a76b4f1e4        dockermetricbeatexample_metricbeat-host               "/usr/local/bin/do..."   2 minutes ago       Up 2 seconds                                           metricbeat-metricbeat-host
27668d971ddf        docker.elastic.co/kibana/kibana:6.0.0                 "/bin/bash /usr/lo..."   2 minutes ago       Up 2 minutes        0.0.0.0:5666->5601/tcp             metricbeat-kibana
af93d8214167        docker.elastic.co/elasticsearch/elasticsearch:6.0.0   "/usr/local/bin/do..."   2 minutes ago       Up 2 minutes        9300/tcp, 0.0.0.0:9222->9200/tcp   metricbeat-elasticsearch

If everything is fine, you should be able to access Kibana, and Monitoring dashboard:

Host Dashboard

Host dashboard

Docker Dashboard

Docker dashboard

Services Monitoring

First of all you need to start the Metricbeat container to monitor services.

make start-monitoring

You should now have an extra Metricbeat container for services (called metricbeat-metricbeat-services).

$ docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                              NAMES
d934ee72db19        dockermetricbeatexample_metricbeat                    "/usr/local/bin/do..."   4 minutes ago       Up 4 minutes                                           metricbeat-metricbeat-services
...

Start/Stop all services

make start-all
make stop-all

Start/Stop services one by one

Apache

4f4cb6e72a39    httpd:2.4   "httpd-foreground"  11 minutes ago  Up About a minute   80/tcp  metricbeat-apache

MongoDB

a379163bc90d    mongo   "docker-entrypoint..."   11 minutes ago    Up About a minute   27017/tcp    metricbeat-mongodb

MySQL

3ae8e0c7c1e3    mysql:8.0   "docker-entrypoint..."   11 minutes ago   Up About a minute   3306/tcp metricbeat-mysql

Nginx

da1a01c36c4c    nginx:1.13.7-alpine "nginx -g 'daemon ..."   11 minutes ago   Up About a minute   80/tcp  metricbeat-nginx

RabbitMq

6c072660a008    bijukunjummen/rabbitmq-server:3.6.5 "/bin/sh -c /opt/r..."   14 minutes ago      Up 2 seconds   4369/tcp, 5672/tcp, 9100-9105/tcp, 25672/tcp, 127.0.0.1:1234->15672/tcp   metricbeat-rabbitmq

Redis

52be9d662f8f    redis:3.2.11-alpine "docker-entrypoint..."   11 minutes ago Up About a minute   6379/tcp    metricbeat-redis

Others

More to come

Clean everything

A simple command to remove all containers related to Metricbeat

$ make clean
d934ee72db19
3ae8e0c7c1e3
a379163bc90d
6c072660a008
52be9d662f8f
da1a01c36c4c
4f4cb6e72a39
a400ee0b914a
43d22a57fde4
c1ca7e0d7e6a
All METRICBEAT containers removed !

(Re)start everything

$ make install

LICENSE

MIT License

Copyright (c) 2017 Yannick Pereira-Reis

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].