All Projects → ApsOps → Filebeat Kubernetes

ApsOps / Filebeat Kubernetes

Licence: mit
Filebeat container, alternative to fluentd used to ship kubernetes cluster and pod logs

Projects that are alternatives of or similar to Filebeat Kubernetes

Log Pilot
Collect logs for docker containers
Stars: ✭ 1,112 (+656.46%)
Mutual labels:  filebeat, logging, container
Elk Hole
elasticsearch, logstash and kibana configuration for pi-hole visualiziation
Stars: ✭ 136 (-7.48%)
Mutual labels:  logstash, filebeat, logging
Docker Elastic
Deploy Elastic stack in a Docker Swarm cluster. Ship application logs and metrics using beats & GELF plugin to Elasticsearch
Stars: ✭ 202 (+37.41%)
Mutual labels:  logstash, filebeat, logging
Json Logging Python
Python logging library to emit JSON log that can be easily indexed and searchable by logging infrastructure such as ELK, EFK, AWS Cloudwatch, GCP Stackdriver
Stars: ✭ 143 (-2.72%)
Mutual labels:  logstash, logging
elk-stack
ELK Stack (Elasticsearch, Logstash & Kibana)
Stars: ✭ 13 (-91.16%)
Mutual labels:  logstash, filebeat
k8s-log
容器日志搜集套件。
Stars: ✭ 15 (-89.8%)
Mutual labels:  logstash, filebeat
Justlog
JustLog brings logging on iOS to the next level. It supports console, file and remote Logstash logging via TCP socket with no effort. Support for logz.io available.
Stars: ✭ 439 (+198.64%)
Mutual labels:  logstash, logging
Logstash Logger
Ruby logger that writes logstash events
Stars: ✭ 442 (+200.68%)
Mutual labels:  logstash, logging
Elk
搭建ELK日志分析平台。
Stars: ✭ 688 (+368.03%)
Mutual labels:  logstash, filebeat
K8s
Deploying Kubernetes High Availability Cluster with Ansible Playbook
Stars: ✭ 125 (-14.97%)
Mutual labels:  pod, container
Logtrail
Kibana plugin to view, search & live tail log events
Stars: ✭ 1,343 (+813.61%)
Mutual labels:  logstash, logging
Vagrant Elastic Stack
Giving the Elastic Stack a try in Vagrant
Stars: ✭ 131 (-10.88%)
Mutual labels:  logstash, filebeat
seahorse
ELKFH - Elastic, Logstash, Kibana, Filebeat and Honeypot (HTTP, HTTPS, SSH, RDP, VNC, Redis, MySQL, MONGO, SMB, LDAP)
Stars: ✭ 31 (-78.91%)
Mutual labels:  logstash, filebeat
rkt-compose
CLI to run Docker Compose file as pod on rkt with integrated service discovery using Consul - EOL
Stars: ✭ 24 (-83.67%)
Mutual labels:  container, pod
Aliware Kafka Demos
提供各种客户端接入阿里云 消息队列 Kafka 的demo工程
Stars: ✭ 279 (+89.8%)
Mutual labels:  logstash, filebeat
filebeat.py
Python 版 Filebeat
Stars: ✭ 48 (-67.35%)
Mutual labels:  logstash, filebeat
docker-elk-stack
The ELK stack Docker containerization (Elasticsearch, Logstash and Kibana)
Stars: ✭ 20 (-86.39%)
Mutual labels:  logstash, filebeat
dissect-tester
Simple API/UI for testing filebeat dissect patterns against a collection of sample log lines.
Stars: ✭ 58 (-60.54%)
Mutual labels:  logstash, filebeat
MeetU
Application that build on Elasticsearch and Spring Boot Microservices (Synchronous Service)
Stars: ✭ 22 (-85.03%)
Mutual labels:  logstash, filebeat
Elkstack
The config files and docker-compose.yml files of Dockerized ELK Stack
Stars: ✭ 96 (-34.69%)
Mutual labels:  logstash, filebeat

filebeat-kubernetes

Docker Pulls

Filebeat container, alternative to fluentd used to ship kubernetes cluster and pod logs

Getting Started

This container is designed to be run in a pod in Kubernetes to ship logs to logstash for further processing. You can provide following environment variables to customize it.

LOGSTASH_HOSTS=example.com:4083,example.com:4084
LOG_LEVEL=info  # log level for filebeat. Defaults to "error".
FILEBEAT_HOST=ip-a-b-c-d # custom "host" field. Refer following manifest to set it to k8s nodeName
CLUSTER_NAME=my_cluster # Kubernetes cluster name to identity if you have multiple clusters. Default value is "default".

The endpoints listed by LOGSTASH_HOSTS should be listening with the Beats input plugin.

This should be run as a Kubernetes Daemonset (a pod on every node).

The updateStrategy will determine how to apply imperative changes, See K8s docs.

Example manifest:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    app: filebeat
spec:
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: filebeat
      name: filebeat
    spec:
      containers:
      - name: filebeat
        image: apsops/filebeat-kubernetes:v0.4
        resources:
          limits:
            cpu: 50m
            memory: 50Mi
        env:
          - name: LOGSTASH_HOSTS
            value: myhost.com:5000
          - name: LOG_LEVEL
            value: info
          - name: CLUSTER_NAME
            value: my_cluster
          - name: FILEBEAT_HOST
            valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
        volumeMounts:
        - name: varlog
          mountPath: /var/log/containers
        - name: varlogpods
          mountPath: /var/log/pods
          readOnly: true
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      volumes:
      - name: varlog
        hostPath:
          path: /var/log/containers
      - name: varlogpods
        hostPath:
          path: /var/log/pods
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Filebeat parses docker json logs and applies multiline filter on the node before pushing logs to logstash.

Make sure you add a filter in your logstash configuration if you want to process the actual log lines.

filter {
  if [type] == "kube-logs" {

    mutate {
      rename => ["log", "message"]
    }

    date {
      match => ["time", "ISO8601"]
      remove_field => ["time"]
    }

    grok {
        match => { "source" => "/var/log/containers/%{DATA:pod_name}_%{DATA:namespace}_%{GREEDYDATA:container_name}-%{DATA:container_id}.log" }
        remove_field => ["source"]
    }
  }
}

This grok pattern would add the fields - pod_name, namespace, container_name and container id to log entry in Elasticsearch.

Contributing

I plan to make this more modular and reliable.

Feel free to open issues and pull requests for bug fixes or features.

Licence

This project is licensed under the MIT License. Refer LICENSE for details.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].