aws-elb-logs-to-logstash
Consumes AWS ELB (Elastic Load Balancer), NLB (Network Load Balancer), ALB (Application Load Balancer) logs from S3 and sends them to logstash for ingestion. The logs are formatted through a LogStash filter.
Requires
s3cmd -and- jq
Configuration
- Edit the script and update variable 'AWS_ACCOUNT_NUMBER' with your account number.
- Edit the script and update variable 'S3_BUCKET_NAME' with your buckets name.
- If your bucket is nested (e.g: loadbalancer-logs/AWSLogs/service/AWSLogs) set S3_BUCKET_NAME to "loadbalancer-logs/AWSLogs"
- Ensure your AWS credentials are configured (e.g: ~/.aws/credentials)
- Execute the script (it will run in a constant loop)
Configuration extended.
Included is a traditional init script and monit config (optional), to use the init script...
- Ensure 'elb-log-consumer.sh' is located here: /opt/elb-consumer
- Alternatively, edit 'elb-consumer-init.sh' and change variable 'THE_PATH'
- Place the init script[elb-consumer-init.sh] in /etc/init.d
- Make init script executable: chmod ug+x elb-consumer-init.sh
- It can then be started as such: /etc/init.d/elb-consumer-init.sh start (or stop)
- The included monit script can be placed in your monits configuration directory and used to control the ELB consumer.
Ingesting CloudTrail logs
See here: https://github.com/vigeek/aws-cloudtrail-to-logstash
Example Dashboard (included)
Some minor details from the dashboard image are obfuscated.