Cloudwatchmetricbeat
Welcome to Cloudwatchmetricbeat. Based on previous work on Cloudwatch Logs and Metrics
Installation
Download a binary, and put it in a good spot on your system. Or use the Dockerfile to build a Docker image.
To run this beat with a given configuration, run:
./cloudwatchmetricbeat -c cloudwatchmetricbeat.yml -e -d "*"
Credentials and permissions
The CloudWatch Exporter uses the
AWS Go SDK,
which offers a variety of ways to provide credentials.
This includes the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment
variables.
The cloudwatch:ListMetrics and cloudwatch:GetMetricStatistics IAM permissions are required.
Configuration
The configuration is in YAML, an example with common options:
---
aws_region: us-east-1
metrics:
- aws_namespace: AWS/ELB
aws_metric_name: RequestCount
aws_dimensions: [AvailabilityZone, LoadBalancerName]
aws_dimension_select:
LoadBalancerName: [myLB]
aws_statistics: [Sum]
| Name | Description |
|---|---|
| aws_region | Required. The AWS region to connect to. |
| metrics | Required. A list of CloudWatch metrics to retrieve and export |
| aws_namespace | Required. Namespace of the CloudWatch metric. |
| aws_metric_name | Required. Metric name of the CloudWatch metric. |
| aws_dimensions | Optional. Which dimension to fan out over. |
| aws_dimension_select | Optional. Which dimension values to filter. Specify a map from the dimension name to a list of values to select from that dimension. |
| aws_statistics | Optional. A list of statistics to retrieve, values can include Sum, SampleCount, Minimum, Maximum, Average. Defaults to Average. |
| delay_seconds | Optional. The newest data to request. Used to avoid collecting data that has not fully converged. Defaults to 300s. |
| range_seconds | Optional. How far back to request data for. Useful for cases such as Billing metrics that are only set every few hours. Defaults to 600s. |
| period_seconds | Optional. Period to request the metric for. Only the most recent data point is used. Defaults to 60s. Can be set globally and per metric. |
CloudWatch has been observed to sometimes take minutes for reported values to
converge. The default delay_seconds will result in data that is at least 5
minutes old being requested to mitigate this.
- Data points with a period of less than 60 seconds are available for 3 hours. These data points are high-resolution metrics and are available only for custom metrics that have been defined with a StorageResolution of 1.
- Data points with a period of 60 seconds (1-minute) are available for 15 days.
- Data points with a period of 300 seconds (5-minute) are available for 63 days.
- Data points with a period of 3600 seconds (1 hour) are available for 455 days (15 months).
Cost
Amazon charges for every API request, see the current charges.
Every metric retrieved requires one API request, which can include multiple
statistics. In addition, when aws_dimensions is provided, the exporter needs
to do API requests to determine what metrics to request. This should be
negligible compared to the requests for the metrics themselves.
If you have 100 API requests every minute, with the price of USD$10 per million requests (as of Jan 2015), that is around $45 per month.
Getting Started with Cloudwatchmetricbeat
Requirements
Init Project
Clone this repository at the following location: ${GOPATH}/github.com/narmi/cloudwatchmetricbeat.
To get running with Cloudwatchmetricbeat and also install the dependencies, run the following command:
make setup
It will create a clean git history for each major step. Note that you can always rewrite the history if you wish before pushing your changes.
To push Cloudwatchmetricbeat in the git repository, run the following commands:
git remote set-url origin https://github.com/narmi/cloudwatchmetricbeat
git push origin master
For further development, check out the beat developer guide.
Build
To build the binary for Cloudwatchmetricbeat run the commands below. This will generate a binary in the same directory with the name cloudwatchmetricbeat.
make collect
make update
make
Run
To run Cloudwatchmetricbeat with debugging output enabled, run:
./cloudwatchmetricbeat -c cloudwatchmetricbeat.yml -e -d "*"
Test
To test Cloudwatchmetricbeat, run the following command:
make testsuite
alternatively:
make unit-tests
make system-tests
make integration-tests
make coverage-report
The test coverage is reported in the folder ./build/coverage/
Update
Each beat has a template for the mapping in elasticsearch and a documentation for the fields which is automatically generated based on etc/fields.yml.
To generate etc/cloudwatchmetricbeat.template.json and etc/cloudwatchmetricbeat.asciidoc
make update
Cleanup
To clean Cloudwatchmetricbeat source code, run the following commands:
make fmt
make simplify
To clean up the build directory and generated artifacts, run:
make clean
Clone
To clone Cloudwatchmetricbeat from the git repository, run the following commands:
mkdir -p ${GOPATH}/github.com/narmi/cloudwatchmetricbeat
cd ${GOPATH}/github.com/narmi/cloudwatchmetricbeat
git clone https://github.com/narmi/cloudwatchmetricbeat
For further development, check out the beat developer guide.
Packaging
The beat frameworks provides tools to crosscompile and package your beat for different platforms. This requires docker and vendoring as described above. To build packages of your beat, run the following command:
make package
This will fetch and create all images required for the build process. The whole process to finish can take several minutes.