AWS Security Group exporter for Prometheus

Motivation | How does it work | Requirements | Docker settings | Metrics | Examples | Grafana | Errors

A dockerized^[1] Prometheus exporter that compares desired/wanted IPv4/IPv6 CIDR against currently applied inbound CIDR rules by protocol and port number in your AWS security group(s) per region.

^[1]: If you want to use this exporter without Docker jump here: Usage without Docker

Motivation

Some IP addresses ranges such as Cloudfront edge nodes or SaaS hosts might change frequently and you possibly want to ensure that those are always in sync with what you have currently defined in your security group. This exporter does exactly this and can easily be hooked up with Alertmanager to trigger alerts in case you get out of sync.

How does it work

Desired/Wanted IP address CIDR

You have to provide a command, which is parsable by bash's eval function and evalutes newline-separated to your desired/wanted IP address CIDR. As a few examples:

# Note that for single IP addresses, AWS requires '/32' to be appended
eval "dig +short nat.travisci.net | xargs -n1 -I% echo \"%/32\""
eval "printf \"10.13.23.23/32\n192.168.0.0/24\n\""

Applied security Group CIDR

You have to provide the following in order to fetch your currently applied sg rules:

• Security group name (The Name tag)^[1]
• AWS region where the security group resides
• Security group rule protocol (e.g.: tcp, udp, icmp, ...)
• Security group rule from port (e.g.: 80, 443, ...)

^[1]: The * wildcard is supported for the name, but you have to ensure to match exactly one security group

Output

The exporter will then output Prometheus readable information as such:

# HELP aws_ec2_sg_compare Determines If CIDR is applied to security group.
# TYPE aws_ec2_sg_compare counter
aws_ec2_sg_compare{name="sg-name",region="us-east-1",proto="tcp",from_port="80",ip="v4",cidr="10.4.1.1/32",sg_id="sg-xxxxx",errno="0",error=""} 1
aws_ec2_sg_compare{name="sg-name",region="us-east-1",proto="tcp",from_port="80",ip="v4",cidr="10.4.1.5/32",sg_id="sg-xxxxx",errno="0",error=""} 0

• A value of 1 means the desired/wanted IP CIDR is applied to the security group
• A value of 0 means the desired/wanted IP CIDR is not applied to the security group

See Metrics for an indepth description.

Requirements

You will need AWS access key and secret with the following permission:

ec2:DescribeSecurityGroups

Docker settings

Tagging

Ensure to use Docker image tags (which are the same as git tags from this repository) to prevent any backwards incompatible changes. The latest tag should only be used for testing purposes.

Additionally do not blindly update Docker image tags before having tested it. Security group rule checks are an important matter and you want to ensure your alerting is reliable.

Environment variables

You can specify up to 4 security group checks: SG1_*, SG2_*, SG3_* and SG4_*.

^[1]: SG*_IP4_CMD and SG*_IP6_CMD are mutually exclusive. Also note that evaluated IP address CIDR are only checked against security group rules that match the protocol (SG*_PROTO) and also match the from port (SG*_FROM_PORT).

Mount points

None - it's fully stateless

Exposed ports

Metrics

This exporter outputs metrics in the following format:

# HELP aws_ec2_sg_compare Determines If CIDR is applied to security group.
# TYPE aws_ec2_sg_compare counter
aws_ec2_sg_compare{name="sg-name",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="10.4.1.1/32",sg_id="sg-xxxxx",errno="0",error=""} 1

The following table describes each of the key/value paris:

• A value of 1 means the desired/wanted IP CIDR is applied to the security group
• A value of 0 means the desired/wanted IP CIDR is not applied to the security group

Examples

Scenario 1 - Travis

Check if your security group named my-sg (in us-east-1) allows all inbound IPv4 addresses from Travis-CI via tcp on https.

Desired/wanted IP CIDR

Ensure you have a working command which can be interpretated by eval and that outputs CIDR (with /[0-9]+ appended) of your desired ranges:

$ eval "dig +short nat.travisci.net | xargs -n1 -I% echo \"%/32\""

35.184.226.236/32
35.188.1.99/32
35.188.73.34/32
35.192.85.2/32
35.192.136.167/32
...

Run aws-ec2-sg-exporter

docker run -it --rm \
	-p 9000:8080 \
	\
	-e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
	-e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
	\
	-e SG1_NAME="my-sg" \
	-e SG1_REGION="us-east-1" \
	-e SG1_PROTO="tcp" \
	-e SG1_FROM_PORT="443" \
	-e SG1_IP4_CMD="dig +short nat.travisci.net | xargs -n1 -I% echo \"%/32\"" \
	cytopia/aws-ec2-sg-exporter

Check output

Check the output via curl:

$ curl localhost:9000`

# HELP aws_ec2_sg_compare Determines If CIDR is applied to security group.
# TYPE aws_ec2_sg_compare counter
aws_ec2_sg_compare{name="my-sg",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="35.184.226.236/32",sg_id="sg-xxxxx",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="35.188.1.99/32",sg_id="sg-xxxxx",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="35.188.73.34/32",sg_id="sg-xxxxx",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="35.192.85.2/32",sg_id="sg-xxxxx",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="35.192.136.167/32",sg_id="sg-xxxxx",errno="0",error=""} 0
...

As you can see, the last line returns a 0, which means this IP CIDR is missing in your security group.

Scenario 2 - Cloudfront

• Check if your security group named my-sg4 (in us-east-1) allows all inbound IPv4 addresses from Cloudfront edge-nodes via tcp on https.
• Check if your security group named my-sg6 (in us-east-1) allows all inbound IPv6 addresses from Cloudfront edge-nodes via tcp on https.

Desired/wanted IP CIDR

Ensure you have a working command which can be interpretated by eval and that outputs CIDR (with /[0-9]+ appended) of your desired ranges:

$ eval "curl -sS https://ip-ranges.amazonaws.com/ip-ranges.json \
	| jq -r '.prefixes \
		| sort_by(.ip_prefix)[] \
		| select( .service | contains(\"CLOUDFRONT\")) \
		| select ( .region | test(\"^(GLOBAL|us-|eu-)\")) \
		| .ip_prefix'"

13.224.0.0/14
13.249.0.0/16
13.32.0.0/15
13.35.0.0/16
13.52.204.0/23
...

$ eval "curl -sS https://ip-ranges.amazonaws.com/ip-ranges.json \
	| jq -r '.ipv6_prefixes \
		| sort_by(.ipv6_prefixes)[] \
		| select( .service | contains(\"CLOUDFRONT\")) \
		| select ( .region | test(\"^(GLOBAL|us-|eu-)\")) \
		| .ipv6_prefix'"

2600:9000:eee::/48
2600:9000:4000::/36
2600:9000:3000::/36
2600:9000:f000::/36
2600:9000:fff::/48
...

Run aws-ec2-sg-exporter

docker run -it --rm \
	-p 9000:8080 \
	\
	-e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
	-e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
	\
	-e SG1_NAME="my-sg4" \
	-e SG1_REGION="us-east-1" \
	-e SG1_PROTO="tcp" \
	-e SG1_FROM_PORT="443" \
	-e SG1_IP4_CMD="curl -sS https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes | sort_by(.ip_prefix)[] | select( .service | contains(\"CLOUDFRONT\")) | select ( .region | test(\"^(GLOBAL|us-|eu-)\")) | .ip_prefix'" \
	\
	-e SG2_NAME="my-sg6" \
	-e SG2_REGION="us-east-1" \
	-e SG2_PROTO="tcp" \
	-e SG2_FROM_PORT="443" \
	-e SG2_IP6_CMD="curl -sS https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.ipv6_prefixes | sort_by(.ipv6_prefixes)[] | select( .service | contains(\"CLOUDFRONT\")) | select ( .region | test(\"^(GLOBAL|us-|eu-)\")) | .ipv6_prefix'" \
	cytopia/aws-ec2-sg-exporter

Check output

Check the output via curl:

$ curl localhost:9000`

# HELP aws_ec2_sg_compare Determines If CIDR is applied to security group.
# TYPE aws_ec2_sg_compare counter
aws_ec2_sg_compare{name="my-sg4",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="13.224.0.0/14",sg_id="sg-xxxxx",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg4",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="13.249.0.0/16",sg_id="sg-xxxxx",errno="0",error=""} 0
aws_ec2_sg_compare{name="my-sg4",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="13.32.0.0/15",sg_id="sg-xxxxx",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg4",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="13.35.0.0/16",sg_id="sg-xxxxx",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg4",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="13.52.204.0/23",sg_id="sg-xxxxx",errno="0",error=""} 1
...
aws_ec2_sg_compare{name="my-sg6",region="us-east-1",proto="tcp",from_port="443",ip="v6",cidr="2600:9000:eee::/48",sg_id="sg-yyyyy",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg6",region="us-east-1",proto="tcp",from_port="443",ip="v6",cidr="2600:9000:4000::/36",sg_id="sg-yyyyy",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg6",region="us-east-1",proto="tcp",from_port="443",ip="v6",cidr="2600:9000:3000::/36",sg_id="sg-yyyyy",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg6",region="us-east-1",proto="tcp",from_port="443",ip="v6",cidr="2600:9000:f000::/36",sg_id="sg-yyyyy",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg6",region="us-east-1",proto="tcp",from_port="443",ip="v6",cidr="2600:9000:fff::/48",sg_id="sg-yyyyy",errno="0",error=""} 0
...

As you can see, the second line ipv4 address returns a 0 and the last ipv6 address returns a 0, which means these IP CIDR are missing in your security groups.

Grafana setup

Graphs

• Align the Min time interval with what you have set UPDATE_TIME to.
• Add you metrics by the name of your specified security group name
• Set the legend to {{ cidr }} to have only the CIDR displayed

Once this is done, your graph will look similar to this one:

Single Stat

• Align the Min time interval with what you have set UPDATE_TIME to.
• Add you metrics by the name of your specified security group name
• sum() gives your the total sum of values (0 and 1) and count() will give you the total number of available IP addresses

Once this is done, your single stat will look similar to this one:

Usage without Docker

Docker is not necessarily required and you can simply use the exporter bash script: aws-ec2-sg-exporter.

By doing so, you need to ensure you have all requirements installed on your system (aws and jq binary as well as bash itself).

Additionally you will have to make sure the script's stdout will somehow be served by a webserver. The recommended method is to have some mechanism which writes the script's output atomically to a static html file and a web server will simply serve it.

aws-ec2-sg-exporter will read all configuration from the shell's environment, so in order to use this script you need to export all values to your env. See Environment variables for possible values.

Error handling

The exporter writes all errors to stderr regardless of using Docker or the standalone aws-ec2-sg-exporter script.

Expected errors

An error occurred (RequestExpired) when calling the DescribeSecurityGroups operation: Request has expired.

In case you are using IAM roles, your session has simply been expired and needs to be renewed. It is recommended to user IAM users instead without session.

[ERR] 2019-08-18 10:55:11 (aws-ec2-sg-exporter): No sg found by name: sg-name22 in region: us-east-1

A security group could not be found by name and region. The exporter will continue to run and output Prometheus metrics, but will mark all desired/wanted IP CIDR as not found in your security group.

[ERR] 2019-08-18 10:56:17 (aws-ec2-sg-exporter): Multiple sg found by name: sg-name-* in region: us-east-1: sg-xxxxx,sg-yyyyy

Multiple security groups have been found by the specified name and region. The exporter will continue to run and output Prometheus metrics, but will mark all desired/wanted IP CIDR as not found in your security group.

Unexpected errors

write error: Broken pipe

This is a very rare condition and will most likely be caused by using broken shell pipes (|) in your commands specified via SG*_IP4_CMD or SG*_IP6_CMD.

In case you are using something like this:

curl http://some-page.tld | grep -E '^[.0-9]+/[0-9]+$';

Consider to add a buffer in between:

curl http://some-page.tld | dd obs=1M 2>/dev/null | grep -E '^[.0-9]+/[0-9]+$';

See here: https://superuser.com/a/642932/705357

License

MIT License

Copyright (c) 2019 cytopia