AWS Msk Kafka Cluster
Terraform module which creates Msk Kafka Cluster on AWS.
These types of resources are supported:
Features
This module create a fully featured Msk Kafka Cluster on Aws. You could configure monitoring, encryption, server options, etc...
Usage
module "msk-cluster" {
source = "angelabad/msk-cluster/aws"
cluster_name = "kafka"
instance_type = "kafka.t3.small"
number_of_nodes = 2
client_subnets = ["subnet-0ab97cbe1bd1406c2", "subnet-0d6cbf60360dbac64"]
kafka_version = "2.6.2"
extra_security_groups = ["sg-019fc0f7d26f6c70f"]
enhanced_monitoring = "PER_BROKER"
s3_logs_bucket = aws_s3_bucket.logs.id
s3_logs_prefix = "msklogs"
prometheus_jmx_exporter = true
prometheus_node_exporter = true
server_properties = {
"auto.create.topics.enable" = "true"
"default.replication.factor" = "2"
}
encryption_in_transit_client_broker = "TLS"
tags = {
Owner = "user"
Environment = "dev"
}
}Requirements
| Name | Version |
|---|---|
| terraform | >= 0.15 |
| aws | >= 4.16 |
| random | >= 2.1 |
Providers
| Name | Version |
|---|---|
| aws | >= 4.16 |
| random | >= 2.1 |
Modules
No modules.
Resources
| Name | Type |
|---|---|
| aws_msk_cluster.this | resource |
| aws_msk_configuration.this | resource |
| aws_msk_scram_secret_association.this | resource |
| aws_security_group.this | resource |
| aws_security_group_rule.jmx-exporter | resource |
| aws_security_group_rule.msk-plain | resource |
| aws_security_group_rule.msk-tls | resource |
| aws_security_group_rule.node_exporter | resource |
| aws_security_group_rule.zookeeper-plain | resource |
| aws_security_group_rule.zookeeper-tls | resource |
| random_id.configuration | resource |
| aws_subnet.this | data source |
Inputs
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| client_authentication_sasl_iam_enabled | Enables IAM client authentication. | bool |
false |
no |
| client_authentication_sasl_scram_secrets_arns | Associates SCRAM secrets stored in the Secrets Manager. You need secret policy. | list(string) |
[] |
no |
| client_authentication_tls_certificate_authority_arns | List of ACM Certificate Authority Amazon Resource Names (ARNs). | list(string) |
[] |
no |
| client_authentication_unauthenticated_enabled | Enables unauthenticated access. | bool |
false |
no |
| client_subnets | A list of subnets to connect to in client VPC | list(string) |
n/a | yes |
| cloudwatch_logs_group | Name of the Cloudwatch Log Group to deliver logs to. | string |
"" |
no |
| cluster_name | Name of the MSK cluster. | string |
n/a | yes |
| encryption_at_rest_kms_key_arn | You may specify a KMS key short ID or ARN (it will always output an ARN) to use for encrypting your data at rest. If no key is specified, an AWS managed KMS ('aws/msk' managed service) key will be used for encrypting the data at rest. | string |
"" |
no |
| encryption_in_transit_client_broker | Encryption setting for data in transit between clients and brokers. Valid values: TLS, TLS_PLAINTEXT, and PLAINTEXT. Default value is TLS_PLAINTEXT. | string |
"TLS_PLAINTEXT" |
no |
| encryption_in_transit_in_cluster | Whether data communication among broker nodes is encrypted. Default value: true. | bool |
true |
no |
| enhanced_monitoring | Specify the desired enhanced MSK CloudWatch monitoring level to one of three monitoring levels: DEFAULT, PER_BROKER, PER_TOPIC_PER_BROKER or PER_TOPIC_PER_PARTITION. See Monitoring Amazon MSK with Amazon CloudWatch. | string |
"DEFAULT" |
no |
| extra_security_groups | A list of extra security groups to associate with the elastic network interfaces to control who can communicate with the cluster. | list(string) |
[] |
no |
| firehose_logs_delivery_stream | Name of the Kinesis Data Firehose delivery stream to deliver logs to. | string |
"" |
no |
| instance_type | Specify the instance type to use for the kafka brokers. e.g. kafka.m5.large. | string |
n/a | yes |
| kafka_version | Specify the desired Kafka software version. | string |
n/a | yes |
| number_of_nodes | The desired total number of broker nodes in the kafka cluster. It must be a multiple of the number of specified client subnets. | number |
n/a | yes |
| prometheus_jmx_exporter | Indicates whether you want to enable or disable the JMX Exporter. | bool |
false |
no |
| prometheus_node_exporter | Indicates whether you want to enable or disable the Node Exporter. | bool |
false |
no |
| provisioned_volume_throughput | Throughput value of the EBS volumes for the data drive on each kafka broker node in MiB per second. The minimum value is 250. The maximum value varies between broker type. See [https://docs.aws.amazon.com/msk/latest/developerguide/msk-provision-throughput.html#throughput-bottlenecks](documentation on throughput bottlenecks). | number |
null |
no |
| s3_logs_bucket | Name of the S3 bucket to deliver logs to. | string |
"" |
no |
| s3_logs_prefix | Prefix to append to the folder name. | string |
"" |
no |
| server_properties | A map of the contents of the server.properties file. Supported properties are documented in the MSK Developer Guide. | map(string) |
{} |
no |
| tags | A mapping of tags to assign to the resource. | map(string) |
{} |
no |
| volume_size | The size in GiB of the EBS volume for the data drive on each broker node. | number |
1000 |
no |
Outputs
| Name | Description |
|---|---|
| arn | Amazon Resource Name (ARN) of the MSK cluster. |
| bootstrap_brokers | A comma separated list of one or more hostname:port pairs of kafka brokers suitable to boostrap connectivity to the kafka cluster. Only contains value if client_broker encryption in transit is set o PLAINTEXT or TLS_PLAINTEXT. |
| bootstrap_brokers_tls | A comma separated list of one or more DNS names (or IPs) and TLS port pairs kafka brokers suitable to boostrap connectivity to the kafka cluster. Only contains value if client_broker encryption in transit is set to TLS_PLAINTEXT or TLS. |
| current_version | Current version of the MSK Cluster used for updates, e.g. K13V1IB3VIYZZH |
| default_security_group | Msk cluster default security group id. |
| encryption_at_rest_kms_key_arn | The ARN of the KMS key used for encryption at rest of the broker data volumes. |
| zookeeper_connect_string | A comma separated list of one or more hostname:port pairs to use to connect to the Apache Zookeeper cluster. |
Authors
Module managed by Angel Abad
License
Apache 2 Licensed. See LICENSE for full details