Airflow AWS Module

Terraform module to deploy an Apache Airflow cluster on AWS, backed by RDS PostgreSQL for metadata, S3 for logs and SQS as message broker with CeleryExecutor

Terraform supported versions:

Usage

You can use this module from the Terraform Registry

module "airflow-cluster" {
  # REQUIRED
  source                   = "powerdatahub/airflow/aws"
  key_name                 = "airflow-key"
  cluster_name             = "my-airflow"
  cluster_stage            = "prod" # Default is 'dev'
  db_password              = "your-rds-master-password"
  fernet_key               = "your-fernet-key" # see https://airflow.readthedocs.io/en/stable/howto/secure-connections.html

  # OPTIONALS
  vpc_id                   = "some-vpc-id"                     # Use default if not provided
  custom_requirements      = "path/to/custom/requirements.txt" # See examples/custom_requirements for more details
  custom_env               = "path/to/custom/env"              # See examples/custom_env for more details
  ingress_cidr_blocks      = ["0.0.0.0/0"]                     # List of IPv4 CIDR ranges to use on all ingress rules
  ingress_with_cidr_blocks = [                                 # List of computed ingress rules to create where 'cidr_blocks' is used
    {
      description = "List of computed ingress rules for Airflow webserver"
      from_port   = 8080
      to_port     = 8080
      protocol    = "tcp"
      cidr_blocks = "0.0.0.0/0"
    },
    {
      description = "List of computed ingress rules for Airflow flower"
      from_port   = 5555
      to_port     = 5555
      protocol    = "tcp"
      cidr_blocks = "0.0.0.0/0"
    }
  ]
  tags                     = {
    FirstKey  = "first-value"                                  # Additional tags to use on resources
    SecondKey = "second-value"
  }
  load_example_dags        = false
  load_default_conns       = false
  rbac                     = true                              # See examples/rbac for more details
  admin_name               = "John"                            # Only if rbac is true
  admin_lastname           = "Doe"                             # Only if rbac is true
  admin_email              = "[email protected]"                 # Only if rbac is true
  admin_username           = "admin"                           # Only if rbac is true
  admin_password           = "supersecretpassword"             # Only if rbac is true
}

Debug and logs

The Airflow service runs under systemd, so logs are available through journalctl.

$ journalctl -u airflow -n 50

Todo

• [x] Run airflow as systemd service
• [x] Provide a way to pass a custom requirements.txt files on provision step
• [ ] Provide a way to pass a custom packages.txt files on provision step
• [x] RBAC
• [ ] Support for Google OAUTH
• [ ] Flower
• [ ] Secure Flower install
• [x] Provide a way to inject environment variables into airflow
• [ ] Split services into multiples files
• [ ] Auto Scalling for workers
• [ ] Use SPOT instances for workers
• [ ] Maybe use the AWS Fargate to reduce costs

Special thanks to villasv/aws-airflow-stack, an incredible project, for the inspiration.

Requirements

Providers

Inputs

{
  "1": "us-east-1a",
  "2": "us-east-1b",
  "3": "us-east-1c",
  "4": "us-east-1d"
}

[
  "0.0.0.0/0"
]

list(object({
    description = string
    from_port   = number
    to_port     = number
    protocol    = string
    cidr_blocks = string
  }))

[
  {
    "cidr_blocks": "0.0.0.0/0",
    "description": "Airflow webserver",
    "from_port": 8080,
    "protocol": "tcp",
    "to_port": 8080
  },
  {
    "cidr_blocks": "0.0.0.0/0",
    "description": "Airflow flower",
    "from_port": 5555,
    "protocol": "tcp",
    "to_port": 5555
  }
]

Outputs