98 Open source projects that are alternatives of or similar to Airflow Testing

Turbine: the bare metals that gets you Airflow

Minikube for big data with Scala and Spark

Workflow engine for Kubernetes

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

triggering a DAG run multiple times

ETLy is an add-on dashboard service on top of Apache Airflow.

Kubernetes custom controller and CRDs to managing Airflow

Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)

Bitnami Docker Image for Apache Airflow Scheduler

Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

afctl helps to manage and deploy Apache Airflow projects faster and smoother.

Docker Apache Airflow

A distributed workflow runner focusing on performance and simplicity.

Yet another helpdesk based on multiple providers

Airflow Deployment on AWS ECS Fargate Using Cloudformation

A plugin for Apache Airflow that allows you to manage the users that can login

Bare minimal Airflow on Kubernetes (Local, EKS, AKS)

A plugin for Apache Airflow that allows you to edit DAGs in browser

Bitnami Docker Image for Apache Airflow

Apache Airflow integration for dbt

Elyra extends JupyterLab Notebooks with an AI centric approach.

Air Data Computer

Airflow training for the crunch conf

Apache DolphinScheduler is a distributed and extensible workflow scheduler platform with powerful DAG visual interfaces, dedicated to solving complex job dependencies in the data pipeline and providing various types of jobs available out of box.

Airflow configuration for Telemetry

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Terraform module to deploy an Apache Airflow cluster on AWS, backed by RDS PostgreSQL for metadata, S3 for logs and SQS as message broker with CeleryExecutor

Dynamically generate Apache Airflow DAGs from YAML configuration files

A Helm chart to install Apache Airflow on Kubernetes

Airflow basics tutorial

Airflow workflow management platform chef cookbook.

A plugin for Apache Airflow that exposes rest end points for the Command Line Interfaces

Fast iterative local development and testing of Apache Airflow workflows

A Data Engineering & Machine Learning Knowledge Hub

Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested data pipelines(DAGs) 🖥 >> [ 🚀, 🚢 ]

No description or website provided.

Airflow plugin to export dag and task based metrics to Prometheus.

Example of an ETL Pipeline using Airflow

Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation, validation and loading of data from S3 -> Redshift -> S3

Astronomer Core Docker Images

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

A production-grade data pipeline has been designed to automate the parsing of user search patterns to analyze user engagement. Extract data from S3, apply a series of transformations and load into S3 and Redshift.

Continuously synchronize directories from remote object store to local filesystem

Cloudformation templates for deploying Airflow in ECS

🐍💻📊 All material from the PyCon.DE 2018 Talk "Beyond Jupyter Notebooks - Building your own data science platform with Python & Docker" (incl. Slides, Video, Udemy MOOC & other References)

Identifies and collects data on cc-licensed content across web crawl data and public apis.

A series of DAGs/Workflows to help maintain the operation of Airflow

Use Airflow to move data from multiple MySQL databases to BigQuery

Udacity Data Engineering Nano Degree (DEND)

A facebook for data

📖 [译] Airflow 中文文档

🐳📊🤓Cookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)

An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR

Example end to end data engineering project.

Automating Your Data Pipeline with Apache Airflow