Top 99 airflow open source projects

Example DAGs using hooks and operators from Airflow Plugins

A web frontend for scheduling Jupyter notebook reports

Curated list of resources about Apache Airflow

A process that runs in unison with Apache Airflow to control the Scheduler process to ensure High Availability

Metric collection, data testing and monitoring for SQL accessible data

Airflow Unit Tests and Integration Tests

📖 [译] Airflow 中文文档

Airflow plugin to export dag and task based metrics to Prometheus.

🐳📊🤓Cookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)

A Helm chart to install Apache Airflow on Kubernetes

Airflow Deployment on AWS ECS Fargate Using Cloudformation

🐍💻📊 All material from the PyCon.DE 2018 Talk "Beyond Jupyter Notebooks - Building your own data science platform with Python & Docker" (incl. Slides, Video, Udemy MOOC & other References)

An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR

Airflow configuration for Telemetry

afctl helps to manage and deploy Apache Airflow projects faster and smoother.

Fast iterative local development and testing of Apache Airflow workflows

Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

Bitnami Docker Image for Apache Airflow

Udacity Data Engineering Nano Degree (DEND)

Example end to end data engineering project.

Airflow training for the crunch conf

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Terraform module to deploy an Apache Airflow cluster on AWS, backed by RDS PostgreSQL for metadata, S3 for logs and SQS as message broker with CeleryExecutor

ETLy is an add-on dashboard service on top of Apache Airflow.

Airflow workflow management platform chef cookbook.

A distributed workflow runner focusing on performance and simplicity.

Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested data pipelines(DAGs) 🖥 >> [ 🚀, 🚢 ]

Workflow engine for Kubernetes

Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation, validation and loading of data from S3 -> Redshift -> S3

Bare minimal Airflow on Kubernetes (Local, EKS, AKS)

Continuously synchronize directories from remote object store to local filesystem

Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/

A series of DAGs/Workflows to help maintain the operation of Airflow

Elyra extends JupyterLab Notebooks with an AI centric approach.

A facebook for data

Automating Your Data Pipeline with Apache Airflow

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Apache DolphinScheduler is a distributed and extensible workflow scheduler platform with powerful DAG visual interfaces, dedicated to solving complex job dependencies in the data pipeline and providing various types of jobs available out of box.

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

Dynamically generate Apache Airflow DAGs from YAML configuration files

Turbine: the bare metals that gets you Airflow

Airflow basics tutorial

Docker Apache Airflow

A plugin for Apache Airflow that exposes rest end points for the Command Line Interfaces

Kubernetes custom controller and CRDs to managing Airflow

A Data Engineering & Machine Learning Knowledge Hub

Yet another helpdesk based on multiple providers

No description or website provided.

Minikube for big data with Scala and Spark

Example of an ETL Pipeline using Airflow

A plugin for Apache Airflow that allows you to manage the users that can login

Astronomer Core Docker Images

Bitnami Docker Image for Apache Airflow Scheduler

A production-grade data pipeline has been designed to automate the parsing of user search patterns to analyze user engagement. Extract data from S3, apply a series of transformations and load into S3 and Redshift.

A plugin for Apache Airflow that allows you to edit DAGs in browser

Cloudformation templates for deploying Airflow in ECS