181 Open source projects that are alternatives of or similar to Airflow Autoscaling Ecs

Example end to end data engineering project.

Blog post on ETL pipelines with Airflow

Example of an ETL Pipeline using Airflow

Metric collection, data testing and monitoring for SQL accessible data

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

Personal Data Engineering Projects

A Data Engineering & Machine Learning Knowledge Hub

Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation, validation and loading of data from S3 -> Redshift -> S3

📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.

dbt adapter for SQL Server and Azure SQL

Continuously synchronize directories from remote object store to local filesystem

Accelerate data science

Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/

A series of DAGs/Workflows to help maintain the operation of Airflow

Elyra extends JupyterLab Notebooks with an AI centric approach.

A simple Spark-powered ETL framework that just works 🍺

Automating Your Data Pipeline with Apache Airflow

A tool for building feature stores.

📊 📋 Dashboards using YAML or JSON files

The Data Engineering Cookbook

The easiest way to automate your data

Workflow engine for Kubernetes

Bitnami Docker Image for Apache Airflow

Quilt is a self-organizing data hub for S3

Collection of open-source Spark tools & frameworks that have made the data engineering and data science teams at Swoop highly productive

Bare minimal Airflow on Kubernetes (Local, EKS, AKS)

Udacity Data Engineering Nano Degree (DEND)

PipelineX: Python package to build ML pipelines for experimentation with Kedro, MLflow, and more

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

Airflow training for the crunch conf

A facebook for data

afctl helps to manage and deploy Apache Airflow projects faster and smoother.

Git-like capabilities for your object storage

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

Example project implementing best practices for PySpark ETL jobs and applications.

An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR

Clean APIs for data cleaning. Python implementation of R package Janitor

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Fast iterative local development and testing of Apache Airflow workflows

Apache DolphinScheduler is a distributed and extensible workflow scheduler platform with powerful DAG visual interfaces, dedicated to solving complex job dependencies in the data pipeline and providing various types of jobs available out of box.

Terraform module to deploy an Apache Airflow cluster on AWS, backed by RDS PostgreSQL for metadata, S3 for logs and SQS as message broker with CeleryExecutor

Data validation and organization of metadata for data frames and database tables

Ansible playbook to deploy distributed technologies

Accumulated knowledge and experience in the field of Data Engineering

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.

Always know what to expect from your data.

Turn complex requirements to workflows without leaving the comfort of your technology stack.

ETLy is an add-on dashboard service on top of Apache Airflow.

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

Dynamically generate Apache Airflow DAGs from YAML configuration files

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks