98 Open source projects that are alternatives of or similar to Data Pipelines With Apache Airflow

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

Apache Airflow CI pipeline

Minikube for big data with Scala and Spark

Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Kubernetes custom controller and CRDs to managing Airflow

Build DataOps platform with Apache Airflow and dbt on AWS

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

A complete development environment setup for working with Airflow

Bitnami Docker Image for Apache Airflow Scheduler

Apache Airflow integration for dbt

Real Time Twitter Sentiment Analysis Product

Docker Apache Airflow

Export Airflow metrics (from mysql) in prometheus format

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Observatory Platform Package

Yet another helpdesk based on multiple providers

Visualize Airflow's schedule by exporting future DAG runs as events to Google Calendar.

Elyra extends JupyterLab Notebooks with an AI centric approach.

TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.

A plugin for Apache Airflow that allows you to manage the users that can login

Apache Airflow - OpenApi Client for Python

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

A lightweight tool to get an AI Infrastructure Stack up in minutes not days. K3ai will take care of setup K8s for You, deploy the AI tool of your choice and even run your code on it.

A plugin for Apache Airflow that allows you to edit DAGs in browser

Identifies and collects data on cc-licensed content across web crawl data and public apis.

Repo that relates to the Medium blog 'Keeping your ML model in shape with Kafka, Airflow' and MLFlow'

Airflow basics tutorial

Use Airflow to move data from multiple MySQL databases to BigQuery

Air Data Computer

A plugin for Apache Airflow that exposes rest end points for the Command Line Interfaces

Personal Data Engineering Projects

A series of DAGs/Workflows to help maintain the operation of Airflow

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

A Data Engineering & Machine Learning Knowledge Hub

Get your MLOps (Level 1) platform started and going fast.

Apache DolphinScheduler is a distributed and extensible workflow scheduler platform with powerful DAG visual interfaces, dedicated to solving complex job dependencies in the data pipeline and providing various types of jobs available out of box.

DBT Cloud Plugin for Airflow

No description or website provided.

Apache Liminals goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production. Liminal provides a Domain Specific Language to build ML workflows on top of Apache Airflow.

Continuously synchronize directories from remote object store to local filesystem

Configuration and definitions of Airflow for OpenTrials

Example of an ETL Pipeline using Airflow

Plugins do Airflow para implementação de pipelines de dados

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Execution of DBT models using Apache Airflow through Docker Compose

Astronomer Core Docker Images

Testbed of AI Systems Quality Management

A facebook for data

Parcel for Apache Airflow

A production-grade data pipeline has been designed to automate the parsing of user search patterns to analyze user engagement. Extract data from S3, apply a series of transformations and load into S3 and Redshift.

Functional Airflow DAG definitions.

Dynamically generate Apache Airflow DAGs from YAML configuration files

Cloudformation templates for deploying Airflow in ECS

Bare minimal Airflow on Kubernetes (Local, EKS, AKS)

Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/

Automating Your Data Pipeline with Apache Airflow

Turbine: the bare metals that gets you Airflow

triggering a DAG run multiple times