118 Open source projects that are alternatives of or similar to opentrials-airflow

Blog post on ETL pipelines with Airflow

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

Airflow plugin to export dag and task based metrics to Prometheus.

Flask-AppBuilder SecurityManager for OpenIDConnect

Fast iterative local development and testing of Apache Airflow workflows

A process that runs in unison with Apache Airflow to control the Scheduler process to ensure High Availability

Apache Airflow Website

🐍💻📊 All material from the PyCon.DE 2018 Talk "Beyond Jupyter Notebooks - Building your own data science platform with Python & Docker" (incl. Slides, Video, Udemy MOOC & other References)

Parcel for Apache Airflow

Udacity Data Engineering Nano Degree (DEND)

💨🥫 A Data Factory system for running data processing pipelines built on AirFlow and tailored to CKAN. Includes evolution of DataPusher and Xloader for loading data to DataStore.

Terraform module to deploy an Apache Airflow cluster on AWS, backed by RDS PostgreSQL for metadata, S3 for logs and SQS as message broker with CeleryExecutor

A web frontend for scheduling Jupyter notebook reports

Saisoku is a Python module that helps you build complex pipelines of batch file/directory transfer/sync jobs.

Airflow Unit Tests and Integration Tests

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

A Helm chart to install Apache Airflow on Kubernetes

Kedro-Airflow makes it easy to deploy Kedro projects to Airflow.

Airflow configuration for Telemetry

Apache Airflow - OpenApi Client for Python

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

Kedro Plugin to support running pipelines on Kubernetes using Airflow.

Airflow training for the crunch conf

Pipeline module for parallel real-time data processing for machine learning models development and production purposes.

Airflow workflow management platform chef cookbook.

🔍 Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript

Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested data pipelines(DAGs) 🖥 >> [ 🚀, 🚢 ]

Example DAGs using hooks and operators from Airflow Plugins

Real Time Twitter Sentiment Analysis Product

Curated list of resources about Apache Airflow

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Metric collection, data testing and monitoring for SQL accessible data

Repo that relates to the Medium blog 'Keeping your ML model in shape with Kafka, Airflow' and MLFlow'

📖 [译] Airflow 中文文档

Execution of DBT models using Apache Airflow through Docker Compose

🐳📊🤓Cookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)

A way for home buyers to know about factors affecting a state

Airflow Deployment on AWS ECS Fargate Using Cloudformation

Data pipeline project

An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR

一个基于浏览器环境的数据采集SDK

afctl helps to manage and deploy Apache Airflow projects faster and smoother.

Plugins do Airflow para implementação de pipelines de dados

Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)

Build and deploy a serverless data pipeline on AWS with no effort.

Bitnami Docker Image for Apache Airflow

A lightweight tool to get an AI Infrastructure Stack up in minutes not days. K3ai will take care of setup K8s for You, deploy the AI tool of your choice and even run your code on it.

Example end to end data engineering project.

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Testbed of AI Systems Quality Management

ETLy is an add-on dashboard service on top of Apache Airflow.

PipelineAI Kubeflow Distribution

A distributed workflow runner focusing on performance and simplicity.

Bulk Stash is a docker rclone service to sync, or copy, files between different storage services. For example, you can copy files either to or from a remote storage services like Amazon S3 to Google Cloud Storage, or locally from your laptop to a remote storage.

Workflow engine for Kubernetes

A Clojure machine learning library

TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.

A complete development environment setup for working with Airflow

Real estate dagster pipeline

Functional Airflow DAG definitions.