512 Open source projects that are alternatives of or similar to AirflowETL

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Example of an ETL Pipeline using Airflow

Example end to end data engineering project.

Metric collection, data testing and monitoring for SQL accessible data

ETLy is an add-on dashboard service on top of Apache Airflow.

A schema-aware Scala library for data transformation

Real-Time Event Streaming & Change Data Capture

ELT Code for your Data Warehouse

A python package to create a database on the platform using our moj data warehousing framework

A list of useful resources to learn Data Engineering from scratch

A client for connecting and running DDLs on hive metastore.

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

DIRECT, the Data Integration Run-time Execution Control Tool, is a data logistics framework that can be used to monitor, log, audit and control data integration / ETL processes.

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs on GCP and AWS.

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

Python library for building Pangeo Forge recipes.

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

A tool for building feature stores.

Airflow Deployment on AWS ECS Fargate Using Cloudformation

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.

Apache DolphinScheduler is a distributed and extensible workflow scheduler platform with powerful DAG visual interfaces, dedicated to solving complex job dependencies in the data pipeline and providing various types of jobs available out of box.

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

Real estate dagster pipeline

Udacity Data Engineering Nano Degree (DEND)

Visualize Airflow's schedule by exporting future DAG runs as events to Google Calendar.

Powerful RDF Knowledge Graph Generation with [R2]RML Mappings

[READ-ONLY] PHP - ETL (Extract Transform Load) data processing library

A production-grade data pipeline has been designed to automate the parsing of user search patterns to analyze user engagement. Extract data from S3, apply a series of transformations and load into S3 and Redshift.

Personal Data Engineering Projects

csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

An Elasticsearch client tailored to data science workflows.

A Kafka Connect source connector that generates data for tests

Rivery CLI

Beneath is a serverless real-time data platform ⚡️

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

A simple Spark-powered ETL framework that just works 🍺

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

www.vixtract.ru

Example project implementing best practices for PySpark ETL jobs and applications.

Enterprise-grade, production-hardened, serverless data lake on AWS

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

Fancy stream processing made operationally mundane

Configuration and definitions of Airflow for OpenTrials

A Data Engineering & Machine Learning Knowledge Hub

Example DAGs using hooks and operators from Airflow Plugins

A web frontend for scheduling Jupyter notebook reports

Bitnami Docker Image for Apache Airflow

MRT Schedule & Locator iOS App built using SwiftUI

Curated list of resources about Apache Airflow

A process that runs in unison with Apache Airflow to control the Scheduler process to ensure High Availability