Top 96 data-engineering open source projects

The practical use-cases of how to make your Machine Learning Pipelines robust and reliable using Apache Airflow.

Personal Data Engineering Projects

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

A schema-aware Scala library for data transformation

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

A Pachyderm deep learning tutorial for conference workshops

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

ETH PhD Program course

Clean APIs for data cleaning. Python implementation of R package Janitor

dbt-sugar is a CLI tool that allows users of dbt to have fun and ease performing actions around dbt models

An Elasticsearch client tailored to data science workflows.

funsies is a lightweight workflow engine 🔧

Python package for Customizable Data Preprocessing Pipelines

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

Real estate dagster pipeline

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Datart is a next generation Data Visualization Open Platform

Road to Azure Data Engineer Part-I: DP-200 - Implementing an Azure Data Solution

Powerful RDF Knowledge Graph Generation with [R2]RML Mappings

Python client for using Prefect Cloud with Saturn Cloud

Useful SQL queries for Blockchain ETL datasets in BigQuery.

DE or DIE meetup made by data engineers for data engineers. Currently in Russian only.

Dummy variable generation with fit/transform capabilities

Easy way to define, execute and store quality rules for your data.

A collection of online resources to help you on your Tech journey.

DEPRECATED: Stream data processing micro-framework

A curated list of big data engineering tools, resources and communities.

Less-Resilient MapReduce framework for Go

Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes

[READ-ONLY] PHP - ETL (Extract Transform Load) data processing library

CSVs sliced, diced & analyzed.

A client for connecting and running DDLs on hive metastore.

Blog post on ETL pipelines with Airflow

A curated list of awesome dbt resources

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.