Top 96 data-engineering open source projects

A daily digest of the articles or videos I've found interesting, that I want to share with you.

Go to: https://github.com/alexklibisz/elastiknn

A convention over configuration workflow orchestrator. Develop locally (Jupyter or your favorite editor), deploy to Airflow or Kubernetes.

A package to easily open an instance of a Google spreadsheet and interact with worksheets through Pandas DataFrames.

Enterprise-grade, production-hardened, serverless data lake on AWS

Metric collection, data testing and monitoring for SQL accessible data

An automatic ML model optimization tool.

Free and open source schema versioning and database migration made natively with .NET Core.

A Clojure dataframe library that runs on Spark

Projects done in the Data Engineering Nanodegree by Udacity.com

Study materials for the Google Cloud Professional Data Engineering Exam

A list of useful resources to learn Data Engineering from scratch

The Accelerator is a tool for fast and reproducible processing of large amounts of data.

Airflow Deployment on AWS ECS Fargate Using Cloudformation

PipelineX: Python package to build ML pipelines for experimentation with Kedro, MLflow, and more

A tool for building feature stores.

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Collection of open-source Spark tools & frameworks that have made the data engineering and data science teams at Swoop highly productive

Accelerate data science

📊 📋 Dashboards using YAML or JSON files

Apache Superset is a Data Visualization and Data Exploration Platform

📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.

Example end to end data engineering project.

A simple Spark-powered ETL framework that just works 🍺

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

Ansible playbook to deploy distributed technologies

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.

Quilt is a self-organizing data hub for S3

dbt adapter for SQL Server and Azure SQL

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

Git-like capabilities for your object storage

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

The easiest way to automate your data

Clean APIs for data cleaning. Python implementation of R package Janitor

Example project implementing best practices for PySpark ETL jobs and applications.

Data validation and organization of metadata for data frames and database tables

Accumulated knowledge and experience in the field of Data Engineering

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

Always know what to expect from your data.

Turn complex requirements to workflows without leaving the comfort of your technology stack.

An Awesome List of Open-Source Data Engineering Projects

📝 A compilation of everything that I learn; Computer Science, Software Development, Engineering, Math, and Coding in General. Read the rendered results here ->

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

Open Metadata and Governance

Fancy stream processing made operationally mundane

A Data Engineering & Machine Learning Knowledge Hub

Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.

Feature Store for Machine Learning

The Data Engineering Cookbook

A python package to create a database on the platform using our moj data warehousing framework

Automate building ML classification pipelines in .NET

Example of an ETL Pipeline using Airflow

Beneath is a serverless real-time data platform ⚡️

Open Source Feature Flagging and A/B Testing Platform

ELT Code for your Data Warehouse

Python library for building Pangeo Forge recipes.

A comprehensive list of 180+ YouTube Channels for Data Science, Data Engineering, Machine Learning, Deep learning, Computer Science, programming, software engineering, etc.

Summary of my projects on kaggle

Deep Neural Network architecture as a predictive optimal controller for {HVAC+Solar cell + battery} disturbance afflicted system vs classic Model Predictive Control

This repo contains commands that data engineers use in day to day work.