95 Open source projects that are alternatives of or similar to Awesome Opensource Data Engineering

A Clojure dataframe library that runs on Spark

Projects done in the Data Engineering Nanodegree by Udacity.com

Study materials for the Google Cloud Professional Data Engineering Exam

A list of useful resources to learn Data Engineering from scratch

The Accelerator is a tool for fast and reproducible processing of large amounts of data.

Airflow Deployment on AWS ECS Fargate Using Cloudformation

PipelineX: Python package to build ML pipelines for experimentation with Kedro, MLflow, and more

A tool for building feature stores.

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Collection of open-source Spark tools & frameworks that have made the data engineering and data science teams at Swoop highly productive

Accelerate data science

📊 📋 Dashboards using YAML or JSON files

Apache Superset is a Data Visualization and Data Exploration Platform

📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.

Example end to end data engineering project.

A simple Spark-powered ETL framework that just works 🍺

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

Ansible playbook to deploy distributed technologies

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.

Quilt is a self-organizing data hub for S3

dbt adapter for SQL Server and Azure SQL

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

Git-like capabilities for your object storage

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

The easiest way to automate your data

Clean APIs for data cleaning. Python implementation of R package Janitor

Example project implementing best practices for PySpark ETL jobs and applications.

Data validation and organization of metadata for data frames and database tables

Accumulated knowledge and experience in the field of Data Engineering

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

Always know what to expect from your data.

Turn complex requirements to workflows without leaving the comfort of your technology stack.

Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.

Feature Store for Machine Learning

The Data Engineering Cookbook