95 Open source projects that are alternatives of or similar to Awesome Opensource Data Engineering

DE or DIE meetup made by data engineers for data engineers. Currently in Russian only.

[READ-ONLY] PHP - ETL (Extract Transform Load) data processing library

A Pachyderm deep learning tutorial for conference workshops

Road to Azure Data Engineer Part-I: DP-200 - Implementing an Azure Data Solution

A convention over configuration workflow orchestrator. Develop locally (Jupyter or your favorite editor), deploy to Airflow or Kubernetes.

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

DEPRECATED: Stream data processing micro-framework

ELT Code for your Data Warehouse

A curated list of awesome dbt resources

dbt-sugar is a CLI tool that allows users of dbt to have fun and ease performing actions around dbt models

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

An automatic ML model optimization tool.

The practical use-cases of how to make your Machine Learning Pipelines robust and reliable using Apache Airflow.

Python client for using Prefect Cloud with Saturn Cloud

Beneath is a serverless real-time data platform ⚡️

Easy way to define, execute and store quality rules for your data.

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

Less-Resilient MapReduce framework for Go

A Data Engineering & Machine Learning Knowledge Hub

A client for connecting and running DDLs on hive metastore.

ETH PhD Program course

A daily digest of the articles or videos I've found interesting, that I want to share with you.

A comprehensive list of 180+ YouTube Channels for Data Science, Data Engineering, Machine Learning, Deep learning, Computer Science, programming, software engineering, etc.

Enterprise-grade, production-hardened, serverless data lake on AWS

funsies is a lightweight workflow engine 🔧

Real estate dagster pipeline

Free and open source schema versioning and database migration made natively with .NET Core.

This repo contains commands that data engineers use in day to day work.

Datart is a next generation Data Visualization Open Platform

Example of an ETL Pipeline using Airflow

Powerful RDF Knowledge Graph Generation with [R2]RML Mappings

Personal Data Engineering Projects

Useful SQL queries for Blockchain ETL datasets in BigQuery.

Fancy stream processing made operationally mundane

Dummy variable generation with fit/transform capabilities

A schema-aware Scala library for data transformation

A collection of online resources to help you on your Tech journey.

Open Source Feature Flagging and A/B Testing Platform

A curated list of big data engineering tools, resources and communities.

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

CSVs sliced, diced & analyzed.

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

Blog post on ETL pipelines with Airflow

Python library for building Pangeo Forge recipes.

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

Clean APIs for data cleaning. Python implementation of R package Janitor

Go to: https://github.com/alexklibisz/elastiknn

A python package to create a database on the platform using our moj data warehousing framework

A package to easily open an instance of a Google spreadsheet and interact with worksheets through Pandas DataFrames.

An Elasticsearch client tailored to data science workflows.

Metric collection, data testing and monitoring for SQL accessible data

Summary of my projects on kaggle

Python package for Customizable Data Preprocessing Pipelines

📝 A compilation of everything that I learn; Computer Science, Software Development, Engineering, Math, and Coding in General. Read the rendered results here ->

Open Metadata and Governance

Automate building ML classification pipelines in .NET

Deep Neural Network architecture as a predictive optimal controller for {HVAC+Solar cell + battery} disturbance afflicted system vs classic Model Predictive Control

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes