526 Open source projects that are alternatives of or similar to Data Engineering Howto

Blog post on ETL pipelines with Airflow

Real estate dagster pipeline

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

A collection of Docker containers for running Blender headless or distributed ✨

Circuit Breaker pattern for .NET

A tool for building feature stores.

A simpler grpc framework

Apache Superset is a Data Visualization and Data Exploration Platform

Pyro 5 - Python remote objects for modern python versions

Distributed, Online, and Outlier Resilient SLAM for Robotic Teams

Resource-adaptive cluster scheduler for deep learning training.

A new kind of virtual event platform 🐧

Feature Store for Machine Learning

Xaynet represents an agnostic Federated Machine Learning framework to build privacy-preserving AI applications.

🚗 Rocket League Distributed Deep Reinforcement Learning Bot

A lua library to provide distributed rate measurement using nginx + redis, you can use it to do a throttling system within many nodes.

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Micro is a distributed cloud operating system

Airflow Deployment on AWS ECS Fargate Using Cloudformation

Hermes: a fault-tolerant replication protocol, implemented over RDMA, guaranteeing linearizability and achieving low latency and high throughput.

distributed transaction service based on reliable msg，基于可靠消息的柔性分布式事务实现方案。

Rucio - Scientific Data Management

Pluggable foundation blocks for building distributed apps.

Orbit - Virtual actor framework for building distributed systems

简单的分布式短链接服务实现

Apache ZooKeeper

ScaleCube Cluster is a lightweight Java VM implementation of SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol. features cluster membership, failure detection, and gossip protocol library.

Testing for collaborative apps and tools

Accelerate data science

Temporal service

📊 📋 Dashboards using YAML or JSON files

PipelineX: Python package to build ML pipelines for experimentation with Kedro, MLflow, and more

Distributed Big Data Orchestration Service

MIT6.824分布式系统（2018秋）

Pegasus Workflow Management System - Automate, recover, and debug scientific computations.

Python Stream Processing. A Faust fork

This is the demo application that i created for my talk 'Microservice Architecture & Implementation with Asp.Net Core' at Dotnet İstanbul Meetup Group.

Examples for the Manning "Vert.x in Action" book

Distributed reliable key-value store for the most critical data of a distributed system

Git Repository for the Mangle tool

Awesome list of distributed systems resources

DC/OS examples

A purely functional library to build distributed and event-driven systems

A High-performance Cluster Computing Engine

Rapid is a scalable distributed membership service

a dynamic configuration framework used in distributed system

Jupiter是一款性能非常不错的, 轻量级的分布式服务框架

Collection of open-source Spark tools & frameworks that have made the data engineering and data science teams at Swoop highly productive

Implementation of the gossip protocol

JavaScript implementation of SWIM membership protocol

Collection of papers in the field of distributed systems, game theory, cryptography, cryptoeconomics, zero knowledge

Sandglass is a distributed, horizontally scalable, persistent, time sorted message queue.

Distributed Time Synchronization Service

COALA IP is a blockchain-ready, community-driven protocol for intellectual property licensing.

Python implementation of the Raft algorithm for distributed consensus

A ZooKeeper client for C++.

xnnyygn's raft implementation

Pure rust building block for distributed systems

Orleans Membership provider for Kubernetes

The Accelerator is a tool for fast and reproducible processing of large amounts of data.