115 Open source projects that are alternatives of or similar to ml-in-production

Example of an ETL Pipeline using Airflow

Beneath is a serverless real-time data platform ⚡️

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

A Pachyderm deep learning tutorial for conference workshops

DEPRECATED: Stream data processing micro-framework

Apache Airflow - OpenApi Client for Python

Gerador de DAGs no Airflow para fazer clipping do Diário Oficial da União.

Blog post on ETL pipelines with Airflow

A complete development environment setup for working with Airflow

Datart is a next generation Data Visualization Open Platform

Go to: https://github.com/alexklibisz/elastiknn

Smart Automation Tool for building modern Data Lakes and Data Pipelines

No description or website provided.

Less-Resilient MapReduce framework for Go

ETH PhD Program course

CSVs sliced, diced & analyzed.

Real estate dagster pipeline

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

A daily digest of the articles or videos I've found interesting, that I want to share with you.

A package to easily open an instance of a Google spreadsheet and interact with worksheets through Pandas DataFrames.

Plugin offering views, operators, sensors, and more developed at Pandora Media.

Metric collection, data testing and monitoring for SQL accessible data

Free and open source schema versioning and database migration made natively with .NET Core.

dbt-sugar is a CLI tool that allows users of dbt to have fun and ease performing actions around dbt models

Python client for using Prefect Cloud with Saturn Cloud

Projects done in the Data Engineering Nanodegree by Udacity.com

Easy way to define, execute and store quality rules for your data.

Python package for Customizable Data Preprocessing Pipelines

A collection of online resources to help you on your Tech journey.

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

A curated list of big data engineering tools, resources and communities.

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes

A schema-aware Scala library for data transformation

[READ-ONLY] PHP - ETL (Extract Transform Load) data processing library

Building data processing pipelines for documents processing with NLP using Apache NiFi and related services

A client for connecting and running DDLs on hive metastore.

Clean APIs for data cleaning. Python implementation of R package Janitor

A curated list of awesome dbt resources

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Personal Data Engineering Projects

A convention over configuration workflow orchestrator. Develop locally (Jupyter or your favorite editor), deploy to Airflow or Kubernetes.

Road to Azure Data Engineer Part-I: DP-200 - Implementing an Azure Data Solution

Enterprise-grade, production-hardened, serverless data lake on AWS

Relational data pipelines for the science lab

An automatic ML model optimization tool.

Powerful RDF Knowledge Graph Generation with [R2]RML Mappings

A Clojure dataframe library that runs on Spark

A list of useful resources to learn Data Engineering from scratch

Study materials for the Google Cloud Professional Data Engineering Exam

Useful SQL queries for Blockchain ETL datasets in BigQuery.

An Elasticsearch client tailored to data science workflows.

The Accelerator is a tool for fast and reproducible processing of large amounts of data.

DE or DIE meetup made by data engineers for data engineers. Currently in Russian only.

Airflow Deployment on AWS ECS Fargate Using Cloudformation

Rivery CLI

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

Functional Airflow DAG definitions.

PipelineX: Python package to build ML pipelines for experimentation with Kedro, MLflow, and more