103 Open source projects that are alternatives of or similar to neon-workshop

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

Beneath is a serverless real-time data platform ⚡️

The practical use-cases of how to make your Machine Learning Pipelines robust and reliable using Apache Airflow.

Example of an ETL Pipeline using Airflow

Dummy variable generation with fit/transform capabilities

A convention over configuration workflow orchestrator. Develop locally (Jupyter or your favorite editor), deploy to Airflow or Kubernetes.

An automatic ML model optimization tool.

Study materials for the Google Cloud Professional Data Engineering Exam

Real estate dagster pipeline

DEPRECATED: Stream data processing micro-framework

PipelineX: Python package to build ML pipelines for experimentation with Kedro, MLflow, and more

A daily digest of the articles or videos I've found interesting, that I want to share with you.

Useful SQL queries for Blockchain ETL datasets in BigQuery.

Enterprise-grade, production-hardened, serverless data lake on AWS

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

A Clojure dataframe library that runs on Spark

Smart Automation Tool for building modern Data Lakes and Data Pipelines

The Accelerator is a tool for fast and reproducible processing of large amounts of data.

dbt-sugar is a CLI tool that allows users of dbt to have fun and ease performing actions around dbt models

Less-Resilient MapReduce framework for Go

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Accelerate data science

Apache Superset is a Data Visualization and Data Exploration Platform

Datart is a next generation Data Visualization Open Platform

[READ-ONLY] PHP - ETL (Extract Transform Load) data processing library

Example end to end data engineering project.

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

Python client for using Prefect Cloud with Saturn Cloud

Go to: https://github.com/alexklibisz/elastiknn

Python package for Customizable Data Preprocessing Pipelines

A package to easily open an instance of a Google spreadsheet and interact with worksheets through Pandas DataFrames.

DE or DIE meetup made by data engineers for data engineers. Currently in Russian only.

Metric collection, data testing and monitoring for SQL accessible data

Relational data pipelines for the science lab

Free and open source schema versioning and database migration made natively with .NET Core.

Easy way to define, execute and store quality rules for your data.

Projects done in the Data Engineering Nanodegree by Udacity.com

Building data processing pipelines for documents processing with NLP using Apache NiFi and related services

A list of useful resources to learn Data Engineering from scratch

A collection of online resources to help you on your Tech journey.

Airflow Deployment on AWS ECS Fargate Using Cloudformation

ETH PhD Program course

A tool for building feature stores.

A curated list of big data engineering tools, resources and communities.

Collection of open-source Spark tools & frameworks that have made the data engineering and data science teams at Swoop highly productive

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

📊 📋 Dashboards using YAML or JSON files

Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes

📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.

An Elasticsearch client tailored to data science workflows.

CSVs sliced, diced & analyzed.

A simple Spark-powered ETL framework that just works 🍺

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

Ansible playbook to deploy distributed technologies

Road to Azure Data Engineer Part-I: DP-200 - Implementing an Azure Data Solution

A client for connecting and running DDLs on hive metastore.

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.

Quilt is a self-organizing data hub for S3

Blog post on ETL pipelines with Airflow

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.