116 Open source projects that are alternatives of or similar to practical-data-engineering

A list of useful resources to learn Data Engineering from scratch

Blog post on ETL pipelines with Airflow

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

Projects done in the Data Engineering Nanodegree by Udacity.com

Metric collection, data testing and monitoring for SQL accessible data

Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes

Bulk Stash is a docker rclone service to sync, or copy, files between different storage services. For example, you can copy files either to or from a remote storage services like Amazon S3 to Google Cloud Storage, or locally from your laptop to a remote storage.

Airflow Deployment on AWS ECS Fargate Using Cloudformation

A client for connecting and running DDLs on hive metastore.

Collection of open-source Spark tools & frameworks that have made the data engineering and data science teams at Swoop highly productive

📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.

A package to easily open an instance of a Google spreadsheet and interact with worksheets through Pandas DataFrames.

A curated list of big data engineering tools, resources and communities.

Free and open source schema versioning and database migration made natively with .NET Core.

DE or DIE meetup made by data engineers for data engineers. Currently in Russian only.

Study materials for the Google Cloud Professional Data Engineering Exam

CSVs sliced, diced & analyzed.

A tool for building feature stores.

Road to Azure Data Engineer Part-I: DP-200 - Implementing an Azure Data Solution

📊 📋 Dashboards using YAML or JSON files

Apache Superset is a Data Visualization and Data Exploration Platform

Easy way to define, execute and store quality rules for your data.

A simple Spark-powered ETL framework that just works 🍺

A curated list of awesome dbt resources

Ansible playbook to deploy distributed technologies

Quilt is a self-organizing data hub for S3

A convention over configuration workflow orchestrator. Develop locally (Jupyter or your favorite editor), deploy to Airflow or Kubernetes.

Saisoku is a Python module that helps you build complex pipelines of batch file/directory transfer/sync jobs.

Enterprise-grade, production-hardened, serverless data lake on AWS

Useful SQL queries for Blockchain ETL datasets in BigQuery.

An automatic ML model optimization tool.

Less-Resilient MapReduce framework for Go

A Clojure dataframe library that runs on Spark

Datart is a next generation Data Visualization Open Platform

[READ-ONLY] PHP - ETL (Extract Transform Load) data processing library

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

The Accelerator is a tool for fast and reproducible processing of large amounts of data.

Pipeline module for parallel real-time data processing for machine learning models development and production purposes.

PipelineX: Python package to build ML pipelines for experimentation with Kedro, MLflow, and more

一个基于浏览器环境的数据采集SDK

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Accelerate data science

Build and deploy a serverless data pipeline on AWS with no effort.

Dummy variable generation with fit/transform capabilities

Example end to end data engineering project.

🔍 Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

Powerful RDF Knowledge Graph Generation with [R2]RML Mappings

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.

A Clojure machine learning library

dbt adapter for SQL Server and Azure SQL

A collection of online resources to help you on your Tech journey.

Git-like capabilities for your object storage

A daily digest of the articles or videos I've found interesting, that I want to share with you.

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Data pipeline project

Python client for using Prefect Cloud with Saturn Cloud

DEPRECATED: Stream data processing micro-framework