8704 Open source projects that are alternatives of or similar to Udacity Data Engineering

Amazon S3 input and output plugin for Fluentd

NoDB isn't a database.. but it sort of looks like one.

Turbine: the bare metals that gets you Airflow

ETL Library for Machine Learning - data pipelines, data munging and wrangling

Kiba enhancement for Ruby ETL.

DDL parase and Convert to BigQuery JSON schema and DDL statements

A lightweight, multi-tenant, scalable and secure gateway that enables Jupyter Notebooks to share resources across distributed clusters such as Apache Spark, Kubernetes and others.

Logical Replication extension for PostgreSQL 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

Data validation and organization of metadata for data frames and database tables

Database migration tool for .NET and .NET Core projects. Inspired by Flyway.

The Hunting ELK

⛈️ Rumble 1.11.0 "Banyan Tree"🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more

A tutorial on the most important features and idioms of Scala that you need to use Spark's Scala APIs.

LabNotebook is a tool that allows you to flexibly monitor, record, save, and query all your machine learning experiments.

Directly upload files to S3 compatible services with Django.

Measure Amazon S3's performance from any location.

Python Helper library for Jupyter Notebooks

Docker images and scripts to deploy to AWS

Code examples on Apache Spark using python

🎓 Jupyter notebooks from UFC data science course

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

A React Starter App that displays how web developers can integrate their front end with AWS on the backend. The App interacts with AWS Cognito, API Gateway, Lambda and DynamoDB on the backend.

Free, open-source SQL client for Windows and Mac 🦅

Reliable PostgreSQL Backup & Restore

An on-line movie recommender using Spark, Python Flask, and the MovieLens dataset

Automating Your Data Pipeline with Apache Airflow

Check read, write permissions on S3 buckets in your account

⚓️ GitHub webhook extension for uploading static pages to AWS S3 directly after commiting to master via Lambda written in Node.js

A Data Engineering & Machine Learning Knowledge Hub

AWS Auto Terminate Idle AWS EMR Clusters Framework is an AWS based solution using AWS CloudWatch and AWS Lambda using a Python script that is using Boto3 to terminate AWS EMR clusters that have been idle for a specified period of time.

An idiomatic Scala wrapper around the AWS Java SDK

Jupyter magics and kernels for working with remote Spark clusters

Detect threats with log data and improve cloud security posture

Scala client for Amazon S3

Vagrant projects for various use-cases with Spark, Zeppelin, IPython / Jupyter, SparkR

Bare minimal Airflow on Kubernetes (Local, EKS, AKS)

Apache Spark - Turbofan Engine Degradation Simulation Data Set example in Apache Spark

A python library to push ethereum blockchain data into an sql database.

Seamless User Interface for replicating data into AWS.

Chai (https://chaijs.com) and Jest (https://jestjs.io/) assertions for testing services built with aws

450+ AWS, Hadoop, Cloud, Kafka, Docker, Elasticsearch, RabbitMQ, Redis, HBase, Solr, Cassandra, ZooKeeper, HDFS, Yarn, Hive, Presto, Drill, Impala, Consul, Spark, Jenkins, Travis CI, Git, MySQL, Linux, DNS, Whois, SSL Certs, Yum Security Updates, Kubernetes, Cloudera etc...

Scrapy pipeline to store chunked items into Amazon S3 or Google Cloud Storage bucket.

🏋️ dbbench is a simple database benchmarking tool which supports several databases and own scripts

ETLy is an add-on dashboard service on top of Apache Airflow.

50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak, TeamCity and DevOps tools built on the major Linux distros: Alpine, CentOS, Debian, Fedora, Ubuntu

Deploy create react app's in AWS S3

This module creates an S3 bucket suitable for receiving logs from other AWS services such as S3, CloudFront, and CloudTrail

☁️ Amazon S3 blob-store

A command line security audit tool for Amazon Web Services

Word2Vec models with Twitter data using Spark. Blog:

Swift wrapper around AWS API

ETL with Python - Taught at DWH course 2017 (TAU)

Python script for AWS resources inventory (cheaper than AWS Config)

Terraform module to deploy an Apache Airflow cluster on AWS, backed by RDS PostgreSQL for metadata, S3 for logs and SQS as message broker with CeleryExecutor

Big Data for Data Engineers Coursera Specialization from Yandex

🌐 Interactive Workshop on GeoAnalysis using PySpark

Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake

Sync data between persistence engines, like ETL only not stodgy