8704 Open source projects that are alternatives of or similar to Udacity Data Engineering

A luigi powered analytics / warehouse stack

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

locopy: Loading/Unloading to Redshift and Snowflake using Python.

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

A guide on how to set up Jupyter with Pyspark painlessly on AWS EC2 clusters, with S3 I/O support

A curated list of awesome Amazon Web Services (AWS) libraries, open source repos, guides, blogs, and other resources. Featuring the Fiery Meter of AWSome.

Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service

Projects done in the Data Engineering Nanodegree by Udacity.com

Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Serverless Platform for the stateful services

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Deploying a Strapi API on AWS (EC2 & RDS & S3)

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

macOS development environment setup: Easy-to-understand instructions with automated setup scripts for developer tools like Vim, Sublime Text, Bash, iTerm, Python data analysis, Spark, Hadoop MapReduce, AWS, Heroku, JavaScript web development, Android development, common data stores, and dev-based OS X defaults.

An idiomatic Scala wrapper around the AWS Java SDK

AWS Auto Terminate Idle AWS EMR Clusters Framework is an AWS based solution using AWS CloudWatch and AWS Lambda using a Python script that is using Boto3 to terminate AWS EMR clusters that have been idle for a specified period of time.

tbls is a CI-Friendly tool for document a database, written in Go.

☁️ Direct Upload to Amazon S3 With CORS demo. Built with Node/Express

Jupyter magics and kernels for working with remote Spark clusters

Vagrant projects for various use-cases with Spark, Zeppelin, IPython / Jupyter, SparkR

Terraform module which creates Redshift resources on AWS

450+ AWS, Hadoop, Cloud, Kafka, Docker, Elasticsearch, RabbitMQ, Redis, HBase, Solr, Cassandra, ZooKeeper, HDFS, Yarn, Hive, Presto, Drill, Impala, Consul, Spark, Jenkins, Travis CI, Git, MySQL, Linux, DNS, Whois, SSL Certs, Yum Security Updates, Kubernetes, Cloudera etc...

Detect threats with log data and improve cloud security posture

Apache Spark - Turbofan Engine Degradation Simulation Data Set example in Apache Spark

Workshop to build a serverless tracking application for your mobile device with an AWS backend

50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak, TeamCity and DevOps tools built on the major Linux distros: Alpine, CentOS, Debian, Fedora, Ubuntu

Continuously synchronize directories from remote object store to local filesystem

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

Scala client for Amazon S3

Deploy website to S3/CloudFront from Python

Spark 2.0 Python Machine Learning examples

Python Helper library for Jupyter Notebooks

Bare minimal Airflow on Kubernetes (Local, EKS, AKS)

A python library to push ethereum blockchain data into an sql database.

Seamless User Interface for replicating data into AWS.

Code examples used in the post "How to Setup Amazon S3 in a Django Project"

DDL parase and Convert to BigQuery JSON schema and DDL statements

Chai (https://chaijs.com) and Jest (https://jestjs.io/) assertions for testing services built with aws

Docker images and scripts to deploy to AWS

Scrapy pipeline to store chunked items into Amazon S3 or Google Cloud Storage bucket.

A simple Spark TDD example

🚚 Agile Data Preparation Workflows made easy with dask, cudf, dask_cudf and pyspark

Kiba enhancement for Ruby ETL.

🏋️ dbbench is a simple database benchmarking tool which supports several databases and own scripts

⛈️ Rumble 1.11.0 "Banyan Tree"🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more

🌐 Interactive Workshop on GeoAnalysis using PySpark

The format of various s3 buckets is convert in one format. for bugbounty and security testing.

Word2Vec models with Twitter data using Spark. Blog:

Deploy create react app's in AWS S3

🎓 Jupyter notebooks from UFC data science course

This module creates an S3 bucket suitable for receiving logs from other AWS services such as S3, CloudFront, and CloudTrail

☁️ Amazon S3 blob-store

ETL with Python - Taught at DWH course 2017 (TAU)

A command line security audit tool for Amazon Web Services

Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake

Python script for AWS resources inventory (cheaper than AWS Config)