1456 Open source projects that are alternatives of or similar to Around Dataengineering

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

A Table of Contents of all Gruntwork Code

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.

A simple Spark-powered ETL framework that just works 🍺

A curated list of resources for Devops

Defcon24 Workshop Contents : Ninja Level Infrastructure Monitoring

Learn and understand Docker technologies, with real DevOps practice!

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

Airflow Deployment on AWS ECS Fargate Using Cloudformation

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

A curated list for DevOps learning resources. Join the slack channel to discuss more.

Ansible and Kubernetes examples from Ansible for Kubernetes Book

Example of an ETL Pipeline using Airflow

Chef Infra, a powerful automation platform that transforms infrastructure into code automating how infrastructure is configured, deployed and managed across any environment, at any scale

🕰️ Monitor your cron jobs

The easiest way to automate your data

50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak, TeamCity and DevOps tools built on the major Linux distros: Alpine, CentOS, Debian, Fedora, Ubuntu

Detect compliance and security violations across Infrastructure as Code to mitigate risk before provisioning cloud native infrastructure.

A daily digest of the articles or videos I've found interesting, that I want to share with you.

Feature Store for Machine Learning

An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

The Fuzzy Labs guide to the universe of open source MLOps

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

Personal Data Engineering Projects

A lightweight tool to get an AI Infrastructure Stack up in minutes not days. K3ai will take care of setup K8s for You, deploy the AI tool of your choice and even run your code on it.

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

A curated collection of publicly available resources on how technology and tech-savvy organizations around the world practice Site Reliability Engineering (SRE)

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

My Docker templates repository 🐳 ☁️ 🐳

Build better AWS infrastructure

A template for maintaining a multiple environments infrastructure with Terraform. This template includes a CI/CD process, that applies the infrastructure in an AWS account.

funsies is a lightweight workflow engine 🔧

🕵️‍♂️ Sanity checking containers, vms, and servers

Terraform Automation and Orchestration Tool (Open Source)

Ansible playbook to deploy distributed technologies

Distributed self-replicating programs in Python

Example project implementing best practices for PySpark ETL jobs and applications.

Data validation and organization of metadata for data frames and database tables

List of Data Science Cheatsheets to rule the world

A robust, and flexible open source User & Entity Behavior Analytics (UEBA) framework used for Security Analytics. Developed with luv by Data Scientists & Security Analysts from the Cyber Security Industry. [PRE-ALPHA]

A Clojure dataframe library that runs on Spark

A repository to keep track of all the code that I end up writing for my blog posts.

Identifies and collects data on cc-licensed content across web crawl data and public apis.

Udacity Data Engineering Nano Degree (DEND)

Example end to end data engineering project.

Metric collection, data testing and monitoring for SQL accessible data

Collection of open-source Spark tools & frameworks that have made the data engineering and data science teams at Swoop highly productive

Blog post on ETL pipelines with Airflow

My presentation at ODSC India 2018 about Deep Learning with Apache Spark

Minikube for big data with Scala and Spark

A Robinhood style COVID-19 🦠 Android tracking app for the US. Open source and built with Kotlin.

Yet another helpdesk based on multiple providers

CLI tool to see redis memory usage by keys in hierarchical way. Think of disk inventory but for redis.

This is a quick script installation for resilient redirector using nginx reverse proxy and letsencrypt compatible with some popular Post-Ex Tools (Cobalt Strike, Empire, Metasploit, PoshC2).

Documentation for Techno Tim YouTube Videos

🚀 INFRA: your infrastructure as a GraphQL service

blog entries