993 Open source projects that are alternatives of or similar to Data-Engineering-Projects

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

Projects done in the Data Engineering Nanodegree by Udacity.com

Metric collection, data testing and monitoring for SQL accessible data

🐳📊🤓Cookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)

PipelineAI Kubeflow Distribution

Airflow Deployment on AWS ECS Fargate Using Cloudformation

Beneath is a serverless real-time data platform ⚡️

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

Compile-time Language Integrated Queries for Scala

Example end to end data engineering project.

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Web-based SQL editor run in your own private cloud. Supports MySQL, Postgres, SQL Server, Vertica, Crate, ClickHouse, Trino, Presto, SAP HANA, Cassandra, Snowflake, BigQuery, SQLite, and more with ODBC

Database migrations. CLI and Golang library.

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Database migrations. CLI and Golang library.

Blog post on ETL pipelines with Airflow

Example of an ETL Pipeline using Airflow

A Data Engineering & Machine Learning Knowledge Hub

Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

Udacity Data Engineering Nano Degree (DEND)

🐍💻📊 All material from the PyCon.DE 2018 Talk "Beyond Jupyter Notebooks - Building your own data science platform with Python & Docker" (incl. Slides, Video, Udemy MOOC & other References)

Road to Azure Data Engineer Part-I: DP-200 - Implementing an Azure Data Solution

Basic Rules of Cassandra Data Modeling

Job tracker & performance platform

Simple implementation of spatial skyline query algorithms

Docker microservice & Crawler by scrapy

Functional F# API for Cassandra

Apache Cassandra / DataStax Enterprise database migration (schema evolution) library

Check if PostgreSQL database is ready

Realtime Angular Admin/CRUD Front End App

Docker Image packaging for Elassandra

API for pokedextracker.com

Scrapy, a fast high-level web crawling & scraping framework for dart and Flutter

Scraping Ebay's products using Scrapy Web Crawling Framework

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

Entity-Relationship diagram code generator library

An utility that help you to generate or add plugin to your Fastify project

Manage postgres schema, triggers, procedures, and views

A bunch of examples of how to deploy things on kubernetes

Cassandra 3 GUI client

An XCode project to compile your own libpq.framework for iOS 11.x

Dockerfile для сборки PostgreSQL под 1С:Предприятие 8

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

Connection-Pooling Compile-Time ORM for Nim

Terraform module to deploy Apache Druid in Kubernetes

Different ways to process data into Cassandra in realtime with technologies such as Kafka, Spark, Akka, Flink

Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.

[DEPRECATED] ⚫ Commando Discord bot built on discord.js-commando.

TypeScript Enum for Postgres Errors with no runtime dependencies. Also compatible with plain JavaScript.

Demonstration of a Hive Input Format for Iceberg

免费 IP 代理池。Scrapy 爬虫框架插件

Metamapper is a data discovery and documentation platform for improving how teams understand and interact with their data.

A schema-aware Scala library for data transformation

Automatic re-connection support for PostgreSQL.

🎊 Design and implement of lightweight crawler framework.

A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.

Entity Framework Core provider for Cassandra