Top 23 data-pipeline open source projects

A list of useful resources to learn Data Engineering from scratch

The enterprise-grade behavioral data engine (web, mobile, server-side, webhooks), running cloud-natively on AWS and GCP

Model complex data transformation pipelines easily

OONI data processing pipeline

Serverless Data Pipeline powered by Kinesis Firehose, API Gateway, Lambda, S3, and Athena

Network traffic data pipeline for real-time predictions and building datasets for deep neural networks

Augmentation pipeline for rendering synthetic paper printing, faxing, scanning and copy machine processes

A Node.js and JavaScript synchronous data pipeline processing, data sharing and stream processing library. Actionable & Transformable Pipeline data processing.

Rivery CLI

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

Automated Tool for Optimized Modelling

Configuration and definitions of Airflow for OpenTrials

Real estate dagster pipeline

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Data pipeline project

Pipeline module for parallel real-time data processing for machine learning models development and production purposes.

Bulk Stash is a docker rclone service to sync, or copy, files between different storage services. For example, you can copy files either to or from a remote storage services like Amazon S3 to Google Cloud Storage, or locally from your laptop to a remote storage.

Saisoku is a Python module that helps you build complex pipelines of batch file/directory transfer/sync jobs.

一个基于浏览器环境的数据采集SDK

Build and deploy a serverless data pipeline on AWS with no effort.

Blog post on ETL pipelines with Airflow

🔍 Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript

A Clojure machine learning library