All Categories → No Category → data-pipelines

Top 13 data-pipelines open source projects

An orchestration platform for the development, production, and observation of data assets.

✭ 4,099

python typescript Jupyter Notebook javascript scala Dockerfile data-science workflow analytics scheduler etl data-pipelines workflow-automation dagster data-orchestrator

Hub

Dataset format for AI. Build, manage, & visualize datasets for deep learning. Stream data real-time to PyTorch/TensorFlow & version-control it. https://activeloop.ai

arakat

ARAKAT - Big Data Analysis and Business Intelligence Application Development Platform

✭ 23

python java shell docker distributed-systems docker-swarm business-intelligence data-pipelines big-data-analytics predictive-maintenance cloud-native-applications

AirflowDataPipeline

Example of an ETL Pipeline using Airflow

✭ 24

python airflow etl postgresql data-engineering data-pipelines

beneath

Beneath is a serverless real-time data platform ⚡️

✭ 65

go typescript python java Jupyter Notebook javascript kubernetes data-science streaming sql etl analytics data-warehouse data-engineering dataops developer-tools data-pipelines mlops beneath

spark-transformers

Spark-Transformers: Library for exporting Apache Spark MLLIB models to use them in any Java application with no other dependencies.

✭ 39

java scala export machine-learning spark apache-spark machine-learning-algorithms transformers mllib machine-learning-library data-pipelines

ml-in-production

The practical use-cases of how to make your Machine Learning Pipelines robust and reliable using Apache Airflow.

✭ 29

python Makefile Dockerfile TSQL shell machine-learning data-engineering data-pipelines apache-airflow

rivery cli

Rivery CLI

✭ 16

python data-science database etl dataops database-management dwh elt data-pipelines data-pipeline dwh-team dataops-platform rivery

versatile-data-kit

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

✭ 144

python java shell data-science sql etl analytics snowflake data-warehouse data-engineering dataops warehouse sqlite3 elt data-pipelines data-quality data-engineer trino data-lineage trinodb

neon-workshop

A Pachyderm deep learning tutorial for conference workshops

✭ 19

python docker kubernetes data-science machine-learning deep-learning containers data-engineering data-pipelines

datajoint-python

Relational data pipelines for the science lab

✭ 140

python Dockerfile mysql s3 databases pipeline-framework scientific-computing cloud-computing data-analysis relational-databases data-pipelines workflow-management datajoint relational-algebra relational-model

CogStack-NiFi

Building data processing pipelines for documents processing with NLP using Apache NiFi and related services

✭ 22

python shell Jupyter Notebook groovy Dockerfile Makefile nlp elasticsearch kibana rest data-integration nifi apache-nifi data-pipelines electronic-health-records

smart-data-lake

Smart Automation Tool for building modern Data Lakes and Data Pipelines

✭ 79

scala java shell spark hive hadoop transform-data data-lake data-pipelines deltalake smart-data-lake

1-13 of 13 data-pipelines projects