Top 205 etl open source projects

Data Making Guidelines
📘 Making Data, the DataMade Way
Aws Etl Orchestrator
A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.
Example Airflow Dags
Example DAGs using hooks and operators from Airflow Plugins
Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch
StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service
Utility that converts an .etl file containing a Windows network packet capture into .pcapng format.
R client for the Elasticsearch HTTP API
Bulk Writer
Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.
A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.
A cross-platform command line tool for parallelised content extraction and analysis.
Mongo Es
A MongoDB to Elasticsearch connector
Metl is a simple, web-based integration platform that allows for several different styles of data integration including messaging, file based Extract/Transform/Load (ETL), and remote procedure invocation via Web Services. Read more at
Aws Serverless Data Lake Framework
Enterprise-grade, production-hardened, serverless data lake on AWS
Linked Data & RDF Manufacturing Tools in Clojure
Bitcoin Etl
ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery
Bender - Serverless ETL Framework
Open Semantic Etl
Python based Open Source ETL tools for file crawling, document processing (text extraction, OCR), content analysis (Entity Extraction & Named Entity Recognition) & data enrichment (annotation) pipelines & ingestor to Solr or Elastic search index & linked data graph database
Mara Example Project 2
An example mini data warehouse for python project stats, template for new projects
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
A visual ETL development and debugging tool for big data
Eel Sdk
Big Data Toolkit for the JVM
Mara Pipelines
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
Kettle Web
基于spring boot通过java代码调用kette
Reddit Detective
Play detective on Reddit: Discover political disinformation campaigns, secret influencers and more
Aws Data Wrangler
Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
A Python stream processing engine modeled after Yahoo! Pipes
Data processing & ETL framework for Ruby
Sentinel Crawler
Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL for Infrastructure 💫 多语言执行器,分布式爬虫
DataX is an open source universal ETL tool that support Cassandra, ClickHouse, DBF, Hive, InfluxDB, Kudu, MySQL, Oracle, Presto(Trino), PostgreSQL, SQL Server
Aws Ecs Airflow
Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks
Kafka Connect
equivalent to kafka-connect 🔧 for nodejs ✨🐢🚀✨
Česká otevřená data
Open Data Etl Utility Kit
Use Pentaho's open source data integration tool (Kettle) to create Extract-Transform-Load (ETL) processes to update a Socrata open data portal. Documentation is available at
LinkedPipes ETL is an RDF based, lightweight ETL tool
(Spatial) data harmonisation with hale studio (formerly HUMBOLDT Alignment Editor)
Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).
Data Story
A visual process builder for Laravel
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
locopy: Loading/Unloading to Redshift and Snowflake using Python.
Sync data between persistence engines, like ETL only not stodgy
Target Postgres
A Target for Postgres
Etl with python
ETL with Python - Taught at DWH course 2017 (TAU)
Stetl, Streaming ETL, is a lightweight geospatial processing and ETL framework written in Python.
ETLy is an add-on dashboard service on top of Apache Airflow.
1-60 of 205 etl projects