Top 205 etl open source projects

📘 Making Data, the DataMade Way

A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.

Example DAGs using hooks and operators from Airflow Plugins

Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service

Utility that converts an .etl file containing a Windows network packet capture into .pcapng format.

R client for the Elasticsearch HTTP API

Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.

A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.

Categorical Query Language IDE

A cross-platform command line tool for parallelised content extraction and analysis.

A MongoDB to Elasticsearch connector

Metl is a simple, web-based integration platform that allows for several different styles of data integration including messaging, file based Extract/Transform/Load (ETL), and remote procedure invocation via Web Services. Read more at www.jumpmind.com/products/metl/overview

Enterprise-grade, production-hardened, serverless data lake on AWS

Linked Data & RDF Manufacturing Tools in Clojure

ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

Bender - Serverless ETL Framework

Linq to database provider.

Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.

Server application to serve U.S. federal spending data via a RESTful API

Python based Open Source ETL tools for file crawling, document processing (text extraction, OCR), content analysis (Entity Extraction & Named Entity Recognition) & data enrichment (annotation) pipelines & ingestor to Solr or Elastic search index & linked data graph database

数据可视化, 数据挖掘, 数据处理 ETL

An example mini data warehouse for python project stats, template for new projects

mito ETL tool

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

A visual ETL development and debugging tool for big data

Big Data Toolkit for the JVM

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

基于spring boot通过java代码调用kette

Play detective on Reddit: Discover political disinformation campaigns, secret influencers and more

Mass processing data with a complete ETL for .net developers

A tool for building feature stores.

Configurable Extract, Transform, and Load

一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

A Python stream processing engine modeled after Yahoo! Pipes

Data processing & ETL framework for Ruby

Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL for Infrastructure 💫 多语言执行器，分布式爬虫

DataX is an open source universal ETL tool that support Cassandra, ClickHouse, DBF, Hive, InfluxDB, Kudu, MySQL, Oracle, Presto(Trino), PostgreSQL, SQL Server

The MarkLogic Data Hub: documentation ==>

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

equivalent to kafka-connect 🔧 for nodejs ✨🐢🚀✨

The CSV to database command line loader

Česká otevřená data

Use Pentaho's open source data integration tool (Kettle) to create Extract-Transform-Load (ETL) processes to update a Socrata open data portal. Documentation is available at http://open-data-etl-utility-kit.readthedocs.io/en/stable

Udacity Data Engineering Nano Degree (DEND)

LinkedPipes ETL is an RDF based, lightweight ETL tool

(Spatial) data harmonisation with hale studio (formerly HUMBOLDT Alignment Editor)

A simple Spark-powered ETL framework that just works 🍺

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

A visual process builder for Laravel

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

locopy: Loading/Unloading to Redshift and Snowflake using Python.

A luigi powered analytics / warehouse stack

Sync data between persistence engines, like ETL only not stodgy

A Singer.io Target for Postgres

Actively curated list of awesome BI tools. PRs welcome!

ETL with Python - Taught at DWH course 2017 (TAU)

Stetl, Streaming ETL, is a lightweight geospatial processing and ETL framework written in Python.

ETLy is an add-on dashboard service on top of Apache Airflow.