Top 205 etl open source projects

Kiba enhancement for Ruby ETL.

PHP ETL (Extract / Transform / Load) library with SOLID principles + almost no dependency.

A python library to push ethereum blockchain data into an sql database.

Public, free to use, repository with diggers configs for scraping / extracting data from various e-commerce websites and online stores

python ETL framework

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

Extract your data from the Yunmai weighing scales cloud API so you can use it elsewhere

AWS Auto Terminate Idle AWS EMR Clusters Framework is an AWS based solution using AWS CloudWatch and AWS Lambda using a Python script that is using Boto3 to terminate AWS EMR clusters that have been idle for a specified period of time.

Detect threats with log data and improve cloud security posture

The backoffice web application of d:swarm (https://github.com/dswarm/dswarm-documentation/wiki)

🐟 A streaming ETL for fish

Monitoring tool to measure flow throughput of data sources and processing components that are part of Data Ingestion and ETL pipelines.

This repository is a getting started guide to Singer.

a go daemon that syncs MongoDB to Elasticsearch in realtime

React components to build CSV files on the fly basing on Array/literal object of data

Example project implementing best practices for PySpark ETL jobs and applications.

A lightweight stream processing library for Go

Data ETL & Analysis on the dataset 'Baby Names from Social Security Card Applications - National Data'.

A hackable data integration & analysis tool to enable non technical users to edit data processing jobs and visualise data on demand.

🔮 Transform, query, and download geospatial data on the web.

A serverless cluster computing system for the Go programming language

SmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

Logical Replication extension for PostgreSQL 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.

The premier open source Data Quality solution

Power of appbase.io via CLI, with nifty imports from your favorite data sources

ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

AIStore: scalable storage for AI applications

WeDataSphere is a financial level one-stop open-source suitcase for big data platforms. Currently the source code of Scriptis and Linkis has already been released to the open-source community. WeDataSphere, Big Data Made Easy!

A simplified, lightweight ETL Framework based on Apache Spark

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

基于web版kettle开发的一套分布式综合调度,管理,ETL开发的用户专业版B/S架构工具

An extensible Java framework for building XML and non-XML streaming applications

An orchestration platform for the development, production, and observation of data assets.

Fancy stream processing made operationally mundane

ETL Library for Machine Learning - data pipelines, data munging and wrangling

A python package to create a database on the platform using our moj data warehousing framework

A SQL-like language for performing ETL transformations.

Monitoring tool to measure flow throughput of data sources and processing components that are part of Data Ingestion and ETL pipelines.

A Go native tabular data extraction package. Currently supports .xls, .xlsx, .csv, .tsv formats.

Basin is a visual programming editor for building Spark and PySpark pipelines. Easily build, debug, and deploy complex ETL pipelines from your browser

Tool and library to convert ETW logs to JSON files

Move your messages from MQTT to Apache Kafka in real-time 🚀

Odoo Integration Addons

Example of an ETL Pipeline using Airflow

dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

C++ implementation of CAN ISO 15765-2 also known as CAN ISO transport protocol. CPP CAN isotp.

Beneath is a serverless real-time data platform ⚡️

ELT Code for your Data Warehouse

Python library for building Pangeo Forge recipes.

Virtual Machine for Desktop Search with Open Semantic Search

Addax is an open source universal ETL tool that supports most of those RDBMS and NoSQLs on the planet, helping you transfer data from any one place to another.

GAMECHANGER aspires to be the Department’s trusted solution for evidence-based, data-driven decision-making across the universe of DoD requirements

OpenRefine is a free, open source power tool for working with messy data and improving it. This repository contains Dockerbuild files for automated builds.

Python3 lib and cli for operating a Cardano Passive Node and using the API's. (PRE-ALPHA)

M-Lab ingestion pipeline

Multi-core processing of 'Gameday' data from Major League Baseball Advanced Media. Additional tools to parallelize large data sets and write them to a database.

ETL for the SPDR ETF holdings XLS documents

The OpenRefine Python Client from Paul Makepeace provides a library for communicating with an OpenRefine server. This fork extends the command line interface (CLI) and is distributed as a convenient one-file-executable (Windows, Linux, Mac). It is also available via Docker Hub, PyPI and Binder.