333 Open source projects that are alternatives of or similar to etl

Powerful RDF Knowledge Graph Generation with [R2]RML Mappings

A schema-aware Scala library for data transformation

An Elasticsearch client tailored to data science workflows.

Example of an ETL Pipeline using Airflow

A lightweight data processing framework for Apache Spark

A client for connecting and running DDLs on hive metastore.

Blog post on ETL pipelines with Airflow

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

A simple Spark-powered ETL framework that just works 🍺

Example project implementing best practices for PySpark ETL jobs and applications.

Fancy stream processing made operationally mundane

Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.

A python package to create a database on the platform using our moj data warehousing framework

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

A tool for building feature stores.

Beneath is a serverless real-time data platform ⚡️

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

ELT Code for your Data Warehouse

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

Python library for building Pangeo Forge recipes.

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

Enterprise-grade, production-hardened, serverless data lake on AWS

Bender - Serverless ETL Framework

ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

📘 Making Data, the DataMade Way

Linq to database provider.

A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.

Server application to serve U.S. federal spending data via a RESTful API

Python based Open Source ETL tools for file crawling, document processing (text extraction, OCR), content analysis (Entity Extraction & Named Entity Recognition) & data enrichment (annotation) pipelines & ingestor to Solr or Elastic search index & linked data graph database

数据可视化, 数据挖掘, 数据处理 ETL

🗺 GIS plugins for Pentaho Data Integration

NBi is a testing framework (add-on to NUnit) for Business Intelligence and Data Access. The main goal of this framework is to let users create tests with a declarative approach based on an Xml syntax. By the means of NBi, you don't need to develop C# or Java code to specify your tests! Either, you don't need Visual Studio or Eclipse to compile y…

Example DAGs using hooks and operators from Airflow Plugins

An example mini data warehouse for python project stats, template for new projects

mito ETL tool

Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

A visual ETL development and debugging tool for big data

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service

Big Data Toolkit for the JVM

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

Utility that converts an .etl file containing a Windows network packet capture into .pcapng format.

基于spring boot通过java代码调用kette

Play detective on Reddit: Discover political disinformation campaigns, secret influencers and more

DataX 是异构数据广泛使用的离线数据同步工具/平台，实现包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、OTS、ODPS 等各种异构数据源之间高效的数据同步功能。

A dbt adapter for Databricks.

A collection of tools for extracting FHIR resources and analytics services on top of that data.

R client for the Elasticsearch HTTP API

Mass processing data with a complete ETL for .net developers

Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.

Configurable Extract, Transform, and Load

Data logistics system enabling real-time pathogen surveillance. Built for the Seattle Flu Study.

A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.

一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中

A Python stream processing engine modeled after Yahoo! Pipes

Categorical Query Language IDE