587 Open source projects that are alternatives of or similar to etlflow

DIRECT, the Data Integration Run-time Execution Control Tool, is a data logistics framework that can be used to monitor, log, audit and control data integration / ETL processes.

csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

www.vixtract.ru

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

Real-Time Event Streaming & Change Data Capture

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中

A model-driven dynamically-configurable framework to acquire data from external sources and save it to your database.

An upgraded and improved version of the Iris automatic GCP-labeling project

Command line tool to run batch jobs concurrently with ETL framework on AWS or other cloud computing resources

CubETL - Framework and tool for data ETL (Extract, Transform and Load) in Python (PERSONAL PROJECT / SELDOM MAINTAINED)

☁️ nodejs kafka connect connector for Google BigQuery

A simplified, lightweight ETL Framework based on Apache Spark

ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

This repository is a getting started guide to Singer.

(Spatial) data harmonisation with hale studio (formerly HUMBOLDT Alignment Editor)

Configurable data bridge for permanent ETL jobs

python ETL framework

一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中

A tool for building feature stores.

bqloader is a simple ETL framework to load data from Cloud Storage into BigQuery.

Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

BETL. Meta data driven ETL generation using T-SQL

Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. What datasets do you want to be added to Ethereum ETL? Vote here: https://blockchain-etl.convas.io.

mito ETL tool

Loads Snowplow enriched events into Google BigQuery

An example mini data warehouse for python project stats, template for new projects

Production Ready Data Integration Product, documentation：

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

A SQL-like language for performing ETL transformations.

A Kafka Connect source connector that generates data for tests

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

seatunnel plugin developing examples.

Starlake is a Spark Based On Premise and Cloud ELT/ETL Framework for Batch & Stream Processing

Stetl, Streaming ETL, is a lightweight geospatial processing and ETL framework written in Python.

Configurable Extract, Transform, and Load

DIY commercial datasets on Google Cloud Platform

Campaign Manager 360 and Display & Video 360 Reports to BigQuery connector

A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.

Google Cloud Platform Machine Learning Samples

Bender - Serverless ETL Framework

A visual ETL development and debugging tool for big data

Blog post on ETL pipelines with Airflow

Hive Storage Handler for interoperability between BigQuery and Apache Hive

A Python framework for data processing on GCP.

Daily updated list of IP addresses / CIDR blocks used by data centers, cloud service providers, servers, etc.

Liquid Application Framework documentation, useful links and sample project

retrive metadata endpoint data with these one liners.

GCP Learning stuff.

Data ETL & Analysis on the global and Mexican datasets of the COVID-19 pandemic.

NR1 Cloud Optimize allows you to Identify right-sizing opportunities and potential savings of your AWS, GCP, and Azure instances across your cloud environment.

Open source tools for Google Cloud Storage and Databases.

A collection of Google Cloud Platform (GCP) plugins

Securely access AWS services from GKE cluster

🔄 Simple tool for ETL. From SQL to Redis.