228 Open source projects that are alternatives of or similar to DIRECT

Real-Time Event Streaming & Change Data Capture

www.vixtract.ru

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs on GCP and AWS.

csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

BETL. Meta data driven ETL generation using T-SQL

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Generic interface exchange format for Data Warehouse Automation and ETL generation.

A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.

ETL flow framework based on Yaml configs in Python

Blog post on ETL pipelines with Airflow

The Taxonomy for ETL Automation Metadata (TEAM) is a metadata management tool for data warehouse automation. It is part of the ecosystem for data warehouse automation, alongside the Virtual Data Warehouse pattern manager and the generic schema for Data Warehouse Automation.

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

AlphaSQL provides Integrated Type and Schema Check and Parallelization for SQL file set mainly for BigQuery

一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中

Configurable Extract, Transform, and Load

(Spatial) data harmonisation with hale studio (formerly HUMBOLDT Alignment Editor)

seatunnel plugin developing examples.

A tool for building feature stores.

A SQL-like language for performing ETL transformations.

Production Ready Data Integration Product, documentation：

Community-focused content to supplement working with BimlFlex.

Treats CSV and JSON files as SQL tables, and exports SQL SELECTs back to CSV or JSON.

A model-driven dynamically-configurable framework to acquire data from external sources and save it to your database.

Bender - Serverless ETL Framework

mito ETL tool

一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中

CubETL - Framework and tool for data ETL (Extract, Transform and Load) in Python (PERSONAL PROJECT / SELDOM MAINTAINED)

Configurable data bridge for permanent ETL jobs

A visual ETL development and debugging tool for big data

A simplified, lightweight ETL Framework based on Apache Spark

ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

This repository is a getting started guide to Singer.

implementing an end-to-end tweets ETL/Analysis pipeline.

A Kafka Connect source connector that generates data for tests

python ETL framework

Stetl, Streaming ETL, is a lightweight geospatial processing and ETL framework written in Python.

Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

Thain is a distributed flow schedule platform.

Big Data Toolkit for the JVM

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

Categorical Query Language IDE

基于spring boot通过java代码调用kette

Play detective on Reddit: Discover political disinformation campaigns, secret influencers and more

📘 Making Data, the DataMade Way

A cross-platform command line tool for parallelised content extraction and analysis.

Mass processing data with a complete ETL for .net developers

A MongoDB to Elasticsearch connector

A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.

Metl is a simple, web-based integration platform that allows for several different styles of data integration including messaging, file based Extract/Transform/Load (ETL), and remote procedure invocation via Web Services. Read more at www.jumpmind.com/products/metl/overview

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

A Python stream processing engine modeled after Yahoo! Pipes

Enterprise-grade, production-hardened, serverless data lake on AWS

Data processing & ETL framework for Ruby

Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL for Infrastructure 💫 多语言执行器，分布式爬虫

A client for connecting and running DDLs on hive metastore.

A collection of tools for extracting FHIR resources and analytics services on top of that data.

Example DAGs using hooks and operators from Airflow Plugins