221 Open source projects that are alternatives of or similar to vixtract

Real-Time Event Streaming & Change Data Capture

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

DIRECT, the Data Integration Run-time Execution Control Tool, is a data logistics framework that can be used to monitor, log, audit and control data integration / ETL processes.

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs on GCP and AWS.

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

BETL. Meta data driven ETL generation using T-SQL

一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中

Bender - Serverless ETL Framework

A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.

A tool for building feature stores.

Configurable Extract, Transform, and Load

Configurable data bridge for permanent ETL jobs

ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

seatunnel plugin developing examples.

A model-driven dynamically-configurable framework to acquire data from external sources and save it to your database.

ETL flow framework based on Yaml configs in Python

A Kafka Connect source connector that generates data for tests

Treats CSV and JSON files as SQL tables, and exports SQL SELECTs back to CSV or JSON.

(Spatial) data harmonisation with hale studio (formerly HUMBOLDT Alignment Editor)

mito ETL tool

Blog post on ETL pipelines with Airflow

一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中

A visual ETL development and debugging tool for big data

A SQL-like language for performing ETL transformations.

A simplified, lightweight ETL Framework based on Apache Spark

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

python ETL framework

Production Ready Data Integration Product, documentation：

CubETL - Framework and tool for data ETL (Extract, Transform and Load) in Python (PERSONAL PROJECT / SELDOM MAINTAINED)

This repository is a getting started guide to Singer.

Stetl, Streaming ETL, is a lightweight geospatial processing and ETL framework written in Python.

Linked Data & RDF Manufacturing Tools in Clojure

Mass processing data with a complete ETL for .net developers

R client for the Elasticsearch HTTP API

ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

A Python stream processing engine modeled after Yahoo! Pipes

Data processing & ETL framework for Ruby

Example DAGs using hooks and operators from Airflow Plugins

Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.

Linq to database provider.

Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL for Infrastructure 💫 多语言执行器，分布式爬虫

DataX is an open source universal ETL tool that support Cassandra, ClickHouse, DBF, Hive, InfluxDB, Kudu, MySQL, Oracle, Presto(Trino), PostgreSQL, SQL Server

Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.

The MarkLogic Data Hub: documentation ==>

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

Server application to serve U.S. federal spending data via a RESTful API

equivalent to kafka-connect 🔧 for nodejs ✨🐢🚀✨

The CSV to database command line loader

Python based Open Source ETL tools for file crawling, document processing (text extraction, OCR), content analysis (Entity Extraction & Named Entity Recognition) & data enrichment (annotation) pipelines & ingestor to Solr or Elastic search index & linked data graph database

Česká otevřená data

Use Pentaho's open source data integration tool (Kettle) to create Extract-Transform-Load (ETL) processes to update a Socrata open data portal. Documentation is available at http://open-data-etl-utility-kit.readthedocs.io/en/stable

📘 Making Data, the DataMade Way

Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch

Categorical Query Language IDE

数据可视化, 数据挖掘, 数据处理 ETL

Udacity Data Engineering Nano Degree (DEND)

An example mini data warehouse for python project stats, template for new projects