1117 Open source projects that are alternatives of or similar to DaFlow

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

A SQL-like language for performing ETL transformations.

ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

Big Data Toolkit for the JVM

Real-Time Event Streaming & Change Data Capture

A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL

python ETL framework

CubETL - Framework and tool for data ETL (Extract, Transform and Load) in Python (PERSONAL PROJECT / SELDOM MAINTAINED)

dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs on GCP and AWS.

www.vixtract.ru

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files

DIRECT, the Data Integration Run-time Execution Control Tool, is a data logistics framework that can be used to monitor, log, audit and control data integration / ETL processes.

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

⛈️ Rumble 1.11.0 "Banyan Tree"🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more

A visual ETL development and debugging tool for big data

Iceberg is a table format for large, slow-moving tabular data

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

Addax is an open source universal ETL tool that supports most of those RDBMS and NoSQLs on the planet, helping you transfer data from any one place to another.

Example MapReduce jobs in Java, Hive, Pig, and Hadoop Streaming that work on Avro data.

网站点击流离线日志分析

Apache Drill is a distributed MPP query layer for self describing data

csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

WeDataSphere is a financial level one-stop open-source suitcase for big data platforms. Currently the source code of Scriptis and Linkis has already been released to the open-source community. WeDataSphere, Big Data Made Easy!

Production Ready Data Integration Product, documentation：

DataX is an open source universal ETL tool that support Cassandra, ClickHouse, DBF, Hive, InfluxDB, Kudu, MySQL, Oracle, Presto(Trino), PostgreSQL, SQL Server

The Hadoop ETL UDFs are the main way to load data from Hadoop into EXASOL

RDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark

An alternative to the "hive standalone" jar for connecting Java applications to Apache Hive via JDBC

Fundamentals of Spark with Python (using PySpark), code examples

an open-source data management platform for knowledge workers (https://github.com/dswarm/dswarm-documentation/wiki)

一个对用户行为日志进行分析的大数据项目

A tool for batch loading data files (json, parquet, csv, tsv) into ElasticSearch

Simple windows desktop application for viewing & querying Apache Parquet files

What's in your data? Extract schema, statistics and entities from datasets

A lib to transform data from jdbc,csv,json to ecah other.

A python library to push ethereum blockchain data into an sql database.

The CSV to database command line loader

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

ETL with Python - Taught at DWH course 2017 (TAU)

Mass processing data with a complete ETL for .net developers

A collection of tools for extracting FHIR resources and analytics services on top of that data.

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

Blog post on ETL pipelines with Airflow

A client for connecting and running DDLs on hive metastore.

大数据学习笔记，学习路线，技术案例整理。

大数据相关内容汇总，包括分布式存储引擎、分布式计算引擎、数仓建设等。关键词：Hadoop、HBase、ES、Kudu、Hive、Presto、Spark、Flink、Kylin、ClickHouse

WASP is a framework to build complex real time big data applications. It relies on a kind of Kappa/Lambda architecture mainly leveraging Kafka and Spark. If you need to ingest huge amount of heterogeneous data and analyze them through complex pipelines, this is the framework for you.

A model-driven dynamically-configurable framework to acquire data from external sources and save it to your database.

Companion to Learning Hadoop and Learning Spark courses on Linked In Learning

Smart Automation Tool for building modern Data Lakes and Data Pipelines

seatunnel plugin developing examples.

同步Hive数据仓库数据到Elasticsearch的小工具

A collection of Apache Parquet add-on modules

How to use Parquet in Flink

BETL. Meta data driven ETL generation using T-SQL

Hive Storage Handler for interoperability between BigQuery and Apache Hive

All code samples, scripts and more in-depth examples for The Apache Ignite Book. Include Apache Ignite 2.6 or above

Make record oriented data to columnar format.