1117 Open source projects that are alternatives of or similar to DaFlow

GCS support for avro-tools, parquet-tools and protobuf

A tool for data sampling, data generation, and data diffing

Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.

A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Make record oriented data to columnar format.

A large-scale entity and relation database supporting aggregation of properties

📁 Extract, Transform, Load (ETL) 👷 refers to a process in database usage and especially in data warehousing. This repository contains a starter kit featuring ETL related work.

Read and write Parquet in Scala. Use Scala classes as schema. No need to start a cluster.

와이빅타 엔지니어링팀의 자료를 정리해두는 곳입니다.

Fundamentals of Spark with Python (using PySpark), code examples

HadoopOffice - Analyze Office documents using the Hadoop ecosystem (Spark/Flink/Hive)

Bender - Serverless ETL Framework

简单的可视化湖州通话数据 假设数据量很大，没法用浏览器直接绘制热力图，把绘制热力图这一步骤放到线下计算分析。使用Apache Spark并行计算数据之后，再使用Apache Spark绘制热力图，然后用leafletjs加载OpenStreetMap图层和热力图图层，以达到良好的交互效果。现在使用Apache Spark实现绘制，可能是Apache Spark不擅长这方面的计算或者是我没有设计好算法，并行计算的速度比不上单机计算。Apache Spark绘制热力图和计算代码在这 https://github.com/yuanzhaokang/ParallelizeHeatmap.git .

Griffon Data Science Virtual Machine

Example projects for 'BigInsights for Apache Hadoop' on IBM Bluemix

an open-source data management platform for knowledge workers (https://github.com/dswarm/dswarm-documentation/wiki)

A tool for batch loading data files (json, parquet, csv, tsv) into ElasticSearch

💎🔥大数据学习笔记

专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...

A luigi powered analytics / warehouse stack

Today I Learned

基于wifi抓取信息的大数据查询分析系统

Cookbook to install Hadoop 2.0+ using Chef

A python library to push ethereum blockchain data into an sql database.

Hadoop Crypto Ledger - Analyzing CryptoLedgers, such as Bitcoin Blockchain, on Big Data platforms, such as Hadoop/Spark/Flink/Hive

RDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

ETL with Python - Taught at DWH course 2017 (TAU)

Mass processing data with a complete ETL for .net developers

mito ETL tool

Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.

🏐 Apache Parquet for modern .NET

WASP is a framework to build complex real time big data applications. It relies on a kind of Kappa/Lambda architecture mainly leveraging Kafka and Spark. If you need to ingest huge amount of heterogeneous data and analyze them through complex pipelines, this is the framework for you.

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

Simple windows desktop application for viewing & querying Apache Parquet files

A client for connecting and running DDLs on hive metastore.

DataX 是异构数据广泛使用的离线数据同步工具/平台，实现包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、OTS、ODPS 等各种异构数据源之间高效的数据同步功能。

How to use Parquet in Flink

Companion to Learning Hadoop and Learning Spark courses on Linked In Learning

A model-driven dynamically-configurable framework to acquire data from external sources and save it to your database.

Smart Automation Tool for building modern Data Lakes and Data Pipelines

BETL. Meta data driven ETL generation using T-SQL

The CSV to database command line loader

Facebook's Hive UDFs

Blog post on ETL pipelines with Airflow

A collection of tools for extracting FHIR resources and analytics services on top of that data.

A collection of Apache Parquet add-on modules

seatunnel plugin developing examples.

Hive Storage Handler for interoperability between BigQuery and Apache Hive

大数据学习笔记，学习路线，技术案例整理。

大数据相关内容汇总，包括分布式存储引擎、分布式计算引擎、数仓建设等。关键词：Hadoop、HBase、ES、Kudu、Hive、Presto、Spark、Flink、Kylin、ClickHouse

Multi docker container images for main Big Data Tools. (Hadoop, Spark, Kafka, HBase, Cassandra, Zookeeper, Zeppelin, Drill, Flink, Hive, Hue, Mesos, ... )

Configurable Extract, Transform, and Load

A tool for building feature stores.

Hive JDBC "uber" or "standalone" jar based on the latest Apache Hive version

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

同步Hive数据仓库数据到Elasticsearch的小工具

All code samples, scripts and more in-depth examples for The Apache Ignite Book. Include Apache Ignite 2.6 or above

Avro Schema Evolution made easy