674 Open source projects that are alternatives of or similar to Hydrograph

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

A simplified, lightweight ETL Framework based on Apache Spark

Project for James' Apache Spark with Scala course

A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.

Methods for the parallel and distributed analysis and mining of the Protein Data Bank using MMTF and Apache Spark.

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

A high-performance, scalable and efficient ShuffleManager plugin for Apache Spark, utilizing UCX communication layer

CubETL - Framework and tool for data ETL (Extract, Transform and Load) in Python (PERSONAL PROJECT / SELDOM MAINTAINED)

Apache Spark on AWS Lambda

A SQL-like language for performing ETL transformations.

A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL

RDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark

DIRECT, the Data Integration Run-time Execution Control Tool, is a data logistics framework that can be used to monitor, log, audit and control data integration / ETL processes.

Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.

Bulletproof Apache Spark jobs with fast root cause analysis of failures.

csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs on GCP and AWS.

curated list of awesome tools and libraries for specific domains

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

Structural Bioinformatics Training Workshop & Hackathon 2018

🏐 Apache Parquet for modern .NET

Serverless proxy for Spark cluster

ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

Griffon Data Science Virtual Machine

一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中

Fundamentals of Spark with Python (using PySpark), code examples

Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch

Simple windows desktop application for viewing & querying Apache Parquet files

No description or website provided.

www.vixtract.ru

A model-driven dynamically-configurable framework to acquire data from external sources and save it to your database.

A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.

Stetl, Streaming ETL, is a lightweight geospatial processing and ETL framework written in Python.

(Spatial) data harmonisation with hale studio (formerly HUMBOLDT Alignment Editor)

Automatic feature engineering using Generative Adversarial Networks using Deeplearning4j and Apache Spark.

一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中

BETL. Meta data driven ETL generation using T-SQL

☁️ nodejs kafka connect connector for Google BigQuery

Configurable data bridge for permanent ETL jobs

Bender - Serverless ETL Framework

简单的可视化湖州通话数据 假设数据量很大，没法用浏览器直接绘制热力图，把绘制热力图这一步骤放到线下计算分析。使用Apache Spark并行计算数据之后，再使用Apache Spark绘制热力图，然后用leafletjs加载OpenStreetMap图层和热力图图层，以达到良好的交互效果。现在使用Apache Spark实现绘制，可能是Apache Spark不擅长这方面的计算或者是我没有设计好算法，并行计算的速度比不上单机计算。Apache Spark绘制热力图和计算代码在这 https://github.com/yuanzhaokang/ParallelizeHeatmap.git .

PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster

Monitoring tool to measure flow throughput of data sources and processing components that are part of Data Ingestion and ETL pipelines.

Apache Spark Course Material

Morpheus brings the leading graph query language, Cypher, onto the leading distributed processing platform, Spark.

An extensible Java framework for building XML and non-XML streaming applications

Real-Time Event Streaming & Change Data Capture

A tool for building feature stores.

python ETL framework

Monitoring tool to measure flow throughput of data sources and processing components that are part of Data Ingestion and ETL pipelines.

A simple Spark-powered ETL framework that just works 🍺

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Simple and Distributed Machine Learning

mito ETL tool

Simple and Distributed Machine Learning

This repository is a getting started guide to Singer.

Big Data Toolkit for the JVM

Configurable Extract, Transform, and Load