1712 Open source projects that are alternatives of or similar to Metorikku

Big Data Toolkit for the JVM

An advanced distributed task flow management on top of Celery

MooseFS – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System (Software-Defined Storage)

An open-source storage layer that brings scalable, ACID transactions to Apache Spark™ and big data workloads.

Building Large-Scale AI Applications for Distributed Big Data

Geo Spatial Data Analytics on Spark

Monitoring tool to measure flow throughput of data sources and processing components that are part of Data Ingestion and ETL pipelines.

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

An on-line movie recommender using Spark, Python Flask, and the MovieLens dataset

A Spark cluster setup running on Docker containers

Example project implementing best practices for PySpark ETL jobs and applications.

A luigi powered analytics / warehouse stack

Big Data for Data Engineers Coursera Specialization from Yandex

Thrill - An EXPERIMENTAL Algorithmic Distributed Big Data Batch Processing Framework in C++

Two-day workshop that covers how to use R to interact databases and Spark

大数据入门指南 ⭐

Feature Store for Machine Learning

Scalable stream processing platform for advanced realtime analytics on top of Kafka and Spark. LogIsland also supports MQTT and Kafka Streams (Flink being in the roadmap). The platform does complex event processing and is suitable for time series analysis. A large set of valuable ready to use processors, data sources and sinks are available.

SparklingGraph provides easy to use set of features that will give you ability to proces large scala graphs using Spark and GraphX.

Realtime calculation distributed system. AKA distributed lodash

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.

An open source indexing subsystem that brings index-based query acceleration to Apache Spark™ and big data workloads.

Kyuubi is a unified multi-tenant JDBC interface for large-scale data processing and analytics, built on top of Apache Spark

Big Data Processing Framework - Unified Data API or SQL on Any Storage

DataFusion has now been donated to the Apache Arrow project

Apache Beam is a unified programming model for Batch and Streaming

Stream computing platform for bigdata

Open-source distributed computation and storage platform

Actively curated list of awesome BI tools. PRs welcome!

A python library to push ethereum blockchain data into an sql database.

Basin is a visual programming editor for building Spark and PySpark pipelines. Easily build, debug, and deploy complex ETL pipelines from your browser

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

Mirror of Apache Calcite - Avatica

Spark SQL index for Parquet tables

A complete (distributed) BigData stack, running in containers

Monitoring tool to measure flow throughput of data sources and processing components that are part of Data Ingestion and ETL pipelines.

ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

Linq to database provider.

Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.

Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.

DIRECT, the Data Integration Run-time Execution Control Tool, is a data logistics framework that can be used to monitor, log, audit and control data integration / ETL processes.

www.vixtract.ru

PySpark Algorithms Book: https://www.amazon.com/dp/B07X4B2218/ref=sr_1_2

Mirror of Apache Phoenix

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

A distributed block-based data storage and compute engine

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs on GCP and AWS.

一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中

Real-Time Event Streaming & Change Data Capture

Configurable data bridge for permanent ETL jobs

简单的可视化湖州通话数据 假设数据量很大，没法用浏览器直接绘制热力图，把绘制热力图这一步骤放到线下计算分析。使用Apache Spark并行计算数据之后，再使用Apache Spark绘制热力图，然后用leafletjs加载OpenStreetMap图层和热力图图层，以达到良好的交互效果。现在使用Apache Spark实现绘制，可能是Apache Spark不擅长这方面的计算或者是我没有设计好算法，并行计算的速度比不上单机计算。Apache Spark绘制热力图和计算代码在这 https://github.com/yuanzhaokang/ParallelizeHeatmap.git .

csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

Bender - Serverless ETL Framework

A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.

Parquet file generator

Enabling queries on compressed data.

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

A SQL-like language for performing ETL transformations.

ETL Library for Machine Learning - data pipelines, data munging and wrangling