Top 369 big-data open source projects

OONI data processing pipeline

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

Drools processor for Apache NiFi

PredictionIO E-Commerce Recommendation Engine Template (Java-based parallelized engine)

简单的可视化湖州通话数据 假设数据量很大，没法用浏览器直接绘制热力图，把绘制热力图这一步骤放到线下计算分析。使用Apache Spark并行计算数据之后，再使用Apache Spark绘制热力图，然后用leafletjs加载OpenStreetMap图层和热力图图层，以达到良好的交互效果。现在使用Apache Spark实现绘制，可能是Apache Spark不擅长这方面的计算或者是我没有设计好算法，并行计算的速度比不上单机计算。Apache Spark绘制热力图和计算代码在这 https://github.com/yuanzhaokang/ParallelizeHeatmap.git .

open source tools for interaction with IBM PAIRS:

Mirror of Apache Lens

ACID Data Source for Apache Spark based on Hive ACID

⚡ Lightning fast density estimation in Julia ⚡

PredictionIO Classification Engine Template (Scala-based parallelized engine)

OLAP and ETL of Big Data

Mirror of Apache VXQuery

Alluxio Python client - Access Any Data Source with Python

❄️ 🐳 Awesome tools and libs for AI, Deep Learning, Machine Learning, Computer Vision, Data Science, Data Analytics and Cognitive Computing that are baked in the oven to be Native on Kubernetes and Docker with Python, R, Scala, Java, C#, Go, Julia, C++ etc

PredictionIO Similar Product Engine Template (Scala-based parallelized engine)

WebGL Heatmap Viewer for Big Data and Bioinformatics

Efficient and scalable parallelism using the message passing interface (MPI) to handle big data and highly computational problems.

Egis - a handy Ruby interface for AWS Athena

Implementation of the k-means algorithm in PyTorch that works for large datasets

Java library that sorts very large files of records by splitting into smaller sorted files and merging

PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster

A collection of tutorials on Hadoop, MapReduce, Spark, Docker

Python library for Bayesian hyper-parameters optimization

A distributed key-value storage that's tolerant to Byzantine fault.

Samples to the Oracle Big Data Lite VM

可视化大屏解决方案, 提供一套可视化编辑引擎, 助力个人或企业轻松定制自己的可视化大屏应用.

Mirror of Apache Falcon

PredictionIO PHP SDK

Apache Spark Course Material

Mirror of Apache CouchDB Mango

Wrangler Transform: A DMD system for transforming Big Data

Mirror of Apache CouchDB

PredictionIO, a machine learning server for developers and ML engineers.

clusterdock is a framework for creating Docker-based container clusters

Data validation library for PySpark 3.0.0

Collaborative Datacenter Simulation and Exploration for Everybody

🔥 One of the most comprehensive open-source data annotation platform.

subsemble R package for ensemble learning on subsets of data

an online learning algorithm library for Storm

Simple and Distributed Machine Learning

Base types for OnlineStats.

Materials for "Machine Learning on Big Data" course

"Byteslice: Pushing the envelop of main memory data processing with a new storage layout" (SIGMOD'15)

基于Vue、three.js、echarts，数据可视化展示项目，包含三维模型导入交互、三维模型标注等功能

Data cleaning, pre-processing, and Analytics on a million movies using Spark and Scala.

TalariaDB is a distributed, highly available, and low latency time-series database for Presto

Ppts, códigos y videos de las meetups, data science days, videollamadas y workshops. Data Science Research es una organización sin fines de lucro que busca difundir, descentralizar y difundir los conocimientos en Ciencia de Datos e Inteligencia Artificial en el Perú, dando oportunidades a nuevos talentos mediante MeetUps, Workshops y Semilleros …

A High-Performance Data Science Toolkit for the Earth Sciences

☁️ nodejs kafka connect connector for Google BigQuery

Apache Arrow DataFusion SQL Query Engine

Win probability predictions for League of Legends matches using neural networks

MATLAB/Octave library for stochastic optimization algorithms: Version 1.0.20

InsightEdge Core

Big Data Visualization

Apache Liminals goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production. Liminal provides a Domain Specific Language to build ML workflows on top of Apache Airflow.

A distributed block-based data storage and compute engine

Service for automatically managing and cleaning up unreferenced data

A high-performance, scalable and efficient ShuffleManager plugin for Apache Spark, utilizing UCX communication layer

An open-source, real-time Security Information & Event Management tool based on big data technologies, providing a scalable, advanced security analytics framework.

rastercube is a python library for big data analysis of georeferenced time series data (e.g. MODIS NDVI)