Top 625 spark open source projects

A complete (distributed) BigData stack, running in containers

A compiler for automatically re-targeting sequential Java code to Apache Spark.

HL7 Apache Spark Datasource

Type System for Data Analysis in Python

Collection of different demo applications using Apache Spark

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

TPC-H queries in Apache Spark SQL using native DataFrames API

Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.

Framework of vectorized and distributed data analytics

Spark Shuffle Optimization with RDMA+AEP

基于Spark2.2新闻网大数据实时系统项目

简单的可视化湖州通话数据 假设数据量很大，没法用浏览器直接绘制热力图，把绘制热力图这一步骤放到线下计算分析。使用Apache Spark并行计算数据之后，再使用Apache Spark绘制热力图，然后用leafletjs加载OpenStreetMap图层和热力图图层，以达到良好的交互效果。现在使用Apache Spark实现绘制，可能是Apache Spark不擅长这方面的计算或者是我没有设计好算法，并行计算的速度比不上单机计算。Apache Spark绘制热力图和计算代码在这 https://github.com/yuanzhaokang/ParallelizeHeatmap.git .

🎼 Docker compose files for various kafka stacks

Apache Spark docker container image (Standalone mode)

Apache Spark Sentry Integration

人生苦短 我用Python

ACID Data Source for Apache Spark based on Hive ACID

Visualize column-level data lineage in Spark SQL

A parallel implementation of word2vec based on Spark

spark on kubernetes

Autoscaling for Google Cloud Dataproc

Online search advertisement platform & Realtime Campaign Monitoring [Maybe Deprecated]

Code Library for My Blog

Apache Spark in your IDE with gradle

low-level helpers for Apache Spark libraries and tests

Identifies and collects data on cc-licensed content across web crawl data and public apis.

Some class materials for a data processing course using PySpark

❄️ 🐳 Awesome tools and libs for AI, Deep Learning, Machine Learning, Computer Vision, Data Science, Data Analytics and Cognitive Computing that are baked in the oven to be Native on Kubernetes and Docker with Python, R, Scala, Java, C#, Go, Julia, C++ etc

Sparkline BI Accelerator provides fast ad-hoc query capability over Logical Cubes. This has been folded into our SNAP Platform(http://bit.ly/2oBJSpP) an Integrated BI platform on Apache Spark.

My presentation at ODSC India 2018 about Deep Learning with Apache Spark

Open-source distribute workflow schedule tools, also support streaming task.

An extensive non-reactive Typescript framework that eases the development experience in Spark AR

Resources for Spark AR

Code examples for my blog posts

Fast Data Cluster (Apache Cassandra, Kafka, Spark, Flink, YARN and HDFS with Vagrant and VirtualBox)

Implementation of Fellegi-Sunter's canonical model of record linkage in Apache Spark, including EM algorithm to estimate parameters

Spark functions to run popular phonetic and string matching algorithms

A Jupyter notebook using some standard techniques for data science and data engineering to analyze data for the 2017 flooding in Houston, TX.

Jupyter kernel for scala and spark

A library for querying Druid data sources with Apache Spark

This is the repo for the Data Engineering 3 - Cloud and Big Data Computing course delivered at the Central European University ceu.edu

A simple grocery list webapplication implemented with the Microframeworks Spark Java, Jodd, Ninja, Javalite, Pippo and Ratpack

PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster

A collection of tutorials on Hadoop, MapReduce, Spark, Docker

Turn your FHIR data set into a powerful API that can be used to develop analytics applications and augment data science workflow.

Examples of spark-lucenerdd

Distributed least squares approximation (dlsa) implemented with Apache Spark

Extensible Python SDK for developing Flyte tasks and workflows. Simple to get started and learn and highly extensible.

Machine Learning Course @ Santa Clara University

GeoTrellis PointCloud library to work with any pointcloud data on Spark

Zoe: Container Analytics as a Service -- mirror of https://gitlab.eurecom.fr/zoe/main/

This code pattern shows you how to train a machine learning model to predict type 2 diabetes using synthesized patient health records.

A boilerplate Spark AR project with Webpack

List of 9500 (and counting) Spark AR Creators. Open an issue or contact me if you want to be added.❤️

Real-Time Data Processing Pipeline & Visualization with Docker, Spark, Kafka and Cassandra

Yelper recommendation system

最好的大数据项目。《Titan数据运营系统》，本项目是一个全栈闭环系统，我们有用作数据可视化的web系统，然后用flume-kafaka-flume进行日志的读取，在hive设计数仓，编写spark代码进行数仓表之间的转化以及ads层表到mysql的迁移，使用azkaban进行定时任务的调度，使用技术：Java/Scala语言，Hadoop、Spark、Hive、Kafka、Flume、Azkaban、SpringBoot，Bootstrap， Echart等；

A Spark Publish/Subscribe NATS Connector

Sparse learning in TensorFlow using data acquired from Spark.

와이빅타 엔지니어링팀의 자료를 정리해두는 곳입니다.