Top 625 spark open source projects

A distributed data processing framework in Haskell.

🐍 Quick reference guide to common patterns & functions in PySpark.

Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs

flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》

Machine Learning Platform and Recommendation Engine built on Kubernetes

A Linux based software package to control led's on Logitech G910, G810, G610 and G410.

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

Source code for James Lee's Aparch Spark with Java course

Splash, a flexible Spark shuffle manager that supports user-defined storage backends for shuffle data storage and exchange

Spark Terasort

FFM (Field-Awared Factorization Machine) on Spark

大数据入门指南 ⭐

A Scala kernel for Jupyter

Scalable stream processing platform for advanced realtime analytics on top of Kafka and Spark. LogIsland also supports MQTT and Kafka Streams (Flink being in the roadmap). The platform does complex event processing and is suitable for time series analysis. A large set of valuable ready to use processors, data sources and sinks are available.

Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.

Relation Extraction using Deep learning(CNN)

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

个人学习知识库涉及到数据仓库建模、实时计算、大数据、Java、算法等。

spark summit 2017 SanFrancisco

🔧 Use dplyr to analyze Big Data 🐘

Spark on Kubernetes infrastructure Helm charts repo

Udacity Data Engineering Nano Degree (DEND)

Run spark calculations from Ammonite

Models and Pipelines for the Spark NLP library

Spark 2.0 Python Machine Learning examples

Google Authenticator support for Laravel Spark

A framework for writing Spark 2.x applications in a pretty way

Webex Bot SDK for Node.js (deprecated in favor of https://github.com/webex/webex-bot-node-framework)

Examples for Deep Learning/Feature Store/Spark/Flink/Hive/Kafka jobs and Jupyter notebooks on Hops

Custom state store providers for Apache Spark

Cookbook to install Hadoop 2.0+ using Chef

Spark job for dependency links

MLeap: Deploy ML Pipelines to Production

Visualize data using relative ordering

Hybrid model of Gradient Boosting Trees and Logistic Regression (GBDT+LR) on Spark

A simple Spark-powered ETL framework that just works 🍺

🚢 Docker image for Apache Spark

ApacheCN 开源组织：公告、介绍、成员、活动、交流方式

Apache Spark Website

type-class based data cleansing library for Apache Spark SQL

List of Data Science Cheatsheets to rule the world

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Educational notes,Hands on problems w/ solutions for hadoop ecosystem

Optimize and improve the Label propagation algorithm

"Sentiment analysis" on a live Twitter feed with Apache Spark and Apache Bahir

Research on distributed system

Next generation tool for decentralized exchange and transformation of semi-structured data

A luigi powered analytics / warehouse stack

Big Data for Data Engineers Coursera Specialization from Yandex

四川大学拓思爱诺用户session行为数据离线分析项目

An improved implementation of the classical feature selection method: minimum Redundancy and Maximum Relevance (mRMR).

Writing application logic for Spark jobs that can be unit-tested without a SparkContext

RSparkling: Use H2O Sparkling Water from R (Spark + R + Machine Learning)

Google BigQuery support for Spark, Structured Streaming, SQL, and DataFrames with easy Databricks integration.

Word2Vec models with Twitter data using Spark. Blog:

Real-time Machine Learning with Apache Spark on Twitter Public Stream

🌐 Interactive Workshop on GeoAnalysis using PySpark

Apache Spark 官方文档中文版

Library for MQL5 (MetaTrader) with Python, Java, Apache Spark, AWS