Top 128 apache-spark open source projects

Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.

The Internals of Spark SQL

A pure Python implementation of Apache Spark's RDD and DStream interfaces.

Infrastructures™ for Machine Learning Training/Inference in Production.

Apache Spark™ and Scala Workshops

pyspark methods to enhance developer productivity 📣 👯 🎉

RDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark

Notes on Apache Spark (pyspark)

Distributed Tensorflow, Keras and PyTorch on Apache Spark/Flink & Ray

Train and run Pytorch models on Apache Spark.

A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL

Apache Spark Connector for Azure Cosmos DB

Profile and monitor your ML data pipeline end-to-end

A Spark Atlas connector to track data lineage in Apache Atlas

📚 Various cheatsheets in PDF

Fundamentals of Spark with Python (using PySpark), code examples

A recommender system for discovering GitHub repos, built with Apache Spark

Simple windows desktop application for viewing & querying Apache Parquet files

Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning

A visual ETL development and debugging tool for big data

Scalable Data Science, course sets in big data Using Apache Spark over databricks and their mathematical, statistical and computational foundations using SageMath.

Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs

Apache Spark on AWS Lambda

Use the TPC-DS benchmark to test Spark SQL performance

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

Griffon Data Science Virtual Machine

Project for James' Apache Spark with Scala course

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

Splash, a flexible Spark shuffle manager that supports user-defined storage backends for shuffle data storage and exchange

Apache Spark docker image

Apache (Py)Spark type annotations (stub files).

A framework for writing Spark 2.x applications in a pretty way

Custom state store providers for Apache Spark

Open source platform for the machine learning lifecycle

A curated list of Pulsar tools, integrations and resources.

When Apache Pulsar meets Apache Spark

PySpark + Scikit-learn = Sparkit-learn

A curated list of awesome Apache Spark packages and resources.

Natural Korean Processor for Apache Spark

(Deprecated) Scikit-learn integration package for Apache Spark

The Internals of Apache Spark

This application comes as Spark2.1-as-Service-Provider using an embedded, Reactive-Streams-based, fully asynchronous HTTP server

Example Maven configuration for a Spark, Scala project

SparkTDA is a package for Apache Spark providing Topological Data Analysis Functionalities.

Spark examples

Distributed Bayesian Entity Resolution in Apache Spark

This is an example of real time stream processing using Spark Streaming, Kafka & Elasticsearch.

Cloud-based SQL engine using SPARK where data is accessible as JDBC/ODBC data source via Spark ThriftServer.

Easy CPU Profiling for Apache Spark applications

Apache Zeppelin notebooks for Recommendation Engines using Keras and Machine Learning on Apache Spark

Plot live-stats as graph from ApacheSpark application using Lightning-viz

Spark Application for analysis of Apache Access logs and detect anamolies! Along with Medium Article.

C# and F# language binding and extensions to Apache Spark

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

R interface for Apache Spark

Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.

Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.

A command-line tool for launching Apache Spark clusters.

Streaming System 相关的论文读物

REST web service for the true real-time scoring (<1 ms) of Scikit-Learn, R and Apache Spark models