Top 625 spark open source projects

Koalas: pandas API on Apache Spark

REST job server for Apache Spark

A daily digest of the articles or videos I've found interesting, that I want to share with you.

Apache Spark testing helpers (dependency free & works with Scalatest, uTest, and MUnit)

Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.

An open source indexing subsystem that brings index-based query acceleration to Apache Spark™ and big data workloads.

Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs

Python clone of Spark, a MapReduce alike framework in Python

Book recommender system using collaborative filtering based on Spark

基于Docker构建的Hadoop开发测试环境，包含Hadoop，Hive，HBase，Spark

☁️ Cloud-scale telemetry ingestion from any stream of data with Azure Event Hubs

The Internals of Spark SQL

Installations for Data Science. Anaconda, RStudio, Spark, TensorFlow, AWS (Amazon Web Services).

Applying Data Science and Machine Learning to Solve Real World Business Problems

▁▂▄▆▇█▇▆▄▂▁

Apache Spark™ and Scala Workshops

Ruby wrapper for Apache Spark

A Spark library for Amazon SageMaker.

A Spark plugin for reading Excel files via Apache POI

Big Data Processing Framework - Unified Data API or SQL on Any Storage

RDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark

MLOps Platform

Spark, Spark Streaming and Spark SQL unit testing strategies

k-Nearest Neighbors algorithm on Spark

Java开发者或者大数据开发者面试知识点整理

Simple and Distributed Machine Learning

Apache Spark (PySpark) Practice on Real Data

Distributed compute platform implemented in Rust, and powered by Apache Arrow.

A scalable nearest neighbor search library in Apache Spark

Realtime calculation distributed system. AKA distributed lodash

Version 1 of Technical Best Practices of Azure Databricks based on real world Customer and Technical SME inputs

This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x

A better compressed bitset in Java

Self-contained examples of Apache Spark streaming integrated with Apache Kafka.

💥 🚀 封装sparkstreaming动态调节batch time(有数据就执行计算)；🚀 支持运行过程中增删topic；🚀 封装sparkstreaming 1.6 - kafka 010 用以支持 SSL。

Unified SQL Analytics Engine Based on SparkSQL

Write your Spark data to Kafka seamlessly

A RPC framework leveraging Spark RPC module

Firely's open source FHIR server

State of the Art Natural Language Processing

Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learni…

TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning

Spark Structured Streaming / Kafka / Cassandra / Elastic

Isolation Forest on Spark

GeoTrellis for PySpark

Apache Spark Connector for Azure Cosmos DB

Spark、Flink等离线任务的调度以及实时任务的监控

Profile and monitor your ML data pipeline end-to-end

Big Data Ecosystem Docker

Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.

Simple and beautiful card component with an elegant spark line, for VueJS.

An open-source toolkit for large-scale genomic analysis

Content for architecting a data science platform for products using Luigi, Spark & Flask.

HandySpark - bringing pandas-like capabilities to Spark dataframes

A Clojure dataframe library that runs on Spark

LearningApacheSpark

Monitor Apache Spark from Jupyter Notebook

Compile-time Language Integrated Queries for Scala

Julia binding for Apache Spark