200 Open source projects that are alternatives of or similar to Awesome Spark

Apache Spark for data engineers

Powerful rapid automatic EDA and feature engineering library with a very easy to use API 🌟

Simple and Distributed Machine Learning

Fundamentals of Spark with Python (using PySpark), code examples

Spark extension for processing large-scale 3D data sets: Astrophysics, High Energy Physics, Meteorology, …

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks

pyspark methods to enhance developer productivity 📣 👯 🎉

Real-world Spark pipelines examples

Apache Spark Connector for Azure Cosmos DB

Structural Bioinformatics Training Workshop & Hackathon 2018

JupyterLab extension that enables monitoring launched Apache Spark jobs from within a notebook

Sentiment Analysis of a Twitter Topic with Spark Structured Streaming

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Asynchronous actions for PySpark

A boilerplate for writing PySpark Jobs

Spark Application for analysis of Apache Access logs and detect anamolies! Along with Medium Article.

Apache (Py)Spark type annotations (stub files).

Routines and data structures for using isarn-sketches idiomatically in Apache Spark

Simple and Distributed Machine Learning

PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster

This is a repo documenting the best practices in PySpark.

Training models with Apache Spark, PySpark for Titanic Kaggle competition

A list about Apache Kafka

This is the development repository of SparkMeasure, a tool for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark task metrics data.

🚚 Agile Data Preparation Workflows made easy with dask, cudf, dask_cudf and pyspark

Demo of PySpark and Jupyter Notebook with the Jupyter Docker Stacks

Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

Distributed Bayesian Entity Resolution in Apache Spark

The Internals of Spark Structured Streaming

C# and F# language binding and extensions to Apache Spark

This application comes as Spark2.1-as-Service-Provider using an embedded, Reactive-Streams-based, fully asynchronous HTTP server

A simple Spark TDD example

酷玩 Spark: Spark 源代码解析、Spark 类库等

This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]

This is an example of real time stream processing using Spark Streaming, Kafka & Elasticsearch.

A library on top of either pex or conda-pack to make your Python code easily available on a cluster

Serverless proxy for Spark cluster

Morpheus brings the leading graph query language, Cypher, onto the leading distributed processing platform, Spark.

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Interactive and Reactive Data Science using Scala and Spark.

Easy to use library to bring Tensorflow on Apache Spark

(Deprecated) Scikit-learn integration package for Apache Spark

Example Maven configuration for a Spark, Scala project

Cloud-based SQL engine using SPARK where data is accessible as JDBC/ODBC data source via Spark ThriftServer.

R interface for Apache Spark

🏐 Apache Parquet for modern .NET

t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark

Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.

A guide on how to set up Jupyter with Pyspark painlessly on AWS EC2 clusters, with S3 I/O support

Spark structured streaming examples with using of version 3.0.0

Easy CPU Profiling for Apache Spark applications

Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.

Automatically setup a productive development environment with Ansible on macOS

Example project implementing best practices for PySpark ETL jobs and applications.

Basin is a visual programming editor for building Spark and PySpark pipelines. Easily build, debug, and deploy complex ETL pipelines from your browser

O'Reilly Book: [Data Algorithms with Spark] by Mahmoud Parsian

SparkTDA is a package for Apache Spark providing Topological Data Analysis Functionalities.