200 Open source projects that are alternatives of or similar to Pyspark Stubs

A curated list of awesome Apache Spark packages and resources.

Spark extension for processing large-scale 3D data sets: Astrophysics, High Energy Physics, Meteorology, …

Routines and data structures for using isarn-sketches idiomatically in Apache Spark

A boilerplate for writing PySpark Jobs

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster

Apache Spark for data engineers

Fundamentals of Spark with Python (using PySpark), code examples

Apache Spark Connector for Azure Cosmos DB

Sentiment Analysis of a Twitter Topic with Spark Structured Streaming

Real-world Spark pipelines examples

Asynchronous actions for PySpark

Simple and Distributed Machine Learning

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

JupyterLab extension that enables monitoring launched Apache Spark jobs from within a notebook

Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks

Simple and Distributed Machine Learning

pyspark methods to enhance developer productivity 📣 👯 🎉

Powerful rapid automatic EDA and feature engineering library with a very easy to use API 🌟

Structural Bioinformatics Training Workshop & Hackathon 2018

Spark Application for analysis of Apache Access logs and detect anamolies! Along with Medium Article.

C# and F# language binding and extensions to Apache Spark

(Deprecated) Scikit-learn integration package for Apache Spark

A simple Spark TDD example

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Word2Vec models with Twitter data using Spark. Blog:

This application comes as Spark2.1-as-Service-Provider using an embedded, Reactive-Streams-based, fully asynchronous HTTP server

Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.

Training models with Apache Spark, PySpark for Titanic Kaggle competition

Natural Korean Processor for Apache Spark

Demo of PySpark and Jupyter Notebook with the Jupyter Docker Stacks

Open source platform for the machine learning lifecycle

A library on top of either pex or conda-pack to make your Python code easily available on a cluster

The Internals of Apache Spark

R interface for Apache Spark

[NOT MAINTAINED] Predicting Bit coin price using Time series analysis and sentiment analysis of tweets on bitcoin

Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.

Example Maven configuration for a Spark, Scala project

Example project implementing best practices for PySpark ETL jobs and applications.

Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.

🌐 Interactive Workshop on GeoAnalysis using PySpark

SparkTDA is a package for Apache Spark providing Topological Data Analysis Functionalities.

A command-line tool for launching Apache Spark clusters.

Streaming System 相关的论文读物

Spark examples

REST web service for the true real-time scoring (<1 ms) of Scikit-Learn, R and Apache Spark models

Haskell on Apache Spark.

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

Spark 2.0 Python Machine Learning examples

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Distributed Bayesian Entity Resolution in Apache Spark

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

This is a repo documenting the best practices in PySpark.

🚚 Agile Data Preparation Workflows made easy with dask, cudf, dask_cudf and pyspark

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

A list about Apache Kafka

A curated list of Pulsar tools, integrations and resources.

This is an example of real time stream processing using Spark Streaming, Kafka & Elasticsearch.

The Internals of Spark Structured Streaming

This is the development repository of SparkMeasure, a tool for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark task metrics data.