200 Open source projects that are alternatives of or similar to Pyspark Stubs

Training models with Apache Spark, PySpark for Titanic Kaggle competition

Automatically setup a productive development environment with Ansible on macOS

Natural Korean Processor for Apache Spark

Demo of PySpark and Jupyter Notebook with the Jupyter Docker Stacks

Open source platform for the machine learning lifecycle

Basic framework utilities to quickly start writing production ready Apache Spark applications

A library on top of either pex or conda-pack to make your Python code easily available on a cluster

Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.

The Internals of Apache Spark

🎼 Docker compose files for various kafka stacks

R interface for Apache Spark

Apache Spark in your IDE with gradle

[NOT MAINTAINED] Predicting Bit coin price using Time series analysis and sentiment analysis of tweets on bitcoin

Some class materials for a data processing course using PySpark

Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.

An Azure Databricks workshop leveraging the New York Taxi and Limousine Commission Trip Records dataset

Example Maven configuration for a Spark, Scala project

Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.

An Apache Spark standalone application using the Spark API in Scala. The application uses Simple Build Tool(SBT) for building the project.

Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.

Distributed least squares approximation (dlsa) implemented with Apache Spark

🌐 Interactive Workshop on GeoAnalysis using PySpark

Generate beautiful documentation for your data pipelines in markdown format

Streaming System 相关的论文读物

Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data sc…

Spark examples

Operator for managing the Spark clusters on Kubernetes and OpenShift.

Haskell on Apache Spark.

A lightweight data processing framework for Apache Spark

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

Apache Spark Course Material

This is a repo documenting the best practices in PySpark.

Apache Pulsar Adapters

🚚 Agile Data Preparation Workflows made easy with dask, cudf, dask_cudf and pyspark

A list about Apache Kafka

Data validation library for PySpark 3.0.0

A curated list of Pulsar tools, integrations and resources.

This is the development repository of SparkMeasure, a tool for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark task metrics data.

Extensible streaming ingestion pipeline on top of Apache Spark

🐬 Apache Parquet for modern .Net

Cloud-based SQL engine using SPARK where data is accessible as JDBC/ODBC data source via Spark ThriftServer.

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

When Apache Pulsar meets Apache Spark

Easy CPU Profiling for Apache Spark applications

酷玩 Spark: Spark 源代码解析、Spark 类库等

Detect common phrases in large amounts of text using a data-driven approach. Size of discovered phrases can be arbitrary. Can be used in languages other than English

A framework for writing Spark 2.x applications in a pretty way

Boilerplate for PySpark on Cloud Kubernetes

This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]

Spark cloud integration: tests, cloud committers and more

A high-performance, scalable and efficient ShuffleManager plugin for Apache Spark, utilizing UCX communication layer

Apache Zeppelin notebooks for Recommendation Engines using Keras and Machine Learning on Apache Spark

Serverless proxy for Spark cluster

Collection of Databricks and Jupyter Notebooks

ETL jobs for Firefox Telemetry

Bulletproof Apache Spark jobs with fast root cause analysis of failures.

PySpark phonetic and string matching algorithms

PySpark + Scikit-learn = Sparkit-learn

Jupyter magics and kernels for working with remote Spark clusters

Morpheus brings the leading graph query language, Cypher, onto the leading distributed processing platform, Spark.