Top 95 pyspark open source projects

Generate beautiful documentation for your data pipelines in markdown format

This repo contains commands that data engineers use in day to day work.

Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data sc…

No description or website provided.

A lightweight data processing framework for Apache Spark

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

Apache Spark for data engineers

Data validation library for PySpark 3.0.0

Simple and Distributed Machine Learning

JupyterLab extension that enables monitoring launched Apache Spark jobs from within a notebook

Detect common phrases in large amounts of text using a data-driven approach. Size of discovered phrases can be arbitrary. Can be used in languages other than English

Boilerplate for PySpark on Cloud Kubernetes

Collection of Databricks and Jupyter Notebooks

ETL jobs for Firefox Telemetry

PySpark phonetic and string matching algorithms

Pyspark in Google Colab: A simple machine learning (Linear Regression) model

Code for my presentation: Using PySpark to Process Boat Loads of Data

Powerful rapid automatic EDA and feature engineering library with a very easy to use API 🌟

This is a place to put s2i images and utilities for spark application builders for openshift

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Anovos - An Open Source Library for Scalable feature engineering Using Apache-Spark

Just a boilerplate for PySpark and Flask

Open Source Contributor Index

PySpark Algorithms Book: https://www.amazon.com/dp/B07X4B2218/ref=sr_1_2

Counting Tweets Per User in Real-Time

Sentiment Analysis of a Twitter Topic with Spark Structured Streaming

Real-world Spark pipelines examples

Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes

jgit-spark-connector is a library for running scalable data retrieval pipelines that process any number of Git repositories for source code analysis.

Routines and data structures for using isarn-sketches idiomatically in Apache Spark

pyspark-cassandra is a Python port of the awesome @datastax Spark Cassandra connector. Compatible w/ Spark 2.0, 2.1, 2.2, 2.3 and 2.4

Spark extension for processing large-scale 3D data sets: Astrophysics, High Energy Physics, Meteorology, …

🚚 Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

Código para workshops Spark com ambiente de desenvolvimento em docker

A connector for Apache Spark and PySpark to Dgraph databases.