216 Open source projects that are alternatives of or similar to Spark-for-data-engineers

pyspark methods to enhance developer productivity 📣 👯 🎉

Spark Application for analysis of Apache Access logs and detect anamolies! Along with Medium Article.

Simple and Distributed Machine Learning

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

Powerful rapid automatic EDA and feature engineering library with a very easy to use API 🌟

Routines and data structures for using isarn-sketches idiomatically in Apache Spark

Real-world Spark pipelines examples

Sentiment Analysis of a Twitter Topic with Spark Structured Streaming

Fundamentals of Spark with Python (using PySpark), code examples

Structural Bioinformatics Training Workshop & Hackathon 2018

Asynchronous actions for PySpark

Spark extension for processing large-scale 3D data sets: Astrophysics, High Energy Physics, Meteorology, …

A curated list of awesome Apache Spark packages and resources.

Apache (Py)Spark type annotations (stub files).

A boilerplate for writing PySpark Jobs

JupyterLab extension that enables monitoring launched Apache Spark jobs from within a notebook

Simple and Distributed Machine Learning

PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster

Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks

Apache Spark Connector for Azure Cosmos DB

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Just a boilerplate for PySpark and Flask

GCModeller: genomics CAD(Computer Assistant Design) Modeller system in .NET language

Open Source Contributor Index

bring sf to spark in production

Extensible streaming ingestion pipeline on top of Apache Spark

The EXASOL package for R provides an interface to the EXASOL database.

Big Data RDF Processing and Analytics Stack built on Apache Spark and Apache Jena http://sansa-stack.github.io/SANSA-Stack/

Anovos - An Open Source Library for Scalable feature engineering Using Apache-Spark

Collection of Databricks and Jupyter Notebooks

Minicurso de introdução à análise de dados

Tutorial on Bayesian optimization in R

PySpark Algorithms Book: https://www.amazon.com/dp/B07X4B2218/ref=sr_1_2

ETL jobs for Firefox Telemetry

Counting Tweets Per User in Real-Time

A converter for the OSM PBFs to Parquet files

Bulletproof Apache Spark jobs with fast root cause analysis of failures.

Old repo for R interface for GraphFrames

Companion to Learning Hadoop and Learning Spark courses on Linked In Learning

PySpark phonetic and string matching algorithms

The Proxima platform.

Overlapping community detection in Large-Scale Networks using BigCLAM model build on Apache Spark

curated list of awesome tools and libraries for specific domains

Repository for integration with Apache Kafka

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/

🐬 Apache Parquet for modern .Net

Detect common phrases in large amounts of text using a data-driven approach. Size of discovered phrases can be arbitrary. Can be used in languages other than English

Kaggle Kernels (Python, R, Jupyter Notebooks)

Materials on the analysis of microarray expression data; focus on re-analysis of public data ( http://tinyurl.com/cruk-microarray)

Pyspark in Google Colab: A simple machine learning (Linear Regression) model

FuseML aims to provide an MLOps framework as the medium dynamically integrating together the AI/ML tools of your choice. It's an extensible tool built through collaboration, where Data Engineers and DevOps Engineers can come together and contribute with reusable integration code.

Big Data Management and Analysis Final Project

Boilerplate for PySpark on Cloud Kubernetes

Spark in Action, 2nd edition - chapter 7 - Ingestion from files

Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes

Methods for the parallel and distributed analysis and mining of the Protein Data Bank using MMTF and Apache Spark.

Code for my presentation: Using PySpark to Process Boat Loads of Data

Astronomy Broker based on Apache Spark

An Apache Spark standalone application using the Spark API in Scala. The application uses Simple Build Tool(SBT) for building the project.

Useless R Functions. That's it