200 Open source projects that are alternatives of or similar to Awesome Spark

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

关注AI模型上线、模型部署

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

A collection of tutorials on Hadoop, MapReduce, Spark, Docker

Distributed Bayesian Entity Resolution in Apache Spark

Machine Learning Course @ Santa Clara University

The Internals of Spark Structured Streaming

This repo contains commands that data engineers use in day to day work.

C# and F# language binding and extensions to Apache Spark

Spark-Transformers: Library for exporting Apache Spark MLLIB models to use them in any Java application with no other dependencies.

Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.

This application comes as Spark2.1-as-Service-Provider using an embedded, Reactive-Streams-based, fully asynchronous HTTP server

Pyspark in Google Colab: A simple machine learning (Linear Regression) model

Spark in Action, 2nd edition - chapter 1 - Introduction

酷玩 Spark: Spark 源代码解析、Spark 类库等

The Internals of Spark SQL

A simple Spark TDD example

PySpark Cheatsheet

Serverless proxy for Spark cluster

Spark in Action, 2nd edition - chapter 7 - Ingestion from files

Code for my presentation: Using PySpark to Process Boat Loads of Data

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/

Morpheus brings the leading graph query language, Cypher, onto the leading distributed processing platform, Spark.

This is a place to put s2i images and utilities for spark application builders for openshift

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

An Apache Spark standalone application using the Spark API in Scala. The application uses Simple Build Tool(SBT) for building the project.

Easy to use library to bring Tensorflow on Apache Spark

(Deprecated) Scikit-learn integration package for Apache Spark

Apache Zeppelin notebooks for Recommendation Engines using Keras and Machine Learning on Apache Spark

Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.

A library that provides useful extensions to Apache Spark and PySpark.

Automatic feature engineering using Generative Adversarial Networks using Deeplearning4j and Apache Spark.

Detect common phrases in large amounts of text using a data-driven approach. Size of discovered phrases can be arbitrary. Can be used in languages other than English

t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark

Spark cloud integration: tests, cloud committers and more

Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.

Collection of Databricks and Jupyter Notebooks

Spark structured streaming examples with using of version 3.0.0

Bulletproof Apache Spark jobs with fast root cause analysis of failures.

Easy CPU Profiling for Apache Spark applications

Overlapping community detection in Large-Scale Networks using BigCLAM model build on Apache Spark

Basin is a visual programming editor for building Spark and PySpark pipelines. Easily build, debug, and deploy complex ETL pipelines from your browser

Example project implementing best practices for PySpark ETL jobs and applications.

O'Reilly Book: [Data Algorithms with Spark] by Mahmoud Parsian

SparkTDA is a package for Apache Spark providing Topological Data Analysis Functionalities.

Anovos - An Open Source Library for Scalable feature engineering Using Apache-Spark

Just a boilerplate for PySpark and Flask

A command-line tool for launching Apache Spark clusters.

Open Source Contributor Index

PySpark Algorithms Book: https://www.amazon.com/dp/B07X4B2218/ref=sr_1_2

bring sf to spark in production

Simple demonstration of how to build a complex real time machine learning visualization tool.

Counting Tweets Per User in Real-Time

Big Data RDF Processing and Analytics Stack built on Apache Spark and Apache Jena http://sansa-stack.github.io/SANSA-Stack/

Jupyter magics and kernels for working with remote Spark clusters

Streaming System 相关的论文读物

Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.

A converter for the OSM PBFs to Parquet files

Old repo for R interface for GraphFrames

REST web service for the true real-time scoring (<1 ms) of Scikit-Learn, R and Apache Spark models