536 Open source projects that are alternatives of or similar to pyspark-cheatsheet

Detect common phrases in large amounts of text using a data-driven approach. Size of discovered phrases can be arbitrary. Can be used in languages other than English

Open Source Contributor Index

The Distributed Computing library for python implemented using PyCOMPSs programming model for HPC.

An open-source, real-time Security Information & Event Management tool based on big data technologies, providing a scalable, advanced security analytics framework.

🔥 One of the most comprehensive open-source data annotation platform.

Repository for integration with Apache Kafka

Mirror of Apache Airavata PHP Gateway

Spark in Action, 2nd edition - chapter 1 - Introduction

Big Data Management and Analysis Final Project

A boilerplate project for Azure Big Data PaaS services

Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes

Basic framework utilities to quickly start writing production ready Apache Spark applications

PredictionIO E-Commerce Recommendation Engine Template (Scala-based parallelized engine)

Apache Beam Site

This repo contains commands that data engineers use in day to day work.

Apache Arrow DataFusion SQL Query Engine

Families In the WIld: A Kinship Recogntion Toolbox.

PredictionIO Python SDK

PySpark phonetic and string matching algorithms

Astronomy Broker based on Apache Spark

Bullet is a streaming query engine that can be plugged into any singular data stream using a Stream Processing framework like Apache Storm, Spark or Flink.

Toolkit for highly memory efficient analysis of single-cell RNA-Seq, scATAC-Seq and CITE-Seq data. Analyze atlas scale datasets with millions of cells on laptop.

Apache Accumulo Testing

PredictionIO Java SDK

PredictionIO, a machine learning server for developers and ML engineers.

Win probability predictions for League of Legends matches using neural networks

A privacy-focused list of alternatives to mainstream services to help the competition.

Spark in Action, 2nd edition - chapter 7 - Ingestion from files

🐬 Apache Parquet for modern .Net

Social Network Facebook Analysis (Python, Networkx)

This project aim is to provide free software to fetch data from plcs (Siemens S7-300/400/1200/1500) and store it. Used stack is completly opensource. I used InfluDB as data storage, so application principle is following Big Data paradigm.

PredictionIO Ruby SDK

A distributed key-value storage that's tolerant to Byzantine fault.

A connector for Apache Spark to access Exasol

Code for my presentation: Using PySpark to Process Boat Loads of Data

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

🦄🌟 Graph Database layer on top of Google Bigtable

Apache Spark Data Source for ROOT File Format

A connector for Apache Spark and PySpark to Dgraph databases.

Apache Pulsar Adapters

Koalas: pandas API on Apache Spark

A distributed, fast open-source graph database featuring horizontal scalability and high availability

"Byteslice: Pushing the envelop of main memory data processing with a new storage layout" (SIGMOD'15)

Boilerplate for PySpark on Cloud Kubernetes

🍉基于Hadoop和HBase的大规模海量数据去重

An open source indexing subsystem that brings index-based query acceleration to Apache Spark™ and big data workloads.

DevOps pipeline for Real Time Social/Web Mining

Apache Trafodion

Mirror of Apache Falcon

An advanced distributed task flow management on top of Celery

Mirror of Apache Airavata Django Portal

整理一些书籍 ,包含 C&C++ 、git 、Java、Keras 、Linux 、NLP 、Python 、Scala 、TensorFlow 、大数据 、推荐系统、数据库、数据挖掘 、机器学习 、深度学习 、算法等。

Data cleaning, pre-processing, and Analytics on a million movies using Spark and Scala.

Samples to the Oracle Big Data Lite VM

Operator for managing the Spark clusters on Kubernetes and OpenShift.

Functions for easily making publication-quality figures with matplotlib.

MATLAB/Octave library for stochastic optimization algorithms: Version 1.0.20

Hazelcast .NET Client

InsightEdge Core

bring sf to spark in production