536 Open source projects that are alternatives of or similar to pyspark-cheatsheet

Spark in Action, 2nd edition - chapter 1 - Introduction

A boilerplate project for Azure Big Data PaaS services

Basic framework utilities to quickly start writing production ready Apache Spark applications

This repo contains commands that data engineers use in day to day work.

Celeborn provides an elastic and high-performance service for shuffle and spilled data.

PySpark phonetic and string matching algorithms

可视化大屏解决方案, 提供一套可视化编辑引擎, 助力个人或企业轻松定制自己的可视化大屏应用.

The Internals of Spark SQL

An Apache Spark standalone application using the Spark API in Scala. The application uses Simple Build Tool(SBT) for building the project.

This code creates a Kinesis Firehose in AWS to send CloudWatch log data to S3.

Toolkit for highly memory efficient analysis of single-cell RNA-Seq, scATAC-Seq and CITE-Seq data. Analyze atlas scale datasets with millions of cells on laptop.

Spark in Action, 2nd edition - chapter 7 - Ingestion from files

🐬 Apache Parquet for modern .Net

This project aim is to provide free software to fetch data from plcs (Siemens S7-300/400/1200/1500) and store it. Used stack is completly opensource. I used InfluDB as data storage, so application principle is following Big Data paradigm.

A distributed key-value storage that's tolerant to Byzantine fault.

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

Code for my presentation: Using PySpark to Process Boat Loads of Data

Apache Spark Data Source for ROOT File Format

Apache Pulsar Adapters

Materials for "Machine Learning on Big Data" course

A distributed in-memory key-value storage for billions of small objects.

A distributed, fast open-source graph database featuring horizontal scalability and high availability

"Byteslice: Pushing the envelop of main memory data processing with a new storage layout" (SIGMOD'15)

Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.

DevOps pipeline for Real Time Social/Web Mining

Mirror of Apache Falcon

PySpark Cheatsheet

基于Vue、three.js、echarts，数据可视化展示项目，包含三维模型导入交互、三维模型标注等功能

Matlab library for gradient descent algorithms: Version 1.0.1

Mirror of Apache Airavata Django Portal

Data cleaning, pre-processing, and Analytics on a million movies using Spark and Scala.

A crawler and API server for Liquor Control Board of Ontario retail data

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

TalariaDB is a distributed, highly available, and low latency time-series database for Presto

This is a place to put s2i images and utilities for spark application builders for openshift

Ppts, códigos y videos de las meetups, data science days, videollamadas y workshops. Data Science Research es una organización sin fines de lucro que busca difundir, descentralizar y difundir los conocimientos en Ciencia de Datos e Inteligencia Artificial en el Perú, dando oportunidades a nuevos talentos mediante MeetUps, Workshops y Semilleros …

Anovos - An Open Source Library for Scalable feature engineering Using Apache-Spark

Automile offers a simple, smart, cutting-edge telematics solution for businesses to track and manage their business vehicles.

Distributed least squares approximation (dlsa) implemented with Apache Spark

Machine Learning Course @ Santa Clara University

Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data sc…

Mirror of Apache CouchDB Mango

A High-Performance Data Science Toolkit for the Earth Sciences

Distributed stream processing model and its implementation

High level linear algebra library for Dlang

☁️ nodejs kafka connect connector for Google BigQuery

Just a boilerplate for PySpark and Flask

swissgeol.ch gives you insight in geoscientific data - above and below the surface.

Wrangler Transform: A DMD system for transforming Big Data

Extensible streaming ingestion pipeline on top of Apache Spark

Deploy a secured, clustered, auto-scaling NiFi service in AWS.

Automile offers a simple, smart, cutting-edge telematics solution for businesses to track and manage their business vehicles.

RECSM-UPF Summer School: Social Media and Big Data Research

Information Inference Service of the OpenAIRE system

Spark-Transformers: Library for exporting Apache Spark MLLIB models to use them in any Java application with no other dependencies.

Mirror of Apache CouchDB

Detect common phrases in large amounts of text using a data-driven approach. Size of discovered phrases can be arbitrary. Can be used in languages other than English

Open Source Contributor Index

PredictionIO E-Commerce Recommendation Engine Template (Scala-based parallelized engine)

Apache Arrow DataFusion SQL Query Engine