154 Open source projects that are alternatives of or similar to Petastorm

Apache Spark is a fast, in-memory data processing engine with elegant and expressive development API's to allow data workers to efficiently execute streaming, machine learning or SQL workloads that require fast iterative access to datasets.This project will have sample programs for Spark in Scala language .

Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

Structural Bioinformatics Training Workshop & Hackathon 2018

JupyterLab extension that enables monitoring launched Apache Spark jobs from within a notebook

Bucketing and partitioning system for Parquet

Boilerplate for PySpark on Cloud Kubernetes

A library that provides useful extensions to Apache Spark and PySpark.

The Hadoop ETL UDFs are the main way to load data from Hadoop into EXASOL

This is a repo documenting the best practices in PySpark.

Asynchronous actions for PySpark

NodeJS module to access apache parquet format files

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

Make record oriented data to columnar format.

Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.

This code creates a Kinesis Firehose in AWS to send CloudWatch log data to S3.

Iceberg is a table format for large, slow-moving tabular data

Powerful rapid automatic EDA and feature engineering library with a very easy to use API 🌟

Efficient training of deep recommenders on cloud.

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Training models with Apache Spark, PySpark for Titanic Kaggle competition

Just a boilerplate for PySpark and Flask

Some class materials for a data processing course using PySpark

Open Source Contributor Index

ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

PySpark Algorithms Book: https://www.amazon.com/dp/B07X4B2218/ref=sr_1_2

An Azure Databricks workshop leveraging the New York Taxi and Limousine Commission Trip Records dataset

Sentiment Analysis of a Twitter Topic with Spark Structured Streaming

GCS support for avro-tools, parquet-tools and protobuf

How to use Parquet in Flink

Kotlin Bigdata Toolkit

CSVs sliced, diced & analyzed.

Apache Parquet

jgit-spark-connector is a library for running scalable data retrieval pipelines that process any number of Git repositories for source code analysis.

Code examples for my blog posts

pyspark-cassandra is a Python port of the awesome @datastax Spark Cassandra connector. Compatible w/ Spark 2.0, 2.1, 2.2, 2.3 and 2.4

A simple Spark TDD example

A collection of tools for extracting FHIR resources and analytics services on top of that data.

A collection of tutorials on Hadoop, MapReduce, Spark, Docker

Código para workshops Spark com ambiente de desenvolvimento em docker

A boilerplate for writing PySpark Jobs

MorphL Community Edition uses big data and machine learning to predict user behaviors in digital products and services with the end goal of increasing KPIs (click-through rates, conversion rates, etc.) through personalization

Machine Learning Course @ Santa Clara University

Big Data Processing Framework - Unified Data API or SQL on Any Storage

🚚 Agile Data Preparation Workflows made easy with dask, cudf, dask_cudf and pyspark

Apache Spark (PySpark) Practice on Real Data

This repo contains commands that data engineers use in day to day work.

Isolation Forest on Spark

A tool for batch loading data files (json, parquet, csv, tsv) into ElasticSearch

Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.

GraphQL service for arrow tables and parquet data sets.

LearningApacheSpark

Parquet file generator

A lightweight data processing framework for Apache Spark

⛈️ Rumble 1.11.0 "Banyan Tree"🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more

A curated list of awesome Apache Spark packages and resources.

Jupyter magics and kernels for working with remote Spark clusters

Apache Parquet

t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark

Apache Spark for data engineers