154 Open source projects that are alternatives of or similar to Petastorm

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

Collection of Databricks and Jupyter Notebooks

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

Optimized Analytics Package for Spark* Platform

My presentation at ODSC India 2018 about Deep Learning with Apache Spark

O'Reilly Book: [Data Algorithms with Spark] by Mahmoud Parsian

Asynchronous actions for PySpark

Final Project for IoT: Big Data Processing and Analytics class. Analyzing U.S nationwide temperature from IoT sensors in real-time

Demo of PySpark and Jupyter Notebook with the Jupyter Docker Stacks

PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster

Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks

Generate beautiful documentation for your data pipelines in markdown format

Basin is a visual programming editor for building Spark and PySpark pipelines. Easily build, debug, and deploy complex ETL pipelines from your browser

Example project implementing best practices for PySpark ETL jobs and applications.

dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

Spark Application for analysis of Apache Access logs and detect anamolies! Along with Medium Article.

🎼 Docker compose files for various kafka stacks

High performance distributed data processing engine

关注AI模型上线、模型部署

Quilt is a self-organizing data hub for S3

异构存储数据迁移

Fast data store for Pandas time-series data

Distributed least squares approximation (dlsa) implemented with Apache Spark

A library on top of either pex or conda-pack to make your Python code easily available on a cluster

Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data sc…

A tool for data sampling, data generation, and data diffing

No description or website provided.

Create full-fledged APIs for static datasets without writing a single line of code.

Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.

Structural Bioinformatics Training Workshop & Hackathon 2018

Bucketing and partitioning system for Parquet

A library that provides useful extensions to Apache Spark and PySpark.

This is a repo documenting the best practices in PySpark.

NodeJS module to access apache parquet format files

Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.

Iceberg is a table format for large, slow-moving tabular data

Efficient training of deep recommenders on cloud.

Training models with Apache Spark, PySpark for Titanic Kaggle competition

Some class materials for a data processing course using PySpark

ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

An Azure Databricks workshop leveraging the New York Taxi and Limousine Commission Trip Records dataset

GCS support for avro-tools, parquet-tools and protobuf

Kotlin Bigdata Toolkit

Apache Parquet

Code examples for my blog posts

A simple Spark TDD example

A collection of tutorials on Hadoop, MapReduce, Spark, Docker

A boilerplate for writing PySpark Jobs

Machine Learning Course @ Santa Clara University

🚚 Agile Data Preparation Workflows made easy with dask, cudf, dask_cudf and pyspark

This repo contains commands that data engineers use in day to day work.

A tool for batch loading data files (json, parquet, csv, tsv) into ElasticSearch

GraphQL service for arrow tables and parquet data sets.

Parquet file generator

A lightweight data processing framework for Apache Spark

🏐 Apache Parquet for modern .NET

⛈️ Rumble 1.11.0 "Banyan Tree"🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more

A curated list of awesome Apache Spark packages and resources.

Jupyter magics and kernels for working with remote Spark clusters

Apache Parquet