Top 63 parquet open source projects

Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files

Manipulate arrays of complex data structures as easily as Numpy.

fully asynchronous, pure JavaScript implementation of the Parquet file format

A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL

A SQLite vtable extension to read Parquet files

Simple windows desktop application for viewing & querying Apache Parquet files

Apache Parquet implementation in Rust

A consistent table management library in python

Big Data Toolkit for the JVM

A large-scale entity and relation database supporting aggregation of properties

Read and write Parquet in Scala. Use Scala classes as schema. No need to start a cluster.

Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)

Go package to read and write parquet files. parquet is a file format to store nested data structures in a flat columnar data format. It can be used in the Hadoop ecosystem and with tools such as Presto and AWS Athena.

Spark SQL index for Parquet tables

Graph-Based Data Science: an abstraction layer in Python for building knowledge graphs, integrated with popular graph libraries – atop Pandas, RDFlib, pySHACL, RAPIDS, NetworkX, iGraph, PyVis, pslpython, pyarrow, etc.

Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.

Apache Parquet

A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.

Read SparkSQL parquet file as RDD[Protobuf]

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

⛈️ Rumble 1.11.0 "Banyan Tree"🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more

GCS support for avro-tools, parquet-tools and protobuf

NodeJS module to access apache parquet format files

Quilt is a self-organizing data hub for S3

Bucketing and partitioning system for Parquet

Parquet file generator

Apache Parquet

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

Iceberg is a table format for large, slow-moving tabular data

High performance distributed data processing engine

ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

Optimized Analytics Package for Spark* Platform

Apache Parquet

Fast data store for Pandas time-series data

A tool for batch loading data files (json, parquet, csv, tsv) into ElasticSearch

A tool for data sampling, data generation, and data diffing

🏐 Apache Parquet for modern .NET

Create full-fledged APIs for static datasets without writing a single line of code.

Apache Drill is a distributed MPP query layer for self describing data

dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

Efficient training of deep recommenders on cloud.

Kotlin Bigdata Toolkit

异构存储数据迁移

Code examples for my blog posts

GraphQL service for arrow tables and parquet data sets.

Fastest and safest Rust implementation of parquet. `unsafe` free. Integration-tested against pyarrow

A custom extractor designed to read parquet for Azure Data Lake Analytics

Apache Spark is a fast, in-memory data processing engine with elegant and expressive development API's to allow data workers to efficiently execute streaming, machine learning or SQL workloads that require fast iterative access to datasets.This project will have sample programs for Spark in Scala language .

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

Julia implementation of Parquet columnar file format reader

WASP is a framework to build complex real time big data applications. It relies on a kind of Kappa/Lambda architecture mainly leveraging Kafka and Spark. If you need to ingest huge amount of heterogeneous data and analyze them through complex pipelines, this is the framework for you.

The Hadoop ETL UDFs are the main way to load data from Hadoop into EXASOL

A command line tool to query an ODBC data source and write the result into a parquet file.

Enables extraction of measurement data from binary files with extension 'raw' used by proprietary software imcFAMOS/imcSTUDIO and facilitates its storage in open source file formats

Collection of Databricks and Jupyter Notebooks

Make record oriented data to columnar format.

This code creates a Kinesis Firehose in AWS to send CloudWatch log data to S3.

Albis: High-Performance File Format for Big Data Systems

A collection of Apache Parquet add-on modules

How to use Parquet in Flink