All Categories â†’ Data Processing → avro

Top 86 avro open source projects

Vscode Data Preview
Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files
Storagetapper
StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service
Jackson Dataformats Binary
Uber-project for standard Jackson binary format backends: avro, cbor, ion, protobuf, smile
Mu Haskell
Mu (μ) is a purely functional framework for building micro services.
Kafkactl
Command Line Tool for managing Apache Kafka
Bigdata Playground
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
Gradle Avro Plugin
A Gradle plugin to allow easily performing Java code generation for Apache Avro. It supports JSON schema declaration files, JSON protocol declaration files, and Avro IDL files.
Avro
Apache Avro is a data serialization system.
Aptos
☀️ A tool for validating data using JSON Schema and converting JSON Schema documents into different data-interchange formats
Kafka Connect Mongodb
**Unofficial / Community** Kafka Connect MongoDB Sink Connector - Find the official MongoDB Kafka Connector here: https://www.mongodb.com/kafka-connector
Rq
Record Query - A tool for doing record analysis and transformation
Avro
A fast Go Avro codec
Abris
Avro SerDe for Apache Spark structured APIs.
Slimmessagebus
Lightweight message bus interface for .NET (pub/sub and request-response) with transport plugins for popular message brokers.
Kebs
Scala library to eliminate boilerplate
Avro Hadoop Starter
Example MapReduce jobs in Java, Hive, Pig, and Hadoop Streaming that work on Avro data.
Schema Registry
A CLI and Go client for Kafka Schema Registry
Schemer
Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.
Magnolify
A collection of Magnolia add-on modules
Kaufmann ex
Kafka backed service library.
Avro4k
Avro support for kotlinx.serialization
Bigdata File Viewer
A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.
Open Bank Mark
A bank simulation application using mainly Clojure, which can be used to end-to-end test and show some graphs.
Avro Builder
Ruby DSL to create Avro schemas
Rumble
⛈️ Rumble 1.11.0 "Banyan Tree"🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more
Gcs Tools
GCS support for avro-tools, parquet-tools and protobuf
Dcos Metrics
The metrics pipeline for DC/OS 1.9-1.11
Go Kafka Avro
A library provides consumer/producer to work with kafka, avro and schema registry
✭ 39
gokafkaavro
Avrocado
Avrocado is a convenience library to handle Avro in Golang
✭ 21
goavro
Avsc
Avro for JavaScript ⚡️
Aptos
☀️ Avro, Protobuf, Thrift on Swagger
Kafka Storm Starter
Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
Pmacct
pmacct is a small set of multi-purpose passive network monitoring tools [NetFlow IPFIX sFlow libpcap BGP BMP RPKI IGP Streaming Telemetry].
Avro4s
Avro schema generation and serialization / deserialization for Scala
Cpp Serializers
Benchmark comparing various data serialization libraries (thrift, protobuf etc.) for C++
Devops Python Tools
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
Iceberg
Iceberg is a table format for large, slow-moving tabular data
Choetl
ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
Schema Registry Ui
Web tool for Avro Schema Registry |
Ratatool
A tool for data sampling, data generation, and data diffing
schema-registry-plugin
Gradle plugin to interact with Confluent Schema-Registry.
spring-kafka-event-sourcing-sampler
Showcases how to build a small Event-sourced application using Spring Boot, Spring Kafka, Apache Avro and Apache Kafka
confluent-spark-avro
Spark UDFs to deserialize Avro messages with schemas stored in Schema Registry.
avro-schema-generator
Library for generating avro schema files (.avsc) based on DB tables structure
spring-cloud-stream-event-sourcing-testcontainers
Goal: create a Spring Boot application that handles users using Event Sourcing. So, whenever a user is created, updated, or deleted, an event informing this change is sent to Kafka. Also, we will implement another application that listens to those events and saves them in Cassandra. Finally, we will use Testcontainers for integration testing.
dotnet-avro
An Avro implementation for .NET
kafka-compose
🎼 Docker compose files for various kafka stacks
avro-typescript
TypeScript Code Generator for Apache Avro Schema Types
avrora
A convenient Elixir library to work with Avro schemas and Confluent® Schema Registry
Insulator
A client UI to inspect Kafka topics, consume, produce and much more
javascript-serialization-benchmark
Comparison and benchmark of JavaScript serialization libraries (Protocol Buffer, Avro, BSON, etc.)
goavro
Goavro translates between Go native types and binary and textual Avro data
✭ 32
goavro
avro ex
An Avro Library that emphasizes testability and ease of use.
1-60 of 86 avro projects