All Projects → beam-community → avro_ex

beam-community / avro_ex

Licence: other
An Avro Library that emphasizes testability and ease of use.

Programming Languages

elixir
2628 projects

Projects that are alternatives of or similar to avro ex

Schema Registry
Confluent Schema Registry for Kafka
Stars: ✭ 1,647 (+3404.26%)
Mutual labels:  schema, avro, avro-schema
avrow
Avrow is a pure Rust implementation of the avro specification https://avro.apache.org/docs/current/spec.html with Serde support.
Stars: ✭ 27 (-42.55%)
Mutual labels:  schema, avro, avro-schema
BeFoR64
BeFoR64, Base64 encoding/decoding library for FoRtran poor men
Stars: ✭ 17 (-63.83%)
Mutual labels:  encoding, decoding
go-webp
Simple and fast webp library for golang
Stars: ✭ 91 (+93.62%)
Mutual labels:  encoding, decoding
registryless-avro-converter
An avro converter for Kafka Connect without a Schema Registry
Stars: ✭ 45 (-4.26%)
Mutual labels:  avro, avro-schema
vorbis aotuv
"aoTuV" is library for encoding and decoding of OggVorbis
Stars: ✭ 35 (-25.53%)
Mutual labels:  encoding, decoding
srclient
Golang Client for Schema Registry
Stars: ✭ 188 (+300%)
Mutual labels:  schema, avro
DjvuNet
DjvuNet is a cross platform fully managed .NET library for working with Djvu documents which can run on Linux, macOS and Windows. Library has been written in C# and targets .NET Core v3.0 and .NET Standard v2.1 or later. We intend to provide DjVu document processing capabilities on par with DjVuLibre reference library (or even better).
Stars: ✭ 54 (+14.89%)
Mutual labels:  encoding, decoding
multibase
multi base encoding/decoding utility
Stars: ✭ 15 (-68.09%)
Mutual labels:  encoding, decoding
avro turf
A library that makes it easier to use the Avro serialization format from Ruby.
Stars: ✭ 130 (+176.6%)
Mutual labels:  schema, avro
darwin
Avro Schema Evolution made easy
Stars: ✭ 26 (-44.68%)
Mutual labels:  avro, avro-schema
cpp-bencoding
A C++ bencoding library supporting both decoding and encoding.
Stars: ✭ 20 (-57.45%)
Mutual labels:  encoding, decoding
xml-avro
Convert XSD -> AVSC and XML -> AVRO
Stars: ✭ 32 (-31.91%)
Mutual labels:  avro, avro-schema
morton-nd
A header-only compile-time Morton encoding / decoding library for N dimensions.
Stars: ✭ 78 (+65.96%)
Mutual labels:  encoding, decoding
sms
A Go library for encoding and decoding SMSs
Stars: ✭ 37 (-21.28%)
Mutual labels:  encoding, decoding
d3coder
Chrome extension for encoding/decoding and hashing text on websites
Stars: ✭ 26 (-44.68%)
Mutual labels:  encoding, decoding
schema-registry
📙 json & avro http schema registry backed by Kafka
Stars: ✭ 23 (-51.06%)
Mutual labels:  schema, avro
sbt-avro
Plugin SBT to Generate Scala classes from Apache Avro schemas hosted on a remote Confluent Schema Registry.
Stars: ✭ 15 (-68.09%)
Mutual labels:  schema, avro
js-multibase
JavaScript implementation of the multibase specification
Stars: ✭ 22 (-53.19%)
Mutual labels:  encoding, decoding
universal-base64
Small universal base64 functions for node.js and browsers
Stars: ✭ 25 (-46.81%)
Mutual labels:  encoding, decoding

AvroEx

An Avro encoding/decoding library written in pure Elixir.

Documentation

The docs can be found on hex.pm

Installation

def deps do
  [{:avro_ex, "~> 2.0"}]
end

Usage

Schema Decoding

Avro uses schemas to define the shape and contract for data. The schemas that your application uses may be defined locally, or may come from a Schema Registry.

In either case, the first step is to decode a schema defined as JSON or Elixir terms into a t:AvroEx.Schema.t/0

iex> AvroEx.decode_schema!(["int", "string"])
%AvroEx.Schema{
  context: %AvroEx.Schema.Context{names: %{}},
  schema: %AvroEx.Schema.Union{
    possibilities: [
      %AvroEx.Schema.Primitive{metadata: %{}, type: :int},
      %AvroEx.Schema.Primitive{metadata: %{}, type: :string}
    ]
  }
}

AvroEx will automatically detect Elixir terms or JSON, so you can decode JSON schemas directly

iex> AvroEx.decode_schema!("[\"int\",\"string\"]")
%AvroEx.Schema{
  context: %AvroEx.Schema.Context{names: %{}},
  schema: %AvroEx.Schema.Union{
    possibilities: [
      %AvroEx.Schema.Primitive{metadata: %{}, type: :int},
      %AvroEx.Schema.Primitive{metadata: %{}, type: :string}
    ]
  }
}

Strict Schema Decoding

When writing an Avro schema, it is helpful to get feedback on unrecognized fields. For this purpose, it is recommended to use the :strict option to provide additional checks. Note that it is not recommended to use this option in production when pulling externally defined schemas, as they may have published a schema with looser validations.

iex> AvroEx.decode_schema!(%{"type" => "map", "values" => "int", "bogus" => "value"}, strict: true)
** (AvroEx.Schema.DecodeError) Unrecognized schema key `bogus` for AvroEx.Schema.Map in %{"bogus" => "value", "type" => "map", "values" => "int"}
    (avro_ex 1.2.0) lib/avro_ex/schema/parser.ex:43: AvroEx.Schema.Parser.parse!/2

Encoding

When publishing Avro data, it first must be encoded using the schema.

iex> schema = AvroEx.decode_schema!(%{
                "type" => "record",
                "name" => "MyRecord",
                "fields" => [
                  %{"name" => "a", "type" => "int"},
                  %{"name" => "b", "type" => "string"},
                ]
              })
iex> AvroEx.encode!(schema, %{a: 1, b: "two"})
<<2, 6, 116, 119, 111>

Decoding

When receiving Avro data, decode it using the schema

iex> AvroEx.decode!(schema, <<2, 6, 116, 119, 111>>)
%{"a" => 1, "b" => "two"}

Schema Encoding

AvroEx also supports encoding schemas back to JSON. This may be needed when registering schemas or serializing them to disk.

iex> AvroEx.encode_schema(schema)
"{\"fields\":[{\"name\":\"a\",\"type\":{\"type\":\"int\"}},{\"name\":\"b\",\"type\":{\"type\":\"string\"}}],\"name\":\"MyRecord\",\"type\":\"record\"}"

Additionally, schemas can be encoded to Parsing Canonical Form using the :canonical option.

iex> AvroEx.encode_schema(schema, canonical: true)
"{\"name\":\"MyRecord\",\"type\":\"record\",\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"string\"}]}"

Testing

For testing convenience, AvroEx.encodable?/2 is exported to check if data can be encoded against the given schema. Note that in production scenarios, it is not recommended to use this function.

defmodule MyModule.Test do
  use ExUnit.Case

  setup do
    data = ...
    schema = ...
    {:ok, %{data: data, schema: schema}}
  end

  describe "my_function/1" do
    test "builds a structure that can be encoded with our avro schema", context do
      result = MyModule.my_function(context.data)

      assert AvroEx.encodable?(context.schema, result)
    end
  end
end
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].