All Projects → salsify → Avro Builder

salsify / Avro Builder

Licence: mit
Ruby DSL to create Avro schemas

Programming Languages

ruby
36898 projects - #4 most used programming language

Projects that are alternatives of or similar to Avro Builder

Jackson Dataformats Binary
Uber-project for standard Jackson binary format backends: avro, cbor, ion, protobuf, smile
Stars: ✭ 221 (+169.51%)
Mutual labels:  hacktoberfest, avro
Mu Haskell
Mu (μ) is a purely functional framework for building micro services.
Stars: ✭ 215 (+162.2%)
Mutual labels:  hacktoberfest, avro
Ign Gazebo
Open source robotics simulator. Through Ignition Gazebo users have access to high fidelity physics, rendering, and sensor models. Additionally, users and developers have multiple points of entry to simulation including a graphical user interface, plugins, and asynchronous message passing and services. Ignition Gazebo is derived from Gazebo, and represents over 16 years of development and experience in robotics and simulation. This library is part of the Ignition Robotics project.
Stars: ✭ 81 (-1.22%)
Mutual labels:  hacktoberfest
Travel Guide
"A travel guide to suggest activities you can do once you arrive to a certain destination. Or you can just browse destinations and check out the different available activities."
Stars: ✭ 82 (+0%)
Mutual labels:  hacktoberfest
Passwd
A beautiful, cross-platform, encrypted password manager 🔐
Stars: ✭ 82 (+0%)
Mutual labels:  hacktoberfest
Flutter web auth
Flutter plugin for authenticating a user with a web service
Stars: ✭ 81 (-1.22%)
Mutual labels:  hacktoberfest
Vue Leaflet
vue-leaflet compatible with vue3
Stars: ✭ 82 (+0%)
Mutual labels:  hacktoberfest
Kde
[MIRROR] KDE team's testing overlay
Stars: ✭ 80 (-2.44%)
Mutual labels:  hacktoberfest
Docker Mycroft
Mycroft Development Environment inside Docker!
Stars: ✭ 81 (-1.22%)
Mutual labels:  hacktoberfest
Rust Onig
Rust bindings for the Oniguruma regex library
Stars: ✭ 81 (-1.22%)
Mutual labels:  hacktoberfest
Fab Speed Dial
Angular Material FAB speed dial
Stars: ✭ 82 (+0%)
Mutual labels:  hacktoberfest
Snackbar
toast-like alert pattern for Android inspired by the Google Material Design Spec
Stars: ✭ 1,234 (+1404.88%)
Mutual labels:  hacktoberfest
Dockest
Docker + Jest integration testing for Node.js
Stars: ✭ 81 (-1.22%)
Mutual labels:  hacktoberfest
Binari
Interactive code editor with a live binary tree visual designed to teach new developers the fundamentals of dynamic programming.
Stars: ✭ 82 (+0%)
Mutual labels:  hacktoberfest
Cookietemple
A collection of best practice cookiecutter templates for all domains and languages with extensive Github support
Stars: ✭ 81 (-1.22%)
Mutual labels:  hacktoberfest
Graphql Doctrine
Automatic GraphQL types from Doctrine entities
Stars: ✭ 81 (-1.22%)
Mutual labels:  hacktoberfest
Commudle Ng
World's first community management platform. And it's free!
Stars: ✭ 81 (-1.22%)
Mutual labels:  hacktoberfest
Noredink Ui
UI widgets we use -- https://noredink-ui.netlify.com/
Stars: ✭ 81 (-1.22%)
Mutual labels:  hacktoberfest
React Builder
⚙ A GUI tool to build your react app in the fastest way with all components and pages with routing.
Stars: ✭ 82 (+0%)
Mutual labels:  hacktoberfest
Acex
ACE3 Extra misc modules and components
Stars: ✭ 82 (+0%)
Mutual labels:  hacktoberfest

Avro::Builder

Build Status Gem Version

Avro::Builder provides a Ruby DSL to create Apache Avro Schemas.

This DSL was created because:

  • The Avro IDL is not supported in Ruby.
  • The Avro IDL can only be used to define Protocols.
  • Schemas can be extracted as JSON from an IDL Protocol but support for imports is still limited.

Additional background on why we developed avro-builder is provided here.

Features

  • The syntax is designed for ease-of-use.
  • Definitions can be imported by name. This includes auto-loading from a configured set of paths. This allows definitions to split across files and even reused between projects.
  • Record definitions can inherit from other record definitions.
  • Schema Store to load files written in the DSL and return Avro::Schema objects.

Limitations

  • Only Avro Schemas, not Protocols are supported.
  • See Issues for functionality that has yet to be implemented.
  • This is beta quality code. There may be breaking changes until version 1.0 is released.

Installation

Add this line to your application's Gemfile:

gem 'avro-builder'

And then execute:

$ bundle

Or install it yourself as:

$ gem install avro-builder

Railtie

When included in a Rails project, #{Rails.root}/avro/dsl is configured as a load path for the DSL.

A rake task is also defined for generating Avro JSON schemas from the DSL.

Usage

To use Avro::Builder, define a schema:

namespace 'com.example'

fixed :password, 8

enum :user_type, :ADMIN, :REGULAR

record :user do
  required :id, :long
  required :user_name, :string
  required :type, :user_type, default: :REGULAR
  required :pw, :password
  optional :full_name, :string
  required :nicknames, :array, items: :string
  required :permissions, :map, values: :bytes
end

The schema definition may be passed as a string or a block to Avro::Builder.build.

This generates the following Avro JSON schema:

{
  "type": "record",
  "name": "user",
  "namespace": "com.example",
  "fields": [
    {
      "name": "id",
      "type": "long"
    },
    {
      "name": "user_name",
      "type": "string"
    },
    {
      "name": "type",
      "type": {
        "name": "user_type",
        "type": "enum",
        "symbols": [
          "ADMIN",
          "REGULAR"
        ],
        "namespace": "com.example"
      },
      "default": "REGULAR"
    },
    {
      "name": "pw",
      "type": {
        "name": "password",
        "type": "fixed",
        "size": 8,
        "namespace": "com.example"
      }
    },
    {
      "name": "full_name",
      "type": [
        "null",
        "string"
      ],
      "default": null
    },
    {
      "name": "nicknames",
      "type": {
        "type": "array",
        "items": "string"
      }
    },
    {
      "name": "permissions",
      "type": {
        "type": "map",
        "values": "bytes"
      }
    }
  ]
}

Required and Optional

Fields for a record are specified as required or optional. Optional fields are implemented as a union in Avro, where null is the first type in the union and the field has a default value of null.

Named Types

fixed and enum fields may be specified inline as part of a record or as standalone named types.

# Either syntax is supported for specifying the size
fixed :f, 4
fixed :g, size: 8

# Either syntax is supported for specifying symbols
enum :e, :X, :Y, :Z
enum :d, symbols: [:A, :B]

# defaults can be set for enums with Ruby Avro v1.10.0
enum :c, symbols: [:A, :B], default: :A

record :my_record_with_named do
  required :f_ref, :f
  required :fixed_inline, :fixed, size: 9
  required :e_ref, :e
  required :enum_inline, :enum, symbols: [:P, :Q]
end

Complex Types

Array, maps and unions can each be embedded within another complex type using methods that match the type name:

record :complex_types do
  required :array_of_unions, :array, items: union(:int, :string)
  required :array_or_map, :union, types: [array(:int), map(:int)]
end

Methods may also be used for complex types instead of separately specifying the type name and options:

record :complex_types do
  required :array_of_unions, array(union(:int, :string))
  required :array_or_map, union(array(:int), map(:int))
end

For more on unions see below.

Nested Records

Nested records may be created by referring to the name of the previously defined record or using the field type :record.

record :sub_rec do
  required :i, :int
end

record :top_rec do
  required :sub, :sub_rec
end

Definining a subrecord inline:

record :my_rec do
  required :nested, :record do
    required :s, :string
  end
end

Nested record types defined without an explicit name are given a generated name based on the name of the field and record that they are nested within. In the example above, the nested record type would have the generated name __my_rec_nested_record:

{
  "type": "record",
  "name": "my_rec",
  "fields": [
    {
      "name": "nested",
      "type": {
        "type": "record",
        "name": "__my_rec_nested_record",
        "fields": [
          {
            "name": "s",
            "type": "string"
          }
        ]
      }
    }
  ]
}

Unions

A union may be specified within a record using required and optional with the :union type:

record :my_record_with_unions do
  required :req_union, :union, types: [:string, :int]
  optional :opt_union, :union, types: [:float, :long]
end

For an optional union, null is automatically added as the first type for the union and the field defaults to null.

Unions may also be defined using the union method instead of specifying the :union type and member types separately:

record :my_record_with_unions do
  required :req_union, union(:string, :int)
  optional :opt_union, union(:float, :long)
end

Logical Types

The DSL supports setting a logical type on any type except a union. The logical types defined in the Avro spec are more limited.

A logical type can be specified for a field using the logical_type attribute:

record :with_timestamp
 required :created_at, :long, logical_type: 'timestamp-micros'
end

Primitive types with a logical type can also be embedded within complex types using either the generic type method:

record :with_date_array
  required :date_array, :array, type(:int, logical_type: date)
end

Or using a primitive type specific method:

record :with_date_array
  required :date_array, :array, int(logical_type: date)
end

Abstract Types

Types can be declared as abstract in the DSL. Declaring a type as abstract prevents the rake task from generating an Avro JSON schema for the type.

A type can be declared as abstract using either an option or a method in the DSL when defining the type:

record :unique_id, abstract: true
  required :uuid, :fixed, size: 38
end

enum :status do
  symbols %w(valid invalid)
  abstract true
end

Type Macros

avro-builder allows type macros to be defined that expand to types that cannot normally be named in Avro schemas. These macro names are not retained in generated schemas but allow definitions to be reused across DSL files:

type_macro :timestamp, long(logical_type: 'timestamp-millis')

record :user do
  required :created_at, :timestamp
  required :updated_at, :timestamp
end

Type macros inherit the namespace from the context where they are defined or an explicit namespace option may be specified:

type_macro :timestamp, long(logical_type: 'timestamp-millis'),
           namespace: 'com.my_company'

Type macros are always marked as abstract and do not generate an Avro JSON schema file when using the rake task.

Auto-loading and Imports

Specify paths to search for definitions:

Avro::Builder.add_load_path('/path/to/dsl/files')

Undefined references are automatically loaded from a file with the same name. The load paths are searched for .rb file with a matching name.

Files may also be explicitly imported using import <filename>.

Extends

A previously defined record may be referenced in the definition of another record using extends <record_name>. This adds all of the fields from the referenced record to the current record. The current record may override fields in the record that it extends.

record :original do
  required :first, :string
  required :second, :int
end

record :extended do
  extends :original
  optional :first, :string
end

Additionally you can provide a namespace to extends if necessary to remove ambiguity.

namespace 'com.newbie'

record :original, namespace: 'com.og' do
  required :first, :string
  required :second, :int
end

record :original do
  required :first, :string
  required :second, :int
end

record :extended do
  extends :original, namespace: 'com.og'
  optional :first, :string
end

Schema Store

The Avro::Builder::SchemaStore can be used to load DSL files and return cached Avro::Schema objects. This schema store can be used as the schema store for avromatic to generate models directly from schemas defined using the DSL.

The schema store must be initialized with the path where DSL files are located:

schema_store = Avro::Builder::SchemaStore.new(path: '/path/to/dsl/files')
schema_store.find('schema_name', 'my_namespace')
#=> Avro::Schema (for file at '/path/to/dsl/files/my_namespace/schema_name.rb')

To configure Avromatic to use this schema store and its Messaging API:

Avromatic.configure do |config|
  config.schema_store = Avro::Builder::SchemaStore.new(path: 'avro/dsl')
  config.registry_url = 'https://builder:[email protected]'
  config.build_messaging!
end

Avro Generate Rake Task

There is a rake task that can be used to generate Avro schemas from all DSL files.

A rake task is automatically defined via a Railtie for Rails projects that uses #{Rails.root}/avro/dsl as the root for Avro DSL files.

Custom rake tasks can also be defined:

require 'avro/builder/rake/avro_generate_task'
Avro::Builder::Rake::AvroGenerateTask.new(name: :custom_gen,
                                          dependencies: [:load_app]) do |task|
  task.filetype = 'avsc' # default option
  task.root = '/path/to/dsl/files'
  task.load_paths << '/additional/dsl/files'
end

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Issues and pull requests are welcome on GitHub at https://github.com/salsify/avro-builder.

License

The gem is available as open source under the terms of the MIT License.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].