All Projects → artur-tamazian → avro-schema-generator

artur-tamazian / avro-schema-generator

Licence: MIT License
Library for generating avro schema files (.avsc) based on DB tables structure

Programming Languages

java
68154 projects - #9 most used programming language

Projects that are alternatives of or similar to avro-schema-generator

schema-registry-php-client
A PHP 7.3+ API client for the Confluent Schema Registry REST API based on Guzzle 6 - http://docs.confluent.io/current/schema-registry/docs/index.html
Stars: ✭ 40 (+5.26%)
Mutual labels:  avro, avro-schema
avrow
Avrow is a pure Rust implementation of the avro specification https://avro.apache.org/docs/current/spec.html with Serde support.
Stars: ✭ 27 (-28.95%)
Mutual labels:  avro, avro-schema
Schema Registry
Confluent Schema Registry for Kafka
Stars: ✭ 1,647 (+4234.21%)
Mutual labels:  avro, avro-schema
Examples
Demo applications and code examples for Confluent Platform and Apache Kafka
Stars: ✭ 571 (+1402.63%)
Mutual labels:  avro, jdbc
registryless-avro-converter
An avro converter for Kafka Connect without a Schema Registry
Stars: ✭ 45 (+18.42%)
Mutual labels:  avro, avro-schema
tamer
Standalone alternatives to Kafka Connect Connectors
Stars: ✭ 42 (+10.53%)
Mutual labels:  avro, jdbc
avro-serde-php
Avro Serialisation/Deserialisation (SerDe) library for PHP 7.3+ & 8.0 with a Symfony Serializer integration
Stars: ✭ 43 (+13.16%)
Mutual labels:  avro, avro-schema
avrora
A convenient Elixir library to work with Avro schemas and Confluent® Schema Registry
Stars: ✭ 59 (+55.26%)
Mutual labels:  avro, avro-schema
xml-avro
Convert XSD -> AVSC and XML -> AVRO
Stars: ✭ 32 (-15.79%)
Mutual labels:  avro, avro-schema
darwin
Avro Schema Evolution made easy
Stars: ✭ 26 (-31.58%)
Mutual labels:  avro, avro-schema
avro ex
An Avro Library that emphasizes testability and ease of use.
Stars: ✭ 47 (+23.68%)
Mutual labels:  avro, avro-schema
avro-typescript
TypeScript Code Generator for Apache Avro Schema Types
Stars: ✭ 19 (-50%)
Mutual labels:  avro, avro-schema
prestashop-shop-creator
Generate random demo data to test your PrestaShop shop.
Stars: ✭ 22 (-42.11%)
Mutual labels:  generator
badaso
The API & platform builder, build your apps 10x faster even more, it's open source & 100% free !
Stars: ✭ 650 (+1610.53%)
Mutual labels:  generator
git-conventional-commits
Git Conventional Commits Util to generate Semantic Version and Markdown Change Log and Validate Commit Messag
Stars: ✭ 58 (+52.63%)
Mutual labels:  generator
kloadgen
KLoadGen is kafka load generator plugin for jmeter designed to work with AVRO and JSON schema Registries.
Stars: ✭ 144 (+278.95%)
Mutual labels:  avro-schema
spring-boot-project-builder
快速构建Spring Boot项目
Stars: ✭ 20 (-47.37%)
Mutual labels:  generator
hexo-generator-index2
Filtered index generator for Hexo
Stars: ✭ 40 (+5.26%)
Mutual labels:  generator
generator-vue-component
📦 Yeoman generator to build your own Vue.js components
Stars: ✭ 32 (-15.79%)
Mutual labels:  generator
incubator-linkis
Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.
Stars: ✭ 2,459 (+6371.05%)
Mutual labels:  jdbc

Build Status Maven Central Codecov Chat in Gitter MIT License

avro-schema-generator

Library for generating avro schema files (.avsc) based on DB tables structure.

How it works:

  1. Prepare a db connection URL, username and a password. Use these to create a DbSchemaExtractor
  2. If you're fancy, configure FormatterConfig and AvroConfig.
  3. Tell schema extractor to give you avro models for tables that you're interested in.
  4. Pass those to SchemaGenerator which will give you perfect avro schemas.

Here's basic example:

// Initialize db crawler that will create avro model objects
DbSchemaExtractor schemaExtractor = new DbSchemaExtractor("jdbc:mysql://localhost:3306", "root", "pass");

AvroConfig avroConfig = new AvroConfig("some.namespace");
// Get avro models for a few tables
List<AvroSchema> schemas = schemaExtractor.getForTables(avroConfig, "mydb", "users", "payments");

for (AvroSchema schema : schemas) {
    // Pass avro model to SchemaGenerator, get schema and print it out.
    String schemaJson = SchemaGenerator.generate(schema);
    System.out.println(schemaJson);
}

More complex example:

DbSchemaExtractor schemaExtractor = new DbSchemaExtractor("jdbc:mysql://localhost:3306", "root", "pass");

// Some of available configuration options
AvroConfig avroConfig = new AvroConfig("some.namespace")
    .setRepresentEnumsAsStrings(true) // use 'string' avro type instead of 'enum' for enums
    .setAllFieldsDefaultNull(true)    // adds default: 'null' to fields definition
    .setNullableTrueByDefault(true)   // makes all fields nullable
    .setUseSqlCommentsAsDoc(true)     // use sql comments to fill 'doc' field
    .setSchemaNameMapper(new ToCamelCase().andThen(new RemovePlural())) // specify table name transformation to be used for schema name
    .setUnknownTypeResolver(type -> "string") // specify what to do with custom and unsupported db types
    .setDateTypeClass(Date.class) // add hint for avro compiler about which class to use for dates
    .setAvroSchemaPostProcessor((schema, table) -> {
        // adding some custom properties to avro schema
        schema.addCustomProperty("db-schema-name", "mydb");
        schema.addCustomProperty("db-table-name", table.getName());
});

// Get avro models for a few tables
List<AvroSchema> schemas = schemaExtractor.getAll(avroConfig);

// You can specify some formatting options by creating a FormatterConfig and passing it to SchemaGenerator.
FormatterConfig formatterConfig = FormatterConfig.builder()
            .setPrettyPrintSchema(true)
            .setPrettyPrintFields(false)
            .setIndent("    ")
            .build();

for (AvroSchema schema : schemas) {
    String schemaJson = SchemaGenerator.generate(schema, formatterConfig);
    System.out.println(schemaJson);
}

Maven dependency

<dependency>
  <groupId>com.github.artur-tamazian</groupId>
  <artifactId>avro-schema-generator</artifactId>
  <version>1.0.10</version>
</dependency>

Supported databases

DB crawling is done using the SchemaCrawler http://www.schemacrawler.com/. So it should work fine with RDBMS mentioned here: http://www.schemacrawler.com/database-support.html.

avro-schema-generator itself was successfully used with MySQL and PostgreSQL.

TODO / Suggested contributions

  • Refactor using Project Lombok
  • Add missing or db-specific types support
  • Migrate to Gradle
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].