GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → jcustenborder → puppet-confluent

jcustenborder / puppet-confluent

Licence: Apache-2.0 license
Puppet Module for installing and configuring the Confluent Platform

Programming Languages

Puppet
460 projects
ruby
36898 projects - #4 most used programming language
HTML
75241 projects

Labels

puppetkafkaschema-registryconfluent

Projects that are alternatives of or similar to puppet-confluent

schema-registry-gitops
Manage Confluent Schema Registry subjects through Infrastructure as code
Stars: ✭ 36 (+157.14%)
Mutual labels:  schema-registry, confluent
Schema Registry
Confluent Schema Registry for Kafka
Stars: ✭ 1,647 (+11664.29%)
Mutual labels:  schema-registry, confluent
avrora
A convenient Elixir library to work with Avro schemas and Confluent® Schema Registry
Stars: ✭ 59 (+321.43%)
Mutual labels:  schema-registry, confluent
Kafka Rest
Confluent REST Proxy for Kafka
Stars: ✭ 1,863 (+13207.14%)
Mutual labels:  schema-registry, confluent
confluent-spark-avro
Spark UDFs to deserialize Avro messages with schemas stored in Schema Registry.
Stars: ✭ 18 (+28.57%)
Mutual labels:  schema-registry, confluent
schema-registry-php-client
A PHP 7.3+ API client for the Confluent Schema Registry REST API based on Guzzle 6 - http://docs.confluent.io/current/schema-registry/docs/index.html
Stars: ✭ 40 (+185.71%)
Mutual labels:  schema-registry, confluent
sbt-avro
Plugin SBT to Generate Scala classes from Apache Avro schemas hosted on a remote Confluent Schema Registry.
Stars: ✭ 15 (+7.14%)
Mutual labels:  schema-registry
kafkaESK
An event-driven monitoring tool that can consume messages from Apache Kafka clusters and display the aggregated data on a dashboard for analysis and maintenance.
Stars: ✭ 79 (+464.29%)
Mutual labels:  confluent
puppet-install-shell
A shell script to install puppet on multiple distros
Stars: ✭ 33 (+135.71%)
Mutual labels:  puppet
vagrant-magento
**DEPRECATED**
Stars: ✭ 12 (-14.29%)
Mutual labels:  puppet
puppet-vault
Puppet module to manage Vault (https://vaultproject.io)
Stars: ✭ 41 (+192.86%)
Mutual labels:  puppet
ksql-jdbc-driver
JDBC driver for Apache Kafka
Stars: ✭ 85 (+507.14%)
Mutual labels:  confluent
puppet-st2
Puppet Module to install and configure StackStorm, IFTTT for Ops
Stars: ✭ 20 (+42.86%)
Mutual labels:  puppet
masterclass-codeexamples
Code examples used in Get into DevOps: The Masterclass
Stars: ✭ 35 (+150%)
Mutual labels:  puppet
puppet-stash
A puppet module to install atlassian stash
Stars: ✭ 19 (+35.71%)
Mutual labels:  puppet
avro-serde-php
Avro Serialisation/Deserialisation (SerDe) library for PHP 7.3+ & 8.0 with a Symfony Serializer integration
Stars: ✭ 43 (+207.14%)
Mutual labels:  confluent
biemond-oradb
Puppet Oracle Database Module
Stars: ✭ 59 (+321.43%)
Mutual labels:  puppet
kafka-connect-fs
Kafka Connect FileSystem Connector
Stars: ✭ 107 (+664.29%)
Mutual labels:  confluent
puppet-archlinux-macbookretina
UNSUPPORTED - Puppet module and accompanying documentation to install/setup Arch linux on a MacBook Pro Retina 11,4
Stars: ✭ 64 (+357.14%)
Mutual labels:  puppet
sonar-puppet
SonarQube plugin for Puppet code.
Stars: ✭ 27 (+92.86%)
Mutual labels:  puppet
View All Similar Projects ➔

Introduction

This puppet module is used to install and configure the Confluent Platform. The documentation is available here.

Note on versioning

Puppet only allows version numbers in <major>.<minor>.<version> format. Due to this limitation the versioning schema matches the major and minor versions of the Confluent Platform. Meaning 3.2.1234 of the puppet module is for 3.2.x of the Confluent platform.

Usage

Zookeeper

Class installation

class{'confluent::zookeeper':
  zookeeper_id => '1',
  heap_size => '4000M'
}

Hiera Installation

include ::confluent::zookeeper
confluent::zookeeper::zookeeper_id: '1'
confluent::zookeeper::config:
  server.1:
    value: 'zookeeper-01.example.com:2888:3888'
  server.2:
    value: 'zookeeper-02.example.com:2888:3888'
  server.3:
    value: 'zookeeper-03.example.com:2888:3888'
confluent::zookeeper::heap_size: '4000M'

Kafka Broker

Class installation

class{'confluent::kafka::broker':
  broker_id => '1',
  zookeeper_connect => [
    'zookeeper-01:2181',
    'zookeeper-02:2181',
    'zookeeper-03:2181'
  ],
  heap_size => '4000M'
}

Heira installation

include ::confluent::kafka::broker
confluent::kafka::broker::broker_id: '1'
confluent::kafka::broker::heap_size: '4000M'
confluent::kafka::broker::zookeeper_connect:
  - 'zookeeper-01:2181',
  - 'zookeeper-02:2181',
  - 'zookeeper-03:2181'
confluent::kafka::broker::data_path: '/var/lib/kafka'

Kafka Connect

Distributed

Class Installation

class{'confluent::kafka::connect::distributed':
  heap_size => '4000M',
  bootstrap_servers => [
    'broker-01:9092',
    'broker-02:9092',
    'broker-03:9092'
  ],
  config => {
    'key.converter' => {
      'value' => 'io.confluent.connect.avro.AvroConverter'
    },
    'value.converter' => {
      'value' => 'io.confluent.connect.avro.AvroConverter'
    },
    'key.converter.schema.registry.url' => {
      'value' => 'http://schema-registry-01:8081'
    },
    'value.converter.schema.registry.url' => {
      'value' => 'http://schema-registry-01:8081'
    },
  },
}

Heira installation

include ::confluent::kafka::connect::distributed
 confluent::kafka::connect::distributed::heap_size: '4000M'
 confluent::kafka::connect::distributed::bootstrap_servers:
  - broker-01:9092
  - broker-02:9092
  - broker-03:9092
 confluent::kafka::connect::distributed::config:
   'key.converter':
     value: 'io.confluent.connect.avro.AvroConverter'
   'value.converter':
     value: 'io.confluent.connect.avro.AvroConverter'
   'key.converter.schema.registry.url':
     value: 'http://schema-registry-01.example.com:8081'
   'value.converter.schema.registry.url':
     value: 'http://schema-registry-01.example.com:8081'

Standalone

Class Installation

class{'confluent::kafka::connect::standalone':
  heap_size => '4000M',
  bootstrap_servers => [
    'broker-01:9092',
    'broker-02:9092',
    'broker-03:9092'
  ],
  config => {
    'key.converter' => {
      'value' => 'io.confluent.connect.avro.AvroConverter'
    },
    'value.converter' => {
      'value' => 'io.confluent.connect.avro.AvroConverter'
    },
    'key.converter.schema.registry.url' => {
      'value' => 'http://schema-registry-01:8081'
    },
    'value.converter.schema.registry.url' => {
      'value' => 'http://schema-registry-01:8081'
    },
  },
}

Heira installation

include ::confluent::kafka::connect::standalone
 confluent::kafka::connect::standalone::heap_size: '4000M'
 confluent::kafka::connect::standalone::bootstrap_servers:
  - broker-01:9092
  - broker-02:9092
  - broker-03:9092
 confluent::kafka::connect::standalone::config:
   'key.converter':
     value: 'io.confluent.connect.avro.AvroConverter'
   'value.converter':
     value: 'io.confluent.connect.avro.AvroConverter'
   'key.converter.schema.registry.url':
     value: 'http://schema-registry-01.example.com:8081'
   'value.converter.schema.registry.url':
     value: 'http://schema-registry-01.example.com:8081'

Schema Registry

Class installation

class {'confluent::schema::registry':
  heap_size => '1024M',
  kafkastore_connection_url => [
      'zookeeper-01:2181',
      'zookeeper-02:2181',
      'zookeeper-03:2181'
  ]
}

Hiera installation

include ::confluent::schema::registry
confluent::schema::registry::heap_size: '1024M'
confluent::schema::registry::kafkastore_connection_url:
  - 'zookeeper-01:2181'
  - 'zookeeper-02:2181'
  - 'zookeeper-03:2181'

Confluent Control Center

Class installation

class {'confluent::control::center':
  heap_size => '6g',
  zookeeper_connect => [
    'zookeeper-01:2181',
    'zookeeper-02:2181',
    'zookeeper-03:2181'
  ],
  bootstrap_servers => [
    'broker-01:9092',
    'broker-02:9092',
    'broker-03:9092'
  ],
  connect_cluster => [
    'kafka-connect-01.example.com:8083',
    'kafka-connect-02.example.com:8083',
    'kafka-connect-03.example.com:8083'
  ]
}

Hiera installation

include ::confluent::control::center
confluent::control::center::heap_size: '6g'
confluent::control::center::zookeeper_connect: 
  - 'zookeeper-01:2181'
  - 'zookeeper-02:2181'
  - 'zookeeper-03:2181'
confluent::control::center::bootstrap_servers: 
  - 'broker-01:9092'
  - 'broker-02:9092'
  - 'broker-03:9092'
confluent::control::center::connect_cluster: 
  - 'kafka-connect-01.example.com:8083'
  - 'kafka-connect-02.example.com:8083'
  - 'kafka-connect-03.example.com:8083'

Run tests

rake spec

Rebuild github pages

rake strings:gh_pages:update
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].