GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → edenhill → Librdkafka

edenhill / Librdkafka

Licence: other
The Apache Kafka C/C++ library

Programming Languages

c
50402 projects - #5 most used programming language
C++
36643 projects - #6 most used programming language
shell
77523 projects
python
139335 projects - #7 most used programming language
CMake
9771 projects
Makefile
30231 projects

Labels

kafkahigh-performanceapache-kafkaconsumerkafka-producerkafka-consumerlibrdkafka

Projects that are alternatives of or similar to Librdkafka

Qbusbridge
The Apache Kafka Client SDK
Stars: ✭ 272 (-95.16%)
Mutual labels:  consumer, kafka, kafka-producer, kafka-consumer
Karafka
Framework for Apache Kafka based Ruby and Rails applications development.
Stars: ✭ 1,223 (-78.23%)
Mutual labels:  apache-kafka, kafka, kafka-producer, kafka-consumer
Kattlo Cli
Kattlo CLI Project
Stars: ✭ 58 (-98.97%)
Mutual labels:  apache-kafka, kafka, kafka-producer, kafka-consumer
Kq
Kafka-based Job Queue for Python
Stars: ✭ 530 (-90.56%)
Mutual labels:  apache-kafka, kafka, kafka-producer, kafka-consumer
Kukulcan
A REPL for Apache Kafka
Stars: ✭ 103 (-98.17%)
Mutual labels:  apache-kafka, kafka, kafka-producer, kafka-consumer
Rafka
Kafka proxy with a simple API, speaking the Redis protocol
Stars: ✭ 49 (-99.13%)
Mutual labels:  kafka, kafka-producer, kafka-consumer
Trubka
A CLI tool for Kafka
Stars: ✭ 296 (-94.73%)
Mutual labels:  kafka, kafka-producer, kafka-consumer
Strimzi Kafka Bridge
Apache Kafka bridge
Stars: ✭ 137 (-97.56%)
Mutual labels:  kafka, kafka-producer, kafka-consumer
Anotherkafkamonitor Akm
Another app which used to monitor the progress of Kafka Producer and Consumer
Stars: ✭ 36 (-99.36%)
Mutual labels:  consumer, kafka, kafka-producer
Apachekafkatutorials
Example Code for Kafka Tutorials @ Learning Journal
Stars: ✭ 155 (-97.24%)
Mutual labels:  apache-kafka, kafka-producer, kafka-consumer
Insulator
A client UI to inspect Kafka topics, consume, produce and much more
Stars: ✭ 53 (-99.06%)
Mutual labels:  consumer, kafka-consumer, apache-kafka
Kafka Ui
Open-Source Web GUI for Apache Kafka Management
Stars: ✭ 230 (-95.91%)
Mutual labels:  apache-kafka, kafka, kafka-producer
Debezium
Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.
Stars: ✭ 5,937 (+5.7%)
Mutual labels:  apache-kafka, kafka, kafka-producer
qwery
A SQL-like language for performing ETL transformations.
Stars: ✭ 28 (-99.5%)
Mutual labels:  kafka-consumer, kafka-producer
Node Sinek
🎩 Most advanced high level Node.js Kafka client
Stars: ✭ 262 (-95.34%)
Mutual labels:  consumer, kafka
kafka-0.11-examples
Code snippets that demonstrate how to leverage the new Kafka 0.11 APIs
Stars: ✭ 17 (-99.7%)
Mutual labels:  kafka-consumer, kafka-producer
Kminion
KMinion is a feature-rich Prometheus exporter for Apache Kafka written in Go. It is lightweight and highly configurable so that it will meet your requirements.
Stars: ✭ 274 (-95.12%)
Mutual labels:  consumer, kafka
Kafka Webview
Full-featured web-based Apache Kafka consumer UI
Stars: ✭ 319 (-94.32%)
Mutual labels:  kafka, kafka-consumer
Racecar
Racecar: a simple framework for Kafka consumers in Ruby
Stars: ✭ 327 (-94.18%)
Mutual labels:  kafka, kafka-consumer
Wirbelsturm
Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.
Stars: ✭ 332 (-94.09%)
Mutual labels:  apache-kafka, kafka
View All Similar Projects ➔

librdkafka - the Apache Kafka C/C++ client library

Copyright (c) 2012-2020, Magnus Edenhill.

https://github.com/edenhill/librdkafka

librdkafka is a C library implementation of the Apache Kafka protocol, providing Producer, Consumer and Admin clients. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.

librdkafka is licensed under the 2-clause BSD license.

KAFKA is a registered trademark of The Apache Software Foundation and has been licensed for use by librdkafka. librdkafka has no affiliation with and is not endorsed by The Apache Software Foundation.

Features

  • Full Exactly-Once-Semantics (EOS) support
  • High-level producer, including Idempotent and Transactional producers
  • High-level balanced KafkaConsumer (requires broker >= 0.9)
  • Simple (legacy) consumer
  • Admin client
  • Compression: snappy, gzip, lz4, zstd
  • SSL support
  • SASL (GSSAPI/Kerberos/SSPI, PLAIN, SCRAM, OAUTHBEARER) support
  • Full list of supported KIPs
  • Broker version support: >=0.8 (see Broker version compatibility)
  • Guaranteed API stability for C & C++ APIs (ABI safety guaranteed for C)
  • Statistics metrics
  • Debian package: librdkafka1 and librdkafka-dev in Debian and Ubuntu
  • RPM package: librdkafka and librdkafka-devel
  • Gentoo package: dev-libs/librdkafka
  • Portable: runs on Linux, MacOS X, Windows, Solaris, FreeBSD, AIX, ...

Documentation

NOTE: The master branch is actively developed, use latest release for production use.

Installation

Installing prebuilt packages

On Mac OSX, install librdkafka with homebrew:

$ brew install librdkafka

On Debian and Ubuntu, install librdkafka from the Confluent APT repositories, see instructions here and then install librdkafka:

$ apt install librdkafka-dev

On RedHat, CentOS, Fedora, install librdkafka from the Confluent YUM repositories, instructions here and then install librdkafka:

$ yum install librdkafka-devel

On Windows, reference librdkafka.redist NuGet package in your Visual Studio project.

For other platforms, follow the source building instructions below.

Installing librdkafka using vcpkg

You can download and install librdkafka using the vcpkg dependency manager:

# Install vcpkg if not already installed
$ git clone https://github.com/Microsoft/vcpkg.git
$ cd vcpkg
$ ./bootstrap-vcpkg.sh
$ ./vcpkg integrate install

# Install librdkafka
$ vcpkg install librdkafka

The librdkafka package in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please create an issue or pull request on the vcpkg repository.

Build from source

Requirements

The GNU toolchain
GNU make
pthreads
zlib-dev (optional, for gzip compression support)
libssl-dev (optional, for SSL and SASL SCRAM support)
libsasl2-dev (optional, for SASL GSSAPI support)
libzstd-dev (optional, for ZStd compression support)

NOTE: Static linking of ZStd (requires zstd >= 1.2.1) in the producer enables encoding the original size in the compression frame header, which will speed up the consumer. Use STATIC_LIB_libzstd=/path/to/libzstd.a ./configure --enable-static to enable static ZStd linking. MacOSX example: STATIC_LIB_libzstd=$(brew ls -v zstd | grep libzstd.a$) ./configure --enable-static

Building

  ./configure
  # Or, to automatically install dependencies using the system's package manager:
  # ./configure --install-deps
  # Or, build dependencies from source:
  # ./configure --install-deps --source-deps-only

  make
  sudo make install

NOTE: See README.win32 for instructions how to build on Windows with Microsoft Visual Studio.

NOTE: See CMake instructions for experimental CMake build (unsupported).

Usage in code

  1. Refer to the examples directory for code using:
  • Producers: basic producers, idempotent producers, transactional producers.
  • Consumers: basic consumers, reading batches of messages.
  • Performance and latency testing tools.

  1. Refer to the examples GitHub repo for code connecting to a cloud streaming data service based on Apache Kafka

  2. Link your program with -lrdkafka (C) or -lrdkafka++ (C++).

Commercial support

Commercial support is available from Confluent Inc

Community support

Only the last official release is supported for community members.

File bug reports and feature requests using GitHub Issues.

Questions and discussions are welcome on the Discussions forum, and on the Confluent Community slack #clients channel.

Language bindings

See Powered by librdkafka for an incomplete list of librdkafka users.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].