uber-go / Kafka Client
Licence: mit
Go client library for Apache Kafka
Stars: ✭ 210
Programming Languages
go
31211 projects - #10 most used programming language
Labels
Projects that are alternatives of or similar to Kafka Client
Kafka Proxy
Proxy connections to Kafka cluster. Connect through SOCKS Proxy, HTTP Proxy or to cluster running in Kubernetes.
Stars: ✭ 186 (-11.43%)
Mutual labels: kafka
Qmq
QMQ是去哪儿网内部广泛使用的消息中间件,自2012年诞生以来在去哪儿网所有业务场景中广泛的应用,包括跟交易息息相关的订单场景; 也包括报价搜索等高吞吐量场景。
Stars: ✭ 2,420 (+1052.38%)
Mutual labels: kafka
Kafka Streams Scala
Thin Scala wrapper around Kafka Streams Java API
Stars: ✭ 192 (-8.57%)
Mutual labels: kafka
Decaton
High throughput asynchronous task processing on Apache Kafka
Stars: ✭ 187 (-10.95%)
Mutual labels: kafka
Thunder
⚡️ Nepxion Thunder is a distribution RPC framework based on Netty + Hessian + Kafka + ActiveMQ + Tibco + Zookeeper + Redis + Spring Web MVC + Spring Boot + Docker 多协议、多组件、多序列化的分布式RPC调用框架
Stars: ✭ 204 (-2.86%)
Mutual labels: kafka
Java Library Examples
💪 example of common used libraries and frameworks, programming required, don't fork man.
Stars: ✭ 204 (-2.86%)
Mutual labels: kafka
Istio Micro
istio 微服务示例代码 grpc+protobuf+echo+websocket+mysql+redis+kafka+docker-compose
Stars: ✭ 194 (-7.62%)
Mutual labels: kafka
Strimzi Kafka Operator
Apache Kafka running on Kubernetes
Stars: ✭ 2,833 (+1249.05%)
Mutual labels: kafka
Hivemq Mqtt Tensorflow Kafka Realtime Iot Machine Learning Training Inference
Real Time Big Data / IoT Machine Learning (Model Training and Inference) with HiveMQ (MQTT), TensorFlow IO and Apache Kafka - no additional data store like S3, HDFS or Spark required
Stars: ✭ 204 (-2.86%)
Mutual labels: kafka
Back End Interview
后端面试题汇总(Python、Redis、MySQL、PostgreSQL、Kafka、数据结构、算法、编程、网络)
Stars: ✭ 188 (-10.48%)
Mutual labels: kafka
Franz Go
franz-go contains a high performance, pure Go library for interacting with Kafka from 0.8.0 through 2.7.0+. Producing, consuming, transacting, administrating, etc.
Stars: ✭ 199 (-5.24%)
Mutual labels: kafka
Pos
Sample Application DDD, Reactive Microservices, CQRS Event Sourcing Powered by DERMAYON LIBRARY
Stars: ✭ 207 (-1.43%)
Mutual labels: kafka
Tech Blog
我的个人技术博客(Python、Django、Docker、Go、Redis、ElasticSearch、Kafka、Linux)
Stars: ✭ 203 (-3.33%)
Mutual labels: kafka
Go Kafka Client Library
A high level Go client library for Apache Kafka that provides the following primitives on top of sarama-cluster:
- Competing consumer semantics with dead letter queue (DLQ)
- Ability to process messages across multiple goroutines
- Ability to Ack or Nack messages out of order (with optional DLQ)
- Ability to consume from topics spread across different kafka clusters
Stability
This library is in alpha. APIs are subject to change, use at your own risk
Contributing
If you are interested in contributing, please sign the License Agreement and see our development guide
Installation
go get -u github.com/uber-go/kafka-client
Quick Start
package main
import (
"os"
"os/signal"
"github.com/uber-go/kafka-client"
"github.com/uber-go/kafka-client/kafka"
"github.com/uber-go/tally"
"go.uber.org/zap"
)
func main() {
// mapping from cluster name to list of broker ip addresses
brokers := map[string][]string{
"sample_cluster": []string{"127.0.0.1:9092"},
"sample_dlq_cluster": []string{"127.0.0.1:9092"},
}
// mapping from topic name to cluster that has that topic
topicClusterAssignment := map[string][]string{
"sample_topic": []string{"sample_cluster"},
}
// First create the kafkaclient, its the entry point for creating consumers or producers
// It takes as input a name resolver that knows how to map topic names to broker ip addrs
client := kafkaclient.New(kafka.NewStaticNameResolver(topicClusterAssignment, brokers), zap.NewNop(), tally.NoopScope)
// Next, setup the consumer config for consuming from a set of topics
config := &kafka.ConsumerConfig{
TopicList: kafka.ConsumerTopicList{
kafka.ConsumerTopic{ // Consumer Topic is a combination of topic + dead-letter-queue
Topic: kafka.Topic{ // Each topic is a tuple of (name, clusterName)
Name: "sample_topic",
Cluster: "sample_cluster",
},
DLQ: kafka.Topic{
Name: "sample_consumer_dlq",
Cluster: "sample_dlq_cluster",
},
},
},
GroupName: "sample_consumer",
Concurrency: 100, // number of go routines processing messages in parallel
}
// Create the consumer through the previously created client
consumer, err := client.NewConsumer(config)
if err != nil {
panic(err)
}
// Finally, start consuming
if err := consumer.Start(); err != nil {
panic(err)
}
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, os.Interrupt)
for {
select {
case msg, ok := <-consumer.Messages():
if !ok {
return // channel closed
}
if err := process(msg); err != nil {
msg.Nack()
} else {
msg.Ack()
}
case <-sigCh:
consumer.Stop()
<-consumer.Closed()
}
}
}
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].