All Projects → ErikJiang → kafka_cluster_example

ErikJiang / kafka_cluster_example

Licence: MIT license
🪲 kafka cluster example

Programming Languages

go
31211 projects - #10 most used programming language
Makefile
30231 projects

Projects that are alternatives of or similar to kafka cluster example

soa-checklist
Microservice Oriented Architecture checklist
Stars: ✭ 92 (+119.05%)
Mutual labels:  cluster
k8s-discovery
Auto discover the running environment and serves the kubernetes clientset interface for interaction
Stars: ✭ 44 (+4.76%)
Mutual labels:  cluster
memshared
Redis-like in-memory database for NodeJS clustered applications
Stars: ✭ 27 (-35.71%)
Mutual labels:  cluster
k3d-action
A GitHub Action to run lightweight ephemeral Kubernetes clusters during workflow. Fundamental advantage of this action is a full customization of embedded k3s clusters. In addition, it provides a private image registry and multi-cluster support.
Stars: ✭ 137 (+226.19%)
Mutual labels:  cluster
urb-k8s
Kubernetes adapter for Universal Resource Broker
Stars: ✭ 19 (-54.76%)
Mutual labels:  cluster
Common
SQL FineBuild provides 1-click install and best-practice configuration on Windows of SQL Server 2019 through to SQL Server 2005
Stars: ✭ 32 (-23.81%)
Mutual labels:  cluster
caddy-tlsconsul
🔒 Consul K/V storage for Caddy Web Server / Certmagic TLS data
Stars: ✭ 89 (+111.9%)
Mutual labels:  cluster
eureka-on-kubernetes
Deploy eureka cluster on kubernetes
Stars: ✭ 61 (+45.24%)
Mutual labels:  cluster
meteor-cluster
worker pool for meteor using node js native `cluster` module
Stars: ✭ 18 (-57.14%)
Mutual labels:  cluster
virgo
Crowdsourced fuzzing cluster. 🚀
Stars: ✭ 21 (-50%)
Mutual labels:  cluster
clusterhat-image
Converts Raspbian/Raspberry Pi OS images to support Cluster HAT
Stars: ✭ 71 (+69.05%)
Mutual labels:  cluster
docker-volume-hetzner
Docker Volume Plugin for accessing Hetzner Cloud Volumes
Stars: ✭ 81 (+92.86%)
Mutual labels:  cluster
traefik-cluster-ecs
This is a tutorial on how to deploy a Traefik Load Balancer in AWS using CloudFormation to load balancer development ECS tasks using hosts (FQDN).
Stars: ✭ 62 (+47.62%)
Mutual labels:  cluster
ha cluster exporter
Prometheus exporter for Pacemaker based Linux HA clusters
Stars: ✭ 63 (+50%)
Mutual labels:  cluster
rbac-tool
Rapid7 | insightCloudSec | Kubernetes RBAC Power Toys - Visualize, Analyze, Generate & Query
Stars: ✭ 546 (+1200%)
Mutual labels:  cluster
neth-proxy
Stratum <-> Stratum Proxy and optimizer for ethminer
Stars: ✭ 35 (-16.67%)
Mutual labels:  cluster
nixos-raspberry-pi-cluster
A user-guide to create a Raspberry Pi (3B+, 4) cluster under NixOS and managed by NixOps
Stars: ✭ 69 (+64.29%)
Mutual labels:  cluster
kubernetes-vmware-autoscaler
Kubernetes autoscaler for vsphere
Stars: ✭ 33 (-21.43%)
Mutual labels:  cluster
PyScholar
A 'supervised' parser for Google Scholar
Stars: ✭ 74 (+76.19%)
Mutual labels:  cluster
WatsonCluster
A simple C# class using Watson TCP to enable a one-to-one high availability cluster.
Stars: ✭ 18 (-57.14%)
Mutual labels:  cluster

🪲 kafka_cluster_example


项目获取:

$ git clone --depth=1 https://github.com/ErikJiang/kafka_cluster_example.git

项目依赖工具:

支持 make 构建:

$ make

Choose a command run in kafka_cluster_example:

Usage: make [target]

Valid target values are:

vendor                  Auto generate go vendor dir.
up                      Docker compose up for src.
down                    Docker compose down for src.
ps                      Docker compose ps for src.
logs                    Docker compose logs for src.
clean                   Clean up docker images for src.
test                    Apache benchmark test for src.
kafka-up                Docker compose up for kafka services.
kafka-down              Docker compose down for kafka services.
kafka-clean             Clean up log and data files for kafka services.
kafka-test              Check running state of the kafka service.
help                    print this help message and exit.

1. 配置 hosts 域名

使用 ifconfig -a 查看本地IP地址;

配置 /etc/hosts 文件,将域名 kfk1、kfk2、kfk3 映射到当前本地 IP 地址,例如;

# 假设本地IP为: 192.168.0.166
192.168.0.166 kfk1 kfk2 kfk3

2. 构建 Kafka 集群

# docker compose 构建方式:
$ docker-compose -f kafka/docker-compose.yml up -d

# 或使用 make 构建方式:
$ make kafka-up

如若在构建下载过程中,出现等待连接超时,可尝试在 docker 的 daemon.json 中添加注册镜像:

{
    "registry-mirrors":["https://docker.mirrors.ustc.edu.cn"]
}

若构建完成,可使用 kafkacat 检测服务是否正常运行:

# 直接进行检测验证:
$ kafkacat -L -b kfk1:19092

# 或使用 make 方式验证:
$ make kafka-test

3. 构建 Produce & Consume 服务

为 produce 和 consume 生成 vendor 依赖:

$ make vendor

构建 Produce & Consume Docker 服务

# 使用 docker compose 直接构建方式:
$ docker-compose -f src/docker-compose.yml up -d

# 或者使用 make 构建方式:
$ make up

4. 最终测试

使用 ApacheBench 进行并发测试(并发数为10,总计100个请求):

# 直接使用 ab 命令进行测试:
$ ab -n100 -c10 -T application/json -p test/ab_post_test.json http://127.0.0.1:9000/api/v1/data

# 或者使用 make 方式测试:
$ make test

文档详见:Wiki

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].