Mosquitto with cluster

中文版自述
In a mosquitto cluster, clients can subscribe to every node, and can also publish to every other node. The cluster will make sure that published messages are forwarded as needed.
The cluster is full decentralized, autonomy system without any leader or key node, to make the system with a high availablity. E.g., each node has a fault rate with 1%, then a decentralized cluster with N nodes has a service availability which is 1-1%^N.

Usage

Install mosquitto on all of the nodes and write the addresses into mosquitto.conf, e.g.,

node_name node1
node_address 192.168.1.1:1883

node_name node2
node_address 192.168.1.2:1883

Then config the loadbalancer, take above adresses as real server address. It is strongly recommend to terminate TLS on the loadbalancer, and use plain TCP inside the cluster.

Installing

> git clone https://github.com/hui6075/mosquitto-cluster.git > cd mosquitto-cluster && vi config.mk

# WITH_BRIDGE:=yes
WITH_CLUSTER:=yes

> make && make install

Cluster Specification

Traffic cycle avoid

In order to avoid infinite PUB/SUB forwarding, the publishes from other brokers will only send to client, the subscription and unsubscription from other brokers will not be forward.

Duplicate subscription avoid

Save local clients' subscriptions and unsubscriptions by a reference counter, only forward the subscription which is fresh for local broker, and only forward the unsubscription which is no longger been subscribed by local client.

Private messages

Some private messages was introduced in order to support cluster session and retain message, i.e.,
PRIVATE SUBSCRIBE
Fix header|PacketID|Topic Filter|QoS|ClientID|SubID

PRIVATE RETAIN
Fix header|Topic|QoS|[PacketID]|ClientID|SubID|OriginalRecvTime|Payload

SESSION REQ
Fix header|ClientID|Clean Session

SESSION RESP
Fix header|ClientID|PacketID|NRsubs|sub1(topic|qos)|...|subN|NRpubs|pub1(topic|state|dir|dup|qos|mid|payload)|...|pubN|

Cluster session support

For cluster session, SESSION REQ would be broadcast for each client CONNECT which has no previous context found in local broker, remote broker which has this client's context will kick-off this client and if clean session set to false, the remote broker would return this client's session include subscription, incomplete publishes with QoS=0/1 inside SESSION RESP. This feature could be disable in mosquitto.conf, in order to endure instantaneous large number of concurrent connections at system startup phase.

Cluster retain message support

For retain message, PRIVATE SUBSCRIBE would be broadcast for each client subscription that is fresh for local broker, and if there exists a retain message, remote broker would return the retain message inside PRIVATE RETAIN. Client will receive the most recent retain message inside the cluster after a frozen window which can be configure inside mosquitto.conf.

QoS support

For QoS, all the publishes set with it's original QoS inside the cluster, and process with QoS=0, to saves the inside traffic. The actual QoS send to client decide by the broker which client connected with.

Other features

All validation, utf-8, ACL checking has disabled for the messages inside the cluster, in order to improve cluster efficiency.

Once CONNECT success with other broker, local broker will send all the local clients' subscription to remote broker, to avoid subscription loss while some brokers down and up.

The cluster current also support node/subscription recover, crash detection.

Design Philosophy

Mosquitto cluster goals

Mosquitto cluster is a distributed implementation of Mosquitto brokers with following goals:
Scalable in a horizontal fashion, which means you can expand cluster capability by increase number of brokers.
Provide continuous service under single point of failure.
Take on the role of one logical MQTT broker for millions of MQTT clients.

Pic1. Mosquitto Cluster Overview

While receive client subscriptin: Notification other brokers that the topic has been subscribed by me,
While receive client publish: route the message to the correct brokers which subscribed this topic.

Pic2. Mosquitto Cluster Private Messages

There's only one logic channel between 2 brokers, in order to reuse this channel for multi clients, some private messages has been introduced, include client id, subscription id, raw timestamp and etc., which help the broker to make a correct route decision.

Pic3. Mosquitto Cluster Internal Message Flow

Forwarding rules:
Forward local client subscriptions to the broker which is a fresh guy for the cluster or just recovered from a fault.
DO NOT forward internal messages to any broker.
Only broadcast local fresh subscription to other brokers.
Broadcast unsubscription until a topic is no longer subscribed by any local client.
Session messages broadcasting can be disable by configuration.
DO NOT forward PUB/SUBs under $SYS.

Benchmark

Using krylovsk/mqtt-benchmark as the benchmark tool to give a simply testing for the cluster. The bandwidth stop increase with 3 or more brokers due to client machine's bottleneck.

Pic4. Mean bandwidth of Mosquitto cluster

n=10k means 10000 messages to send per client, c=100 means 100 clients to start. MsgSize=1000bytes, QoS=2.

A more detailed test report is available under: https://github.com/hui6075/mosquitto/tree/develop/benchmark

Study Material

Pic5. Mosquitto code analyze(benchmark/mosquitto_code_analyze.jpg, w/o cluster)

Other Projects

• MQTT v5.0 Chinese Translation: https://github.com/hui6075/mqtt_v5
• MQTT broker forward latency measurement tool: https://github.com/hui6075/mqtt-bm-latency