HARAQA - High Availability Routing And Queueing Application
haraqa is designed to be a developer friendly, scalable message queue for data persistence and real-time data streaming between microservices. Haraqa provides high-throughput, low-latency, fault-tolerant pipelines for architectures of any size.
Table of Contents
About the Project
Overview
Haraqa is meant for handling and persisting data in a distributed system. One or more servers can be used to send and receive messages. Each server has a set of 'topics', a set of messages stored in the order received.
A Haraqa client can produce and/or consume from a server's topics. Clients can produce messages one at a time or in batches. Clients consume messages by making a request for a specific offset and limit. The messages can be consumed one at a time or in batches.
Persistence and Replication
Each server, after receiving a message from a producer, can save the message to multiple volumes. These volumes are meant to be distributed in the architecture, such as having multiple PersistentVolumes in a Kubernetes cluster, EBS in AWS, or Persistent Disks in Google Cloud. The server reads messages from the last volume when sending to consumer clients.
If a volume is removed or corrupted during a restart the server repopulates the data from the other volumes.
Getting started
API Docs
The docker server also includes local api documentation at the /docs/redocs and /docs/swagger endpoints.
Server
The recommended deployment strategy is to use Docker
docker run -it -p 4353:4353 -v $PWD/vol1:/vol1 haraqa/haraqa /vol1
docker run -it [port mapping] [volume mounts] haraqa/haraqa [flags] [volumes]
To run from source, navigate to cmd/server and run the main.go file.
cd cmd/server
go run main.go vol1
Flags:
-http uint Port to listen on (default 4353)
-cache boolean Enable queue file caching (default true)
-cors boolean Enable CORS (default true)
-docs boolean Enable Docs pages (default true)
-entries integer The number of msg entries per queue file before creating a new file (default 5000)
-limit integer Default batch limit for consumers (default -1)
-ballast integer Garbage collection memory ballast size in bytes (default 1073741824)
-prometheus boolean Enable prometheus metrics (default true)
Volumes:
Volumes will be written to in the order given and recovered from in the reverse order. Consumer requests are read from the last volume. For this reason it's recommended to use a local volume last.
For instance, given
docker run haraqa/haraqa /vol1 /vol2 /vol3
When a client produces a message, it will be written to /vol1, then /vol2, then /vol3. When a client consumes a message, it will be read from /vol3.
During recovery, if data exists in /vol3 it will be replicated to volumes /vol1 and /vol2. If /vol3 is empty, /vol2 will be replicated to /vol1 and /vol3.
Client
go get github.com/haraqa/haraqa
Contributing
We want this project to be the best it can be and all feedback, feature requests or pull requests are welcome.
License
MIT © 2019 haraqa and contributors. See LICENSE for more information.