RabbitMQ on Kubernetes
RabbitMQ is an open-source message broker software that implements a handful of messaging protocols, originally the AMQP (Advanced Message Queuing Protocol), and also includes web-based ones such as STOMP (Simple Text Orientated Messaging Protocol), MQTT (Message Queuing Telemetry Transport), and WebSockets to decouple applications that share asynchronous data. RabbitMQ not only serves as an attractive messaging system choice due to its robustness and well-maintained open-source nature but also stands out for its ease of use and configuration. Before creating our first RabbitMQ instance and cluster, let's explore some fundamental concepts around messaging and check out some common use cases.
βΉοΈ As of writing the latest RabbitMQ version is
3.13.2
. See change log here.
Fundamentalsβ
Sync vs Async messaging πβ
In synchronous messaging, the sender waits for a response before proceeding, which can lead to bottlenecks or inefficiencies. Asynchronous messaging, on the other hand, frees up the sender by allowing messages to be sent without an immediate response, breaking the synchronous relationship between apps or microservices is known as decoupling
.
Messaging systems excel in scenarios requiring asynchronous communication. Decoupled applications that exchange high message volumes and depend on reliable delivery are prime candidates to use it. From distributed systems
to microservices architectures
and event-driven setups
, messaging brokers like RabbitMQ facilitate seamless communication across multipe diverse technological architectures.
Producers and consumers π²β
In its simplest form, the messaging graph looks like this: Producer - Queue - Consumer
Messaging Brokers ποΈβ
The Messaging broker is the core component of the messaging system, orchestrating the management of various messaging core entities, such as connections, channels, queues, and exchanges among others.
As the central hub, the broker encompasses all RabbitMQ features, ranging from clustering and mirroring to GUI and plugin management. It facilitates communication between clients and the broker itself, adhering to the open standard AMQP protocol for message transportation.
Other key components π§©β
A producer application establishes a connection
with the messaging broker via a TCP connection
. Within this connection, multiple channels
can be created, serving to logically separate data flows. Each channel facilitates the transmission of data to a queue
. These queues act as intermediary storage, from which consumer apps
connect to consume messages from. If the Pub/Sub messaging pattern
is used, an exchange
might also be present.
RabbitMQ also introduces the concept of virtual hosts (vhosts)
to partition and isolate messaging resources within the broker, enabling distinct environments or applications to operate independently within the same RabbitMQ instance, and enhancing security and manageability.
In scenarios demanding heightened fault tolerance and message reliability, messaging replicas
come into play. These replicas, used in quorum queues, ensure data redundancy and resilience by maintaining synchronized copies of messages across multiple nodes or clusters.
Messaging patterns πβ
Publish / Subscribeβ
The Pub/Sub pattern is used to broadcast messages to multiple subscribers who are interested in receiving notification updates. Here, instead of publishing a message to a single queue, messages are processed by an exchange which then routes the messages to multiple queues based on predetermined routing rules and bindings. Each queue is bound to the exchange with a routing key.
Worker queuesβ
Worker queues are a way to distribute and parallelize the processing of tasks or messages across multiple workers or consumers. It can be simpler than the Pub/Sub pattern since it doesnβt have the added abstraction of an exchange agent. In this pattern, tasks or messages are published to a single queue. Multiple workers consume messages from this queue, with each worker processing tasks independently.
Oh by the wayβ
If you are a fan of supporting Open Source projects working to make Kubernetes package management better for everyone then please consider supporting Glasskube, by giving us a Star on GitHub π
Benefits of messaging systems πβ
Scalingβ
Queues can be added to expand capacity
if needed. Even multiple broker nodes can be coupled to form messaging clusters
.
Batchingβ
Batching involves grouping multiple messages into a single batch or bundle before transmitting them over the network. This can positively impact performance, efficiency, and resource utilization.
Architecture decouplingβ
Applications can be developed and scaled independently, optimizing resource usage and interoperability.
Reliability and Persistenceβ
By clustering
, replicating
and node federation
, no node or storage volume is fully responsible for data persistence. No node need be a single point of failure.
Use casesβ
So which architectures or apps might benefit the most from messaging systems? Think of those that require asynchronous communication, scalability, reliability, and decoupling between components. Solutions like social media apps, financial services, IoT products, telecommunication apps, and supply chain management tools are just a few.
Messaging app example: π¬
For instance, every time you receive a message on WhatsApp or via text, the message payload is processed asynchronously using queues.
Financial app example: π
The same goes for when you use a financial app to put a buy order on your favorite stock. The buy details will be delivered from the front end of the producer app to the queue so that a backend consumer app can pick it up and continue processing your buy order.
+-------------+
| Financial |
| App |
+------+------+ +-----------------+
| | |
v | RabbitMQ |
+-----------+-----------+ | Broker |
| | | |
| User Initiates Buy +--------->+ Queue |
| Order | | |
| | +-----------------+
+-----------+-----------+ |
| |
v |
+-------------+ |
| Consumer | |
| App |<------------------------+
+-------------+
Features of RabbitMQ π§°β
Ease of useβ
Compared to other messaging systems, RabbitMQβs simplicity of configuration and use stands out among the rest.
Delivery Acknowledgement and consumer confirmsβ
Typical of synchronous communication but less present in async is a received acknowledgment when a message was successfully consumed. RabbitMQ has a feature that can be enabled to achieve this.
Distributed networkβ
RabbitMQ as a messaging broker fits into distributed architectures, using producer and consumer apps connecting from remote hosts, this makes it desirable for the RabbitMQ broker to also be distributed there are three ways to accomplish this: clustering, federations, or the shovel plugin.
Tools + pluginsβ
RabbitMQ has a series of tools and plugins to make managment and configuration much easier. Just as the management plugin which delivers a useful GUI from which mange management and monitoring tasks can be done as well as deployment plugins. Some useful plugins
-
rabbitmq_management: for management console access
-
rabbitmq_prometheus: exports metrics in Prometheus-compatible format, facilitating monitoring and observability.
-
rabbitmq_federation: enables cluster and exchange formation and data replication
-
rabbitmq_peer_discovery_k8s: used for automatic node discovery (cluster formation) in k8s environments
-
RabbitMQ comes with some useful CLI tools
rabbitmqcli
,rabbitmq-plugins
,rabbitmqadmin
andrabbitmq-queues
. To be executed inside the RabbitMQ nodes they enable cluster management, plugin configurations, and an array of customizations.
RabbitMQ Installation methodsβ
Docker π³β
# latest RabbitMQ 3.13
docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.13-management
βΉοΈ Check out a great in-depth installation guide from Marcel Dempers using Docker
Open Source RabbitMQ Server on multiple operating systemsβ
RabbitMQ Cluster Kubernetes Operator βΈοΈβ
We will focus on this installation method since itβs the recommended method for working with RabbitMQ in Kubernetes environments.
Install the RabbitMQ Cluster Kubernetes Operatorβ
The RabbitMQ Cluster Kubernetes Operator automates the provisioning, management, and operations of RabbitMQ clusters running on Kubernetes.
Installationβ
Installing the RabbitMQ cluster operator can easily be achieved with the Glasskube package manager.
Install Glasskubeβ
If you already installed glasskube
you can skip this step.
If not, glasskube
can easily be installed the way you usually install packages for your operating system.
- macOS
- Linux
- Windows
- NixOS/Nixpkgs
On macOS, you can use Homebrew to install and update Glasskube.
brew install glasskube/tap/glasskube
You can install Glasskube using one of the package managers below.
RPM-based installation (RedHat/CentOS/Fedora)
dnf install https://releases.dl.glasskube.dev/glasskube_v0.20.1_amd64.rpm
DEB-based installation (Ubuntu/Debian)
curl -LO https://releases.dl.glasskube.dev/glasskube_v0.20.1_amd64.deb
sudo dpkg -i glasskube_v0.20.1_amd64.deb
APK-based installation (Alpine)
curl -LO https://releases.dl.glasskube.dev/glasskube_v0.20.1_amd64.apk
apk add --allow-untrusted glasskube_v0.20.1_amd64.apk
If you are using a distribution that does not use one of the package managers above, or require a 32-bit binary, check out additional download options attached to our latest release.
Download the windows archive from our latest Release and unpack it using Windows Explorer.
You can either use the package temporarily in a nix-shell:
nix-shell -p glasskube
Or install it globally by adding pkgs.glasskube
to your environment.systemPackages
.
After installing Glasskube on your local machine, make sure to install the necessary components in your Kubernetes cluster by running glasskube bootstrap
.
For more information, check out our bootstrap guide.
Install the RabbitMQ cluster operatorβ
- GUI π₯οΈ
- CLI π§βπ»
Start the UI via the command line:
glasskube serve
Install RabbitMQ cluster operator via the Glasskube UI.
A package can be installed with a simple command.
glasskube install rabbitmq-operator
The process will wait until the package got successfully installed.
You will need access to a Kubernetes cluster running Kubernetes 1.19
or later (You can easily create a local cluster by using Minikube) kubectl
isnβt strictly speaking a dependency for installing packages via Glasskube, but it is the recommended way to interact with the cluster. Therefore, it is highly recommended. Installation instructions are available for macOS, Linux, and Windows.
βΉοΈ Glasskube will automatically install the cluster operator in a newly created namespace called
rabbitmq-system
Confirm Service Availabilityβ
Before configuring your app to use RabbitMQ Cluster Kubernetes Operator, ensure that RabbitmqCluster
Custom Resource is deployed to your Kubernetes cluster and is available.
To confirm this availability, run
kubectl get customresourcedefinitions.apiextensions.k8s.io
Create a RabbitMQ Instanceβ
The RabbitMQ Cluster Kubernetes Operator creates the necessary resources, such as Services
, StatefulSets
, Secrets
and ConfigMaps
in the same namespace in which the RabbitmqCluster
was defined.
First, create a YAML file to define a RabbitmqCluster
resource named definition.yaml.
βΉοΈ Note: The YAML file can have any name, but the steps that follow assume it is called
definition.yaml
.
Then copy and paste the below snippet into the file and save it:
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: definition
Next, apply the definition by running:
kubectl apply -f definition.yaml
Then verify that the process was successful by running:
kubectl get all -l app.kubernetes.io/name=definition
Configure a RabbitMQ Instanceβ
To configure a RabbitMQ instance in other words the cluster itself, open definition.yaml or edit the configuration in place by running:
kubectl edit rabbitmqcluster definition
By default all of the required manifests and objects to support your RabbitMQ have been created, to edit them you simply have to edit the Kubernetes manifest files.
Access the RabbitMQ dashboard on Kubernetesβ
You can access the RabbitMQ management dashboard in multiple ways but editing the service object and using a load balancer. The quickest way that does not require you to change any default configuration is by port-forwarding the definition-server-0
pod.
kubectl port-forward -n rabbitmq-system definition-server-0 8080:15672
Then you can access the dashboard login page to access the console. To find the credentials to access your deployment you will have to decode the username and password in the provisioned Kubernetes secret.
kubectl get secret definition-default-user -n rabbitmq-system -o yaml
The output will be something like this:
apiVersion: v1
data:
default_user.conf: ZGVmYXVsdF91c2VYXVsdF91c2VyXzhMZklqOGFFQmROZmcKZGVmYXVsdF9wYXNzID0gd3IyWTl3MVJLMUtOdWN2S2ptS0FaX2R3dlRDSEhiYWEK
host: ZGVmaW5pdGlvbi5yYWJiaXRtcS1zeXN0ZW0uc3Zj
password: d3IyWTl3MVJLMUtOdWN2S2ptR3dlRDSEhiYWE=
port: NTYMg==
provider: cmFiml0bXE=
type: cmFiYmlbXE=
username: ZGVmYXVsdF91c2VMZklqFFQmRsbHd4b1BOZmc=
Decode the base64 password
and username
values and pass output into the login page
# Password
β ~ echo -n d3IyWTl3MVJLMUtOdW2S2ptS0FaX2R3dlRDSEhiYWE= | base64 --decode
wr2Y9w1RK1KNucvKjmKAZ_dwvTCHHba%
# Username
β ~ echo -n ZGVmYXVsdF91c2VyXhMZklqOGFFQmRsbHd4b1BOZmc= | base64 --decode
default_user_8LfIj8aEBdllwxoPNf%
βΉοΈ To monitor your RabbitMQ instance check out this documentation.
Create a queue πβ
There are multiple ways to declare queues in RabbitMQ, depending on your workflow and preferences.
Option 1: Queue Declaration in a Producer Appβ
In most cases, you'll declare a queue within the producer app written in your preferred programming language. When the app publishes its first message, the queue declaration will automatically create the queue if it doesn't already exist. This approach allows for dynamic queue creation as needed.
Option 2: RabbitMQ Configuration Filesβ
Alternatively, you can use RabbitMQ configuration files to declare multiple types of RabbitMQ objects, including vhosts, channels, exchanges, and queues. This method provides a more structured approach to managing RabbitMQ resources and configurations.
Option 3: Management UIβ
For manual management, you can create queues directly from the RabbitMQ Management UI. It provides a user-friendly interface for interacting with RabbitMQ, allowing you to perform administrative tasks without using the command line.
User and Virtual Host Creation (Optional)β
Before creating queues, it's recommended to set up users and virtual hosts to organize messaging workflows effectively. For example:
rabbitmqctl add_user jake jakepassword
rabbitmqctl set_user_tags jake administrator
rabbitmqctl set_permissions -p / jake ".*" ".*" ".*"
rabbitmqctl add_vhost vhost-1
rabbitmqctl set_permissions -p vhost-1 jake ".*" ".*" ".*"
Queue Creation via Management UIβ
-
Access RabbitMQ Management Console: Open the RabbitMQ Management UI in your web browser.
-
Login with Credentials: Use the credentials of the user you created (e.g., username: jake, password: jakepassword) to log in to the Management UI.
-
Navigate to Queues Section: Click on the "Queues" tab in the Management UI to view existing queues.
-
Add Queue: Scroll to the bottom of the page and click on the "Add a new queue" button to create a new queue.
-
Specify the queue name, type, and other desired properties like node preference.
-
Optionally, configure queue bindings, policies, and other advanced settings based on your requirements.
-
-
Save Changes: Click on the "Add queue" or "Save" button to confirm and create the new queue.
By following these steps, you can easily create and manage queues in RabbitMQ according to your specific needs.
The queue is now ready:
Build a RabbitMQ cluster πβ
When building a production application with a messaging component to its architecture, considerations around high availability must be made. RabbitMQ offers a clustering feature that consists of joining multiple RabbitMQ brokers and replicating the configuration and the data contained inside the message queues.
βΉοΈ To understand how clusters are formed we will need to understand a few concepts. By using the operator, much of the concepts we explore below will be configured for you by default. Nonetheless, itβs still valuable to understand what is happening under the hood.
Authenticationβ
For a RabbitMQ node to join forces and form a cluster with another node that need to share an Erlang cookie environment variable. This will ensure that nodes share the configuration data needed to consider themselves a cluster. This doesnβt ensure queue data replication though. This is where the concept of mirroring comes in.
Quorum queuesβ
The RabbitMQ quorum queue is a modern queue type, which implements a durable, replicated FIFO queue based on the Raft consensus algorithm. Previously Classic Mirrored Queues were recommended but they have since been deprecated. Quorum queues are implemented much the same way yet they offer high availability via replication and focus on data safety. Policies can be applied to queues and itβs through these policies that we can configure the mirroring specification.
Queue declaration
In the past, replication of queues was specified by using policies in conjunction with Classic Queues. Quorum queues are created differently but should be compatible with all client applications which allow you to provide arguments when declaring a queue. The x-queue-type argument needs to be provided with the value quorum when creating the queue. For example, using the Elixir AMQP client1, declaring a Quorum Queue is as follows:
Queue.declare(publisher_chan, "my-quorum-queue", durable: true, arguments: [ "x-queue-type": "quorum" ])
βΉοΈ Manage quorum queues easily by using the
rabbitmq-queues
CLI. Here are some useful CLI commands.
Automatic Failoverβ
Each quorum queue is made up by a primary replica, known as the leader
in Raft terminology, along with potentially multiple secondary replicas, referred to as followers
.
Initially, a leader is elected when the cluster is formed, and subsequently, if the current leader becomes unavailable.
Suppose a node hosting the leader of a quorum queue fails or stops for any reason. In that case, another node hosting one of the followers for that quorum queue will assume leadership and resume operations.
When failed or rejoining followers come back online, they undergo a process of resynchronization, commonly known as "catching up," with the leader. Unlike traditional mirrored queues, where a temporary replica failure requires a full resynchronization from the current leader, only the differential changes are transferred if a rejoining replica lags behind the leader.
Fault toleranceβ
For optimal performance in RabbitMQ clusters, it's beneficial to constrain the quorum queue size to a smaller, odd number
of nodes.
It's also worth mentioning that performance tends to degrade noticeably with quorum queue node sizes that exceed 5. Therefore, itβs strongly advised against deploying quorum queues on more than 7 RabbitMQ nodes.
Add instances to the cluster πͺ΄β
βΉοΈ Upon deployment, the cluster operator automatically generates a set of Kubernetes manifest files defining the default configurations for the RabbitMQ instance. You can inspect these configurations using the following command: Run
k get all -n rabbitmq-admin
to see all created objects. These default configurations can be customized and updated to better align with your specific requirements and preferences.
The RabbitMQ Kubernetes operator simplifies the setup process by handling essential tasks such as Mirroring, Failover, and Node federation automatically. This means that creating a RabbitMQ cluster is as straightforward as adjusting the number of replicas in the RabbitmqCluster
definition file. With these capabilities built-in, managing RabbitMQ clusters becomes much more seamless and hassle-free.
Here I updated the RabbitmqCluster
manifest to scale to 3
replicas:
βΉοΈ When specifying the number of replicas for the
RabbitmqCluster
object an even number of replicas is highly discouraged. Odd numbers (1, 3, 5, 7, and so on) must be used.
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"rabbitmq.com/v1beta1","kind":"RabbitmqCluster","metadata":{"annotations":{},"name":"definition","namespace":"rabbitmq-system"}}
creationTimestamp: "2024-05-07T17:04:53Z"
finalizers:
- deletion.finalizers.rabbitmqclusters.rabbitmq.com
generation: 6
name: definition
namespace: rabbitmq-system
resourceVersion: "2729957"
uid: f9b4cad8-68b4-489e-b7a1-baa3b91882d5
spec:
delayStartSeconds: 30
image: rabbitmq:3.13.1-management
override: {}
persistence:
storage: 10Gi
rabbitmq:
additionalConfig: |
load_definitions = /etc/rabbitmq/definitions.json
replicas: 3 # added three replicas which will for the RabbitMQ cluster
resources:
limits:
cpu: "2"
memory: 2Gi
requests:
cpu: "1"
memory: 2Gi
secretBackend:
externalSecret: {}
service:
type: ClusterIP
terminationGracePeriodSeconds: 604800
...
Not only have the nodes been added to the cluster but the queue has been automatically replicated to all cluster nodes too
Messaging brokers such as RabbitMQ play a central role in event-driven architectures, offering flexibility
for task processing
, workflow orchestration
, and service integrations
while offering a selection of messaging patterns
to choose from to best suit your architecture. An obvious choice when prioritizing delivery reliability
and fault tolerance
, due to its native acknowledgment and message persistence features.
Getting started with RabbitMQ is easier than ever especially now that Glasskube integrates the cluster controller for easy installation and management. This guide aims to offer the necessary insights for setting up a functional instance or RabbitMQ cluster within a Kubernetes environment. While there's so much more to learn regarding RabbitMQ such as this and that, I hope this primer provides ample groundwork to kickstart your RabbitMQ journey.
If you like this sort of content and would like to see more of it, please consider supporting us by giving us a Star on GitHub π