The Secrets of Message-Queues in Practice
There is a lot of talk about microservice systems. Message queues play a major role in these, but are rarely discussed.
Dr. Matthias Boldt - Much is written and discussed about cloud software and cloud computing using microservices. Within the architectures required for these systems, message queues play a central role. Unfortunately, they rarely appear in the general discussion - wrongly, because there are “secrets” that can only be revealed in a practical deployment.
Modern software systems are just like cooking recipes: You can’t just stir all the ingredients together and hope that the result has a good taste. Let’s take a lemon dish, for example. To prepare it, you have to perform a little trick of whipped egg whites and gelatin in between. It’s the same with cloud applications, especially those that use microservices. Without whipped egg whites, of course, but with message queues.
What is the role of message queues?
In a system constructed according to the Microservices Architecture pattern, many different service modules are responsible for processing the data. These must be connected in the correct/desired order. Message queues can be used here. They not only connect the individual microservices, but also transmit the data between them.
Message queues are, of course, only one possible way of switching messages between the processing units of a cloud software system. However, they offer many advantages over, for example, the use of simple REST calls that services exchange with each other.
What are message queues used for?
Of course, the transport of messages and data between different services is the most important task of a message queue. However, other goals can also be implemented with this technology. It can support a “clean” architecture of the entire system. Below is a small selection of these goals:
abstraction from the general workflow
An often desired additional benefit is the separation of concerns. In combination with message queues and workflow control for the processing sequences within the cloud system, this separation can be implemented. In this way, the individual, processing services are not overloaded with knowledge about the existence of other services and the overall process. This also has another, positive effect: the processing services can be quickly and easily linked together in new workflows in other ways. This enables the services to be used universally.
asynchronous decoupling of services
The individual modules/services of a software system require different times for processing data. In many cases, it is undesirable for parts of a system to wait for the completion of a processing task they have commissioned and to be blocked during this time, i.e. unable to execute any further jobs. Message queues can be used to implement such an asynchronous “fire-and-forget” mode. If the job is stored as a message in the queue, all work is done for the ordering module and it is free for further tasks.
point-by-point scaling
Systems using microservices (in contrast to classic, monolithic systems) can scale very specifically in individual parts/functional groups/services. This means that additional services can be started and involved in the processing of data at precisely those points where there is an increased demand. Users thus experience no time delay in completing their work orders, even though the load on the system has increased. This load can be measured in many different ways. The use of a message queue provides an elegant solution that also distributes the work to the newly started services. By measuring the length of a queue, an extremely reliable signal about the load on the system is available at any time. It can be used excellently for up- and down-scaling of the coupled services.
Is any message queue system suitable?
A large number of message queue systems are available today for use in productive practice. So you have to choose. It is a good advised to collect a list of criteria, according to which you would like to narrow down the field of candidates. Some of these criteria are listed and explained below:
Technology/Framework
Is the basic technology used by the message queue (e.g. programming language, framework, …) already used by the development and operations team? Is knowledge of it available, or does it first need to be built up through training and education? The answers to these questions influence the overall costs and the ramp-up phase of the project. New is exciting and interesting, but often expensive to use.
Client-Library
A message queue must be connected to the existing parts of a system as a software component. This is normally done by using client libraries, which encapsulate the communication processes and make them available as handy functions. Such libraries must be found for the programming language used in the existing system. This should be well maintained and used by a larger community. It will prevent surprises in the future.
If there are no such libraries, an adapter for the communication protocol used by the message queue can of course be developed in-house. In most cases, this is not complicated, but it requires additional time and thus has a negative impact on the ramp-up phase of the project … not to mention the costs.
Message schemas
The processes implemented by message queue systems are quite different. There are many different schemes/patterns, each of which can be used for specific tasks. The “system of choice” must of course be able to implement the messaging patterns that should be used in your existing software system. So a detailed analysis is worthwhile here to avoid running into unsolvable problems during the implementation. In this case, you would have to start all over again and a large part of your previous work would be “scrap” … not a nice thought.
Size of the data blocks
The amount of data to be transmitted and the size of the data blocks should play a major role in the selection. Message queues are in many cases only intended as “letter carriers”. They often have problems with “packets”. If larger amounts of data are to be transmitted with the messages, this limits the selection. Unfortunately, practical experiments are also necessary in this case to test the effects on the speed of the switching.
Messages per second
Speed is not magic, not even with message queue systems. In the end, these are also just software and not magic. The speed of data transmission and the amount of data that can be processed are decisive criteria for a selection.
In the end, the selection of a message queue system is like in real life: Only a practical test can help to find the right solution.