They enable distributed development of highly scalable applications and unleashing the full potential of public cloud infrastructure.
But there can be no good without evil.
Microservices are DIFFICULT to understand and troubleshoot.
When done right, microservices are asynchronous, which means each service can operate without relying on other services, and the communication between services does not depend on any particular order of events, making it extremely challenging to understand and track.
Services can obviously communicate with one another directly but by doing so, you are basically creating a dependency that will affect the robustness of your system and how complicated it is to add functionality.
So when dealing with microservices, it’s not enough to understand the state of a particular service, you need also to understand the interactions between its different components.
To put it differently, without the proper setup, you lack the timeline of requests between services and the cause and effect of different events in your system.
You lack context.
The popular way to achieve asynchronous relations between services is by having services communicating by exchanging messages - aka asynchronous messaging or message brokers.
Async messaging plays a crucial role in distributed systems architecture, making it easier to build complex systems using microservices.
Communicating over a message broker minimizes the mutual awareness that services should have of each other in order to be able to exchange messages, effectively implementing decoupling which is the root idea of microservices.
But now, understanding the path of each message, where it goes, why, and when, is both critical, and here is the hard truth, a nightmare. (more on that later).
In short, when it comes to microservices and distributed systems, you need to see the full picture and the small details.
That's the only way you could gain that visibility into what happens (or doesn't) in your systems and how to deal with it.
But how do you get visibility?
OpenTelemetry to The Rescue 🔭
OpenTelemetry is a set of tools built to enable instrumentation, gathering and generating telemetry data. It is an open source and a member of the CNCF - the Cloud Native Computing Foundation.
It provides us with the missing piece of this microservices communication puzzle - context - by allowing us to gather data on each request that happens in our distributed system.
Think of it as a middleware, enabled via an sdk (in each microservice), that collects data about the calls the service is executing. Whether the call is to a messaging queue, database or another service.
Its power comes from its ability to maintain context propagation - we can store context on top of each request, thus enabling us to group related operations together, and create these graphs automatically.
The Fun Part - Gaining Visibility Using ExpressJS, Kafka, OpenTelemetry and Zipkin
Here’s how you actually implement OpenTelemetry in microservices to understand async messaging in your system.
There are many great async messaging tools out there that you can use for your services communication (AWS SQS, Kafka, RabbitMQ, just to name a few).
Due to its popularity and ease of use we have selected Kafka for this tutorial.
Step 1 - Create a docker-compose.yml file to run Kafka
You’ll have to install both kafkajs and express packages. In addition, you’ll need a kafkajs plugin for OpenTelemetry. This plugin was created at Aspecto as part of our ongoing contribution to OpenTelemtry.
The kafkajs plugin captures producer and consumer operations and creates spans according to the semantic conventions for Messaging Systems. You can read more about this plugin here or check it out at the github repo.