Whatever the industry, business workflows often rely on distributed software systems. That’s why it is crucial to establish the interconnectivity and interoperability between these systems. It’s now getting more common for application designs to be built on event-driven architecture. Why? Because this is the way to enable efficient, real-time communication between different components without them being aware of one another.
In this article, we’re going to talk about event-driven architecture (EDA) and its most commonly used messaging pattern: publish/subscribe (pub/sub). We’ll explain how things work within such systems, what differentiates them from the so-called traditional methods, and when they are a good fit.
Understanding event-driven architecture and pub/sub
To grasp the idea behind event-driven architecture and pub/sub messaging, it’s worth defining all the key terms first.
Event-driven architecture is a design model that connects distributed software systems and allows for efficient communication. EDA makes it possible to exchange information in real time or near real time. It is common in designing apps that rely on microservices (when every service runs its own process.)
The concept of event-driven architecture is mainly realized through the publish/subscribe communication model.
Publish/subscribe is a flexible messaging pattern that allows disparate system components to interact with one another asynchronously.
The key point here is that pub/sub enables computers to communicate and react to data updates as they occur. This is in contrast to the traditional request/response messaging pattern where data is updated at intervals, as a response to a user-initiated request. There are always two participants — a client and a server. The client makes a call over the HTTP protocol and waits for a server to respond with the requested content.
Simplified illustration of how traditional request/response messaging works
The request/response approach to connecting distributed software systems always requires some waiting time for a response to be provided. While it’s still frequently used in a variety of cases, pub/sub messaging is becoming quite a relevant solution too.
What are events and notifications?
In event-driven systems, components interact by exchanging notifications about the occurrence of events. What are these?
An event is a state change or an update within the system that triggers the action of other systems. It can be anything from a transaction and sensor input to a mouse click and a photo upload, etc. Events may vary in complexity and size and originate from both internal and external sources.
Events shouldn't be confused with notifications, though.
A notification is a message created by one component to inform about the occurrence of an event and describe it to other components. Notifications contain information about events and the context of their occurrences such as location or time.
Event-driven architecture and pub/sub components
Event-based architectures usually comprise three core components:
event producers that generate or detect events and transmit them to event managers;
event managers that act as middleware and are responsible for asynchronous filtering, processing, and routing of received events; and
event consumers that receive events and act on them.
Publish/subscribe systems follow the typical event-driven structure, yet with slightly different names of the components. From the pub/sub perspective, the producers of event notifications act as publishers and the consumers as corresponding subscribers, hence the name. But the idea of interaction between systems remains unchanged.
The middleware may contain multiple message brokers that transfer events and push them out to all interested consumers or subscribers (as it’s shown in the picture below.)
Key components of event-driven architectures
Subscribing services inform a broker about their interest in specific event notifications by creating a subscription based on filters that indicate the notification relevance. But it falls to the broker to decide whether an event matches a given filter.
Within the pub/sub system, there are a few subscription models, yet the most commonly used one is called topic-based, meaning event notifications have associated topics (or subjects) under which they are published.
As far as the roles of event publishers and subscribers, they aren't static or inflexible — components can change them or play both. Say, a consumed event may evoke an event inside a subscribing component, so the latter publishes its state change as a new event for other components to act on.
Within such architectures, services that produce events neither know nor care about services that consume events and vice versa. The latter just perform tasks required as a result of the produced event. Owing to this, EDA and pub/sub are perfect for building dynamic environments because it is easy to extend the whole ecosystem by flexibly adding new components without modifying the existing ones.
Key principles of the pub/sub paradigm of event-driven architecture
There are several principles, which can also be considered as advantages, underlying event-driven pub/sub architectures and differentiating them from other system communication patterns.
Scalability. EDAs allow for great horizontal scalability as one event may trigger responses from multiple systems with different needs and providing different results.
Loose coupling. Producers and consumers are unaware of each other. There is an intermediary that receives events, processes them, and sends them to the systems interested in specific events. This allows for loose coupling of services and facilitates their modifying, testing, and deployment. And unlike point-to-point system integrations, components can be easily added to or removed from a system.
Asynchronous eventing. Event notifications are broadcast asynchronously, meaning events are published as they happen. A service can consume or process a published event later if it’s unavailable. And it doesn’t affect or block a producing service.
Fault tolerance. Owing to systems being loosely coupled, the failure of one won't make any difference to the working of other systems.
All of the aforementioned allows businesses of different scale and sphere of interest to meet customer demand for real-time, personalized products and services.
Now that you know the basic concepts, we’ll try to explain how the pub/sub model works with an example.
How the pub/sub pattern of event-driven architecture works
So, to prove that publish/subscribe is an elegant, simple yet powerful system communication paradigm enabling event-driven applications and architectures, we'll draw a picture of its working principle. Let’s take a simple pizza ordering scenario, presented in the picture below. While there may be a greater number of microservices involved in the process, we’ll cover the three key ones. They include
a User Profile Service where a customer can place a pizza order;
a Food Delivery Service that schedules delivery time and assigns deliverymen to orders, and
a Restaurant Partner Service that takes a pizza order and notifies when it’s ready.
The working principle of the pub/sub system
So, here’s a rough picture of the event flow:
The user places an order for a pizza via the User Profile Service (a mobile app UI). The service captures such data as the user’s name, current location, contact info, etc., and publishes the pizza order event.
The Food Delivery Service (like Uber Eats) subscribes to the pizza order event so it reacts to it by publishing the take pizza order event.
The Restaurant Partner Service, which is subscribed to the take pizza order event, fulfills the order and publishes the pizza order ready event.
The Food Delivery Service sends the allocate nearest deliveryman and schedule delivery time events, respectively. It can now monitor the location of the order and provide ETA (Estimated Time of Arrival) notifications for the user.
All the microservices are linked by a system’s message broker that can be implemented with the help of platforms like Apache Kafka. The broker acts as an intermediate that delivers events from publishers to subscribers and vice versa.
Key tools to enable the pub/sub and event-driven architecture
The data exchange logic between event producers and consumers can be executed with the help of different event processing platforms. Your IT team needs to consider which software fits the bill when creating an event-powered solution. Below we’re covering the key players that may help you bring the event-driven architecture and pub/sub concepts to life.
Apache Kafka is the heavyweight in the world of distributed data streaming. The open-source platform allows for publishing, storing, processing, and getting subscribed to the streams of events in real time. Kafka provides its pub/sub functionality in a highly scalable, fault-tolerant, and secure fashion, allowing for handling trillions of events a day. Besides, it can be deployed on-premises and in the cloud.
Pulsar is another Apache family system for distributed publish/subscribe messaging. Initially designed as a message queuing system, Pulsar has added event streaming functionality in recent releases. Highly scalable, the system isolates producing and consuming operations and allows for flexible messaging models.
ActiveMQ is a flexible, open-source message broker presented by Apache Software Foundation. It provides the functionality of the pub/sub messaging through topics between multiple producers and consumers. ActiveMQ is relatively easy to deploy in complex structures; the system shows high throughput and great reliability.
Redis is an open-source, in-memory data structure store used as a message broker and a database. Redis is also a popular choice for pub/sub applications, enabling processing millions of requests.
Of course, there are a bunch of other tools: These four are just the tip of an iceberg. Moreover, creating a functional event-driven architecture requires complementary services, say, cloud data storages.
Building an event streaming and analytics solution from scratch carries a lot of advantages, especially for large companies. Smaller businesses can benefit from Infrastructure-as-a-Service (IaaS) or Platform-as-a-Service (PaaS) solutions that provide a wide array of resources to implement pub/sub such as pre-built data infrastructures, software development kits (SDKs), Kafka connectors, and many more. Here are a few options to consider:
Next, let’s focus on domains where EDA and pub/sub find successful application.
Common use cases of event-driven architecture and pub/sub
Since event-driven applications allow for making real-time business decisions by using the most up-to-date information, they are common for a wide variety of use cases. We’ll cover a few below.
Internet of Things. IoT solutions power different industries from transportation to healthcare. Events can be used to capture IoT device outputs such as sensor data streams, temperature readings, location points, and other state changes. EDA allows for processing huge volumes of data from IoT-based environments and devices the moment it is generated. As such manual or automated decisions can be taken to address issues and plan proactive measures.
Online shopping. Since there are a lot of services involved in the eCommerce processes, EDA is a great way to bring them all together. Events are broadcast at every stage from order placement to shipment, allowing for real-time visibility and immediate reaction to customer interactions and orders.
Healthcare. With a massive amount of data generated by IoMT devices, wearables, and other medical equipment in real time, it is necessary to act on new information in a timely fashion and be aware of critical moments faster. The event-driven approach to processing this data provides opportunities for remote patient monitoring, preventive maintenance of medical diagnostics equipment, and more proactive healthcare in general.
Online banking. With event-driven architecture, it is possible to monitor transactions in real time, detect fraudulent activity as it happens, and notify a cardholder of a suspected card fraud over a phone immediately
Event-driven architecture isn’t a magic bullet
Event-driven architectures are ideal when system agility, interoperability, and interconnectivity are of top priority. That's why they are commonly found in modern applications that use microservices or rely on disjointed components.
From this perspective, EDA and pub/sub is a good choice when:
you rely on multiple systems that need to operate in response to the same events;
you need to continuously monitor resource state and receive notifications about any changes and updates with minimal time lag;
you want to improve your existing systems and make them more scalable and responsive;
you deal with a high volume and high velocity of data, for instance, IoT data; or
your systems must share information without tight coupling, etc.
At the same time, EDA isn't a universal solution that fits all scenarios without exception. If your systems are simple and unlikely to scale up, there’s no need in overcomplicating things. Request/response models haven’t lost their relevance yet and still can be used in a variety of cases.
