Event-driven microservices are a powerful tool, but they come with caveats that are worth considering before you fully commit to using them.
Based on a distributed log (probably Kafka), an event-driven microservice subscribes to several topics and publishes to several more. It processes data asynchronously and batches its I/O. A Kafka broker gives you service discovery, load balancing, horizontal scaling, fault tolerance, guaranteed delivery, and best-in-class throughput; it's a pretty good deal.
All these benefits come bundled together, however, which makes event streaming somewhat contagious, in the way that async functions are contagious. In order for an event-driven service to processes messages at speed, it can't perform a separate set of network round-trips for each one. It shouldn't be calling APIs or querying a database; rather, it should be ingesting streamed versions of those things. Any synchronous operations will block the pipeline and throw the service's performance off a cliff. The service's output also has to be asynchronous, which propagates all these constraints to downstream services, too.
And you may not want to fully commit to streams. For every problem stream processing solves, it creates a new one. A synchronous API will respond to clients with errors if it gets too busy, but an asynchronous pipeline will gladly fill its disks with more work than it can chew, unless you implement a robust backpressure system. And Kafka forces you to handle messages in order, so if you encounter a bad message, you have to handle it immediately. In order to retry at a later time, you have to forward that message to a "retry" topic which circles back to an earlier stage in your pipeline. Now the same message can end up in multiple different topics: the normal topic and the retry topic.
This is my biggest problem with event streams, actually: too many representations of the same entity in different places. For a conventional database-backed service, you can check the current state of an entity by querying the relevant database table. Meanwhile, in order to completely understand the status of an entity in a ten-step streaming pipeline, you have to look through nine separate message logs. Most of the messages in those logs will be stale, too, so make sure to take processing time into account.
To be clear, I like Kafka. My point is that legibility is as much of a consideration when designing a system as performance is. Asynchronous systems perform better, but synchronous systems are often easier to reason about, because they can maintain a single source of truth for each entity in their domain. You can balance these properties and get the benefits of both, but that requires deliberate boundaries between the synchronous and asynchronous parts of your system.
Patterns for fitting synchronous and asynchronous systems together:
-
Workflow engine + worker nodes. A scheduler with a database can serve as a synchronous source of truth while dispatching, monitoring, and retrying asynchronous tasks across a cluster of workers. This centralizes application state while remaining highly scalable.
Also, a dynamic scheduler will easily be able to describe workflow topologies and error-handling flows that would be too complex to represent with static Kafka topics.
-
If you do want a conventional streaming pipeline, a synchronous control plane can be a great enhancement. The control plane can handle observability and configuration concerns, trigger replays, and even answer debugging queries about the internal states of stream operators.