Event-driven microservices: what the diagrams leave out

costas 21 July 2026 0 architecture, microservices, event-driven Technology

Every architecture talk about event-driven microservices has the same slide: a handful of services, a message broker in the middle, and arrows flowing neatly between them. Services are decoupled, they scale independently, everyone is happy.

After several years of building and running systems like this, I still like the approach. But the diagram leaves out most of the work. Here are the parts that don't fit on the slide.

1. Messages will be delivered more than once

Most brokers give you at-least-once delivery. That means every consumer must be idempotent: processing the same event twice must have the same effect as processing it once. In practice that means giving every event a unique ID, recording what you've already handled, and designing writes so they can safely be repeated.

2. Order is not guaranteed (unless you pay for it)

"Order updated" can arrive before "order created". You can get ordering per key with partitioning, but only within a partition, and only if producers publish consistently. Consumers should be ready for events that arrive early, late, or after a newer version has already been processed.

3. "Save to the database and publish an event" is two operations

If the service crashes between the two, you either lose the event or publish one for data that was never saved. The transactional outbox pattern solves this: write the event to an outbox table in the same transaction as your data, and let a separate process publish it. It's one of those patterns you only appreciate after debugging the alternative.

4. Failed messages need a home

Some messages will fail no matter how many times you retry them. Without a dead-letter queue and a way to inspect and replay its contents, they either block the queue or silently disappear. Treat the dead-letter queue as a first-class part of the system, with alerts and a runbook.

5. Event schemas are public APIs

The moment another team consumes your event, its shape is a contract. Add fields rather than change them, version when you must, and never assume you know every consumer.

6. Debugging needs a thread to pull

When one user action fans out across five services, logs from each service are useless on their own. Propagate a correlation ID through every event and every log line, and invest in distributed tracing early. It turns "something is wrong somewhere" into "this event failed in this service at this step".

7. Eventual consistency is a product decision

If a user saves something and the screen they land on is fed by another service, they may not see their change yet. That's not a bug to fix in the backend; it's a behaviour to design for together with product and frontend.

Is it worth it?

For systems with many independent workflows and teams, yes. The decoupling is real, and so is the resilience. But it's worth going in with eyes open: the arrows on the diagram are where most of the engineering happens.

costas 21 July 2026 0 architecture, microservices, event-driven Technology

costas

papanicolaou