Module 5: Asynchronous Messaging and Event-Driven Architecture

Learning Path

Monday, September 28, 2026, 6:00–8:30 PM. This 34-slide session follows messaging middleware from enterprise integration to Kafka and modern streaming runtimes. Documented company cases are dated and sourced; the fictional campus marketplace anchors the two live demos. All IDs and orders are synthetic. These exercises do not send notifications or charge money.

Required Preparation

Review Module 4’s commands, domain events, and duplicate-event example. Read the RabbitMQ acknowledgement/prefetch sections, Kafka main concepts, LinkedIn’s 2011 announcement, and the short architecture discussion in Iggy’s February 2026 post linked below. Predict what happens if a consumer performs its effect but crashes before recording progress.

For hands-on participation, install and start Docker with Compose v2 and download the classroom kit before class. The kit runs clients inside containers; no host Python or Java setup is required. Students without Docker can pair with a teammate or use the paper activity. The instructor should complete the kit’s rehearsal checklist before Monday; image downloads need internet access.

Message-Oriented Middleware and Its Evolution

Message-oriented middleware (MOM) sits between applications to manage message exchange. A sender addresses a destination and the middleware handles delivery according to configured rules. Store-and-forward can let a receiver process work after an outage; persistence, routing, access control, and acknowledgements make that exchange manageable. The business still needs an owner for unresolved work.

Teaching example: a branch submits a transfer instruction while a back-office application is unavailable. A queue preserves pending work, but acceptance into that queue does not mean funds moved. Account invariants, duplicate prevention, and reconciliation remain application responsibilities. This is an illustrative design, not a description of a particular bank.

These are overlapping pressures, not a sequence in which each generation eliminates the previous one. IBM’s MQ overview and history grounds the early enterprise-integration milestone. Later sections use original Kafka and Iggy sources for their histories.

Keep the Vocabulary Separate

Using a broker does not automatically create a well-designed EDA. A chain of services can remain tightly coupled through required event sequences and shared schemas.

Event-Driven Architecture Beyond the Broker

flowchart LR
 O[Ordering owns confirmation] -->|OrderConfirmed| B[Event distribution]
 B --> P[Pickup subscription]
 B --> A[Analytics subscription]
 P --> W[Prepare pickup]
 A --> R[Update report]

A domain event is a statement by its owner about a business transition. Consumers own their reactions. New subscriptions can add behavior without changing the producer’s list of callers, but event meaning, compatibility, and permissions still need coordination. Users may see intermediate states while consumers catch up. See the event-driven architecture guide.

Choreography lets participants react and publish subsequent facts. Orchestration assigns workflow progress to a coordinator that issues commands and records outcomes. Both can use messaging. In the marketplace, analytics can react independently to OrderConfirmed; a checkout coordinator may be preferable for tracking stock reservation, payment uncertainty, and compensation. Ask who detects that a workflow has stopped progressing and who owns its recovery.

Event-driven design is also distinct from event sourcing. Publishing an event after a database update does not make the event log the authoritative record from which all state is reconstructed. That is a separate persistence decision with its own evolution and replay obligations.

A Second Topology: Independent Subscribers

flowchart LR
 P[Event producer] --> X[Fanout exchange]
 X --> Q1[Pickup queue]
 X --> Q2[Analytics queue]
 Q1 --> W1[Pickup worker]
 Q2 --> W2[Analytics worker]

Here pickup and analytics own different queues. With a fanout exchange, each bound queue receives the publication. Adding another worker to pickup shares pickup work; it does not create a new subscriber history. The queue/binding lifecycle matters: a subscriber that did not have a bound queue at publication time cannot assume it can fetch earlier events. The RabbitMQ pub/sub tutorial demonstrates this topology.

Use Cases: Who Coordinates the Work?

Choreography: a marketplace publishes OrderConfirmed. Pickup prepares a package while analytics updates a sales view. Neither waits for the other. Each owns retries and exposes its own status. This fits independent reactions to one fact.

Orchestration: a fictional travel-package coordinator requests a flight hold, then a hotel reservation, and records the result of each step. If the hotel is unavailable, it releases the flight hold. If a provider times out, it queries the uncertain result before compensating. This fits a multi-step goal that needs a visible owner for deadlines and recovery. The coordinator can use RPC and messaging together; it is not the same component as the broker.

Expedia/Kayak-Style Travel: Why RPC and Messaging Both Matter

Think of the user experience of comparing flights on Expedia or KAYAK. The following is an illustrative architecture, not a claim about either company’s internal implementation. Metasearch may redirect a traveler to a supplier that owns the actual booking.

Without precise contracts, a service may treat “request received” as “ticket issued,” retry a timed-out reservation as a new purchase, or confuse an old search price with a committed fare. A correlation ID joins the entire trip attempt across RPC and events. An event ID identifies one fact; a stable operation key identifies one requested booking. Schema versions preserve meaning when producers change fields.

Class prompt: the provider reserved the flight, but the RPC response disappeared. Should the UI say failed, confirmed, or checking? Expected: checking/unknown until lookup or reconciliation resolves the outcome. Publish BookingConfirmed only when the authoritative result is known. Messaging makes delayed completion observable; it does not make an unknown RPC result certain.

Commands, Events, and Meaning

PreparePickup is a request to perform work. OrderConfirmed records a completed business transition. A consumer must not infer that every event means payment succeeded: the producer’s contract must define that meaning. Keep the message small and exclude customer credentials or student information.

{
  "id": "e1",
  "type": "OrderConfirmed",
  "schemaVersion": 1,
  "orderId": "o42"
}

id stays stable when retrying the same logical event; orderId connects different events for the same order. A schema version supports explicit compatibility rules. Adding an optional field may be compatible; changing a field’s meaning silently is not. Real contracts also need time semantics, ownership, validation, and trace context.

Asynchronous processing allows a producer to finish before a worker completes its task. It introduces pending states: “order confirmed” and “pickup ready” are different facts. Broker downtime, finite storage, and backlog growth still require a policy.

RabbitMQ: Share Pending Work

flowchart LR
  P[Job producer] --> E[Default exchange]
  E --> Q[Durable pickup queue]
  Q --> A[Worker A]
  Q --> B[Worker B]
  A -->|ACK after effect| Q
  B -->|ACK after effect| Q

In this example, a producer routes PreparePickup messages through the default exchange to the durable pickup-jobs queue. Two workers compete for deliveries. Each has prefetch 1, so it holds at most one unacknowledged delivery. Distribution depends on readiness and processing time; do not assume an exact alternating sequence.

The producer uses persistent messages and publisher confirms. A confirm concerns broker acceptance; a consumer acknowledgement concerns a delivery handled by the worker. Acknowledgement after processing permits recovery of unacknowledged work when a connection closes, but also permits repeated effects. See RabbitMQ’s confirmation guide.

Sharing Work Versus Publish/Subscribe

If both notifications and analytics must receive a fact, two workers on the same queue are the wrong arrangement. A fanout exchange can route to two separately bound queues, with one queue per logical subscriber. Workers within each subscriber’s queue can then share its work. Exchange routing and queue storage are separate responsibilities. See the RabbitMQ publish/subscribe tutorial.

RabbitMQ also offers streams; this module compares its conventional work queues with Kafka’s retained log. Product names alone do not determine the architecture.

Why Kafka Was Born at LinkedIn

LinkedIn’s January 11, 2011 open-source announcement describes activity data such as page views, searches, and ad impressions. These facts mattered to product relevance and engagement analysis. The challenge was to collect a large continuous stream and make it useful to downstream readers with different processing schedules.

The 2011 Kafka paper (PDF), by Jay Kreps, Neha Narkhede, and Jun Rao, framed the problem around log processing. Its design joined durable collection with online consumption: readers tracked their own positions in a partitioned log. Historical Kafka was not identical to today’s Kafka; use the paper to understand motivation rather than copy its early guarantees or configuration.

From Activity Pipeline to Shared Infrastructure

In October 2019, LinkedIn reported seven trillion messages per day and described its Kafka customizations and release branches. This is a dated report about a particular deployment, not a current global benchmark. The instructional lesson is how one reusable stream can serve many teams, making quotas, compatibility, upgrades, and operational ownership central concerns.

Worked design, illustrative: a marketplace publishes order facts once. An operational consumer updates a pickup view, a streaming job computes rolling demand, and a warehouse connector ingests history. A corrected report can read retained data again. Each consumer has independent progress; a shared event definition prevents each team from inventing incompatible meanings for “confirmed.”

Why Kafka Gained Ground Against Conventional Queue Designs

Kafka matched workloads where many applications needed the same history. Retention separates record lifetime from one consumer’s completion. Groups separate application progress. Partitioning enables parallel reads and writes. Append-oriented storage, batching, compression, and use of the OS page cache improve movement of large volumes of records. These mechanisms are described in the Kafka design documentation.

The benefit has a price. Larger batches may add latency. More partitions create more metadata and recovery work. A heavily used key can concentrate traffic. Replicated retained history costs storage and bandwidth. Teams still need contract ownership, backpressure, and incident response. Kafka’s adoption is understandable from these workload advantages without claiming every queue system was slower or became obsolete.

Kafka: Retain Facts for Independent Readers

flowchart LR
  P[Order producer] --> T[Orders topic - partition 0]
  T --> G[Pickup group - own offset]
  T --> A[Analytics group - own offset]
  T --> R[New replay group - earliest retained offset]

The orders topic starts with one partition. It receives three keyed records. Pickup and analytics use different group IDs, so each application can read the history at its own pace. A second member in the same group shares partition ownership rather than creating another independent subscription. With one partition, only one member of that group reads it in steady state.

An offset is a position within a partition. After processing offsets 0, 1, and 2, the next position is 3. A committed group offset records a restart position; it does not prove an external side effect happened. Rebalances can move partition ownership when group membership changes.

Retention controls how long records remain available. A fresh group starting at the earliest available offset can replay retained history. --from-beginning does not override an existing valid committed group offset. Different groups are independent readers; reading a record does not delete it. See the Kafka introduction.

With a stable key partitioner and partition count, an order’s key routes its records to the same partition. The log orders records within that partition, not across the topic. Concurrent producers still need a business ordering policy, and changing partition count may remap future keys. The optional two-partition exercise explores parallelism without promising even traffic distribution.

Real-World Enterprise Cases

These are documented historical examples, not claims about the companies’ complete current architectures. Use the source dates when discussing them.

Uber: Many Consumers of Operational Data

Uber’s August 2016 uReplicator report describes collecting rider/driver events and logs into Kafka for real-time and batch pipelines. Metrics, debugging, alerts, and dashboards share upstream data. Its replication tooling addressed reliability and operational control across Kafka clusters.

Discussion: copying a record to another cluster and applying a business effect are separate steps. Which progress marker would prove each one? The lesson is that a messaging backbone requires operational tooling, not just producer and consumer code. Do not infer from this report that every ride-dispatch or payment operation follows the same path.

Netflix: Isolate Ingestion from Consumer Load

Netflix’s April 2016 Keystone report describes fronting Kafka clusters for collection and buffering, with routed topics in consumer clusters for downstream processing. Separating those duties protected ingestion from unpredictable readers. The report also makes an explicit tradeoff among acceptable data loss, application availability, and cost.

Discussion: would that loss policy fit a payment ledger? Explain why the business invariant matters more than the broker name. Treat the topology as a historical case; its old configuration flags are not prescriptions for modern Kafka. Kafka was infrastructure for this event-data pipeline, not the transport for streaming movie bytes to viewers.

What an Enterprise Backbone Actually Needs

The case studies illustrate reusable data distribution. An enterprise platform additionally needs clear event ownership, compatible schema evolution, access policy, retention limits, consumer onboarding, quotas, lineage, and recovery procedures. A centralized broker without these practices can simply centralize failure and confusion.

Illustrative retailer design: fulfillment commands enter a work queue; completed-sale facts enter a retained stream for inventory views, analytics, and report rebuilding. The two systems serve different purposes. Before adding both, weigh whether the team can operate two platforms and whether one existing product already supplies the needed models.

This comparison concerns the specific models taught, not every feature of every product. RabbitMQ offers streams, and Kafka’s capabilities evolve. Assess the selected version. Neither “Kafka took over all MQ” nor “Iggy replaces Kafka” is a sound architectural conclusion by itself.

Apache Iggy and the Next Efficiency Questions

Iggy is a Rust-based persistent streaming project. Its February 2026 runtime article explains a move toward thread-per-core resource ownership and Linux io_uring. The goal is to reduce contention and improve locality while submitting asynchronous I/O through completion queues. Rust avoids a tracing garbage collector; it does not eliminate storage latency, network faults, or the need for replication protocols.

Small systems example: instead of many workers repeatedly locking one shared partition structure, assign partition work to an owning core and pass requests to it. This can reduce contention, but a hot partition can still overload its owner. Efficient batching and careful partition placement remain important. A faster single-node log does not settle distributed recovery correctness.

The project’s August 24, 2026 announcement dates Apache Top-Level Project graduation to August 19. That is a governance milestone. The same announcement lists clustering release work and a Kafka protocol gateway among next steps. Those statements are dated: verify the exact release’s implemented and tested capabilities before choosing it. Do not assume an existing Kafka client or connector will work unchanged.

Classroom Evaluation: What Would Convince You?

Compare equal workloads and guarantees before judging a benchmark. Hold message size, batching, compression, storage medium, retention, replication, and acknowledgement policy constant. Measure throughput alongside p95/p99 latency, CPU/memory use, backlog recovery, and behavior during a node failure. Record the version and hardware so another team can reproduce the comparison.

Then assess SDK/connector coverage, security controls, observability, rolling upgrades, support, and migration cost. A promising runtime design warrants experiments; it is not evidence that Iggy has already displaced Kafka across enterprises. This module makes no new Iggy performance measurements and requires no Iggy installation for Monday.

Can a Crash Lose Work or Repeat It?

Consider these failure windows for both brokers:

At-most-once processing can omit work; at-least-once processing can repeat work. Claims of exactly-once effects need an explicit transactional boundary. Kafka transactions can coordinate Kafka records and offsets in a supported processing workflow; they cannot automatically include an external email or payment API. The console demo illustrates consumption and progress, not exactly-once business processing.

For a local database effect, use one transaction:

BEGIN
  insert message ID into processed_messages with a unique constraint
  if the ID was newly inserted:
    update the business record
COMMIT
acknowledge the delivery / commit the next offset

On a duplicate, skip the effect after confirming the earlier transaction committed. Concurrent duplicates need database-enforced uniqueness. Keep processed IDs for the supported redelivery/replay window. External providers require a stable idempotency key or reconciliation; a local ID record cannot make a remote action part of the database transaction.

The producer has another gap: updating an order and publishing its event separately can leave one without the other. An outbox saves state and a pending event together; a relay publishes later and may publish more than once. That is why outbox publication still needs safe repeat handling downstream.

Worked Example: The Same Booking Approval Arrives Twice

Approval e17 authorizes proposal p42 version 1. For a local itinerary update, a unique e17 record and the update commit in one transaction. A repeated e17 then skips the update. Acknowledge only after that commit. Writing the ID and the update separately leaves a crash window.

For a remote booking, the provider might create booking b9 and then lose the response. Persist operation key p42:v1 before calling it, reuse that key on retries, and query the provider after uncertainty. A new event ID or a new operation key on each retry can create duplicates. A Kafka commit or RabbitMQ ACK cannot atomically include the external provider.

Classroom Examples

Standalone from scratch: no earlier lab, project repository, host Python, or host Java installation is required. Install/start Docker with Compose v2, download the ZIP, unzip it, and open terminals in the extracted module-05 folder. The kit includes all files and exact commands; the 10-minute demo assumes images have already downloaded and brokers are ready.

Download and unzip the Module 5 classroom kit (ZIP: Compose configuration, Python worker, and walkthrough). Read the step-by-step walkthrough (Markdown) alongside the demonstration. It includes every terminal command, expected observations, cleanup, troubleshooting, and a rehearsal checklist.

1. RabbitMQ, shared workers: start A and B, publish six pickup jobs, and count distinct acknowledged IDs. Stop workers, publish a backlog, then restart one.
2. RabbitMQ, failed worker: stop normal workers, run the crash worker, and publish one job. Recovery must show the same ID with redelivered=True. Explain the repeated simulated effect.
3. Kafka, independent readers and replay: publish three events, let pickup and analytics each read them, and start a fresh audit group to read retained history.
4. Kafka, same group: run two pickup members with one partition. Publish another event after assignment and identify the active member. Stop it and observe takeover.

The RabbitMQ application uses a short Python client so the acknowledgement boundary is visible. Kafka uses its bundled CLI so the first encounter focuses on topic, group, partition, and offset rather than client-library setup. Both brokers are single-node teaching systems; the examples do not test replication, durable host storage, authentication design, or production failover.

What to Record

For the 10-minute RabbitMQ demo, start both workers before the teaching segment; use the backlog step if time permits. The 12-minute Kafka replay and 8-minute group exercises also assume prestarted brokers and prepared terminal tabs. Full setup and the optional two-partition exercise belong before or after class.

For each run, write down the topology, message/event ID, receiving worker or group, and the observed progress signal. Compare observed output with the prediction. A repeat ID matters more than two lines with the same order ID, since an order can legitimately have several events.

The kit’s output examples are predictions, not captured execution evidence. Before classroom use, run its rehearsal checklist on a Docker-enabled machine.

Paper Activity and Team Discussion

Classroom discussion, 4 minutes; no software needed. Choose either the queue or log exercise below in pairs; complete the other after class. In pairs, draw one queue with three job cards. Give each of two workers one card, leaving it marked unacknowledged. Let worker A record its effect and fail before acknowledgement. Return A’s card to pending work. Have B handle it and identify the repeated effect.

Next draw a Kafka partition containing e1, e2, e3. Give pickup and analytics separate bookmarks. Advance pickup to position 3 while analytics stays at 0. Let analytics read all three, then add a fresh audit bookmark at 0. Erase e1 to model retention and explain why a new reader cannot recover it. Add a second pickup worker and decide who owns the single partition.

Discuss: A report contains a bug and must be rebuilt from last week’s events. What history is available, what retention is required, and which effects must be disabled during replay? A queue draining successfully does not by itself provide a replayable archive.

Expected reasoning: queue workers share pending work; independent groups keep separate log positions. Redelivery and replay can both repeat effects for different reasons. Rebuilding a derived report can be appropriate; resending every pickup notification usually is not.

Operating the Workflow

Queue depth alone hides how long a user has waited. Track oldest pending age, ready/unacknowledged deliveries, worker failures, and processing latency. For Kafka, track lag per group/partition and time since successful processing. Zero lag alone does not establish business correctness.

If arrivals are 12 jobs/second and workers complete 8/second, backlog grows by 4/second: 240 additional jobs in one minute. Add capacity where useful, limit ingress, or shed optional work. More retries can increase overload. A dead-letter path isolates messages that repeatedly fail; bound attempts, preserve diagnostic context, and assign an owner for investigation and replay.

Use synthetic data here. Production design additionally needs least-privilege access, encrypted connections, schema validation, storage/retention budgets, and appropriate replication. Neither a durable queue on one node nor replication factor 1 survives every broker or disk failure.

Optional Project Extension

Choose one asynchronous workflow in your project. State the command/event contract and one invariant. Draw the publisher’s state/publication boundary and the consumer’s effect/progress boundary. Propose an outbox or deduplication strategy where needed, plus one failure experiment and a measurable recovery condition. Official assessed requirements, deadlines, grades, and submissions remain in Canvas.

Connecting the Dots: Event Streams for Agentic AI

An agentic travel assistant observes a cancellation, retrieves alternatives through tools, proposes a recovery plan, and reacts to the approved outcome. These are familiar distributed-systems interactions with an additional decision-making component. Kafka can serve as the event transport and retained history connecting data sources, workers, and audit readers. RPC remains useful for immediate searches, model inference, and tool calls. Durable workflow state determines what is pending or complete.

An event log is not a reasoning engine, an authorization decision, or a replacement for a database. Agents still face stale context, retries, duplicated input, unknown external outcomes, and schema changes. Replaying events for analysis must not silently repeat real-world actions.

IBM, Confluent, and Enterprise AI Infrastructure

IBM announced completion of its Confluent acquisition on March 17, 2026, linking event streaming and governed real-time data to enterprise applications and AI. IBM acquired Confluent; Apache Kafka remains an Apache open-source project.

Confluent’s Streaming Agents documentation describes agents running on Flink within event streams. This is one commercial implementation of the broader pattern. It supports the argument that streaming infrastructure can provide current context and coordinate agent activity; it does not establish that Kafka is required for every AI system or that delivery guarantees make model decisions correct.

flowchart LR
 D[Trip disrupted] --> K1[Kafka disruption topic]
 K1 --> A[Planner agent]
 A -->|RPC| S[Mock search API]
 A --> P[Proposal topic and stored plan]
 P --> H[Human approval]
 H --> K2[Kafka approval topic]
 K2 --> E[Executor]
 E -->|RPC with operation key| B[Mock booking API]
 E --> O[Kafka outcome topic]
 O --> U[Status and audit]

End-to-end trace: trip t42 is disrupted → planner queries mock alternatives by RPC → proposal p42:v1 is stored and published → a human approves that exact version → executor calls the mock provider using operation key p42:v1 → a confirmed or unresolved outcome updates the traveler view and audit timeline. Each arrow needs an owner, a contract, and a recovery rule.

Agentic Assignment: Travel Recovery Assistant

Read the complete assignment brief (Markdown: architecture, contracts, implementation steps, and acceptance experiments).

Build from a new project; no previous course implementation is required. Use the classroom broker kit as infrastructure and implement your own Trip API, planner, approval API, executor, mock provider, and independent audit consumer. The supplied kit is not a completed agent application. A deterministic planner is the required baseline and must be labeled a mock agent; adding an LLM is optional. No paid AI service or real travel account is required.

Goal: recover one synthetic cancelled trip with a valid alternative. Invariant: create at most one mock booking per approved proposal version and none without approval of that exact version. Revalidate budget and arrival constraints before execution. A changed proposal requires new approval.

Implement disruption → proposal → approval → booking → outcome. Test duplicate approval, provider success followed by response loss, executor restart, stale approval, and search failure. Rebuild the audit timeline from a new consumer group without increasing booking count. Provide code, reproducible startup commands, a sequence diagram, event contracts, and expected/observed evidence. Canvas remains the official source for deadlines, grading, submissions, and any assessed requirements.

The progression is the lesson: MOM manages asynchronous exchange; Kafka adds reusable event history; agents become new consumers and producers of that history. RPC tools, explicit approval, durable state, and safe repeat handling turn that flow into a recoverable application.