Message-oriented middleware
Infrastructure mediating message exchange, routing, storage, and delivery between applications.
CMPE273 · Module 5
Asynchronous Messaging and Event-Driven Architecture
Trace message-oriented middleware and event-driven architecture from enterprise queues to Kafka at LinkedIn and Apache Iggy, with real-world cases and small RabbitMQ/Kafka demos.
Learning Objectives
- Explain the evolution from enterprise messaging to retained event streams.
- Distinguish commands, events, competing consumers, and independent subscribers.
- Trace RabbitMQ publisher confirms, consumer acknowledgements, and redelivery.
- Explain Kafka partitions, offsets, consumer groups, retention, and replay.
- Locate the failure window between a business effect and progress recording.
- Choose a messaging design and define recovery and observability requirements.
- Use documented enterprise cases and implementation evidence to evaluate Kafka, queues, and Iggy.
- Combine synchronous tools, event streams, and human approval in a recoverable agentic workflow.
Core Concepts
Event-driven architecture
Services publish facts and react through explicit event contracts and subscriptions.
Queue
Pending deliveries shared by competing workers in the RabbitMQ example.
Exchange and binding
RabbitMQ routing rules connecting publication to one or more queues.
Topic and partition
Kafka records organized into ordered logs within a named topic.
Consumer group
Kafka consumers sharing partition ownership and committed progress for one application.
Idempotent effect
Repeating a logical operation leaves the same intended business result.
Diagrams
Independent reactions to OrderConfirmed
flowchart LR O[Ordering owns confirmation] -->|OrderConfirmed| B[Event distribution] B --> P[Pickup subscription] B --> A[Analytics subscription] P --> W[Prepare pickup] A --> R[Update report]
Ordering publishes a completed business transition. Pickup and analytics have independent subscriptions and reactions. Their completion times differ; the customer should see the real workflow state.
RabbitMQ pickup workers share one queue
flowchart LR P[Job producer] --> E[Default exchange] E --> Q[Durable pickup queue] Q --> A[Worker A] Q --> B[Worker B] A -->|ACK after effect| Q B -->|ACK after effect| Q
The producer routes jobs to one queue. A and B compete for its deliveries. Each worker acknowledges after its simulated effect; loss before acknowledgement permits redelivery.
Each subscriber owns a queue
flowchart LR P[Event producer] --> X[Fanout exchange] X --> Q1[Pickup queue] X --> Q2[Analytics queue] Q1 --> W1[Pickup worker] Q2 --> W2[Analytics worker]
The exchange routes a publication to two bound queues. Pickup and analytics each receive a logical copy; multiple workers attached to either queue share that subscriber’s work.
Independent groups read retained order events
flowchart LR P[Order producer] --> T[Orders topic - partition 0] T --> G[Pickup group - own offset] T --> A[Analytics group - own offset] T --> R[New replay group - earliest retained offset]
Each group reads the same retained partition with independent progress. A new group can start at the earliest available record. Reads do not delete records, but retention can remove old history.
Event-driven travel recovery with approved actions
flowchart LR D[Trip disrupted] --> K1[Kafka disruption topic] K1 --> A[Planner agent] A -->|RPC| S[Mock search API] A --> P[Proposal topic and stored plan] P --> H[Human approval] H --> K2[Kafka approval topic] K2 --> E[Executor] E -->|RPC with operation key| B[Mock booking API] E --> O[Kafka outcome topic] O --> U[Status and audit]
A disruption triggers planning. The planner calls a mock search API and publishes a proposal. A human approves its exact version. The executor calls the mock booking API with a stable operation key and publishes an outcome for status and audit. Durable application state supports recovery at every step.
Examples, Discussions, and Project Ideas
Slide 2: Message-oriented middleware: why it exists
MOM lets applications exchange messages through managed infrastructure.
- Store-and-forward bridges applications that are not available at the same time.
- Routing, persistence, acknowledgements, and access control move into shared infrastructure.
- A broker reduces temporal coupling; schemas and business meaning still couple applications.
Slide 3: The evolution follows changing workloads
Several generations coexist because they solve different problems.
- 1990s: enterprise queues connect heterogeneous systems; MQSeries arrives in 1993.
- 2000s: APIs, protocols, pub/sub, and integration tooling make messaging more accessible.
- 2010s–2020s: retained logs, stream processing, cloud services, and efficient runtimes expand the design space.
Slide 4: Architecture, API, protocol, and product
These names answer different questions.
- EDA describes how services react to events; MOM provides message transport and mediation.
- JMS/Jakarta Messaging defines a Java API; AMQP defines a wire protocol.
- IBM MQ, RabbitMQ, Kafka, and Iggy implement different messaging models and tradeoffs.
Slide 5: Event-driven architecture: publish a business fact
A producer owns the fact; each subscriber owns its reaction.
Slide 6: Who coordinates the work? Two use cases
Use choreography for independent reactions; orchestration for a tracked multi-step goal.
- Choreography: OrderConfirmed triggers pickup preparation and analytics independently.
- Orchestration: a travel coordinator holds a flight, reserves a hotel, and tracks both results.
- If the hotel fails, the coordinator releases the flight hold and records recovery.
Slide 7: Travel search needs RPC; booking updates need events
Think of an Expedia/Kayak-style experience: search now, track changes over time.
- RPC: query flight/hotel providers for current options, with deadlines and partial results.
- Command: ReserveFlight requests an action; a timeout does not prove it failed.
- Event: BookingConfirmed reports a fact that alerts, itinerary views, and analytics can consume.
Slide 8: Two workers share a queue
Competing consumers divide work from one RabbitMQ queue.
- One delivery goes to one worker; failures can lead to redelivery.
- Two independent subscribers need separate queues bound to an exchange.
Slide 9: Pub/sub requires independent subscriptions
Two workers and two subscribers are different topologies.
Slide 10: Predict the failure window
A worker prints the pickup result and then loses its connection before ACK.
- Will the queue deliver the job again?
- Which identifier lets us recognize the repeat?
- What changes if ACK happens before the work?
Slide 11: RabbitMQ demo: six pickup jobs
Standalone lab: no earlier module, repository, Python, or Java installation required.
- Download and unzip the classroom kit; open three terminals in its module-05 folder.
- Start Docker with Compose v2; run the README setup and wait for broker readiness.
- Start workers A and B, publish six jobs, then inspect IDs and queue counts.
Slide 12: RabbitMQ demo: crash before ACK
Run example B with every normal worker stopped.
- Start CRASH, send one job, and observe the intentional failure.
- Start RECOVERY; compare IDs and the redelivered flag.
- The simulated effect repeats because no deduplication exists.
Slide 13: One booking even when a message arrives twice
Approval e17 for proposal p42:v1 is delivered again after a worker restart.
- Local example: save e17 and update the itinerary in one database transaction; then acknowledge.
- On redelivery, the stored e17 prevents a second itinerary update.
- Remote example: retry the booking with the same key p42:v1; query an uncertain result.
Slide 14: Break
Resume with Kafka’s origins and the retained log.
- Leave brokers running; stop RabbitMQ workers when their queues drain.
Slide 15: Kafka was born from LinkedIn activity data
LinkedIn open-sourced Kafka on January 11, 2011.
- Page views, searches, and ad activity generated a continuous stream of facts.
- The same data needed to reach product analytics and other downstream systems.
- Kafka connected high-volume collection with consumers that ran at different speeds.
Slide 16: The original Kafka design question
Can one system support both log collection and online consumption?
- The 2011 paper is by Jay Kreps, Neha Narkhede, and Jun Rao at LinkedIn.
- A persistent log lets readers track positions rather than delete records after reading.
- Partitioned storage and batched transfer target high-volume log processing.
Slide 17: From one pipeline to an enterprise backbone
LinkedIn reported seven trillion messages per day in October 2019.
- Treat that number as a dated engineering report, not a current benchmark.
- Many producer and consumer teams turn the shared log into integration infrastructure.
- At that scale, quotas, upgrades, ownership, and operations become product requirements.
Slide 18: Why Kafka gained ground in data pipelines
Retained history made repeated use of the same data practical.
- Independent groups support live applications, batch ingestion, and rebuilding derived views.
- Append-oriented I/O, page cache, batching, and compression improve data movement efficiency.
- Partitions create parallelism; retention and replication add storage and operational costs.
Slide 19: Kafka keeps a partitioned history
Each consumer group tracks its own position in the retained log.
- A record has a key, value, partition, and offset.
- Consumption does not remove records; retention limits replay.
- Ordering is within a partition.
Slide 20: Kafka demo: publish and replay
Run example C using one topic partition and three synthetic events.
- Pickup reads live events; analytics starts later and reads history.
- Inspect committed offset 3 after offsets 0, 1, and 2.
- Use a fresh audit group to replay; compare with restarting pickup.
Slide 21: Kafka demo: group membership
Run example D with two members of pickup.
- One partition means one active member in that group in steady state.
- Produce e4 after the rebalance; identify the receiving member.
- Stop that member and observe takeover; repeat reads are possible.
Slide 22: Uber: a message bus for real-time and batch data
Uber’s 2016 engineering report describes Kafka feeding multiple pipelines.
- Rider/driver events and application logs feed downstream consumers.
- Real-time users include metrics, debugging, alerts, and dashboards.
- uReplicator addressed Kafka replication reliability and operational control.
Slide 23: Netflix: buffering and isolation in Keystone
The April 2016 report separates fronting and consumer Kafka clusters.
- Fronting clusters collect events; downstream routing feeds consumer clusters.
- The separation protects ingestion from unpredictable consumer workloads.
- The report explicitly balanced data loss, availability, and cost for that pipeline.
Slide 24: Kafka and traditional MQ coexist
Choose the delivery model before choosing the product.
- Work dispatch: per-message completion, routing, and retry handling often favor a queue.
- Shared event history: independent readers and replay often favor a retained log.
- A retailer can use both: fulfillment commands on queues, sales facts in streams.
Slide 25: Can a crash lose work or repeat it?
The order of “do the work” and “record completion” changes recovery.
- Record completion first, then crash: the booking may never happen.
- Book first, then crash before recording it: a retry may book twice.
- Use durable state and the provider’s stable operation key; Kafka alone cannot make a remote call atomic.
Slide 26: Apache Iggy: a new implementation of the streaming log
Rust and modern I/O target the cost of moving and retaining messages.
- Thread-per-core ownership aims to reduce shared-state contention and improve locality.
- Linux io_uring supports submission and completion of asynchronous I/O.
- Apache Iggy graduated to a Top-Level Project in August 2026.
Slide 27: Evaluate Iggy with evidence, not a speed headline
An efficient runtime is one dimension of an enterprise platform.
- Compare equal message sizes, batching, persistence, replication, and acknowledgement settings.
- Measure tail latency, CPU, backlog recovery, and behavior during node failure.
- Check released clustering, SDKs, connectors, security, upgrades, and Kafka protocol compatibility.
Slide 28: Choose a design and defend it
A marketplace needs pickup jobs, analytics, and a corrected report.
- Choose a model for fulfillment commands, analytics history, and report replay.
- Defend it using ordering, retention, recovery, and operating cost.
- Use the paper activity if brokers are unavailable.
Slide 29: Operate the asynchronous workflow
A backlog is a promise of future work that needs a bound.
- Watch oldest pending job age, ready/unacked counts, and consumer lag.
- Bound retries; isolate invalid messages and alert an owner.
- Use an outbox for the database/publication gap; consumers still handle repeats.
Slide 30: From business events to agentic AI
Kafka can carry fresh context and agent outcomes across an enterprise.
- A flight cancellation wakes a recovery agent; new prices and policy facts update its context.
- Agents use RPC tools for immediate lookups and publish proposals or outcomes as events.
- Kafka transports and retains events; agent logic, authorization, and workflow state live elsewhere.
Slide 31: IBM + Confluent: real-time data for agents
IBM completed its Confluent acquisition on March 17, 2026.
- Confluent adds a commercial streaming platform and integrations around the open-source ecosystem.
- Its Streaming Agents documentation describes event-driven agents running on Flink over streams.
- The opportunity: current business context, independent agents, and observable outcomes.
Slide 32: End to end: a travel recovery assistant
An event starts the workflow; RPC tools supply options; a person approves the action.
Slide 33: Assignment: prove recovery, not just the happy path
Build the complete local travel workflow with synthetic data and a mock planner.
- Create disruption → proposal → approval → mock booking → observable outcome.
- Test duplicate approval, lost booking response, executor restart, and stale proposal approval.
- Replay audit history without booking again; submit code, contracts, a diagram, and evidence.
Slide 34: Connect the dots: messages, streams, and agents
The same distributed-systems rules apply when the consumer is an AI agent.
- MOM decouples applications; retained logs let new consumers reuse business history.
- RPC gets an immediate answer; events carry facts and outcomes across time.
- An agent still needs contracts, durable state, idempotency, approval, and recovery.
Canvas remains the official source for project requirements, deadlines, submissions, and grading.
Key Takeaways
- Sharing work and giving every application a copy require different subscription arrangements.
- Broker acceptance, consumer progress, and business completion are distinct facts.
- Failures between effects and progress recording can cause repeated work.
- Kafka replay depends on retained records and consumer positions.
- Single-node classroom examples do not demonstrate production resilience.
- Kafka’s adoption reflects workload fit; traditional queues remain useful.
- Iggy explores runtime efficiency; evaluate release capabilities and failure behavior separately.
Lab Links
- Classroom examples and setup
Small RabbitMQ workers and Kafka console exercises, with expected observations.