1 / 34
Sections
Slide 1
From enterprise messaging to event streaming
Module 5 · September 28 · 6:00–8:30 PM
- Why messaging became core distributed-systems infrastructure.
- From enterprise queues to Kafka at LinkedIn and Apache Iggy.
- Predict, run, fail, recover, and choose a design.
Slide 2
Message-oriented middleware: why it exists
MOM lets applications exchange messages through managed infrastructure.
- Store-and-forward bridges applications that are not available at the same time.
- Routing, persistence, acknowledgements, and access control move into shared infrastructure.
- A broker reduces temporal coupling; schemas and business meaning still couple applications.
Slide 3
The evolution follows changing workloads
Several generations coexist because they solve different problems.
- 1990s: enterprise queues connect heterogeneous systems; MQSeries arrives in 1993.
- 2000s: APIs, protocols, pub/sub, and integration tooling make messaging more accessible.
- 2010s–2020s: retained logs, stream processing, cloud services, and efficient runtimes expand the design space.
Slide 4
Architecture, API, protocol, and product
These names answer different questions.
- EDA describes how services react to events; MOM provides message transport and mediation.
- JMS/Jakarta Messaging defines a Java API; AMQP defines a wire protocol.
- IBM MQ, RabbitMQ, Kafka, and Iggy implement different messaging models and tradeoffs.
Slide 5
Event-driven architecture: publish a business fact
A producer owns the fact; each subscriber owns its reaction.
flowchart LR O[Ordering owns confirmation] -->|OrderConfirmed| B[Event distribution] B --> P[Pickup subscription] B --> A[Analytics subscription] P --> W[Prepare pickup] A --> R[Update report]
Ordering publishes a completed business transition. Pickup and analytics have independent subscriptions and reactions. Their completion times differ; the customer should see the real workflow state.
Slide 6
Who coordinates the work? Two use cases
Use choreography for independent reactions; orchestration for a tracked multi-step goal.
- Choreography: OrderConfirmed triggers pickup preparation and analytics independently.
- Orchestration: a travel coordinator holds a flight, reserves a hotel, and tracks both results.
- If the hotel fails, the coordinator releases the flight hold and records recovery.
Slide 7
Travel search needs RPC; booking updates need events
Think of an Expedia/Kayak-style experience: search now, track changes over time.
- RPC: query flight/hotel providers for current options, with deadlines and partial results.
- Command: ReserveFlight requests an action; a timeout does not prove it failed.
- Event: BookingConfirmed reports a fact that alerts, itinerary views, and analytics can consume.
Slide 8
Two workers share a queue
Competing consumers divide work from one RabbitMQ queue.
- One delivery goes to one worker; failures can lead to redelivery.
- Two independent subscribers need separate queues bound to an exchange.
flowchart LR P[Job producer] --> E[Default exchange] E --> Q[Durable pickup queue] Q --> A[Worker A] Q --> B[Worker B] A -->|ACK after effect| Q B -->|ACK after effect| Q
The producer routes jobs to one queue. A and B compete for its deliveries. Each worker acknowledges after its simulated effect; loss before acknowledgement permits redelivery.
Slide 9
Pub/sub requires independent subscriptions
Two workers and two subscribers are different topologies.
flowchart LR P[Event producer] --> X[Fanout exchange] X --> Q1[Pickup queue] X --> Q2[Analytics queue] Q1 --> W1[Pickup worker] Q2 --> W2[Analytics worker]
The exchange routes a publication to two bound queues. Pickup and analytics each receive a logical copy; multiple workers attached to either queue share that subscriber’s work.
Slide 10
Predict the failure window
A worker prints the pickup result and then loses its connection before ACK.
- Will the queue deliver the job again?
- Which identifier lets us recognize the repeat?
- What changes if ACK happens before the work?
Slide 11
RabbitMQ demo: six pickup jobs
Standalone lab: no earlier module, repository, Python, or Java installation required.
- Download and unzip the classroom kit; open three terminals in its module-05 folder.
- Start Docker with Compose v2; run the README setup and wait for broker readiness.
- Start workers A and B, publish six jobs, then inspect IDs and queue counts.
Slide 12
RabbitMQ demo: crash before ACK
Run example B with every normal worker stopped.
- Start CRASH, send one job, and observe the intentional failure.
- Start RECOVERY; compare IDs and the redelivered flag.
- The simulated effect repeats because no deduplication exists.
Slide 13
One booking even when a message arrives twice
Approval e17 for proposal p42:v1 is delivered again after a worker restart.
- Local example: save e17 and update the itinerary in one database transaction; then acknowledge.
- On redelivery, the stored e17 prevents a second itinerary update.
- Remote example: retry the booking with the same key p42:v1; query an uncertain result.
Slide 14
Break
Resume with Kafka’s origins and the retained log.
- Leave brokers running; stop RabbitMQ workers when their queues drain.
Slide 15
Kafka was born from LinkedIn activity data
LinkedIn open-sourced Kafka on January 11, 2011.
- Page views, searches, and ad activity generated a continuous stream of facts.
- The same data needed to reach product analytics and other downstream systems.
- Kafka connected high-volume collection with consumers that ran at different speeds.
Slide 16
The original Kafka design question
Can one system support both log collection and online consumption?
- The 2011 paper is by Jay Kreps, Neha Narkhede, and Jun Rao at LinkedIn.
- A persistent log lets readers track positions rather than delete records after reading.
- Partitioned storage and batched transfer target high-volume log processing.
Slide 17
From one pipeline to an enterprise backbone
LinkedIn reported seven trillion messages per day in October 2019.
- Treat that number as a dated engineering report, not a current benchmark.
- Many producer and consumer teams turn the shared log into integration infrastructure.
- At that scale, quotas, upgrades, ownership, and operations become product requirements.
Slide 18
Why Kafka gained ground in data pipelines
Retained history made repeated use of the same data practical.
- Independent groups support live applications, batch ingestion, and rebuilding derived views.
- Append-oriented I/O, page cache, batching, and compression improve data movement efficiency.
- Partitions create parallelism; retention and replication add storage and operational costs.
Slide 19
Kafka keeps a partitioned history
Each consumer group tracks its own position in the retained log.
- A record has a key, value, partition, and offset.
- Consumption does not remove records; retention limits replay.
- Ordering is within a partition.
flowchart LR P[Order producer] --> T[Orders topic - partition 0] T --> G[Pickup group - own offset] T --> A[Analytics group - own offset] T --> R[New replay group - earliest retained offset]
Each group reads the same retained partition with independent progress. A new group can start at the earliest available record. Reads do not delete records, but retention can remove old history.
Slide 20
Kafka demo: publish and replay
Run example C using one topic partition and three synthetic events.
- Pickup reads live events; analytics starts later and reads history.
- Inspect committed offset 3 after offsets 0, 1, and 2.
- Use a fresh audit group to replay; compare with restarting pickup.
Slide 21
Kafka demo: group membership
Run example D with two members of pickup.
- One partition means one active member in that group in steady state.
- Produce e4 after the rebalance; identify the receiving member.
- Stop that member and observe takeover; repeat reads are possible.
Slide 22
Uber: a message bus for real-time and batch data
Uber’s 2016 engineering report describes Kafka feeding multiple pipelines.
- Rider/driver events and application logs feed downstream consumers.
- Real-time users include metrics, debugging, alerts, and dashboards.
- uReplicator addressed Kafka replication reliability and operational control.
Slide 23
Netflix: buffering and isolation in Keystone
The April 2016 report separates fronting and consumer Kafka clusters.
- Fronting clusters collect events; downstream routing feeds consumer clusters.
- The separation protects ingestion from unpredictable consumer workloads.
- The report explicitly balanced data loss, availability, and cost for that pipeline.
Slide 24
Kafka and traditional MQ coexist
Choose the delivery model before choosing the product.
- Work dispatch: per-message completion, routing, and retry handling often favor a queue.
- Shared event history: independent readers and replay often favor a retained log.
- A retailer can use both: fulfillment commands on queues, sales facts in streams.
Slide 25
Can a crash lose work or repeat it?
The order of “do the work” and “record completion” changes recovery.
- Record completion first, then crash: the booking may never happen.
- Book first, then crash before recording it: a retry may book twice.
- Use durable state and the provider’s stable operation key; Kafka alone cannot make a remote call atomic.
Slide 26
Apache Iggy: a new implementation of the streaming log
Rust and modern I/O target the cost of moving and retaining messages.
- Thread-per-core ownership aims to reduce shared-state contention and improve locality.
- Linux io_uring supports submission and completion of asynchronous I/O.
- Apache Iggy graduated to a Top-Level Project in August 2026.
Slide 27
Evaluate Iggy with evidence, not a speed headline
An efficient runtime is one dimension of an enterprise platform.
- Compare equal message sizes, batching, persistence, replication, and acknowledgement settings.
- Measure tail latency, CPU, backlog recovery, and behavior during node failure.
- Check released clustering, SDKs, connectors, security, upgrades, and Kafka protocol compatibility.
Slide 28
Choose a design and defend it
A marketplace needs pickup jobs, analytics, and a corrected report.
- Choose a model for fulfillment commands, analytics history, and report replay.
- Defend it using ordering, retention, recovery, and operating cost.
- Use the paper activity if brokers are unavailable.
Slide 29
Operate the asynchronous workflow
A backlog is a promise of future work that needs a bound.
- Watch oldest pending job age, ready/unacked counts, and consumer lag.
- Bound retries; isolate invalid messages and alert an owner.
- Use an outbox for the database/publication gap; consumers still handle repeats.
Slide 30
From business events to agentic AI
Kafka can carry fresh context and agent outcomes across an enterprise.
- A flight cancellation wakes a recovery agent; new prices and policy facts update its context.
- Agents use RPC tools for immediate lookups and publish proposals or outcomes as events.
- Kafka transports and retains events; agent logic, authorization, and workflow state live elsewhere.
Slide 31
IBM + Confluent: real-time data for agents
IBM completed its Confluent acquisition on March 17, 2026.
- Confluent adds a commercial streaming platform and integrations around the open-source ecosystem.
- Its Streaming Agents documentation describes event-driven agents running on Flink over streams.
- The opportunity: current business context, independent agents, and observable outcomes.
Slide 32
End to end: a travel recovery assistant
An event starts the workflow; RPC tools supply options; a person approves the action.
flowchart LR D[Trip disrupted] --> K1[Kafka disruption topic] K1 --> A[Planner agent] A -->|RPC| S[Mock search API] A --> P[Proposal topic and stored plan] P --> H[Human approval] H --> K2[Kafka approval topic] K2 --> E[Executor] E -->|RPC with operation key| B[Mock booking API] E --> O[Kafka outcome topic] O --> U[Status and audit]
A disruption triggers planning. The planner calls a mock search API and publishes a proposal. A human approves its exact version. The executor calls the mock booking API with a stable operation key and publishes an outcome for status and audit. Durable application state supports recovery at every step.
Slide 33
Assignment: prove recovery, not just the happy path
Build the complete local travel workflow with synthetic data and a mock planner.
- Create disruption → proposal → approval → mock booking → observable outcome.
- Test duplicate approval, lost booking response, executor restart, and stale proposal approval.
- Replay audit history without booking again; submit code, contracts, a diagram, and evidence.
Slide 34
Connect the dots: messages, streams, and agents
The same distributed-systems rules apply when the consumer is an AI agent.
- MOM decouples applications; retained logs let new consumers reuse business history.
- RPC gets an immediate answer; events carry facts and outcomes across time.
- An agent still needs contracts, durable state, idempotency, approval, and recovery.