CMPE273 · Module 9

Making Multi-Service Workflows Reliable

Distributed transactions, sagas, and idempotency explained through online orders, safe retries, and an AI booking agent.

Learning Objectives

Explain why one database transaction cannot roll back independent services.
Compare two-phase commit with sagas and identify their tradeoffs.
Design compensating actions and durable recovery for a failed workflow.
Use idempotency keys and atomic duplicate detection to make retries safe.
Explain how outbox and inbox records close message-processing failure windows.
Apply these guarantees to an agent that executes approved tool calls.

Core Concepts

Distributed transaction

One logical operation that changes state across multiple networked participants.

Saga

A workflow of committed local transactions with compensating actions for business recovery.

Compensation

A new business action that offsets an earlier committed action; it may fail or require intervention.

Idempotency

Repeating the same logical operation preserves the intended effect of a single execution.

Transactional outbox

Save a state change and its outgoing event in the same local database transaction.

Diagrams

One database can protect its own changes

flowchart LR
  C[Checkout] --> T[Orders DB transaction]
  T --> O[Order row]
  T --> L[Line items]
  C --> P[Remote payment service]

The order and line items share a local commit boundary. Rolling back that transaction does not undo a payment at another service.

Three services, three independent outcomes

flowchart LR
  O[Order O-42] --> I[Inventory: reserved]
  I --> P[Payment: charged USD 80]
  P --> D[Delivery: rejected address]
  D --> R[Recovery needed]

Delivery failure leaves both a reservation and a charge. Throwing an exception in checkout cannot erase those committed effects.

Coordinated decision across compatible databases

flowchart TD
  C[Coordinator] -->|1. Prepare| A[Inventory DB]
  C -->|1. Prepare| B[Payment ledger DB]
  A -->|Prepared vote| V[Collect all votes]
  B -->|Prepared vote| V
  V --> D[Durably record decision]
  D -->|2. Commit or abort| A
  D -->|2. Commit or abort| B

All participants must support the protocol. After prepare, a participant cannot safely guess the decision when the coordinator is unreachable.

O-42 forward actions and recovery

flowchart LR
  I[Reserve stock] --> P[Charge USD 80]
  P --> D[Book delivery]
  D -->|Rejected| R[Refund USD 80]
  R --> S[Release stock]
  S --> C[Order cancelled]

Successful earlier steps remain committed until separate compensation transactions run. The customer may see recovery pending before cancellation completes.

Two ways to coordinate the same order

flowchart TD
  W[Durable orchestrator] --> I[Inventory]
  W --> P[Payment]
  W --> D[Delivery]
  E[OrderCreated event] --> H[Inventory handler]
  H --> R[StockReserved event]
  R --> Q[Payment handler]

The upper blocks show explicit commands from a coordinator. The lower blocks show the start of an event chain without a central workflow driver.

Lost response after a successful charge

flowchart LR
  C[Checkout] -->|Charge key O-42-pay| P[Payment commits]
  P --> X[Response lost]
  X --> T[Checkout times out]
  T -->|Same key| R[Lookup or safe retry]
  R --> P

The network failure hides the result, not necessarily the effect. Reusing the operation identity lets the payment service recognize a repeat.

Two attempts, one charge

flowchart LR
  A[Attempt 1: O-42-pay] --> G[Idempotency gate]
  B[Attempt 2: O-42-pay] --> G
  G --> L[One ledger charge]
  G --> R[Recorded result returned]

The gate deduplicates by authenticated caller, operation, and key. It must protect concurrent requests as well as later retries.

Atomic handling of a local ledger operation

flowchart LR
  R[Request] --> T[Begin local transaction]
  T --> K[Claim unique key]
  K --> B[Update ledger]
  B --> S[Save result]
  S --> C[Commit together]

A crash before commit leaves neither a new ledger effect nor a completed key record. After commit, a retry retrieves the result instead of applying another effect.

Durable publication and duplicate-safe consumption

flowchart LR
  T[Local transaction] --> O[Order row]
  T --> E[Outbox row]
  E --> R[Relay]
  R --> B[Broker]
  B --> C[Consumer transaction]
  C --> I[Unique inbox event ID]
  C --> S[Local state change]

The outbox closes the state-versus-publication gap. A relay crash after publishing can cause redelivery; the inbox and business update prevent a repeated local effect.

Enforce the stock invariant at its owner

flowchart LR
  A[Order A] --> I[Inventory atomic reservation]
  B[Order B] --> I
  I --> W[One reservation succeeds]
  I --> F[Other request: out of stock]

The inventory service protects the last item locally. A later compensation cannot justify overselling during concurrent sagas.

An agent proposes a trip; controlled tools carry it out

flowchart LR
  A[AI proposes trip] --> H[Human approves exact proposal]
  H --> G[Tool gateway validates approval]
  G --> W[Durable booking saga]
  W --> T[Hotel and payment tools]
  T --> R[Results and reconciliation]
  R --> W

The model proposes actions, while the gateway checks authority and the workflow owns durable execution. A repeated model turn must not create another booking for the same approved action.

Examples, Discussions, and Project Ideas

Slide 1: One Order, Several Services — What Can Go Wrong?

Learn distributed transactions, sagas, and idempotency by making an online order reliable.

Reserve one item, collect payment, and arrange delivery.
Explain partial success, recover from failure, and retry safely.
Finish by giving an AI booking agent the same safeguards.

Slide 2: A Local Transaction Has a Boundary

Inside one database, related changes can commit or roll back together.

An order row and its line items can share one ACID transaction.
A remote payment call is outside that database transaction.

Slide 3: Payment Succeeds, Delivery Fails

A workflow can fail after earlier services have committed successfully.

O-42 reserves the last headset and pays $80.
Delivery rejects the address; the overall order cannot finish.

Discuss: What should the customer see while recovery is still running?

Slide 4: Two-Phase Commit — Agree Before Finishing

A coordinator asks participating transaction managers to prepare, then records a commit or abort decision.

Phase 1: every participant must durably promise it can commit.
Phase 2: commit only after all prepare; otherwise decide abort.
A prepared participant may wait for recovery while holding resources.

Slide 5: A Saga — Finish in Steps, Recover in Steps

Each service commits locally; failures trigger explicit compensating actions.

Reserve stock, charge payment, then book delivery.
If delivery fails, refund payment and release stock.

Slide 6: Compensation Is a Business Action

A refund offsets a charge, but does not erase history or reverse every consequence.

Reserve stock → release the reservation; charge → request a refund.
Already shipped? Use a return process; an email cannot be unsent.
Refund unavailable? Persist RECOVERY_PENDING, retry, and escalate.

Discuss: Should a non-refundable ticket be purchased before or after reversible reservations?

Slide 7: Who Drives the Saga?

Orchestration uses an explicit workflow owner; choreography lets services react to events.

Orchestration: one durable workflow tracks progress and recovery.
Choreography: event handlers own reactions; trace the whole chain.
Both need duplicate handling, timeouts, and observable state.

Slide 8: A Timeout Means “We Do Not Know Yet”

The payment may have committed even when checkout never received the response.

A new charge request could charge the customer twice.
Query status or retry the same operation with the same key.

Slide 9: Idempotency — Retry Without Repeating the Effect

One logical operation keeps one stable key across network retries and worker restarts.

Same key and same payload: return the recorded outcome.
Same key, different amount: reject the mismatch.
New intended purchase: use a new key.

Slide 10: Close the Crash Window in the Database

Commit duplicate detection, the local business change, and the result in one transaction.

Enforce a unique constraint on caller, operation, and key.
Store a request fingerprint and result beside the business change.
A separate “check then charge” leaves races and crash gaps.

Slide 11: Save the Order and Its Event Together

An outbox prevents a committed order from silently losing its outgoing event.

Save the order change and outbox event in one local transaction.
A relay publishes committed outbox rows and retries after failures.
Consumers commit the event ID and local effect together.

Slide 12: Two Customers Want the Last Item

Sagas do not isolate concurrent workflows from each other.

Inventory must atomically reserve only when available stock is positive.
Use reservation IDs and controlled expiry; release only your reservation.
Persist saga state and version-check transitions after worker restarts.

Slide 13: Choose the Smallest Reliable Boundary

Choose from business requirements and participant capabilities.

One database: prefer one local transaction.
Compatible transaction managers and atomic commit required: consider 2PC and its recovery cost.
Independent services with acceptable intermediate states: use a saga.
Every retryable effect: define idempotency and reconciliation.

Slide 14: Tabletop Lab — Break Checkout, Then Recover

In pairs, trace O-42 through failures and prove that recovery preserves the intended effects.

Lose the payment response; retry with the same key. Count charges.
Reject delivery; fail the first refund. Show pending recovery and resume.
Redeliver one event; race two last-item reservations.
Record workflow state, operation keys, event IDs, and final balances.

Open the scenario worksheet

Slide 15: Agentic Finale — An AI Can Plan; Tools Must Execute Safely

A travel agent books a hotel, payment, and transport using a durable saga.

Bind approval to the exact itinerary, price limit, and proposal version.
Recheck permissions and availability; persist one key per approved action.
On timeout, inspect or retry that action; compensate or escalate failures.

Canvas remains the official source for project requirements, deadlines, submissions, and grading.

Key Takeaways

A timeout leaves the outcome unknown; inspect or retry using the same operation identity.
Two-phase commit coordinates a common decision; a saga exposes intermediate committed states.
Compensation needs explicit business rules, durable progress, and recovery ownership.
Duplicate detection and the local business change must commit together.
Outbox relays may redeliver; consumers still need durable deduplication.
Agent tool calls need the same retry guarantees plus authorization bound to an immutable proposal.