Making Multi-Service Workflows Reliable

Distributed transactions, sagas, and idempotency

A customer wants one outcome: an accepted order. Inventory, payment, and delivery each own their data and can succeed or fail independently. This module follows order O-42 to explain how to finish that work, recover when it cannot finish, and avoid repeating effects when messages or requests are retried.

Learning Objectives

Explain why one database transaction cannot roll back independent services.
Compare two-phase commit with sagas and identify their tradeoffs.
Design compensating actions and durable recovery for a failed workflow.
Use idempotency keys and atomic duplicate detection to make retries safe.
Explain how outbox and inbox records close message-processing failure windows.
Apply these guarantees to an agent that executes approved tool calls.

Required preparation

Review Module 5’s message delivery and duplicate handling and Module 8’s consistency and concurrency tradeoffs. Be able to explain a local database commit and why a remote timeout does not reveal the remote outcome. Read the three resources marked Required below; the Stripe API example is optional. No cloud account or payment credentials are needed for the tabletop activity.

Core Concepts

Distributed transaction

One logical operation that changes state across multiple networked participants.

Saga

A workflow of committed local transactions with compensating actions for business recovery.

Compensation

A new business action that offsets an earlier committed action; it may fail or require intervention.

Idempotency

Repeating the same logical operation preserves the intended effect of a single execution.

Transactional outbox

Save a state change and its outgoing event in the same local database transaction.

Fifteen-slide learning path

The same sections power Presentation Mode and the printable student handout. Diagrams describe scenarios, and each explanation states what the arrows mean.

1. One Order, Several Services — What Can Go Wrong?

Learn distributed transactions, sagas, and idempotency by making an online order reliable.

Reserve one item, collect payment, and arrange delivery.
Explain partial success, recover from failure, and retry safely.
Finish by giving an AI booking agent the same safeguards.

Worked explanation: Ask students whether a failed checkout screen proves that no payment happened. Keep order O-42 as the running example.

2. A Local Transaction Has a Boundary

Inside one database, related changes can commit or roll back together.

An order row and its line items can share one ACID transaction.
A remote payment call is outside that database transaction.

One database can protect its own changes

flowchart LR
  C[Checkout] --> T[Orders DB transaction]
  T --> O[Order row]
  T --> L[Line items]
  C --> P[Remote payment service]

The order and line items share a local commit boundary. Rolling back that transaction does not undo a payment at another service.

Worked explanation: ACID means atomicity, consistency, isolation, and durability. Isolation depends on the database isolation level. Avoid holding a local transaction open while waiting on a slow remote call.

3. Payment Succeeds, Delivery Fails

A workflow can fail after earlier services have committed successfully.

O-42 reserves the last headset and pays $80.
Delivery rejects the address; the overall order cannot finish.

Three services, three independent outcomes

flowchart LR
  O[Order O-42] --> I[Inventory: reserved]
  I --> P[Payment: charged USD 80]
  P --> D[Delivery: rejected address]
  D --> R[Recovery needed]

Delivery failure leaves both a reservation and a charge. Throwing an exception in checkout cannot erase those committed effects.

Worked explanation:

Discuss: What should the customer see while recovery is still running?

4. Two-Phase Commit — Agree Before Finishing

A coordinator asks participating transaction managers to prepare, then records a commit or abort decision.

Phase 1: every participant must durably promise it can commit.
Phase 2: commit only after all prepare; otherwise decide abort.
A prepared participant may wait for recovery while holding resources.

Coordinated decision across compatible databases

flowchart TD
  C[Coordinator] -->|1. Prepare| A[Inventory DB]
  C -->|1. Prepare| B[Payment ledger DB]
  A -->|Prepared vote| V[Collect all votes]
  B -->|Prepared vote| V
  V --> D[Durably record decision]
  D -->|2. Commit or abort| A
  D -->|2. Commit or abort| B

All participants must support the protocol. After prepare, a participant cannot safely guess the decision when the coordinator is unreachable.

Worked explanation: A missing vote prevents a commit decision. An uncertain prepared participant needs the recorded decision during recovery. Atomic commit does not by itself define global isolation. An ordinary external payment API usually cannot join this protocol.

5. A Saga — Finish in Steps, Recover in Steps

Each service commits locally; failures trigger explicit compensating actions.

Reserve stock, charge payment, then book delivery.
If delivery fails, refund payment and release stock.

O-42 forward actions and recovery

flowchart LR
  I[Reserve stock] --> P[Charge USD 80]
  P --> D[Book delivery]
  D -->|Rejected| R[Refund USD 80]
  R --> S[Release stock]
  S --> C[Order cancelled]

Successful earlier steps remain committed until separate compensation transactions run. The customer may see recovery pending before cancellation completes.

Worked explanation: Prefer reverse dependency order where appropriate. This scenario assumes delivery rejection is definitive and no booking occurred. A timeout requires reconciliation before choosing a compensation path.

6. Compensation Is a Business Action

A refund offsets a charge, but does not erase history or reverse every consequence.

Reserve stock → release the reservation; charge → request a refund.
Already shipped? Use a return process; an email cannot be unsent.
Refund unavailable? Persist RECOVERY_PENDING, retry, and escalate.

Worked explanation: Compensation can incur fees, take time, or be impossible. Give refunds their own stable operation keys. Preserve the audit trail. Do not label an order fully cancelled until required compensations are confirmed.

Discuss: Should a non-refundable ticket be purchased before or after reversible reservations?

7. Who Drives the Saga?

Orchestration uses an explicit workflow owner; choreography lets services react to events.

Orchestration: one durable workflow tracks progress and recovery.
Choreography: event handlers own reactions; trace the whole chain.
Both need duplicate handling, timeouts, and observable state.

Two ways to coordinate the same order

flowchart TD
  W[Durable orchestrator] --> I[Inventory]
  W --> P[Payment]
  W --> D[Delivery]
  E[OrderCreated event] --> H[Inventory handler]
  H --> R[StockReserved event]
  R --> Q[Payment handler]

The upper blocks show explicit commands from a coordinator. The lower blocks show the start of an event chain without a central workflow driver.

Worked explanation: An orchestrator can be replicated with durable state and concurrency control. Choreography still has dependencies and recovery ownership; events do not remove coordination.

8. A Timeout Means “We Do Not Know Yet”

The payment may have committed even when checkout never received the response.

A new charge request could charge the customer twice.
Query status or retry the same operation with the same key.

Lost response after a successful charge

flowchart LR
  C[Checkout] -->|Charge key O-42-pay| P[Payment commits]
  P --> X[Response lost]
  X --> T[Checkout times out]
  T -->|Same key| R[Lookup or safe retry]
  R --> P

The network failure hides the result, not necessarily the effect. Reusing the operation identity lets the payment service recognize a repeat.

Worked explanation: A timeout is not a definitive rejection. If the provider offers no idempotency or lookup guarantee, hold the workflow for reconciliation instead of blindly issuing another charge.

9. Idempotency — Retry Without Repeating the Effect

One logical operation keeps one stable key across network retries and worker restarts.

Same key and same payload: return the recorded outcome.
Same key, different amount: reject the mismatch.
New intended purchase: use a new key.

Two attempts, one charge

flowchart LR
  A[Attempt 1: O-42-pay] --> G[Idempotency gate]
  B[Attempt 2: O-42-pay] --> G
  G --> L[One ledger charge]
  G --> R[Recorded result returned]

The gate deduplicates by authenticated caller, operation, and key. It must protect concurrent requests as well as later retries.

Worked explanation: A correlation ID connects logs; it does not automatically deduplicate operations. Retain keys for the full retry and reconciliation window. After expiry, a retry may become a new operation. Provider-specific response and error caching contracts vary.

10. Close the Crash Window in the Database

Commit duplicate detection, the local business change, and the result in one transaction.

Enforce a unique constraint on caller, operation, and key.
Store a request fingerprint and result beside the business change.
A separate “check then charge” leaves races and crash gaps.

Atomic handling of a local ledger operation

flowchart LR
  R[Request] --> T[Begin local transaction]
  T --> K[Claim unique key]
  K --> B[Update ledger]
  B --> S[Save result]
  S --> C[Commit together]

A crash before commit leaves neither a new ledger effect nor a completed key record. After commit, a retry retrieves the result instead of applying another effect.

Worked explanation: Concurrent duplicates wait or receive a documented in-progress response. A failed transaction rolls back the key claim. For a remote provider, this local transaction cannot include the remote effect; persist the operation intent and reuse the provider key while reconciling its status.

11. Save the Order and Its Event Together

An outbox prevents a committed order from silently losing its outgoing event.

Save the order change and outbox event in one local transaction.
A relay publishes committed outbox rows and retries after failures.
Consumers commit the event ID and local effect together.

Durable publication and duplicate-safe consumption

flowchart LR
  T[Local transaction] --> O[Order row]
  T --> E[Outbox row]
  E --> R[Relay]
  R --> B[Broker]
  B --> C[Consumer transaction]
  C --> I[Unique inbox event ID]
  C --> S[Local state change]

The outbox closes the state-versus-publication gap. A relay crash after publishing can cause redelivery; the inbox and business update prevent a repeated local effect.

Worked explanation: Acknowledge the broker after the consumer commits. Preserve per-order sequence or validate versions when order matters. Inbox deduplication does not protect a remote side effect unless that provider also supports a stable operation key.

12. Two Customers Want the Last Item

Sagas do not isolate concurrent workflows from each other.

Inventory must atomically reserve only when available stock is positive.
Use reservation IDs and controlled expiry; release only your reservation.
Persist saga state and version-check transitions after worker restarts.

Enforce the stock invariant at its owner

flowchart LR
  A[Order A] --> I[Inventory atomic reservation]
  B[Order B] --> I
  I --> W[One reservation succeeds]
  I --> F[Other request: out of stock]

The inventory service protects the last item locally. A later compensation cannot justify overselling during concurrent sagas.

Worked explanation: Reservation expiry can race with payment. Confirm the reservation before finalizing; if it expired, reconcile and compensate payment as needed. Deduplicate compensation and prevent stale workers from advancing an obsolete workflow version.

13. Choose the Smallest Reliable Boundary

Choose from business requirements and participant capabilities.

One database: prefer one local transaction.
Compatible transaction managers and atomic commit required: consider 2PC and its recovery cost.
Independent services with acceptable intermediate states: use a saga.
Every retryable effect: define idempotency and reconciliation.

Worked explanation: If a business invariant cannot tolerate intermediate states and participants cannot coordinate atomically, redesign the boundary or use reservations and explicit pending states. A saga is not a drop-in replacement for an ACID transaction.

14. Tabletop Lab — Break Checkout, Then Recover

In pairs, trace O-42 through failures and prove that recovery preserves the intended effects.

Lose the payment response; retry with the same key. Count charges.
Reject delivery; fail the first refund. Show pending recovery and resume.
Redeliver one event; race two last-item reservations.
Record workflow state, operation keys, event IDs, and final balances.

Worked explanation: Expected outcomes are one charge, one eventual refund, one stock release, no repeated consumer effect, and at most one last-item winner. Run this first as a paper trace, then apply the same cases to the semester project. Do not claim exactly-once delivery from these observations.

15. Agentic Finale — An AI Can Plan; Tools Must Execute Safely

A travel agent books a hotel, payment, and transport using a durable saga.

Bind approval to the exact itinerary, price limit, and proposal version.
Recheck permissions and availability; persist one key per approved action.
On timeout, inspect or retry that action; compensate or escalate failures.

An agent proposes a trip; controlled tools carry it out

flowchart LR
  A[AI proposes trip] --> H[Human approves exact proposal]
  H --> G[Tool gateway validates approval]
  G --> W[Durable booking saga]
  W --> T[Hotel and payment tools]
  T --> R[Results and reconciliation]
  R --> W

The model proposes actions, while the gateway checks authority and the workflow owns durable execution. A repeated model turn must not create another booking for the same approved action.

Worked explanation: Use separate keys for booking, payment, and cancellation, all bound to the approved workflow and action. A changed itinerary requires a new proposal and approval; idempotency is not authorization. If a hotel call times out, reconcile before purchasing a replacement. Ask students to identify the same lost-response failure from slide 8.

Scenario worksheet: recover order O-42

Use a paper trace or your project services. Start each independent case with one headset available, no charge, no refund, and no delivery. Let O-42 cost $80. Use a durable workflow record with the order ID, state, version, completed steps, operation keys, and unresolved outcomes. A worker should be able to resume from that record after a restart.

Failure to inject	Next safe action	Evidence to inspect
Charge committed; response lost	Look up or retry `O-42-pay` with unchanged payload	One $80 charge and the same provider reference
Delivery definitively rejects address	Refund `O-42-refund`, then release reservation `O-42-stock`	One refund, stock restored, order cancelled
Refund temporarily fails	Persist recovery pending and retry the same refund operation	No false cancellation; eventual single refund and release
Relay publishes, then crashes before marking sent	Republish the same event ID	One consumer state change despite two deliveries
Two orders race for one headset	Atomically enforce available stock at inventory	One successful reservation; stock never negative
Saga worker restarts after payment	Reconcile the persisted payment operation before advancing	No new key and no second charge

For the first case, trace this sequence: save payment intent → provider commits → response disappears → worker restarts → worker loads the saved key → provider returns the existing charge → workflow records payment confirmed. Contrast that with generating a new key after restart: the provider would see a new intended operation and could charge again.

For compensation, trace STOCK_RESERVED → PAYMENT_CONFIRMED → RECOVERY_PENDING → REFUND_CONFIRMED → CANCELLED. Release stock before declaring cancellation complete. If the refund’s outcome is unknown, reconcile it; a timeout alone is not a reason to generate a second refund key. Give unresolved recovery an owner, an alert threshold, and a manual resolution path.

Project practice

Choose one workflow in your semester project. Document its invariant, local transaction boundaries, action and compensation pairs, retry keys, retention window, and response to unknown outcomes. Demonstrate a lost response and a worker restart, then explain the final state using logs and durable records. This is public learning guidance; Canvas remains the official system for submissions, grades, quizzes, exams, announcements, and individualized feedback.

Knowledge Check

Does a payment timeout mean the customer was not charged?

No. The remote charge may have committed. Reconcile or retry with the same stable key under the provider contract.

Does a saga give the same isolation as one database transaction?

No. Intermediate local commits can be visible. Protect invariants using reservations, conditional updates, versions, or a different transaction boundary.

Why save the deduplication key and ledger update in the same transaction?

Separate commits leave a crash window where a retry either repeats the effect or is incorrectly treated as already complete.

Does an outbox prevent duplicate messages?

No. It makes publication recoverable after a local commit. A relay may publish twice, so consumers need atomic inbox deduplication and local effects.

Can an approved AI agent retry with a fresh booking key after a timeout?

That could create a second booking. Reuse the persisted key for the approved action and reconcile the existing outcome; validate approval independently.

Key Takeaways

A timeout leaves the outcome unknown; inspect or retry using the same operation identity.
Two-phase commit coordinates a common decision; a saga exposes intermediate committed states.
Compensation needs explicit business rules, durable progress, and recovery ownership.
Duplicate detection and the local business change must commit together.
Outbox relays may redeliver; consumers still need durable deduplication.
Agent tool calls need the same retry guarantees plus authorization bound to an immutable proposal.

Additional Resources

PostgreSQL two-phase transactions documentation
Required: prepare, commit, and rollback across cooperating participants.
Saga orchestration pattern documentation
Required: compensation, idempotency, and lack of saga isolation.
Transactional outbox pattern documentation
Required: atomic state-and-event storage and duplicate delivery.
Stripe idempotent requests documentation
Optional API example: key reuse, parameter matching, and retention limits.

Module 9: Making Multi-Service Workflows Reliable