1 / 15
Sections
Slide 1
One Order, Several Services — What Can Go Wrong?
Learn distributed transactions, sagas, and idempotency by making an online order reliable.
- Reserve one item, collect payment, and arrange delivery.
- Explain partial success, recover from failure, and retry safely.
- Finish by giving an AI booking agent the same safeguards.
Slide 2
A Local Transaction Has a Boundary
Inside one database, related changes can commit or roll back together.
- An order row and its line items can share one ACID transaction.
- A remote payment call is outside that database transaction.
flowchart LR C[Checkout] --> T[Orders DB transaction] T --> O[Order row] T --> L[Line items] C --> P[Remote payment service]
The order and line items share a local commit boundary. Rolling back that transaction does not undo a payment at another service.
Slide 3
Payment Succeeds, Delivery Fails
A workflow can fail after earlier services have committed successfully.
- O-42 reserves the last headset and pays $80.
- Delivery rejects the address; the overall order cannot finish.
flowchart LR O[Order O-42] --> I[Inventory: reserved] I --> P[Payment: charged USD 80] P --> D[Delivery: rejected address] D --> R[Recovery needed]
Delivery failure leaves both a reservation and a charge. Throwing an exception in checkout cannot erase those committed effects.
Slide 4
Two-Phase Commit — Agree Before Finishing
A coordinator asks participating transaction managers to prepare, then records a commit or abort decision.
- Phase 1: every participant must durably promise it can commit.
- Phase 2: commit only after all prepare; otherwise decide abort.
- A prepared participant may wait for recovery while holding resources.
flowchart TD C[Coordinator] -->|1. Prepare| A[Inventory DB] C -->|1. Prepare| B[Payment ledger DB] A -->|Prepared vote| V[Collect all votes] B -->|Prepared vote| V V --> D[Durably record decision] D -->|2. Commit or abort| A D -->|2. Commit or abort| B
All participants must support the protocol. After prepare, a participant cannot safely guess the decision when the coordinator is unreachable.
Slide 5
A Saga — Finish in Steps, Recover in Steps
Each service commits locally; failures trigger explicit compensating actions.
- Reserve stock, charge payment, then book delivery.
- If delivery fails, refund payment and release stock.
flowchart LR I[Reserve stock] --> P[Charge USD 80] P --> D[Book delivery] D -->|Rejected| R[Refund USD 80] R --> S[Release stock] S --> C[Order cancelled]
Successful earlier steps remain committed until separate compensation transactions run. The customer may see recovery pending before cancellation completes.
Slide 6
Compensation Is a Business Action
A refund offsets a charge, but does not erase history or reverse every consequence.
- Reserve stock → release the reservation; charge → request a refund.
- Already shipped? Use a return process; an email cannot be unsent.
- Refund unavailable? Persist RECOVERY_PENDING, retry, and escalate.
Slide 7
Who Drives the Saga?
Orchestration uses an explicit workflow owner; choreography lets services react to events.
- Orchestration: one durable workflow tracks progress and recovery.
- Choreography: event handlers own reactions; trace the whole chain.
- Both need duplicate handling, timeouts, and observable state.
flowchart TD W[Durable orchestrator] --> I[Inventory] W --> P[Payment] W --> D[Delivery] E[OrderCreated event] --> H[Inventory handler] H --> R[StockReserved event] R --> Q[Payment handler]
The upper blocks show explicit commands from a coordinator. The lower blocks show the start of an event chain without a central workflow driver.
Slide 8
A Timeout Means “We Do Not Know Yet”
The payment may have committed even when checkout never received the response.
- A new charge request could charge the customer twice.
- Query status or retry the same operation with the same key.
flowchart LR C[Checkout] -->|Charge key O-42-pay| P[Payment commits] P --> X[Response lost] X --> T[Checkout times out] T -->|Same key| R[Lookup or safe retry] R --> P
The network failure hides the result, not necessarily the effect. Reusing the operation identity lets the payment service recognize a repeat.
Slide 9
Idempotency — Retry Without Repeating the Effect
One logical operation keeps one stable key across network retries and worker restarts.
- Same key and same payload: return the recorded outcome.
- Same key, different amount: reject the mismatch.
- New intended purchase: use a new key.
flowchart LR A[Attempt 1: O-42-pay] --> G[Idempotency gate] B[Attempt 2: O-42-pay] --> G G --> L[One ledger charge] G --> R[Recorded result returned]
The gate deduplicates by authenticated caller, operation, and key. It must protect concurrent requests as well as later retries.
Slide 10
Close the Crash Window in the Database
Commit duplicate detection, the local business change, and the result in one transaction.
- Enforce a unique constraint on caller, operation, and key.
- Store a request fingerprint and result beside the business change.
- A separate “check then charge” leaves races and crash gaps.
flowchart LR R[Request] --> T[Begin local transaction] T --> K[Claim unique key] K --> B[Update ledger] B --> S[Save result] S --> C[Commit together]
A crash before commit leaves neither a new ledger effect nor a completed key record. After commit, a retry retrieves the result instead of applying another effect.
Slide 11
Save the Order and Its Event Together
An outbox prevents a committed order from silently losing its outgoing event.
- Save the order change and outbox event in one local transaction.
- A relay publishes committed outbox rows and retries after failures.
- Consumers commit the event ID and local effect together.
flowchart LR T[Local transaction] --> O[Order row] T --> E[Outbox row] E --> R[Relay] R --> B[Broker] B --> C[Consumer transaction] C --> I[Unique inbox event ID] C --> S[Local state change]
The outbox closes the state-versus-publication gap. A relay crash after publishing can cause redelivery; the inbox and business update prevent a repeated local effect.
Slide 12
Two Customers Want the Last Item
Sagas do not isolate concurrent workflows from each other.
- Inventory must atomically reserve only when available stock is positive.
- Use reservation IDs and controlled expiry; release only your reservation.
- Persist saga state and version-check transitions after worker restarts.
flowchart LR A[Order A] --> I[Inventory atomic reservation] B[Order B] --> I I --> W[One reservation succeeds] I --> F[Other request: out of stock]
The inventory service protects the last item locally. A later compensation cannot justify overselling during concurrent sagas.
Slide 13
Choose the Smallest Reliable Boundary
Choose from business requirements and participant capabilities.
- One database: prefer one local transaction.
- Compatible transaction managers and atomic commit required: consider 2PC and its recovery cost.
- Independent services with acceptable intermediate states: use a saga.
- Every retryable effect: define idempotency and reconciliation.
Slide 14
Tabletop Lab — Break Checkout, Then Recover
In pairs, trace O-42 through failures and prove that recovery preserves the intended effects.
- Lose the payment response; retry with the same key. Count charges.
- Reject delivery; fail the first refund. Show pending recovery and resume.
- Redeliver one event; race two last-item reservations.
- Record workflow state, operation keys, event IDs, and final balances.
Slide 15
Agentic Finale — An AI Can Plan; Tools Must Execute Safely
A travel agent books a hotel, payment, and transport using a durable saga.
- Bind approval to the exact itinerary, price limit, and proposal version.
- Recheck permissions and availability; persist one key per approved action.
- On timeout, inspect or retry that action; compensate or escalate failures.
flowchart LR A[AI proposes trip] --> H[Human approves exact proposal] H --> G[Tool gateway validates approval] G --> W[Durable booking saga] W --> T[Hotel and payment tools] T --> R[Results and reconciliation] R --> W
The model proposes actions, while the gateway checks authority and the workflow owns durable execution. A repeated model turn must not create another booking for the same approved action.