1 / 25
Sections
Slide 1
Fault Tolerance + Observability
Module 9 · November 9, 2026 · CMPE273
- Design for partial failure; use evidence to explain the outcome.
- Follow one request from user symptom to a bounded recovery decision.
- Instana walkthroughs and paired exercises connect resilience patterns to telemetry.
Slide 2
What you will be able to do
Choose a recovery policy and show how you would verify it.
- Explain timeouts, retries, circuit breakers, bulkheads, and backpressure.
- Explain Logs, Traces, Metrics; analyze time series and trace requests across different languages.
- Produce a short incident report with evidence, uncertainty, mitigation, and validation.
Slide 3
Preparation and session plan
Browser + worksheet; no local agent installation is required.
- Required: review the lost-response and idempotency example from the transaction session.
- Open the Instana playground; IBM registration or existing access may be required.
- Use the supplied synthetic packet if access or relevant telemetry is unavailable.
Slide 4
Three pillars: Logs, Traces, Metrics
Start with events, request paths, and measurements over time.
- Logs: structured, timestamped events explain what happened; correlate by trace and operation ID.
- Traces: related spans show where one request spent time across service boundaries.
- Metrics: numerical time series show how broadly users and resources are affected.
Slide 5
Time series analysis: detect and compare change
A time series is timestamped values for a metric and a fixed dimension set.
- Look for trends, spikes, step changes, seasonality, and lag using aligned baseline and incident windows.
- A counter rising by 600 in 60 seconds means 10 requests/s; 12 errors among those requests means 2%.
- Check resets, missing data, chart rollups, and units; do not average instance p95 values.
Slide 6
Built-in metrics: a useful starting point
Supported sensors and runtime instrumentation expose signals before custom business instrumentation.
- Use call rate, error ratio, and latency to locate affected services and intervals.
- Inspect available CPU, memory, restarts, JVM/GC, or event-loop metrics to test resource hypotheses.
- Coverage depends on configuration and supported versions; add business metrics for checkout success and safe fallbacks.
Slide 7
One trace across Node.js, Java, and Python
Language-independent context connects instrumented service spans.
- Inject and extract compatible trace context at every HTTP or messaging boundary.
- Keep the trace ID across the request; each span has its own ID and service identity.
- Missing spans may mean lost context, unsupported instrumentation, sampling, or export gaps.
flowchart LR N[Node.js gateway] -->|trace context| J[Java checkout] J -->|trace context| P[Python payment] N -. spans .-> B[Tracing backend] J -. spans .-> B P -. spans .-> B
In this illustrative path, an instrumented Node.js gateway calls Java checkout, which calls Python payment. Compatible context injection and extraction connect spans under one trace ID, while each service emits its own spans to the backend. The languages are examples, not verified sandbox services.
Slide 8
A timeout is an unknown outcome
The caller stopped waiting; the downstream operation may still commit.
- Distinguish a crash, a slow dependency, a dropped response, and resource exhaustion.
- Retrying a payment requires the same operation identity and durable deduplication.
- A healthy process does not prove that a user journey is succeeding.
Slide 9
One failure, several affected services
An entry service can become unavailable while its dependency is merely slow.
- Requests occupy connection pools and worker slots while waiting.
- Queues grow; callers retry; useful work competes with repeated work.
- Find the failing boundary before blaming every service with elevated latency.
flowchart LR U[Client] --> G[Gateway] G --> C[Checkout] C --> P[Slow payment dependency] C --> I[Inventory] G -. metrics and traces .-> O[Telemetry] C -. metrics logs traces .-> O P -. spans when instrumented .-> O
A slow payment dependency holds checkout requests and can exhaust gateway capacity. Telemetry from each boundary helps distinguish upstream impact from the downstream source; missing instrumentation leaves uncertainty.
Slide 10
Start with the user journey and SLO
Define good requests before selecting a dashboard.
- Example SLI: eligible checkout requests that succeed within 1 second / all eligible checkout requests.
- Example SLO: 99% good requests over a rolling 7-day window; define exclusions explicitly.
- For 10,000 eligible requests, 99% allows 100 bad requests; 160 bad requests exceed that budget by 60.
Slide 11
Deadlines bound waiting
Give a whole operation one time budget and propagate the remaining budget.
- Example: 1,000 ms total; reserve 150 ms for gateway and checkout work.
- Two 350 ms attempts plus up to 100 ms backoff fit within the remaining 850 ms.
- Actual work, queueing, and network time consume the budget; stop if the remaining time is insufficient.
Slide 12
Safe retries need a budget
Retry only selected transient failures when the operation is safe to repeat.
- Use a bounded attempt count, exponential backoff, and jitter.
- Put retry ownership at one deliberate layer; avoid multiplicative retries.
- Three layers with three attempts each can send 27 calls to one dependency.
Slide 13
Circuit breakers fail fast
A breaker temporarily rejects calls to an unhealthy dependency.
- Closed: calls run; a defined failure threshold opens the breaker.
- Open: reject calls quickly; after a cooldown permit limited probes.
- Half-open: recover after successful probes or reopen after failure.
stateDiagram-v2 [*] --> Closed Closed --> Open: Failure threshold Open --> HalfOpen: Cooldown elapsed HalfOpen --> Closed: Successful probes HalfOpen --> Open: Probe fails
The breaker starts closed, opens after the configured failure threshold, and allows limited half-open probes after cooldown. Successful probes restore calls; failed probes reopen it. Thresholds and probe limits are policy choices, and a breaker does not repair a dependency.
Slide 14
Bulkheads and backpressure protect capacity
Isolate work and bound admission rather than allowing unlimited queues.
- Bulkhead: separate worker or connection budgets by dependency or workload.
- Backpressure: bound queues, reject excess work, or slow producers.
- Measure queue depth, queue age, in-flight work, saturation, and rejection rate.
Slide 15
Instrument the recovery policy itself
Show whether protection reduced user impact or hid the failure.
- Record attempts, deadline exceeded, breaker state, rejections, and fallback results.
- Propagate trace context; correlate logs without using trace IDs as metric labels.
- Restrict telemetry access; redact tokens, personal data, and tool prompt content.
Slide 16
Walkthrough 1: establish a baseline in Instana
Choose an available application and freeze one populated time window.
- Open Applications or application monitoring; select a populated application perspective.
- Compare equal baseline and incident windows; record built-in metrics, units, aggregation, and resolution.
- Open Services or dependencies; draw one entry → downstream path and note missing visibility.
Slide 17
Walkthrough 2: from symptom to trace
Compare one slow or erroneous call with one successful call in the same scope.
- Open the selected service; use Analyze Calls or the available call/trace view.
- Filter by service and endpoint; inspect a slow or erroneous call and its trace timeline.
- Record trace/span relationships, service/runtime metadata where visible, and any propagation gaps.
Slide 18
Walkthrough 3: correlate infrastructure and logs
Test a hypothesis while preserving the time window.
- From the service or trace, open related infrastructure when available.
- Compare built-in resource/runtime time series; distinguish trend, spike, lag, and missing data.
- Inspect correlated logs if available; otherwise identify the event fields needed to test the hypothesis.
Slide 19
Exercise 1: diagnose before prescribing
Pairs: choose a request path and produce a falsifiable incident hypothesis.
- Record two observations across the three pillars, including one time series comparison with units.
- Separate affected service from suspected dependency; identify missing evidence.
- Propose one bounded mitigation and one observation that would disprove your diagnosis.
Slide 20
Exercise 2: calculate a safe recovery policy
Design exercise; these settings are not changes to the shared sandbox.
- Allocate the 1,000 ms deadline; choose retry ownership, attempt limit, and idempotency rule.
- Choose a breaker probe policy, bounded queue, and a dependency bulkhead.
- Predict which telemetry would change and which user outcome must remain correct.
Slide 21
Synthetic incident: degraded checkout
Fallback packet only; these values were not collected from Instana.
- 10,000 eligible requests: 9,840 good; 160 bad; 98.4% good against a 99% SLO.
- Checkout trace: 920 ms root; payment attempts at 100–450 ms and 520–870 ms.
- Payment CPU stays flat while connections saturate; this does not prove a CPU bottleneck.
Slide 22
Mitigate, then verify recovery
A quieter dashboard is insufficient without user-outcome evidence.
- Compare equivalent before/after windows, traffic, endpoints, and SLI definitions.
- Check good-request ratio, tail latency, attempt amplification, queue age, and rejected work.
- Use a time-bounded experiment with rollback criteria; watch for stale or misleading fallbacks.
Slide 23
Transfer the method to your project
Apply one pattern to one dependency and capture evidence.
- In your own local environment: establish baseline, inject bounded latency or outage, then restore it.
- Keep traffic and observation windows comparable; capture traces and recovery behavior.
- For agent tool calls: bound retries and concurrency, preserve operation identity, and require approval for consequential recovery actions.
Slide 24
Debrief: defend the evidence
Explain what you know, what you infer, and what you would check next.
- Why can a timeout still produce a committed payment?
- Why can a mean hide a latency spike, and why might a cross-language trace be incomplete?
- How could a breaker reduce latency while the good-request ratio stays poor?
Slide 25
Takeaways and next meeting
Reliable systems contain failures and make recovery observable.
- Bound time, repeated work, concurrency, and queue growth.
- Connect user impact to a request path; validate claims with aligned telemetry.
- November 16: Service Networking with Hashicorp Consul - A workshop. November 23: No class (Thanksgiving holiday).