Fault Tolerance, Observability, and Operational Diagnosis

Slide 2: What you will be able to do

Choose a recovery policy and show how you would verify it.

• Explain timeouts, retries, circuit breakers, bulkheads, and backpressure.
• Explain Logs, Traces, Metrics; analyze time series and trace requests across different languages.
• Produce a short incident report with evidence, uncertainty, mitigation, and validation.

Slide 3: Preparation and session plan

Browser + worksheet; no local agent installation is required.

• Required: review the lost-response and idempotency example from the transaction session.
• Open the Instana playground; IBM registration or existing access may be required.
• Use the supplied synthetic packet if access or relevant telemetry is unavailable.

Open Instana playground

Slide 4: Three pillars: Logs, Traces, Metrics

Start with events, request paths, and measurements over time.

• Logs: structured, timestamped events explain what happened; correlate by trace and operation ID.
• Traces: related spans show where one request spent time across service boundaries.
• Metrics: numerical time series show how broadly users and resources are affected.

Slide 5: Time series analysis: detect and compare change

A time series is timestamped values for a metric and a fixed dimension set.

• Look for trends, spikes, step changes, seasonality, and lag using aligned baseline and incident windows.
• A counter rising by 600 in 60 seconds means 10 requests/s; 12 errors among those requests means 2%.
• Check resets, missing data, chart rollups, and units; do not average instance p95 values.

Slide 6: Built-in metrics: a useful starting point

Supported sensors and runtime instrumentation expose signals before custom business instrumentation.

• Use call rate, error ratio, and latency to locate affected services and intervals.
• Inspect available CPU, memory, restarts, JVM/GC, or event-loop metrics to test resource hypotheses.
• Coverage depends on configuration and supported versions; add business metrics for checkout success and safe fallbacks.

Slide 7: One trace across Node.js, Java, and Python

Language-independent context connects instrumented service spans.

• Inject and extract compatible trace context at every HTTP or messaging boundary.
• Keep the trace ID across the request; each span has its own ID and service identity.
• Missing spans may mean lost context, unsupported instrumentation, sampling, or export gaps.

Slide 8: A timeout is an unknown outcome

The caller stopped waiting; the downstream operation may still commit.

• Distinguish a crash, a slow dependency, a dropped response, and resource exhaustion.
• Retrying a payment requires the same operation identity and durable deduplication.
• A healthy process does not prove that a user journey is succeeding.

Slide 9: One failure, several affected services

An entry service can become unavailable while its dependency is merely slow.

• Requests occupy connection pools and worker slots while waiting.
• Queues grow; callers retry; useful work competes with repeated work.
• Find the failing boundary before blaming every service with elevated latency.

Slide 10: Start with the user journey and SLO

Define good requests before selecting a dashboard.

• Example SLI: eligible checkout requests that succeed within 1 second / all eligible checkout requests.
• Example SLO: 99% good requests over a rolling 7-day window; define exclusions explicitly.
• For 10,000 eligible requests, 99% allows 100 bad requests; 160 bad requests exceed that budget by 60.

Slide 11: Deadlines bound waiting

Give a whole operation one time budget and propagate the remaining budget.

• Example: 1,000 ms total; reserve 150 ms for gateway and checkout work.
• Two 350 ms attempts plus up to 100 ms backoff fit within the remaining 850 ms.
• Actual work, queueing, and network time consume the budget; stop if the remaining time is insufficient.

Slide 12: Safe retries need a budget

Retry only selected transient failures when the operation is safe to repeat.

• Use a bounded attempt count, exponential backoff, and jitter.
• Put retry ownership at one deliberate layer; avoid multiplicative retries.
• Three layers with three attempts each can send 27 calls to one dependency.

Slide 13: Circuit breakers fail fast

A breaker temporarily rejects calls to an unhealthy dependency.

• Closed: calls run; a defined failure threshold opens the breaker.
• Open: reject calls quickly; after a cooldown permit limited probes.
• Half-open: recover after successful probes or reopen after failure.

Slide 14: Bulkheads and backpressure protect capacity

Isolate work and bound admission rather than allowing unlimited queues.

• Bulkhead: separate worker or connection budgets by dependency or workload.
• Backpressure: bound queues, reject excess work, or slow producers.
• Measure queue depth, queue age, in-flight work, saturation, and rejection rate.

Slide 15: Instrument the recovery policy itself

Show whether protection reduced user impact or hid the failure.

• Record attempts, deadline exceeded, breaker state, rejections, and fallback results.
• Propagate trace context; correlate logs without using trace IDs as metric labels.
• Restrict telemetry access; redact tokens, personal data, and tool prompt content.

Slide 16: Walkthrough 1: establish a baseline in Instana

Choose an available application and freeze one populated time window.

• Open Applications or application monitoring; select a populated application perspective.
• Compare equal baseline and incident windows; record built-in metrics, units, aggregation, and resolution.
• Open Services or dependencies; draw one entry → downstream path and note missing visibility.

Baseline walkthrough and worksheet

Slide 17: Walkthrough 2: from symptom to trace

Compare one slow or erroneous call with one successful call in the same scope.

• Open the selected service; use Analyze Calls or the available call/trace view.
• Filter by service and endpoint; inspect a slow or erroneous call and its trace timeline.
• Record trace/span relationships, service/runtime metadata where visible, and any propagation gaps.

Trace walkthrough

Slide 18: Walkthrough 3: correlate infrastructure and logs

Test a hypothesis while preserving the time window.

• From the service or trace, open related infrastructure when available.
• Compare built-in resource/runtime time series; distinguish trend, spike, lag, and missing data.
• Inspect correlated logs if available; otherwise identify the event fields needed to test the hypothesis.

Correlation walkthrough

Slide 19: Exercise 1: diagnose before prescribing

Pairs: choose a request path and produce a falsifiable incident hypothesis.

• Record two observations across the three pillars, including one time series comparison with units.
• Separate affected service from suspected dependency; identify missing evidence.
• Propose one bounded mitigation and one observation that would disprove your diagnosis.

Exercise 1 instructions

Slide 20: Exercise 2: calculate a safe recovery policy

Design exercise; these settings are not changes to the shared sandbox.

• Allocate the 1,000 ms deadline; choose retry ownership, attempt limit, and idempotency rule.
• Choose a breaker probe policy, bounded queue, and a dependency bulkhead.
• Predict which telemetry would change and which user outcome must remain correct.

Exercise 2 and instructor checks

Slide 21: Synthetic incident: degraded checkout

Fallback packet only; these values were not collected from Instana.

• 10,000 eligible requests: 9,840 good; 160 bad; 98.4% good against a 99% SLO.
• Checkout trace: 920 ms root; payment attempts at 100–450 ms and 520–870 ms.
• Payment CPU stays flat while connections saturate; this does not prove a CPU bottleneck.

Synthetic packet and questions

Slide 22: Mitigate, then verify recovery

A quieter dashboard is insufficient without user-outcome evidence.

• Compare equivalent before/after windows, traffic, endpoints, and SLI definitions.
• Check good-request ratio, tail latency, attempt amplification, queue age, and rejected work.
• Use a time-bounded experiment with rollback criteria; watch for stale or misleading fallbacks.

Slide 23: Transfer the method to your project

Apply one pattern to one dependency and capture evidence.

• In your own local environment: establish baseline, inject bounded latency or outage, then restore it.
• Keep traffic and observation windows comparable; capture traces and recovery behavior.
• For agent tool calls: bound retries and concurrency, preserve operation identity, and require approval for consequential recovery actions.

Slide 24: Debrief: defend the evidence

Explain what you know, what you infer, and what you would check next.

• Why can a timeout still produce a committed payment?
• Why can a mean hide a latency spike, and why might a cross-language trace be incomplete?
• How could a breaker reduce latency while the good-request ratio stays poor?

Slide 25: Takeaways and next meeting

Reliable systems contain failures and make recovery observable.

• Bound time, repeated work, concurrency, and queue growth.
• Connect user impact to a request path; validate claims with aligned telemetry.
• November 16: Service Networking with Hashicorp Consul - A workshop. November 23: No class (Thanksgiving holiday).