Module 9: Fault Tolerance, Observability, and Operational Diagnosis

Session goal and preparation

This combined lesson covers fault tolerance and observability in one session. The 25-slide presentation allocates 140 minutes to concepts, walkthroughs, exercises, and discussion, plus a 10-minute break after slide 15. Use Present Module for classroom teaching and Student Handout for printable slide notes, diagrams, and the exercise links.

Required preparation: Review timeout ambiguity and idempotency in the transaction reference session. Bring a browser and a place to record evidence. No local Instana installation is required. Work in pairs, alternating driver and investigator.

Access check: The Instana playground redirected to an IBM Instana Observability Sandbox registration form when checked on November 9, 2026. Arrange access before class or follow the instructor’s shared screen. Registration is a student/instructor choice; use the synthetic packet below if access is unavailable. The authenticated demo’s current service names, scenarios, and controls have not been verified. These walkthroughs follow linked IBM documentation; labels may vary. No sandbox fault injection or configuration changes are required.

Three pillars of observability: Logs, Traces, Metrics

Start with three complementary views of the same system. They are commonly called the three pillars; collecting all three is useful only when they help explain a user outcome. A dashboard alone does not answer every question.

For a slow checkout, begin with the latency and error time series to locate the affected interval. Open a trace from that interval to find the delayed dependency. Follow the trace ID into structured logs to inspect the attempt outcome. Return to metrics to determine whether many users were affected and whether recovery worked. Keep service, endpoint, instance, and time window aligned.

Time series analysis: from measurements to a hypothesis

A metric time series is a sequence of timestamped values for one metric and one dimension set, such as CPU usage for one service instance. Changing service, instance, or region selects a different series. Start with a question, choose the relevant series, and record units, collection interval, aggregation, and missing-data behavior.

• Trend: Is queue age growing steadily, suggesting arrivals exceed completed work?
• Spike or step change: Did error ratio change suddenly near a deployment or dependency event? That timing suggests a hypothesis, not proof of cause.
• Seasonality: Is this the normal daily demand peak? Compare like-for-like workload periods before declaring an anomaly.
• Lag: Does pool saturation rise before upstream latency? Inspect a trace or event to test the proposed chain.

Worked example: counters, gauges, and latency distributions

The following values are synthetic teaching data. A cumulative request counter increases from 12,000 to 12,600 over 60 seconds: (12,600 - 12,000) / 60 = 10 requests/second. If an error counter increases by 12 over the same interval and scope, the error ratio is 12 / 600 = 2%. Do not divide cumulative lifetime errors by the current minute’s requests. A counter reset after restart needs reset-aware handling; a negative delta is not negative traffic.

Connection pool usage of 100 out of 100 is a gauge, a current quantity. Request duration is a distribution; p95 highlights a slow tail that a mean can hide. Do not average instance p95 values to obtain a service p95: combine compatible distribution data or use a correctly aggregated service view. Likewise, compute fleet error ratio from summed errors and summed requests rather than averaging per-instance percentages.

Compare equal-duration baseline and incident windows with the same endpoint, instance grouping, traffic mix, timezone, and chart resolution. A five-minute rollup may hide a brief outage visible in smaller buckets. Missing points mean unknown collection or availability, not necessarily zero. Use historical baselines when available; one ten-minute window cannot establish seasonality.

How built-in metrics help

Built-in metrics from supported infrastructure sensors and application/runtime instrumentation give a starting point before adding custom business telemetry. With the necessary agents, sensors, or tracers configured, Instana associates supported measurements with the monitored entity and provides service and infrastructure views. Coverage depends on the technology, version, configuration, and access; inspect what the environment actually exposes. See IBM’s supported technologies and infrastructure metrics documentation.

Use these signals to narrow the investigation, then add custom measurements for what the platform cannot infer: successful checkout, duplicate-payment prevention, fallback correctness, attempts per logical operation, or remaining workflow budget. Keep metric labels bounded; put individual order and trace IDs in logs or traces. Automatic collection reduces setup effort but does not replace business instrumentation or validation.

Traces across applications written in multiple languages

flowchart LR
  N[Node.js gateway] -->|trace context| J[Java checkout]
  J -->|trace context| P[Python payment]
  N -. spans .-> B[Tracing backend]
  J -. spans .-> B
  P -. spans .-> B

Consider an illustrative path: Node.js gateway → Java checkout → Python payment. Language boundaries do not require separate logical traces. A shared trace ID and correct span relationships connect the instrumented operations. Each service has its own span IDs and identifies itself with a distinct service name and relevant runtime/version attributes.

1. The gateway starts a server span and its outgoing client span injects trace context into the HTTP request.
2. Checkout extracts that context, starts a child server span, and injects context on its call to payment.
3. Payment extracts context and creates its own spans; related structured logs include trace and span IDs.
4. Instrumentation exports spans to a compatible backend. Consistent context propagation and identity allow the backend to reconstruct the request path across languages.

For OpenTelemetry with W3C Trace Context, the traceparent header carries the trace ID, parent span ID, and trace flags; tracestate may carry additional vendor state. traceparent is tracing context, not authentication. The HTTP libraries and propagators on both sides must agree; do not assume instrumentation for one service automatically instruments its peers. See OpenTelemetry context propagation.

Instana supports tracing for multiple technologies and can integrate OpenTelemetry spans with its native tracing. Verify supported language/runtime and library versions, instrumentation settings, and propagation compatibility before claiming an end-to-end trace. See Instana tracing and OpenTelemetry integration. Choose the instrumentation approach deliberately to avoid duplicate spans from overlapping agents.

A missing Python span could mean uninstrumented code, a stripped header, an unsupported client library, lost context across an asynchronous task, sampling, or an export problem. It does not prove that payment was never called. Across a queue, propagate context through supported message metadata; consumer processing may use a parent relationship or span links depending on the messaging model. Do not infer causality solely by sorting wall-clock timestamps from different hosts.

Mini-check: If gateway and checkout share a trace ID but payment starts a new trace, where would you inspect first? Check the outgoing request’s propagation, the receiving library’s extraction, and async context handling before blaming the language boundary. In the live sandbox, record languages only when runtime metadata shows them; use this illustrative path if the available demo is not multilingual.

Core concepts: contain failure, then explain it

A payment can commit even after checkout stops waiting. A timeout therefore means an unknown outcome, not a guaranteed rollback. Preserve the same idempotency key for a repeated logical payment, and make duplicate detection atomic with the business effect. A new key can cause a second charge.

Use a whole-request deadline and propagate remaining time. Retry only transient failures when replay is safe, with one retry owner, bounded attempts, exponential backoff, and jitter. A circuit breaker bounds exposure to a persistently unhealthy dependency. Bulkheads isolate worker and connection capacity; backpressure bounds incoming work and queue growth. A fallback must preserve business meaning: showing a stale catalog is different from claiming an unconfirmed payment succeeded.

flowchart LR
  U[Client] --> G[Gateway]
  G --> C[Checkout]
  C --> P[Slow payment dependency]
  C --> I[Inventory]
  G -. metrics and traces .-> O[Telemetry]
  C -. metrics logs traces .-> O
  P -. spans when instrumented .-> O

stateDiagram-v2
  [*] --> Closed
  Closed --> Open: Failure threshold
  Open --> HalfOpen: Cooldown elapsed
  HalfOpen --> Closed: Successful probes
  HalfOpen --> Open: Probe fails

Worked example: deadline and error budget

For a 1,000 ms deadline, reserve 150 ms for gateway/checkout work. The remaining 850 ms can accommodate two attempts of at most 350 ms plus at most 100 ms of backoff, leaving 50 ms of margin. Queueing, network overhead, and elapsed work also consume the budget; compute the remaining time before each attempt. An exhausted deadline prevents a retry even if the attempt limit has not been reached. Cancellation does not undo a committed write.

Define a good request as an eligible checkout request that completes successfully within 1 second. With a 99% SLO and 10,000 eligible requests, the budget is 100 bad requests. If 160 requests are bad, the good-request ratio is 98.4%; the window exceeds its budget by 60 requests. Count each bad request once even if it is both slow and unsuccessful. A single exercise window does not establish a rolling 7-day SLO result.

Metrics locate impact; traces reveal request paths; structured logs explain events. Compare the same service, endpoint, time window, timezone, and units. Parent span time includes child execution and waiting; overlapping spans cannot simply be added. Redact sensitive payloads and tokens, and use trace IDs for correlation rather than high-cardinality metric labels.

Walkthrough 1: baseline

Time: 10 minutes. Goal: establish the scope before investigating a symptom.

1. Open the playground. With access available, navigate to Applications or the application monitoring area and select a populated application perspective. Record the actual name rather than assuming a sample app exists.
2. Select a recent populated interval, then freeze an absolute start and end time. Record the UI timezone. If the interval is empty, select another populated interval before collecting evidence.
3. Record request/call volume and its aggregation, erroneous-call rate, and latency with the exact displayed statistic and unit. A mean is not p95; a rate is not a total count.
4. Open Services, Dependencies, or the available dependency view. Identify an entry service and one downstream service. Sketch the path and note which boundaries lack telemetry.
5. Compare an equal-duration baseline interval using the same filters and chart resolution. Identify a trend, spike, or step change, or report none. Record collection gaps; do not substitute zero. Compute a request rate or error ratio from compatible counts if available. Restore the incident interval before proceeding.

Expected result: a reproducible scope and two connected services. A dashboard showing errors identifies impact; it does not yet identify a cause.

Walkthrough 2: trace investigation

Time: 10 minutes. Goal: connect a user symptom to a specific request path.

1. From the selected service, open Analyze Calls or the available calls/traces view. Filter by the chosen service and endpoint; confirm the interval stayed fixed.
2. Choose a slow or erroneous call and open its trace. Record trace ID, timestamp, root duration, service path, status/error detail, and the longest relevant child span. If no erroneous calls exist, use a slow successful call.
3. Compare a typical successful call for the same endpoint and interval. Explain whether the difference is in downstream waiting, local work, or a visibility gap. Record service names, trace IDs, span/parent relationships, and runtime languages only where metadata confirms them. If a boundary starts a disconnected trace, list propagation and instrumentation checks; do not assume a language mismatch.
4. Look for repeated calls to the same dependency. Label them as possible retries until operation identity or an explicit attempt event confirms the relationship.
5. Write one observation, one hypothesis, one alternative explanation, and one missing measurement. Do not infer an unseen breaker state from short latency alone.

Expected result: two referenced traces and a bounded hypothesis. If trace details are unavailable, record that limit and use the synthetic packet; do not claim a live diagnosis.

Walkthrough 3: correlation

Time: 10 minutes. Goal: test rather than merely repeat a hypothesis.

1. Follow a service/trace link to related infrastructure if it is available. Confirm the same interval and service instance before comparing CPU, memory, restarts, or connection/thread saturation.
2. Select one available built-in resource/runtime metric and compare its baseline and incident time series, including units, aggregation, chart resolution, and gaps. Explain which custom business metric would still be needed to establish successful checkout. Decide whether the resource signal supports your hypothesis. Normal CPU does not rule out a connection bottleneck, network latency, or remote dependency wait.
3. Open correlated logs if exposed. Find the matching trace or operation ID and order relevant events. If logs are unavailable, write the fields you would need: timestamp, service/instance, trace ID, operation ID, attempt, outcome, remaining deadline, and redacted error category.
4. Record one corroborating signal and one contradiction or missing signal. An event occurring at the same time is not proof of cause.

Expected result: a trace plus a second signal, or a precise instrumentation gap. Shared demo data may not expose logs, queue metrics, or breaker state.

Exercise 1: evidence-based diagnosis

Time: 12 minutes, pairs. Use the chosen live path or the labeled synthetic packet. Spend 4 minutes collecting evidence, 4 minutes testing a hypothesis, and 4 minutes writing a six-line incident brief.

1. User symptom and affected endpoint.
2. Application/service names, absolute window, and timezone.
3. Two observations from different pillars, with units, trace/event references, and one baseline-to-incident time series comparison.
4. Suspected failure boundary, alternative explanation, and confidence.
5. One bounded mitigation and its correctness risk.
6. One falsifying check and one recovery measurement.

Success check: another pair can reproduce your scope, distinguish observations from inferences, and explain what would change your conclusion. Choosing “insufficient evidence” with a specific next check is valid.

Exercise 2: bounded recovery

Time: 10 minutes, pairs. This is a policy design exercise, not a request to modify the shared Instana environment.

Design recovery for checkout calling payment under a 1,000 ms deadline. Allocate local time, attempts, backoff, and margin; choose the retry owner and operation identity. Add a breaker policy with a failure window, minimum sample size, cooldown, and bounded probes. Add an explicit payment concurrency limit and queue bound. Explain what to reject or defer when capacity is exhausted. Choose one safe fallback or explain why no success fallback is valid.

Predict three signals that would change: attempts per logical operation, queue age/concurrency, and good-request ratio. Explain how you would distinguish reduced retry amplification from reduced incoming demand. State rollback criteria before proposing a change.

Instructor checks: the time budget adds up; no retry starts after deadline exhaustion; repeated payments retain the same operation identity; exactly one layer owns retries; rejection is observable and counts according to the SLI; a breaker cannot fabricate payment success. Breaker thresholds and capacity limits require workload evidence, so numerical choices here are hypotheses to test.

Synthetic incident packet

Fallback only: all values below are invented for instruction and were not observed in Instana. Use the same steps and worksheet as the live exercises.

Scope: application teaching-checkout, endpoint POST /checkout, interval 18:00–18:10 America/Los_Angeles. There are 10,000 eligible logical requests: 9,840 succeed within 1 second and 160 are bad by the SLI. Mean latency is 240 ms; p95 is 760 ms; p99 is 1,400 ms. These percentiles alone do not determine the good-request ratio.

The baseline window is 17:50–18:00 in the same timezone. Traffic volume is held equal in this teaching example; request mix and all other factors still require validation in a real comparison.

Trace demo-slow-01: root checkout span 0–920 ms; payment attempt 1 at 100–450 ms ends with a deadline error; attempt 2 at 520–870 ms succeeds. Trace demo-fast-02: root 0–180 ms; payment at 60–130 ms succeeds. A synthetic log links both slow-trace attempts to operation order-42 and the same payment idempotency key. No breaker state or lock-wait metric is provided.

Questions: What proves repeated work? What indicates saturation? Is CPU exhaustion supported? Is a retry storm alone proven? What evidence would distinguish lock contention, a remote delay, and a connection leak? Would you retry another non-idempotent payment?

Reasoning check: attempt amplification rises from 1.01 to 1.50 at unchanged logical volume. The pool and queue signals support saturation; nearly flat CPU does not support CPU exhaustion. Explicit attempt logs confirm a retry for the selected operation, not that retries caused every incident request. A DB wait trace, dependency latency, connection lifecycle metrics, and broader attempt distribution would test competing explanations. The 920 ms successful root is good under this SLI; the full population still misses the example 99% target.

Evidence worksheet

Copy one worksheet per pair. Mark each field live, synthetic, or unavailable.

Project transfer and optional extension

In your own local project environment, measure a baseline, inject bounded latency or a temporary dependency outage, then restore normal service. Keep traffic, endpoint, and observation windows comparable. Record attempts per logical operation, deadline behavior, queue age, error ratio, and traces before and after recovery. Limit the blast radius and stop if the expected recovery does not occur. This extension does not require changing the shared Instana sandbox.

An agent making a tool call needs the same deadlines, concurrency bounds, and operation identity. Trace the approved operation without logging credentials or sensitive prompts. Keep consequential remediation behind the project’s approval boundary.

The exercises are learning activities. Canvas remains the official system for grades, submissions, quizzes, exams, announcements, and individualized feedback.

Resources

Required foundations: OpenTelemetry context propagation, Instana infrastructure metrics, and Instana tracing.

Required: IBM’s service dashboard guide and trace/call analysis guide explain the views used above. Optional: infrastructure correlation supports the third walkthrough.