Discovery
Resolve a stable name or registry entry into candidate network endpoints.
CMPE273 · Module 6
Discovery, Load Balancing, and Cloud-Native Runtime Patterns
Trace a request through service discovery, balancing, health checks, and orchestration. Explore Gateway API, modern data planes, and a recoverable agent runtime.
Learning Objectives
- Identify containers, nodes, Pods, Deployments, and Services in a small Kubernetes application.
- Distinguish discovery, routing, load balancing, authentication, and authorization.
- Explain Service-to-Pod routing and why connection reuse can skew traffic.
- Design health checks, graceful shutdown, and scaling around failure behavior.
- Explain why a service mesh is useful and compare Istio sidecar and ambient placement.
- Apply durable state and constrained tool discovery to an agent workflow.
Core Concepts
Balancing unit
The connection, request, or stream selected for assignment to a backend.
Readiness
An instance’s eligibility to receive normal traffic; distinct from a restart decision.
Reconciliation
A control loop that repeatedly moves observed resources toward desired state.
Workload identity
A verifiable identity used to authenticate a running service, separate from its address.
Durable workflow
Recoverable progress and operation identity shared across disposable workers.
Diagrams
From your declaration to a running application
flowchart LR A["You + kubectl"] -->|"submit YAML"| C["Kubernetes API<br/>desired state"] C --> B["Controllers + scheduler"] B --> N["Node: kubelet + runtime"] N --> P["Pod<br/>application container"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d class C control
A cluster contains a control plane and worker nodes. Control-plane components arrange the work; node software starts containers. Application requests do not pass through kubectl.
A failed Pod triggers a repair loop
flowchart LR C["Controller<br/>desired 3, sees 2"] --> N["Create Pod"] N --> S["Scheduler<br/>selects node"] S --> K["Kubelet<br/>starts container"] K --> R["Ready<br/>join endpoints"] R -. "observed state" .-> C classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class C,S control
Reconciliation is asynchronous. A new Pod needs scheduling, image startup, and readiness before it can receive traffic. No step restores a discarded local checkout session.
Two places to select a backend
flowchart LR R["Registry or DNS"] -. "endpoint updates" .-> C["Client library"] C --> A["Service replica A"] R -. "endpoint updates" .-> P["Proxy or gateway"] U["Simple client"] --> P P --> B["Service replica B"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class R,P control
The upper path puts endpoint selection in the client; the lower path delegates it to a proxy. Both must handle stale membership and failed connections.
Names, membership, and traffic are different paths
flowchart LR C["Client"] -->|"DNS lookup"| D["Cluster DNS"] D -. "Service IP" .-> C C -->|"new connection"| V["Service virtual IP<br/>data plane"] V --> P["Ready Pod"] E["EndpointSlice<br/>addresses + conditions"] -. "updates" .-> V K["Pod labels and readiness"] -.-> E classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class D,E,K control
DNS locates the Service; it is not in the path of every request. EndpointSlice updates inform routing, while the data plane carries application traffic.
Same three requests, different selection units
flowchart LR C["Client<br/>requests 1, 2, 3"] -->|"one TCP connection"| L["L4 balancer"] L --> A["Backend A<br/>all three requests"] D["Client<br/>requests 1, 2, 3"] -->|"one HTTP/2 connection"| P["L7 proxy"] P --> B["Backend A: 1, 3<br/>Backend B: 2"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d class L,P control
Illustrative routing, not a guaranteed distribution. The L7 proxy uses separate upstream connections; it does not split one TCP connection across servers. An active stream stays with its selected backend.
Probe outcomes and their consequences
flowchart LR S["Startup passes"] --> R["Readiness fails"] S --> L["Liveness fails<br/>past threshold"] R --> N["Remove from<br/>normal routing"] L --> X["Restart<br/>container"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class N,X warning
A configured startup probe gates the other probes until success. Failing readiness does not by itself restart a container; liveness failure can.
Locate, authenticate, then authorize
flowchart LR S["Runtime credential<br/>restricted delivery"] --> C["Ordering"] C -->|"locate"| D["Inventory<br/>Service name"] D -->|"request"| P["Verify identity<br/>authorize operation"] P -->|"allowed"| B["Business rules"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d class S,P control
Service discovery and authorization are separate controls. Runtime credentials identify the caller; Inventory still checks permission and enforces stock invariants.
Illustrative rollout: keep useful capacity available
flowchart LR A["v1 serves"] --> B["v2 starts"] B --> C["v2 ready"] C --> D["Terminate v1"] D --> E["Routing updates<br/>and draining"] E --> F["v1 exits"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class D,E warning
This is a desired rollout sequence, not an atomic handoff. Existing connections and delayed routing updates require explicit shutdown behavior and testing.
Scaling is a feedback loop
flowchart LR W["Work arrives"] --> P["Ready replicas"] P --> M["Load metrics"] M --> H["Autoscaler<br/>desired count"] H --> D["Scheduling<br/>startup delay"] D --> P classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class M,H control
Metrics describe past load; useful capacity arrives later. Set bounds and stabilization, and check whether the database or provider quota is the true bottleneck.
Platform ownership and application routing
flowchart LR R["HTTPRoute<br/>matches + weights"] -. "configures" .-> G["Gateway<br/>selected GatewayClass"] U["Client"] --> G G -->|"90 weight"| A["Stable Service"] G -->|"10 weight"| B["Canary Service"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d class R,G control
The weights illustrate a canary policy, not exact counts in a small sample. The selected controller reconciles these resources into actual routing behavior.
The control plane configures; the data plane carries requests
flowchart LR A["Ordering app"] --> P["Proxy"] --> Q["Proxy"] --> B["Inventory app"] C["Istio control plane"] -. "identity + policy" .-> P C -. "identity + policy" .-> Q classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d class C control
The proxies authenticate peers, enforce configured policy, and measure traffic. Istio’s control plane distributes configuration; business requests flow through the data plane, not through that control plane.
Two ways to place the mesh data plane
flowchart LR A["Sidecar mode<br/>application Pod"] --> B["Local Envoy<br/>L4 + L7"] --> D["Destination proxy + app"] E["Ambient mode<br/>application Pod"] --> C["Node ztunnel<br/>L4 secure overlay"] --> W["Waypoint when needed<br/>L7 policy"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d class C control
These are simplified placement paths. Ambient also uses destination-side ztunnel handling; a waypoint is not required for the L4-only path. Feature support and traffic enrollment must be checked for the deployed version.
Observe three different states
flowchart LR C["Client"] --> S["Stable Service"] S --> A["Pod A<br/>ready"] S -. "routing removed" .-> B["Pod B<br/>not ready"] D["Controller"] --> R["Replacement<br/>after deletion"] R -. "when ready" .-> S classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class B warning class D control
Readiness failure changes traffic eligibility without replacing the Pod. Deletion triggers replacement. A Service name stays stable across both experiments.
Model eligibility before endpoint selection
flowchart LR A["Agent request<br/>model + token budget"] --> G["Compatible inference gateway"] G --> P["Endpoint picker<br/>eligible model replicas"] M["Serving metrics"] -.-> P P --> R1["Model replica A"] P --> R2["Model replica B"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class G,P,M control
The picker chooses among eligible model servers. Cache reuse may save work, but sending everything to one warm replica can worsen queueing.
Three gates before a tool call
flowchart LR A["Agent runtime"] --> L["Locate server"] L --> C["Inspect tools<br/>or Agent Card"] C --> P["Verify identity<br/>check permission"] P --> T["Invoke tool"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class C,P control
Capabilities describe available interfaces. Identity and policy determine allowed actions; untrusted descriptions or tool results cannot grant new permissions.
The agent proposes; the execution boundary enforces
flowchart LR P["Disruption<br/>planner reacts"] --> V["Store proposal<br/>exact-version approval"] V --> X["Executor<br/>check constraints"] X --> T["Booking tool<br/>operation key"] T --> R["Record result<br/>or reconcile"] R -. "durable progress" .-> V classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class V,X control class R data
Discovery and balancing keep workers reachable. Durable state, concurrency control, approval checks, and provider repeat handling preserve the booking invariant.
Examples, Discussions, and Project Ideas
Slide 2: Mini Kubernetes: where the technology fits
Kubernetes (K8s) runs containerized applications across a cluster of machines.
- An image packages an application; a container runtime starts it. Kubernetes schedules, replaces, and connects running instances across nodes.
- kubectl is your command-line client. Use kind, minikube, or Docker Desktop for local practice; managed cloud clusters provide the same core Kubernetes APIs.
Slide 3: Mini Kubernetes: declare, inspect, connect
Four objects make the lab readable: Namespace, Deployment, Pod, and Service.
- A Namespace groups resources. A Deployment maintains the requested replicas; each Pod is a scheduled unit containing one or more containers.
- A Service gives selected Pods a stable network entry point. Apply the lab YAML, inspect Pods and the Service, then send a request from the client Pod.
Slide 4: Discovery: where should the client connect?
Names outlive instances; discovery information can still become stale.
- DNS maps a name to an address or set of addresses. Caches and connection reuse delay how quickly a caller sees change.
- Client-side discovery lets a library choose from endpoints. Proxy-side discovery centralizes selection, while adding a component to operate.
Slide 5: Kubernetes Services and EndpointSlices
A Service gives changing Pods a stable access point.
- A normal ClusterIP Service has a virtual IP. DNS resolves its service name; the data plane forwards connections to eligible endpoints.
- Label selectors connect a Service to Pods; EndpointSlices track their addresses and conditions. A headless Service exposes endpoint addresses without a virtual IP.
Slide 6: L4 connections versus L7 requests
A connection is a reusable channel; a request is one operation sent through it.
- L4 = transport layer: select a backend using connection information such as IP addresses and TCP ports. Keep that connection on its selected backend.
- L7 = application layer: understand HTTP paths, headers, or gRPC methods. Select a backend for each request, even when requests share a client connection.
Slide 7: Choose a balancing policy for the work
Equal request counts do not imply equal cost or equal latency.
- Round robin rotates eligible targets: simple when requests and instances have similar cost and capacity.
- Least-request policies favor less busy targets. Request counts help, but a long model generation may cost much more than a short lookup.
- Weighted routing allocates traffic shares, useful for canaries or unequal capacity. A small sample will not exactly match the configured percentages.
- Consistent hashing improves affinity to a key. Hot keys can overload one target; affinity does not replicate session state.
Discuss: Replica A has one 30-second generation; B has three 10-millisecond reads. Is active-request count enough?
Slide 8: Startup, readiness, and liveness
Each probe answers a different operational question.
- Startup asks whether initialization finished. Readiness decides whether to receive normal Service traffic; liveness can trigger a container restart.
- Do not restart every replica because one shared provider is unavailable. Make liveness about local progress and readiness about the work this instance can serve.
Slide 9: Retries must fit a deadline and a budget
Failure handling can overload the very service it is trying to recover.
- Three layers making up to three attempts each can generate 27 backend attempts for one original request.
- Assign retry ownership, preserve the remaining deadline, and use backoff with jitter. Retry mutations only with a safe repeat contract.
- Bound concurrency and queues; reject excess work visibly. A circuit breaker temporarily avoids a failing dependency while a bulkhead protects other work.
- A timeout means the result may be unknown. Keep the operation key and reconcile, as in the payment workflow from Module 4.
Slide 10: Configuration and workload identity
A discovered address tells you where; verified identity tells you who.
- Keep ordinary configuration separate from the image. Deliver secrets at runtime with restricted access and rotation; base64 is not encryption.
- Authenticate workloads with verified credentials, then authorize the specific operation. A DNS name or caller-supplied header is not proof of identity.
Slide 11: Rolling updates and graceful draining
A new version must become ready before an old replica disappears.
- Rolling updates trade spare capacity against availability through maxSurge and maxUnavailable. Readiness keeps a warming replica out of normal traffic.
- During termination, endpoint updates and shutdown overlap. Handle termination signals, stop accepting new work, and finish or checkpoint within the grace period.
Slide 12: Autoscaling follows signals, with delay
Add replicas for the bottleneck you can actually relieve.
- HPA adjusts desired replicas from metrics. A simplified example: 3 replicas at 80% CPU against a 60% target suggest ceil(3 × 80/60) = 4.
- KEDA can activate event-driven workers from external signals. Pending work, oldest-item age, and safe concurrency often explain worker pressure better than CPU alone.
Slide 13: Gateway API separates routing responsibilities
Traffic policy is an API contract implemented by a controller and data plane.
- GatewayClass identifies an implementation. A Gateway describes listeners; routes attach application traffic rules to those listeners.
- HTTPRoute can express HTTP matching and weighted backends. Confirm controller support and route status; installing API definitions alone does not forward traffic.
Slide 14: Why a service mesh? Why Istio?
Make service-to-service security, traffic policy, and telemetry consistent.
- Without shared infrastructure, every team must configure credentials, timeouts, and network telemetry in each language. A mesh moves selected controls into managed proxies.
- Istio configures those proxies and workload identities. Use it for consistent mTLS, access policy, and traffic control; weigh its operational cost against your actual needs.
Slide 15: Istio deployment choices: sidecar and ambient
The mesh capabilities you need determine where proxies must run.
- Sidecar mode places an Envoy proxy alongside the application in each enrolled Pod. It can handle both transport security and application-aware traffic policy.
- Ambient mode uses a per-node ztunnel for L4 security. Add waypoint proxies for workloads needing L7 policy and routing; sidecar-free still includes proxies.
Slide 16: Lab: remove a replica from traffic
Predict what changes before running the failure experiment.
- Send repeated requests through one Service name to two Pods; record the returned Pod names without expecting exact alternation.
- Fail readiness on one Pod, then restore it. Delete a Pod next: compare routing removal with controller replacement and recovery time.
Slide 17: Inference routing needs workload context
A model request may occupy an accelerator far longer than a normal API lookup.
- Route only to replicas that serve the requested model. Queue pressure and cache locality can inform selection when the serving stack exposes them.
- Gateway API Inference Extension adds inference-oriented selection to compatible gateways. Evaluate the exact release and implementation, not just the project name.
Slide 18: Agent discovery includes capabilities and trust
Finding an endpoint and finding a useful tool are different discovery problems.
- DNS or a Service locates the server. MCP tools/list describes tools and input schemas; A2A Agent Cards advertise agent capabilities and connection details.
- An advertised capability is not permission to use it. Validate the server, authorize each operation, and keep an explicit approved tool set.
Slide 19: A recoverable travel-agent runtime
Use dynamic reasoning inside a workflow with durable, enforceable boundaries.
- A planner proposes an alternative; approval applies to that exact proposal version. An executor revalidates constraints before a booking call.
- Replicas share durable workflow state and a stable provider operation key. A lost response enters reconciliation rather than a fresh booking attempt.
Slide 20: Architecture review
A good diagram explains where a request goes and what happens when that path fails.
- Show the stable name, discovery owner, balancing unit, health signals, and shared failure domains for one service.
- Explain rollout and draining, scaling signals, workload permissions, and the durable state needed when a replica disappears.
- For the agent extension, remove an executor after provider success but before its response is recorded. Show how the outcome is recovered without a second booking.
- Bring an annotated architecture and expected-versus-observed evidence. Keep deadlines, grading, and submissions in Canvas.
Discuss: Which runtime control improves availability, and which application rule preserves correctness? Identify both for your failure experiment.
Canvas remains the official source for project requirements, deadlines, submissions, and grading.
Key Takeaways
- Stable names hide changing instances, but discovery and health updates take time.
- Balance at the right unit: replicas do not redistribute an existing connection.
- Readiness, restarts, draining, and scaling have distinct jobs.
- Gateway API, eBPF, and mesh are implementation choices with explicit prerequisites.
- Agents need authorized capabilities and durable execution rules in addition to healthy endpoints.
Lab Links
- Runtime lab and paper alternative
Observe two replicas, readiness removal, replacement, and recovery.
- Architecture review
Connect runtime choices to a measurable failure experiment.