1 / 20
Sections
Slide 1
Discovery, Load Balancing, and Cloud-Native Runtime Patterns
Module 6 · CMPE273 · October 5, 2026
- How does a request reach a healthy service when instances keep changing?
- Follow the campus marketplace into Kubernetes, then extend the same runtime ideas to an agent that uses tools.
Slide 2
Mini Kubernetes: where the technology fits
Kubernetes (K8s) runs containerized applications across a cluster of machines.
- An image packages an application; a container runtime starts it. Kubernetes schedules, replaces, and connects running instances across nodes.
- kubectl is your command-line client. Use kind, minikube, or Docker Desktop for local practice; managed cloud clusters provide the same core Kubernetes APIs.
flowchart LR A["You + kubectl"] -->|"submit YAML"| C["Kubernetes API<br/>desired state"] C --> B["Controllers + scheduler"] B --> N["Node: kubelet + runtime"] N --> P["Pod<br/>application container"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d class C control
A cluster contains a control plane and worker nodes. Control-plane components arrange the work; node software starts containers. Application requests do not pass through kubectl.
Slide 3
Mini Kubernetes: declare, inspect, connect
Four objects make the lab readable: Namespace, Deployment, Pod, and Service.
- A Namespace groups resources. A Deployment maintains the requested replicas; each Pod is a scheduled unit containing one or more containers.
- A Service gives selected Pods a stable network entry point. Apply the lab YAML, inspect Pods and the Service, then send a request from the client Pod.
flowchart LR C["Controller<br/>desired 3, sees 2"] --> N["Create Pod"] N --> S["Scheduler<br/>selects node"] S --> K["Kubelet<br/>starts container"] K --> R["Ready<br/>join endpoints"] R -. "observed state" .-> C classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class C,S control
Reconciliation is asynchronous. A new Pod needs scheduling, image startup, and readiness before it can receive traffic. No step restores a discarded local checkout session.
Slide 4
Discovery: where should the client connect?
Names outlive instances; discovery information can still become stale.
- DNS maps a name to an address or set of addresses. Caches and connection reuse delay how quickly a caller sees change.
- Client-side discovery lets a library choose from endpoints. Proxy-side discovery centralizes selection, while adding a component to operate.
flowchart LR R["Registry or DNS"] -. "endpoint updates" .-> C["Client library"] C --> A["Service replica A"] R -. "endpoint updates" .-> P["Proxy or gateway"] U["Simple client"] --> P P --> B["Service replica B"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class R,P control
The upper path puts endpoint selection in the client; the lower path delegates it to a proxy. Both must handle stale membership and failed connections.
Slide 5
Kubernetes Services and EndpointSlices
A Service gives changing Pods a stable access point.
- A normal ClusterIP Service has a virtual IP. DNS resolves its service name; the data plane forwards connections to eligible endpoints.
- Label selectors connect a Service to Pods; EndpointSlices track their addresses and conditions. A headless Service exposes endpoint addresses without a virtual IP.
flowchart LR C["Client"] -->|"DNS lookup"| D["Cluster DNS"] D -. "Service IP" .-> C C -->|"new connection"| V["Service virtual IP<br/>data plane"] V --> P["Ready Pod"] E["EndpointSlice<br/>addresses + conditions"] -. "updates" .-> V K["Pod labels and readiness"] -.-> E classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class D,E,K control
DNS locates the Service; it is not in the path of every request. EndpointSlice updates inform routing, while the data plane carries application traffic.
Slide 6
L4 connections versus L7 requests
A connection is a reusable channel; a request is one operation sent through it.
- L4 = transport layer: select a backend using connection information such as IP addresses and TCP ports. Keep that connection on its selected backend.
- L7 = application layer: understand HTTP paths, headers, or gRPC methods. Select a backend for each request, even when requests share a client connection.
flowchart LR C["Client<br/>requests 1, 2, 3"] -->|"one TCP connection"| L["L4 balancer"] L --> A["Backend A<br/>all three requests"] D["Client<br/>requests 1, 2, 3"] -->|"one HTTP/2 connection"| P["L7 proxy"] P --> B["Backend A: 1, 3<br/>Backend B: 2"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d class L,P control
Illustrative routing, not a guaranteed distribution. The L7 proxy uses separate upstream connections; it does not split one TCP connection across servers. An active stream stays with its selected backend.
Slide 7
Choose a balancing policy for the work
Equal request counts do not imply equal cost or equal latency.
- Round robin rotates eligible targets: simple when requests and instances have similar cost and capacity.
- Least-request policies favor less busy targets. Request counts help, but a long model generation may cost much more than a short lookup.
- Weighted routing allocates traffic shares, useful for canaries or unequal capacity. A small sample will not exactly match the configured percentages.
- Consistent hashing improves affinity to a key. Hot keys can overload one target; affinity does not replicate session state.
Slide 8
Startup, readiness, and liveness
Each probe answers a different operational question.
- Startup asks whether initialization finished. Readiness decides whether to receive normal Service traffic; liveness can trigger a container restart.
- Do not restart every replica because one shared provider is unavailable. Make liveness about local progress and readiness about the work this instance can serve.
flowchart LR S["Startup passes"] --> R["Readiness fails"] S --> L["Liveness fails<br/>past threshold"] R --> N["Remove from<br/>normal routing"] L --> X["Restart<br/>container"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class N,X warning
A configured startup probe gates the other probes until success. Failing readiness does not by itself restart a container; liveness failure can.
Slide 9
Retries must fit a deadline and a budget
Failure handling can overload the very service it is trying to recover.
- Three layers making up to three attempts each can generate 27 backend attempts for one original request.
- Assign retry ownership, preserve the remaining deadline, and use backoff with jitter. Retry mutations only with a safe repeat contract.
- Bound concurrency and queues; reject excess work visibly. A circuit breaker temporarily avoids a failing dependency while a bulkhead protects other work.
- A timeout means the result may be unknown. Keep the operation key and reconcile, as in the payment workflow from Module 4.
Slide 10
Configuration and workload identity
A discovered address tells you where; verified identity tells you who.
- Keep ordinary configuration separate from the image. Deliver secrets at runtime with restricted access and rotation; base64 is not encryption.
- Authenticate workloads with verified credentials, then authorize the specific operation. A DNS name or caller-supplied header is not proof of identity.
flowchart LR S["Runtime credential<br/>restricted delivery"] --> C["Ordering"] C -->|"locate"| D["Inventory<br/>Service name"] D -->|"request"| P["Verify identity<br/>authorize operation"] P -->|"allowed"| B["Business rules"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d class S,P control
Service discovery and authorization are separate controls. Runtime credentials identify the caller; Inventory still checks permission and enforces stock invariants.
Slide 11
Rolling updates and graceful draining
A new version must become ready before an old replica disappears.
- Rolling updates trade spare capacity against availability through maxSurge and maxUnavailable. Readiness keeps a warming replica out of normal traffic.
- During termination, endpoint updates and shutdown overlap. Handle termination signals, stop accepting new work, and finish or checkpoint within the grace period.
flowchart LR A["v1 serves"] --> B["v2 starts"] B --> C["v2 ready"] C --> D["Terminate v1"] D --> E["Routing updates<br/>and draining"] E --> F["v1 exits"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class D,E warning
This is a desired rollout sequence, not an atomic handoff. Existing connections and delayed routing updates require explicit shutdown behavior and testing.
Slide 12
Autoscaling follows signals, with delay
Add replicas for the bottleneck you can actually relieve.
- HPA adjusts desired replicas from metrics. A simplified example: 3 replicas at 80% CPU against a 60% target suggest ceil(3 × 80/60) = 4.
- KEDA can activate event-driven workers from external signals. Pending work, oldest-item age, and safe concurrency often explain worker pressure better than CPU alone.
flowchart LR W["Work arrives"] --> P["Ready replicas"] P --> M["Load metrics"] M --> H["Autoscaler<br/>desired count"] H --> D["Scheduling<br/>startup delay"] D --> P classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class M,H control
Metrics describe past load; useful capacity arrives later. Set bounds and stabilization, and check whether the database or provider quota is the true bottleneck.
Slide 13
Gateway API separates routing responsibilities
Traffic policy is an API contract implemented by a controller and data plane.
- GatewayClass identifies an implementation. A Gateway describes listeners; routes attach application traffic rules to those listeners.
- HTTPRoute can express HTTP matching and weighted backends. Confirm controller support and route status; installing API definitions alone does not forward traffic.
flowchart LR R["HTTPRoute<br/>matches + weights"] -. "configures" .-> G["Gateway<br/>selected GatewayClass"] U["Client"] --> G G -->|"90 weight"| A["Stable Service"] G -->|"10 weight"| B["Canary Service"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d class R,G control
The weights illustrate a canary policy, not exact counts in a small sample. The selected controller reconciles these resources into actual routing behavior.
Slide 14
Why a service mesh? Why Istio?
Make service-to-service security, traffic policy, and telemetry consistent.
- Without shared infrastructure, every team must configure credentials, timeouts, and network telemetry in each language. A mesh moves selected controls into managed proxies.
- Istio configures those proxies and workload identities. Use it for consistent mTLS, access policy, and traffic control; weigh its operational cost against your actual needs.
flowchart LR A["Ordering app"] --> P["Proxy"] --> Q["Proxy"] --> B["Inventory app"] C["Istio control plane"] -. "identity + policy" .-> P C -. "identity + policy" .-> Q classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d class C control
The proxies authenticate peers, enforce configured policy, and measure traffic. Istio’s control plane distributes configuration; business requests flow through the data plane, not through that control plane.
Slide 15
Istio deployment choices: sidecar and ambient
The mesh capabilities you need determine where proxies must run.
- Sidecar mode places an Envoy proxy alongside the application in each enrolled Pod. It can handle both transport security and application-aware traffic policy.
- Ambient mode uses a per-node ztunnel for L4 security. Add waypoint proxies for workloads needing L7 policy and routing; sidecar-free still includes proxies.
flowchart LR A["Sidecar mode<br/>application Pod"] --> B["Local Envoy<br/>L4 + L7"] --> D["Destination proxy + app"] E["Ambient mode<br/>application Pod"] --> C["Node ztunnel<br/>L4 secure overlay"] --> W["Waypoint when needed<br/>L7 policy"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d class C control
These are simplified placement paths. Ambient also uses destination-side ztunnel handling; a waypoint is not required for the L4-only path. Feature support and traffic enrollment must be checked for the deployed version.
Slide 16
Lab: remove a replica from traffic
Predict what changes before running the failure experiment.
- Send repeated requests through one Service name to two Pods; record the returned Pod names without expecting exact alternation.
- Fail readiness on one Pod, then restore it. Delete a Pod next: compare routing removal with controller replacement and recovery time.
flowchart LR C["Client"] --> S["Stable Service"] S --> A["Pod A<br/>ready"] S -. "routing removed" .-> B["Pod B<br/>not ready"] D["Controller"] --> R["Replacement<br/>after deletion"] R -. "when ready" .-> S classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class B warning class D control
Readiness failure changes traffic eligibility without replacing the Pod. Deletion triggers replacement. A Service name stays stable across both experiments.
Slide 17
Inference routing needs workload context
A model request may occupy an accelerator far longer than a normal API lookup.
- Route only to replicas that serve the requested model. Queue pressure and cache locality can inform selection when the serving stack exposes them.
- Gateway API Inference Extension adds inference-oriented selection to compatible gateways. Evaluate the exact release and implementation, not just the project name.
flowchart LR A["Agent request<br/>model + token budget"] --> G["Compatible inference gateway"] G --> P["Endpoint picker<br/>eligible model replicas"] M["Serving metrics"] -.-> P P --> R1["Model replica A"] P --> R2["Model replica B"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class G,P,M control
The picker chooses among eligible model servers. Cache reuse may save work, but sending everything to one warm replica can worsen queueing.
Slide 18
Agent discovery includes capabilities and trust
Finding an endpoint and finding a useful tool are different discovery problems.
- DNS or a Service locates the server. MCP tools/list describes tools and input schemas; A2A Agent Cards advertise agent capabilities and connection details.
- An advertised capability is not permission to use it. Validate the server, authorize each operation, and keep an explicit approved tool set.
flowchart LR A["Agent runtime"] --> L["Locate server"] L --> C["Inspect tools<br/>or Agent Card"] C --> P["Verify identity<br/>check permission"] P --> T["Invoke tool"] classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class C,P control
Capabilities describe available interfaces. Identity and policy determine allowed actions; untrusted descriptions or tool results cannot grant new permissions.
Slide 19
A recoverable travel-agent runtime
Use dynamic reasoning inside a workflow with durable, enforceable boundaries.
- A planner proposes an alternative; approval applies to that exact proposal version. An executor revalidates constraints before a booking call.
- Replicas share durable workflow state and a stable provider operation key. A lost response enters reconciliation rather than a fresh booking attempt.
flowchart LR P["Disruption<br/>planner reacts"] --> V["Store proposal<br/>exact-version approval"] V --> X["Executor<br/>check constraints"] X --> T["Booking tool<br/>operation key"] T --> R["Record result<br/>or reconcile"] R -. "durable progress" .-> V classDef control fill:#fff3da,stroke:#9b6b16,color:#35260d classDef warning fill:#ffebe7,stroke:#bd5949,color:#542b24 classDef data fill:#e8eefc,stroke:#556bb4,color:#24345e class V,X control class R data
Discovery and balancing keep workers reachable. Durable state, concurrency control, approval checks, and provider repeat handling preserve the booking invariant.
Slide 20
Architecture review
A good diagram explains where a request goes and what happens when that path fails.
- Show the stable name, discovery owner, balancing unit, health signals, and shared failure domains for one service.
- Explain rollout and draining, scaling signals, workload permissions, and the durable state needed when a replica disappears.
- For the agent extension, remove an executor after provider success but before its response is recorded. Show how the outcome is recovered without a second booking.
- Bring an annotated architecture and expected-versus-observed evidence. Keep deadlines, grading, and submissions in Canvas.