# Module 6: Discovery and Load Balancing Lab

**Goal:** send requests through a stable Service name, observe multiple backends, and distinguish an unready Pod from a deleted Pod. Use a disposable local cluster. This example uses synthetic responses only and exposes no external load balancer.

Download `runtime-lab.yaml` alongside these instructions.

The manifest uses `python:3.13-slim` for a standard-library HTTP server and an in-cluster client. Image retrieval requires registry access; a version-family tag is convenient for teaching but not an immutable production pin. Record the resolved image ID when comparing results. The server returns its Pod name, exposes `/live` and `/ready`, and supports readiness failure through a local marker file. It does not implement a real Inventory database, an authentication system, or graceful shutdown.

### 1. Establish a baseline

Save the download as `runtime-lab.yaml`, inspect your current Kubernetes context, and apply it only to your disposable cluster:

```bash
kubectl config current-context
kubectl apply -f runtime-lab.yaml
kubectl -n cmpe273-module06 rollout status deployment/inventory --timeout=120s
kubectl -n cmpe273-module06 wait --for=condition=Ready pod/client --timeout=120s
kubectl -n cmpe273-module06 get pods -o wide
kubectl -n cmpe273-module06 get endpointslices -l kubernetes.io/service-name=inventory
kubectl -n cmpe273-module06 exec client -- python /lab/client.py
```

The client makes fresh HTTP connections through `inventory:8080` and prints per-Pod counts plus errors. You should normally see both replica names over repeated runs; do not expect exact alternation or a guaranteed 50/50 split. **Do not use `kubectl port-forward service/inventory` to test balancing**: port forwarding selects a backing Pod rather than exercising the ordinary Service path.

### 2. Fail readiness without restarting

```bash
LAB_POD=$(kubectl -n cmpe273-module06 get pod -l app=inventory -o jsonpath='{.items[0].metadata.name}')
kubectl -n cmpe273-module06 exec "$LAB_POD" -- touch /tmp/not-ready
kubectl -n cmpe273-module06 wait --for=condition=Ready=false "pod/$LAB_POD" --timeout=60s
kubectl -n cmpe273-module06 get endpointslices -l kubernetes.io/service-name=inventory -o yaml
kubectl -n cmpe273-module06 exec client -- python /lab/client.py
kubectl -n cmpe273-module06 get pods
```

Compare the endpoint conditions and returned names. After readiness detection and routing propagation, new connections should use the other ready Pod. The unready Pod still exists; its liveness endpoint passes, so the experiment should not increase its restart count. Record timestamps instead of assuming the transition is instantaneous.

### 3. Restore, then replace

```bash
kubectl -n cmpe273-module06 exec "$LAB_POD" -- rm -f /tmp/not-ready
kubectl -n cmpe273-module06 wait --for=condition=Ready "pod/$LAB_POD" --timeout=60s
kubectl -n cmpe273-module06 exec client -- python /lab/client.py
kubectl -n cmpe273-module06 delete pod "$LAB_POD"
kubectl -n cmpe273-module06 rollout status deployment/inventory --timeout=120s
kubectl -n cmpe273-module06 get pods -o wide
kubectl -n cmpe273-module06 exec client -- python /lab/client.py
```

Deletion is the deliberate fault injection. The Deployment creates a replacement with a new identity. The Service name remains the same. To capture transient errors and recovery time, keep running the client in a second terminal while you delete the Pod. A completed rollout does not measure the interruption window by itself.

### Record evidence

| Experiment            | Prediction                                             | Evidence to capture                                |
| --------------------- | ------------------------------------------------------ | -------------------------------------------------- |
| Baseline              | Both replicas can receive fresh connections            | Pod names, counts, errors, image IDs               |
| Readiness failure     | One Pod leaves normal routing after propagation        | Endpoint conditions, request counts, restart count |
| Readiness restoration | The same Pod can rejoin                                | Ready transition and returned name                 |
| Pod deletion          | A new Pod eventually joins under the same Service name | Old/new names, timestamps, transient errors        |

For cleanup, verify that the namespace contains only this exercise, then run `kubectl delete namespace cmpe273-module06`. This deletes all resources in that lab namespace.

### Troubleshooting

- **ImagePullBackOff:** inspect `kubectl -n cmpe273-module06 describe pod POD_NAME` and confirm registry access.
- **No ready endpoints:** check Pod readiness and the Service selector with `kubectl -n cmpe273-module06 describe service inventory`.
- **DNS or connection errors:** inspect cluster DNS and network policy; run the client inside the lab namespace, where the short name `inventory` resolves.
- **Only one backend appears:** first confirm both Pods are ready. Repeat the sample and inspect connection reuse before assuming a broken balancer; a short sample is not a fairness guarantee.

### Paper alternative

Sketch two replicas behind one Service. At time 0 both are ready; at time 1 B fails readiness; at time 2 membership updates reach the data plane; at time 3 an old connection to B closes. Mark where a fresh connection and an existing connection may go at each time. Then replace B and show which name remains stable. Use this to distinguish health detection, propagation, and connection lifetime without guessing an exact outage duration.