# Module 8 classroom lab

Replication, Consistency, CAP, and PACELC. Work in pairs, predict the result, run the command, then explain what the observation establishes. All services are open source. Python exercises use only the standard library. The container exercises use actual PostgreSQL 17.6, etcd 3.6.0, and Apache CouchDB 3.4.3 servers.

## Preparation

Extract the ZIP into its own folder and open a terminal there. Install Python 3.10 or newer, a running Docker-compatible engine with the Docker CLI, and Docker Compose v2. On Linux Docker Engine works directly. On macOS an open-source VM runtime such as Colima can supply the engine. Allow time for image downloads before class. Run one profile at a time to limit laptop resource use. Models are illustrative Python programs, not substitutes for testing a database's guarantees.

```sh
python3 lab.py doctor
docker compose --profile pg up -d --wait
python3 lab.py pg status
```

Wait for the primary and standby health checks. The `pg status` helper should show `1|10|1` on both, and `standby1|streaming|async` on the primary. All SQL runs through `docker compose exec`, so no local PostgreSQL installation or exposed database port is required. Use this Compose folder throughout; the network name is `cmpe273-m08`.

These are disposable teaching instances with a fixed `labonly` password. CouchDB binds only to localhost ports 15984 and 25984. etcd has no published host port and uses plaintext on the lab network. Use sample data only. These settings are deliberately minimal and are not a production deployment guide.

## Slides 1 through 7

1. **Replication and consistency:** `python3 lab.py doctor`. Expect Python readiness plus Compose/engine details, or a clear missing-Docker message. Inspect `compose.yaml` and identify the profiles before starting containers.
2. **Copies of one record:** `python3 lab.py pg status`. Compare the row on two database processes. Identify the primary's streaming sender.
3. **Log based replication:** `python3 lab.py pg positions`. Trace the transaction engine, durable WAL, sender, receiver, local WAL, replay process, and data pages. Compare `sent_lsn`, `write_lsn`, `flush_lsn`, and `replay_lsn` on the primary with received/replayed positions on the standby. LSN means log sequence number, a byte position in the log. The values can match on an idle cluster. Repeat after pausing replay with the stale-read exercise to separate receipt from application. Statistics reflect reports and separate queries, not one atomic cross-server snapshot. A replica starts from a base backup before consuming subsequent WAL.
4. **Replication modes:** `python3 lab.py pg sync`. The helper configures the named standby and uses `synchronous_commit=remote_apply` for one update. The completed update is visible on the configured standby. Inspect the helper's SQL and distinguish `local`, `on` (remote flush when configured), and `remote_apply`. This setup has only one standby and no automatic failover.
5. **Stale reads:** `python3 lab.py pg stale`. Replay pauses before the primary increments the row's version and decrements remaining stock. The primary shows the newer version while the standby retains the prior value. Initial values vary if you repeat the experiment. A paused replay is a deterministic lag injection, not a full network partition. Keep it brief because retained WAL can grow.
6. **Eventual consistency:** `python3 lab.py pg catchup`. The helper resumes replay and polls until versions match. Observe convergence after repair. This experiment supplies no universal staleness bound.
7. **Read your writes:** repeat `python3 lab.py pg stale`. The helper's primary read sees its completed update while its standby read does not. Discuss routing the writer's next read to the primary, or carrying a WAL position and waiting for a sufficiently advanced replica. Run `python3 lab.py pg catchup` before leaving PostgreSQL.

## Slides 8 through 10

Stop the PostgreSQL profile without deleting its data and start etcd:

```sh
docker compose --profile pg stop
docker compose --profile etcd up -d
python3 lab.py etcd seed
python3 lab.py etcd status
```

The seed helper waits for the cluster to accept `stock=10`, then reads it through another member. Status lists three members and a leader. Startup may take several seconds. Each member has its own persistent volume.

8. **Linearizable reads:** inspect `python3 lab.py etcd seed`. The successful write finishes before the later read starts. Read the etcd guarantees reference. One matching result does not prove a consistency model.
9. **Sequential consistency:** `python3 lab.py model histories`. Place each write and read into a candidate total order. Explain the ordering cycle in the four-operation counterexample. Sequential consistency preserves each process's program order but need not preserve cross-process real-time order. Also run `python3 lab.py model causal`, draw the message-to-reply dependency, and explain why causal consistency can leave independent edits unordered. This model is not an etcd causal-read setting.
10. **Quorum overlap:** `python3 lab.py model quorum`. The helper enumerates all nine pairs of two-member read/write sets for N=3. Change the local copy to enumerate one-member read sets and find a disjoint pair. R+W>N presumes a fixed replica set and appropriate completed acknowledgements. It does not by itself select the newest concurrent version or provide linearizability. etcd uses Raft rather than this simple R/W quorum algorithm.

## Slides 11 through 14

```sh
python3 lab.py etcd isolate
python3 lab.py model cap
python3 lab.py etcd reads
python3 lab.py etcd heal
```

11. **Network partitions:** `isolate` disconnects member 3 from the Compose network, while its process remains alive and reachable through `docker exec` and loopback. Members 1 and 2 accept stock 9. Member 3's local read still returns stock 10, while its linearizable read should fail at the three-second client deadline. The helper polls the majority to allow a leader election. This is an asymmetric access demonstration: the local diagnostic client can reach member 3 while peers cannot.
12. **CAP:** the standard-library `model cap` exercise gives the two-execution argument. If the isolated member returns an older value after a completed write elsewhere, linearizability fails. Waiting forever or returning an error fails CAP availability for that operation. The theorem does not define availability as an uptime percentage or require a fixed latency bound.
13. **Availability choices:** `etcd reads` compares both modes on connected member 1 and then on isolated member 3. Expect majority reads to work, member 3's `--consistency=s` to return old data, and its `--consistency=l` to fail. etcd calls the local mode “serializable”; do not equate this flag with SQL transaction serializability. A weaker read does not enable writes on a minority or turn every etcd operation into an availability-first service.
14. **Recovery:** `heal` reconnects member 3 with its DNS alias, then waits for a linearizable read to return 9. This step assumes the isolation exercise wrote stock 9. If repeating the sequence, heal first, then seed again. Do not run seed between isolate and heal because it would overwrite the expected value.

## Slide 15

```sh
docker compose --profile etcd stop
docker compose --profile pg up -d --wait
python3 lab.py pg latency
python3 lab.py pg async
```

**PACELC:** The helper measures five updates with local commit acknowledgement and five with remote replay acknowledgement. It prints median client elapsed time in milliseconds, including Docker and psql process overhead. This localhost test has no injected network delay, so noise may mask or reverse a small observed difference. Report the actual numbers without claiming a universal performance ratio. For a deeper optional comparison, put the standby on another machine with a known measured network delay and repeat. The stronger acknowledgement waits for a remote condition even when there is no partition. `pg async` restores asynchronous replication and resumes replay.

## Slides 16 and 17

```sh
docker compose --profile pg stop
docker compose --profile couch up -d
python3 lab.py couch seed
python3 lab.py couch conflict
python3 lab.py couch resolve
```

16. **Conflicting writes:** seed creates `cart/cart-1` on server A with a book, then replicates it to server B. With no continuous replication, A adds a pen and B independently adds a notebook from the shared revision. The helper performs one-shot replication in both directions and prints the winning body plus `_conflicts`, then fetches the other branch. The two independent edits both succeed locally. The default winning revision is not a semantic merge. This demonstrates application-controlled disconnected replication, not a measured physical network partition.
17. **Conflict resolution:** resolve unions the additions, updates the winning revision, deletes losing leaf revisions, and replicates both ways. Expect `['book', 'notebook', 'pen']` and no `_conflicts` on both servers. This rule is valid only for this additions-only exercise. Do not use union for arbitrary cart removals, money, inventory limits, or ordered workflow transitions. Concurrent writers must stop during this simple resolution exercise. Tombstones remain part of replication history.

For a clean repeat without deleting unrelated databases, use a fresh extracted kit/Compose project with a changed project and network name, or perform the explicitly scoped full lab reset below. `couch seed` intentionally does not erase existing work.

## Slides 18 through 20

18. **Mergeable state:** `python3 lab.py model merge`. Compare lost updates in naive shared-counter code with per-writer components merged by maximum. Check commutativity and idempotence assertions. Add a third distinct writer as an extension. The toy G-counter supports only increments and cannot enforce bounded stock.
19. **Stale agent context:** `python3 lab.py model agent`. A search result refers to proposal 2 while authoritative approval refers to 3. The executor rejects the mismatch and must fetch the approved immutable version. Explain why retrieval freshness does not grant approval. This model does not start Milvus or call any real booking or payment provider.
20. **Project consistency choices:** `python3 lab.py model decision`. Complete `consistency-memo.md` for one project operation. Include a concrete failure sequence, selected guarantee, and a recovery observation from this lab. Discuss in pairs. Canvas remains the official system for submissions and grades.

## Recovery and cleanup

Always heal etcd before stopping or recreating its containers. Resume PostgreSQL replay and reset its synchronous policy before stopping the standby. Do not simply stop the standby while required synchronous commits are in progress.

```sh
python3 lab.py etcd heal
python3 lab.py pg async
docker compose --profile pg --profile etcd --profile couch down
```

Run each recovery helper only while its profile is running and after its corresponding exercise. If interrupted, restore that profile first. For an intentional **full reset of this disposable lab**, the following command removes only this Compose project's named volumes and containers, including all lab records:

```sh
docker compose --profile pg --profile etcd --profile couch down -v
```

Troubleshooting: run `docker compose --profile pg --profile etcd --profile couch ps -a` and `docker compose logs SERVICE`. If PostgreSQL initialization fails, inspect primary logs and the mounted init files. Changed init scripts do not rerun against existing volumes. If CouchDB ports are occupied, change the published ports and matching values in `lab.py`. For image-pull failures, check engine connectivity and registry availability. The pinned tags provide a common teaching baseline and are not a claim about the newest releases.

## References

- [PostgreSQL streaming and synchronous replication](https://www.postgresql.org/docs/17/warm-standby.html): follow a write from primary to standby.
- [PostgreSQL commit settings](https://www.postgresql.org/docs/17/runtime-config-wal.html): compare local durability, remote flush, and remote apply.
- [PostgreSQL replay controls](https://www.postgresql.org/docs/17/functions-admin.html): find pause, resume, and replay position functions.
- [etcd guarantees](https://etcd.io/docs/v3.6/learning/api_guarantees/): distinguish linearizable and member-local reads.
- [etcd container guide](https://etcd.io/docs/v3.6/op-guide/container/): understand cluster bootstrap parameters.
- [CouchDB conflicts](https://docs.couchdb.org/en/stable/replication/conflicts.html): inspect revision branches and resolution.
- [Gilbert and Lynch CAP proof](https://groups.csail.mit.edu/tds/papers/Gilbert/Brewer6.pdf): optional formal definitions and proof.
- [Abadi on PACELC](https://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-little.html): read the PACELC paragraph before the product examples.
- [Consistency and baseball](https://www.microsoft.com/en-us/research/wp-content/uploads/2011/10/ConsistencyAndBaseballReport.pdf): optional consistency examples.
- [CRDT introduction](https://crdt.tech/): optional counter merge properties.