Module 8: Replication, Consistency, CAP, and PACELC

Learning goals

Checkout updates stock from 10 to 9, yet a customer still sees 10. Two disconnected devices both edit the same cart. An agent retrieves an older itinerary after a newer one receives approval. This module uses these concrete situations to ask which copy can answer, which writes can succeed, and how the system recovers.

Required preparation

Review timeouts in Module 3, duplicate delivery in Module 5, and causal order and fencing in Module 7. Bring Python 3.10 or newer and a working container engine with Docker Compose v2. Download images before class. Run one profile at a time. No cloud account, paid service, Python package installation, or real customer data is needed.

Read the selected sections of PostgreSQL replication, etcd guarantees, and CouchDB conflicts. The lab instructions identify the relevant sections. Formal proofs and the deeper CRDT readings are optional.

The 20 slides allow about 140 minutes for guided teaching and exercises, leaving 10 minutes in the class meeting for a break and questions. Use the Python models if container startup is unavailable, then complete the actual database experiments later. Models demonstrate reasoning rules and do not validate a database implementation.

Download the classroom kit ZIP

with Compose configuration, Python helper, instructions, and a project memo template.

Read the lab instructions Markdown

for commands, expected output, troubleshooting, and recovery.

Guided exercises

Predict the output before each command, compare it with the actual result, and record what the observation does and does not establish. The same teaching sections drive Presentation Mode and the printable handout.

flowchart TD
  P[Primary accepts stock update]
  P -->|WAL records| R[Read replica]
  R -->|Possibly stale stock| C[Reader]

%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 32, "padding": 6}, "themeVariables": {"fontSize": "22px"}}}%%
flowchart LR
  C[Writer] -->|Write request| P
  subgraph P[Primary]
    direction TB
    E[Transaction engine] -->|Generate WAL| L[Durable WAL log]
    L -->|Log records| T[WAL sender]
    E -->|Flush pages after WAL| D[Primary data pages]
  end
  P <-->|WAL to replica / progress to primary| S
  subgraph S[Replica]
    direction TB
    R[WAL receiver] -->|Write and flush| W[Local WAL log]
    W -->|Records in order| A[Replay process]
    A -->|Apply changes| B[Replica data pages]
  end
  S -->|Read query result| Q[Reader]

flowchart TD
  A[etcd1 accepts stock 9] <-->|Peer traffic| B[etcd2 stock 9]
  B -.->|Network disconnected| C[etcd3 local stock 10]

flowchart TD
  S[Cart contains book] -->|Writer A adds pen| A[Branch A]
  S -->|Writer B adds notebook| B[Branch B]
  A -->|Replicate revision| C[CouchDB retains both branches]
  B -->|Replicate revision| C

Lab sequence and expected results

Start from the extracted kit directory:

python3 lab.py doctor
docker compose --profile pg up -d --wait
python3 lab.py pg status
python3 lab.py pg positions
python3 lab.py pg sync
python3 lab.py pg stale
python3 lab.py pg catchup

The two database processes initially agree. The positions exercise shows the primary sender’s progress alongside the replica’s received and replayed log positions. An LSN is a position in PostgreSQL’s write-ahead log. A received or flushed record may still await replay, so its presence alone does not establish read visibility. With replay paused, the primary advances while the standby returns the older version. After resuming replay, the helper polls until the versions match. The read-your-writes exercise repeats the stale-read experiment and compares primary and standby reads. A primary read is appropriate only under this lab’s single-primary assumptions. Failover requires additional routing and recovery guarantees.

Before slide 8, switch profiles:

docker compose --profile pg stop
docker compose --profile etcd up -d
python3 lab.py etcd seed
python3 lab.py etcd status

The history and quorum activities on slides 9–10 use standard-library Python. The quorum helper enumerates fixed replica subsets. It is not an implementation of etcd’s Raft protocol, and quorum overlap alone does not establish linearizability.

For slides 11–14:

python3 lab.py etcd isolate
python3 lab.py model cap
python3 lab.py etcd reads
python3 lab.py etcd heal

The helper isolates a live member’s network, writes through the connected majority, and compares the isolated member’s old local read with its unsuccessful linearizable read. A three-second client timeout bounds the classroom wait. CAP availability itself has no fixed response-time bound. An error is a response but does not successfully implement the requested read or write. Heal before stopping the cluster or repeating seed.

For PACELC on slide 15:

docker compose --profile etcd stop
docker compose --profile pg up -d --wait
python3 lab.py pg latency
python3 lab.py pg async

Compare the actual elapsed medians and the acknowledgement conditions. Localhost measurements include process overhead and may not expose a significant latency difference. Acknowledging remote replay still requires a remote condition. Avoid assigning an unconditional CAP or PACELC label to an entire database product.

For the conflicting-write exercises:

docker compose --profile pg stop
docker compose --profile couch up -d
python3 lab.py couch seed
python3 lab.py couch conflict
python3 lab.py couch resolve

Expect two revision branches after independent edits, then one resolved additions-only cart containing a book, a notebook, and a pen. Inspect both branches before resolving. Replication selects a default winning revision but cannot infer your business merge rule. A last-write-wins rule chooses a version using an ordering rule, often a timestamp, and can discard another user’s valid change. Clock skew makes physical-time choices especially risky. A revision precondition can reject a stale edit on one server, while disconnected accepted branches still need a resolution policy. This exercise uses manual one-shot replication between two actual servers, with no continuous sync between edits.

Finish with:

python3 lab.py model merge
python3 lab.py model agent
python3 lab.py model decision

An additions-only cart and a grow-only counter can use simple merge rules. Seat inventory, money, removals, and approval transitions require their own invariants and concurrency controls. A convergent result can still violate a business rule. An AI executor must check authoritative state and exact-version approval even when its retrieval tool offers stronger freshness options.

Recovery and project practice

Follow the kit’s recovery steps after each experiment. Use the

to justify one operation’s policy with observed evidence. This is public learning material. Canvas remains the official system for graded submissions, exams, announcements, and individualized feedback.