1 / 16
Sections
Module 1
Building Systems That Work When Things Go Wrong
Distributed systems are built for scale, reach, and resilience, but their real test is how they behave under uncertainty.
- Enterprise systems cross machines, teams, regions, and trust boundaries.
- The course theme is useful behavior despite failure.
Slide 2
Before We Start: Ungraded Survey
Quick pulse check on experience with cloud, APIs, databases, message brokers, observability, and AI tooling.
- What distributed systems have you already used or built?
- Which failure stories have you personally encountered?
- Which tools do you want more confidence with this semester?
Slide 3
What Makes a System Distributed?
Independent components communicate over a network and coordinate to provide one larger service.
- Components have separate processes, memory, clocks, and failure modes.
- Communication crosses network boundaries.
- Users expect one coherent experience.
Slide 4
Why Distributed Systems Matter
Distribution is how enterprises scale capability, geography, teams, and resilience.
- Cloud platforms scale across zones and regions.
- Microservices let teams own focused capabilities.
- Financial and streaming systems need continuous operation.
- Agentic systems coordinate models, tools, memory, and policies.
Slide 5
Partial Failure
In a distributed system, one component can fail while the rest of the system keeps running.
- Single-node failure is often obvious.
- Partial failure is ambiguous and localized.
- Clients may see timeouts, stale data, or degraded features.
flowchart LR
subgraph singleNode[Single node]
c1[Client] --> app[Application and data]
app -.-> down[Whole system unavailable]
end
subgraph distributedSystem[Distributed system]
c2[Client] --> gw[API Gateway]
gw --> svcA[Service A healthy]
gw --> svcB[Service B unavailable]
svcA --> db[Shared data store]
svcB -.-> db
end
The left side shows a single-node application where one server failure stops the whole system. The right side shows a distributed application where Service B is unavailable while clients, the gateway, Service A, and the database may still be running. The system is neither fully healthy nor fully down.
Slide 6
Network Failure Is Not Process Failure
A timeout does not prove that the remote operation failed.
- Service B sends a request.
- Service A processes it successfully.
- The response is lost.
- Service B times out and cannot determine whether the operation occurred.
sequenceDiagram participant B as Service B participant N as Network participant A as Service A participant D as Data Store B->>N: createOrder(requestId=42) N->>A: deliver request A->>D: commit order 42 D-->>A: commit ok A-->>N: response ok Note over N: response is lost N-->>B: no response B->>B: timeout, outcome unknown
Service B sends a create request to Service A. Service A commits the operation, but the response is lost on the network. Service B only observes a timeout, so it cannot know whether the operation happened without an idempotency key, status lookup, or reconciliation flow.
Slide 7
Ordering and Consistency
Services can observe events in different orders and temporarily disagree.
- Each service owns a local view.
- Messages may arrive late or out of order.
- Consistency is a design choice, not a default.
flowchart LR orders[Order service state paid] shipping[Shipping service state pending] event[Payment authorized event delayed] orders --> event event -.-> shipping orders -.-> userA[Customer support view paid] shipping -.-> userB[Fulfillment view pending]
The order service has recorded an order as paid, while the shipping service still sees it as pending because the payment event has not arrived or has not been processed. The diagram is understandable by reading the state labels, not by relying on color.
Slide 8
Failures Are Often Silent
Distributed systems often fail through delay, omission, duplication, or inconsistent observations rather than a clean error.
- A dependency can be slow instead of down.
- A message can be duplicated after retry.
- A write can succeed while the caller sees failure.
Slide 9
Debugging Requires Reconstruction
No single process has the whole story; engineers reconstruct behavior from logs, metrics, and traces.
- Logs explain local events.
- Metrics reveal trends and symptoms.
- Traces connect one request across services.
flowchart LR client[Client] --> gateway[API Gateway] gateway --> svcA[Service A] svcA --> broker[Message Broker] broker --> svcB[Service B] svcB --> db[Database] gateway -.-> obs[Observability platform] svcA -.-> obs broker -.-> obs svcB -.-> obs db -.-> obs
A request moves from client to gateway to services and data stores. Each component emits telemetry. Traces connect the path, logs explain local decisions, and metrics show aggregate health.
Slide 10
The Eight Fallacies of Distributed Computing
Classic false assumptions that make distributed systems fragile.
- The network is reliable.
- Latency is zero.
- Bandwidth is infinite.
- The network is secure.
- Topology does not change.
- There is one administrator.
- Transport cost is zero.
- The network is homogeneous.
Slide 11
What Students Will Build
The course builds toward services that communicate, persist state, handle failure, expose telemetry, and justify tradeoffs.
- APIs and service boundaries.
- Messaging and event-driven workflows.
- Distributed data and consistency decisions.
- Observability, security, and resilience.
- Responsible use of coding agents as engineering assistants.
Slide 12
Course Roadmap
The semester moves from foundations to communication, data, resilience, operations, security, and AI-enabled architectures.
- Foundations and communication.
- Coordination, replication, partitioning, and consistency.
- Event-driven systems and microservices.
- Observability, resilience, security, and final project integration.
Slide 13
Enterprise-Grade Distributed Systems Stack
Enterprise systems combine client, edge, service, data, event, platform, observability, and governance layers.
flowchart TB clients[Users clients partner systems] edge[Edge DNS CDN load balancer API gateway] services[Services domain APIs workflows AI agents] events[Events broker streams queues] data[Data relational document cache search lake] platform[Platform containers orchestration cloud regions] observe[Operations logs metrics traces alerts] govern[Governance security policy compliance cost] clients --> edge --> services services --> events services --> data events --> data platform --> services services --> observe platform --> observe govern -.-> edge govern -.-> services govern -.-> data
The stack moves from users and clients through edge routing, services, events, data, platform infrastructure, observability, and governance. Each layer introduces responsibilities that must be designed and operated explicitly.
Slide 14
Technologies and Responsible Coding-Agent Use
Tools accelerate implementation, but engineers remain responsible for correctness, security, and operational behavior.
- Use agents to explore, scaffold, test, and explain.
- Verify generated code, dependency choices, and security assumptions.
- Never outsource architectural accountability.
- Document where agent assistance changed the work.
Slide 15
Week 1 Lab
Build and observe a tiny two-service interaction so failure is visible.
- Service B calls Service A.
- Add request IDs and logs.
- Simulate timeout, retry, and duplicate request scenarios.
- Explain what Service B knows and does not know.
flowchart LR student[Student browser or curl] serviceB[Service B caller] serviceA[Service A worker] store[Service A local store] logs[Console logs with request IDs] student --> serviceB serviceB --> serviceA serviceA --> store serviceB -.-> logs serviceA -.-> logs serviceA -.-> serviceB
Service B calls Service A over HTTP and includes a request ID. Service A records work in its local store and both services emit logs. The lab varies response delay and dropped responses so students can see how uncertainty appears at Service B.
Slide 16
What’s Next
Module 2 moves from identifying distributed boundaries to designing communication and coordination between services.
- Read Module 1 again after class and update your failure examples.
- Watch for hidden distributed systems in everyday tools.
- Next: communication patterns, coordination costs, and request semantics.