CMPE273 Enterprise Distributed Systems

Module 1 · Published

Module 1: Introduction to Enterprise Distributed Systems

Define distributed systems, contrast them with parallel systems, and examine why enterprise systems become difficult once communication crosses network boundaries.

Learning Objectives

  1. Explain what makes a system distributed and how it differs from a parallel system.
  2. Describe core distributed systems goals including scalability, availability, reliability, transparency, and fault tolerance.
  3. Identify partial failure, latency, coordination, and network uncertainty as central sources of design difficulty.
  4. Recognize distributed systems patterns in cloud platforms, microservices, financial systems, streaming systems, and agentic systems.

Why This Topic Matters

Enterprise software is rarely a single program running on one machine. A checkout flow may cross an API gateway, identity provider, inventory service, payment processor, fraud model, message broker, data warehouse, and observability pipeline. Each boundary adds power, but each also adds uncertainty.

Distributed systems let organizations scale teams, workloads, data, and geography. They also introduce the central engineering problem of the course: how to build systems that remain understandable and useful when machines fail, networks delay messages, and independent services must coordinate.

Required Preparation

  • Read this module page before class.
  • Skim the additional resources listed at the end; focus on the abstracts or introductions.
  • Bring one example of a distributed system you use regularly, such as banking, streaming media, delivery tracking, cloud storage, or an AI assistant.

Core Concepts

Distributed System

A set of independent computers that coordinate over a network and appear to users or applications as one coherent system.

Parallel System

A system focused on dividing computation across processors, usually with tighter control over communication, memory, and execution.

Scalability

The ability to handle growth in users, requests, data volume, or geographic reach without redesigning the entire system.

Availability

The ability of a system to remain usable when components fail, traffic spikes, or dependencies become degraded.

Fault Tolerance

The ability to continue operating correctly, possibly in a reduced mode, despite hardware, software, or network failures.

Partial Failure

A failure mode where one part of the system fails or becomes unreachable while other parts continue running.

Distributed Systems Versus Parallel Systems

A parallel system divides work to complete computation faster. A distributed system coordinates independent components to provide a larger service. The two can overlap, but their design pressures differ.

Parallel computing usually asks, “How do we split this computation efficiently?” Distributed systems ask, “How do independently running components communicate, coordinate, fail, recover, and still provide useful behavior?”

Why Distributed Systems Are Difficult

The network is not just a cable between services. It is a source of delay, packet loss, reordering, overload, security boundaries, and ambiguous failure. A service call that times out may mean the request was never received, the response was lost, the dependency is overloaded, or the operation succeeded but the client did not hear back.

This uncertainty affects every enterprise domain: cloud platforms replicate state across regions, microservices coordinate business workflows, financial systems guard correctness under failure, streaming systems process events continuously, and agentic systems combine tools, models, memory, and policies across multiple services.

Enterprise Case Study

A Checkout Flow Under Pressure

Context
An online retailer runs separate services for identity, catalog, orders, payments, recommendations, and fulfillment.
System tension
During a promotion, payment requests slow down while order events continue to enter the message broker. Some users retry, some payments complete late, and dashboards show rising latency but not a single obvious failure.
Lesson
The architecture needs timeouts, idempotency, retry limits, durable messaging, service-level observability, and clear ownership of consistency boundaries.

Simple Distributed Application

flowchart LR
  clients[Clients web mobile partner APIs]
  gateway[API Gateway]
  identity[Identity Service]
  orders[Order Service]
  payments[Payment Service]
  agents[AI Agent Orchestrator]
  broker[Message Broker]
  ordersDb[Orders Database]
  paymentsDb[Payments Database]
  events[Event Store]
  obs[Observability logs metrics traces]

clients --> gateway
gateway --> identity
gateway --> orders
gateway --> payments
gateway --> agents
orders --> ordersDb
payments --> paymentsDb
orders --> broker
payments --> broker
agents --> broker
broker --> events
identity -.-> obs
orders -.-> obs
payments -.-> obs
agents -.-> obs
broker -.-> obs

Clients enter through an API gateway, which routes requests to independent services. Services own separate data stores and publish events through a broker. Observability receives telemetry from each component so engineers can understand behavior across service boundaries.

Teaching Diagrams

The following diagrams are used in Presentation Mode and are included here for review. Each diagram is paired with a textual explanation so the concept does not depend on color or visual layout alone.

Single-Node Failure Versus Partial Failure

flowchart LR
  subgraph singleNode[Single node]
    c1[Client] --> app[Application and data]
    app -.-> down[Whole system unavailable]
  end
  subgraph distributedSystem[Distributed system]
    c2[Client] --> gw[API Gateway]
    gw --> svcA[Service A healthy]
    gw --> svcB[Service B unavailable]
    svcA --> db[Shared data store]
    svcB -.-> db
  end

The left side shows a single-node application where one server failure stops the whole system. The right side shows a distributed application where Service B is unavailable while clients, the gateway, Service A, and the database may still be running. The system is neither fully healthy nor fully down.

Lost Response After Successful Processing

sequenceDiagram
  participant B as Service B
  participant N as Network
  participant A as Service A
  participant D as Data Store
  B->>N: createOrder(requestId=42)
  N->>A: deliver request
  A->>D: commit order 42
  D-->>A: commit ok
  A-->>N: response ok
  Note over N: response is lost
  N-->>B: no response
  B->>B: timeout, outcome unknown

Service B sends a create request to Service A. Service A commits the operation, but the response is lost on the network. Service B only observes a timeout, so it cannot know whether the operation happened without an idempotency key, status lookup, or reconciliation flow.

Inconsistent State Across Two Services

flowchart LR
  orders[Order service state paid]
  shipping[Shipping service state pending]
  event[Payment authorized event delayed]
  orders --> event
  event -.-> shipping
  orders -.-> userA[Customer support view paid]
  shipping -.-> userB[Fulfillment view pending]

The order service has recorded an order as paid, while the shipping service still sees it as pending because the payment event has not arrived or has not been processed. The diagram is understandable by reading the state labels, not by relying on color.

Distributed Request Path With Logs, Metrics, and Traces

flowchart LR
  client[Client] --> gateway[API Gateway]
  gateway --> svcA[Service A]
  svcA --> broker[Message Broker]
  broker --> svcB[Service B]
  svcB --> db[Database]
  gateway -.-> obs[Observability platform]
  svcA -.-> obs
  broker -.-> obs
  svcB -.-> obs
  db -.-> obs

A request moves from client to gateway to services and data stores. Each component emits telemetry. Traces connect the path, logs explain local decisions, and metrics show aggregate health.

Enterprise Distributed Systems Stack

flowchart TB
  clients[Users clients partner systems]
  edge[Edge DNS CDN load balancer API gateway]
  services[Services domain APIs workflows AI agents]
  events[Events broker streams queues]
  data[Data relational document cache search lake]
  platform[Platform containers orchestration cloud regions]
  observe[Operations logs metrics traces alerts]
  govern[Governance security policy compliance cost]
  clients --> edge --> services
  services --> events
  services --> data
  events --> data
  platform --> services
  services --> observe
  platform --> observe
  govern -.-> edge
  govern -.-> services
  govern -.-> data

The stack moves from users and clients through edge routing, services, events, data, platform infrastructure, observability, and governance. Each layer introduces responsibilities that must be designed and operated explicitly.

Week 1 Service A and Service B Lab Architecture

flowchart LR
  student[Student browser or curl]
  serviceB[Service B caller]
  serviceA[Service A worker]
  store[Service A local store]
  logs[Console logs with request IDs]
  student --> serviceB
  serviceB --> serviceA
  serviceA --> store
  serviceB -.-> logs
  serviceA -.-> logs
  serviceA -.-> serviceB

Service B calls Service A over HTTP and includes a request ID. Service A records work in its local store and both services emit logs. The lab varies response delay and dropped responses so students can see how uncertainty appears at Service B.

Worked Example

Consider a request to place an order:

  1. The client sends POST /orders to the API gateway.
  2. The gateway verifies identity and forwards the request to the order service.
  3. The order service stores a pending order and publishes an OrderCreated event.
  4. The payment service consumes the event and attempts authorization.
  5. The payment service records its result and publishes a PaymentAuthorized or PaymentFailed event.
  6. Observability tools collect logs, metrics, and traces across the gateway, services, broker, and databases.

The single user action becomes a distributed workflow. To make it robust, the system needs idempotent operations, stable message schemas, timeouts, retry policies, and a plan for compensating when later steps fail.

In-Class Discussion Prompts

  • Where do you see partial failure in systems you use every day?
  • When should a system retry, and when should it stop retrying?
  • Which is more important for a payment system: availability or consistency? What about a streaming recommendation system?
  • What should an AI agent do when one of its tools is slow, unavailable, or returns conflicting information?

Hands-On Activity: Trace a Distributed Request

In small groups, sketch the services involved in one familiar workflow: ride sharing, food delivery, online banking, streaming video, or an AI coding assistant. Identify:

  • clients and entry points
  • services or components
  • data stores
  • network calls or events
  • likely partial failures
  • telemetry needed to debug the workflow

The goal is not a perfect architecture. The goal is to recognize boundaries, communication paths, and failure modes.

Knowledge Check

Why is partial failure harder than total failure?

With total failure, the system is clearly unavailable. With partial failure, some components continue responding while others are slow, unreachable, or inconsistent, so clients and services must decide whether to retry, fail over, wait, or degrade.

What is one practical difference between distributed and parallel systems?

Parallel systems usually emphasize speeding up computation with tightly coordinated processors. Distributed systems emphasize coordination across independent networked components that may fail, lag, or disagree.

Why does observability matter from the first module?

Distributed systems fail across service boundaries. Logs, metrics, and traces are how engineers reconstruct what happened when no single process has the whole story.

Key Takeaways

  • A distributed system is defined by independent components coordinating through communication, usually across network boundaries.
  • Scalability, availability, reliability, transparency, and fault tolerance are goals that often trade off against one another.
  • Latency, partial failure, and network uncertainty make distributed systems fundamentally different from single-process applications.
  • Enterprise systems depend on distributed design in cloud platforms, microservices, payments, streaming pipelines, and AI-enabled workflows.

Additional Resources

Connection to the Final Project

Your final project should not only “use services.” It should demonstrate deliberate distributed systems thinking: clear service boundaries, communication choices, data ownership, failure handling, observability, and security assumptions. Module 1 gives you the vocabulary to describe those choices before you implement them.