Module 4: Architecture Styles for Enterprise Distributed Systems

Learning Path

The September 21 session uses 32 slides, including discussions and a team workshop. The suggested pacing totals 142 minutes with an 8-minute break after the duplicate-event discussion. All marketplace requirements, identifiers, and failure scenarios are teaching examples.

Required Preparation

Review Module 3’s REST contracts, deadlines, and uncertain outcomes. Read the introductions and tradeoffs in the N-tier, microservices, event-driven architecture, and Bulkhead references below. Bring a candidate team-project problem and a way to draw a diagram. The required workshop needs no new software.

Optional: read the transactional outbox and anti-corruption layer references. Bring the two-service application from Module 3 if you want to implement a failure experiment after the design exercise.

The Running Example

A fictional campus marketplace lets a buyer reserve the last item, pay through a mock payment provider, and receive pickup instructions. Inventory must prevent overselling. Payments must prevent a repeated request from charging twice. A notification may be delayed without changing whether an order is confirmed. These requirements give us a way to evaluate architecture choices.

Layered Systems

A layered design separates request handling, application coordination, domain rules, and persistence. For a stock reservation, the handler validates input, domain code checks the requested quantity, and persistence enforces the stock invariant under concurrency. Reading stock and later decrementing it without protection allows two buyers to reserve the same item.

A worked reservation transaction can conditionally decrement stock only if sufficient quantity remains, verify that the update succeeded, and insert the reservation before committing. Failure to insert the reservation rolls back the decrement. A stable request key with a uniqueness constraint helps repeated requests return the existing reservation. The exact transaction mechanism depends on the database.

flowchart LR
 U[HTTP handler] --> A[Checkout application]
 A --> D[Domain rules]
 A --> R[Repository]
 R --> DB[(Database)]

A logical layer does not require a separate server. One application process can contain every layer. A deployment tier introduces a runtime boundary, such as the network between application and database. See the N-tier architecture guide.

A modular monolith groups business capabilities inside one deployable application. Internal interfaces and ownership rules can keep catalog, ordering, and inventory understandable. This can fit a small team with a short delivery window. Shared process capacity and a single release still connect their operational fate.

Microservices and Data Ownership

A microservice owns a cohesive business capability and its write rules. Other services use its public contract rather than updating its tables. Independent evolution depends on compatible contracts and operational practices, not on how many containers exist. See the microservices guide.

flowchart TB
 UI[Marketplace client] --> O[Ordering]
 O --> I[Inventory]
 O --> P[Payments]
 O --> OD[(Order data)]
 I --> ID[(Stock data)]
 P --> PD[(Payment data)]

Inventory enforces stock constraints. Payments manages charge identity and provider reconciliation. Ordering tracks progress across both. When these owners use separate transactions, ordering must handle intermediate states. A timeout after requesting a charge cannot safely mean either “paid” or “not paid.”

A service per table often fragments business rules into chatty calls. A shared database server can host separately owned data, but resource contention and outages remain shared risks. Direct cross-service writes further couple schema changes and business semantics.

Event-Driven Systems

A command asks an owner to attempt an action. An event reports a completed fact. A query reads state. For example, ReserveStock requests a reservation, while StockReserved records success. The meaning of OrderPlaced requires a contract: it might mean received rather than paid.

flowchart LR
 O[Ordering] -->|OrderConfirmed| B[Broker]
 B --> N[Notification consumer]
 B --> A[Analytics consumer]

Notification and analytics can react after the order commits. This reduces immediate waiting but introduces lag, duplicate handling, and monitoring of unfinished work. Each subscriber needs its own logical delivery if both must observe the event. Competing workers on one queue usually divide work instead. See the event-driven architecture guide.

Reliable Publication and Repeated Effects

Writing order state and separately publishing an event creates a failure window. With an outbox, one local transaction writes both the state and a pending event. A relay publishes pending events and retries when necessary. Publication followed by a crash before recording completion can produce duplicates. See the AWS transactional outbox pattern.

flowchart LR
 O[Ordering] --> T[Local transaction]
 T --> D[(Order and outbox rows)]
 D --> R[Relay]
 R --> B[Broker]
 B --> C[Consumer]

For local consumer effects, atomically record the processed event ID and update business state. For an external charge, local deduplication alone cannot close the gap between a provider action and the local record. Use a stable provider idempotency key, query the provider’s result when possible, and reconcile uncertainty. Define deduplication retention to cover the supported retry or replay window.

Workflow Recovery

stateDiagram-v2
  direction LR
 [*] --> Pending
 Pending --> Reserved: stock reserved
 Reserved --> Confirmed: payment succeeds
 Reserved --> ReleasePending: payment declined
 ReleasePending --> Cancelled: stock released
 Reserved --> Reconciling: payment outcome unknown
 Reconciling --> Confirmed: charge found
 Reconciling --> ReleasePending: confirmed no charge

A known payment decline can initiate stock release. An unknown payment result enters reconciliation. Compensation is an additional business action that may need retries and an operator-visible pending state. It cannot erase every external effect. Reservation expiration racing with a late successful payment requires a policy, such as refunding a charge if fulfillment is no longer possible.

Discussion: An email sender crashes after sending but before acknowledging an event. Can your design guarantee exactly one email without help from the provider? Explain the failure window and the user experience you would accept.

Integration Boundaries

For every edge, name the owner, operation, schema, business meaning, identity, permission, deadline, and repeat behavior. State which system is authoritative and how the caller learns the final outcome after a timeout.

flowchart LR
 I[Inventory] -->|quantity in units| A[Supplier adapter]
 A -->|quantity in cases| S[Legacy supplier]

The supplier adapter translates both representation and meaning. A supplier reporting three cases does not report three units. Record pack size, handle invalid conversions, and alert on inconsistent contracts. An adapter adds code and latency, so use it where semantic differences justify that cost. See the anti-corruption layer pattern.

Failure Domains and Containment

flowchart TB
 subgraph H[One host]
 O[Ordering container]
 I[Inventory container]
 end
 O --> DB[(Shared database server)]
 I --> DB
 P[Payments on another host] --> DB

Draw shared hosts, database servers, brokers, zones, credentials, quotas, and deployment mechanisms. Two replicas can still fail together. Separate service ownership does not remove a shared dependency.

flowchart LR
 P[Payment slows] --> W[Checkout workers wait]
 W --> Q[Requests queue]
 Q --> T[Clients time out]
 T --> R[Retries add load]
 R --> W

A deadline bounds waiting. A circuit breaker temporarily stops attempts to a failing dependency. A bulkhead limits the capacity one dependency can consume. Backpressure slows incoming work when downstream capacity is insufficient. These controls complement one another. See the Bulkhead pattern.

For an illustrative steady-state calculation, 40 requests per second with 2 seconds average residence time require about 80 in-flight requests on average. A pool limited to 50 cannot sustain that workload. Measure latency distributions, queue age, saturation, and completion rates rather than assuming more retries help.

A safe fallback preserves business truth. Hiding recommendations is acceptable when that optional service fails. Reporting a payment as successful merely because it timed out changes the meaning of confirmation. Keep an explicit processing state and a path to reconciliation.

Architecture Workshop

Required, 15 minutes: Work in teams with a paper or digital diagram. Choose an initial architecture for the marketplace and explain why it fits your team.

1. Spend five minutes drawing capabilities, data owners, and integration edges. Label each edge as a command, query, or event.
2. Spend four minutes tracing success and a payment timeout. Record the state visible to the buyer at each step.
3. Spend three minutes reviewing another team’s design. Identify one shared failure domain or uncertain outcome they have missed.
4. Spend three minutes revising your diagram. Record the tradeoff behind one changed boundary.

Your design should answer: who prevents overselling, how does a repeated checkout avoid duplicate effects, and what remains usable when notification stops? These are learning prompts. Canvas remains the official source for any assessed requirements.

Optional Implementation Experiment

Extend the Module 3 services with a deterministic mock payment provider and a persistent request key. First record the successful checkout result and its correlation ID. Then make the provider record a successful charge but delay its response beyond the caller’s deadline. Retry with the same key and query the provider’s recorded result. Verify one charge and a final order state consistent with that result.

Next stop the notification consumer. If your design uses durable messaging, confirm that order completion and notification progress remain distinct. Restart the consumer and inspect recovery. Record the number of unique business effects, unresolved orders, and the age of pending work. A diagram-only team can conduct the same exercise as a state-transition table.

Team Project Ideas

These are candidate scopes and experiment suggestions, not new official assignment rules. Use synthetic data and mock external systems. Follow Final Project for public guidance and Canvas for official requirements, submissions, deadlines, and grading.

1. Campus Marketplace

Problem: Campus sellers risk losing buyers’ trust when a scarce item sells twice or a retried checkout charges twice. Help student buyers receive a reliable purchase and pickup commitment.

Build one checkout workflow with inventory, ordering, and a mock payment provider. Add notification as an asynchronous consumer if it serves the design. Demonstrate a buyer reserving and purchasing one item.

• Invariant: available stock never becomes negative, and a checkout key produces at most one charge.
• Experiments: race two buyers for the last unit, then delay a successful payment response and retry.
• Evidence: reservation records, provider charge count, final order state, and a trace across the workflow.
• Extension: reservation expiration with a late payment and a compensating refund.

2. Event Ticketing

Problem: Event organizers need to prevent conflicting bookings while allowing people to finish a purchase. Help attendees obtain a clear seat commitment and return abandoned holds to availability.

Build seat holds with expiration and purchase confirmation. Use a clearly defined owner for each seat. A browser UI is optional if an API script demonstrates the entire workflow.

• Invariant: at most one confirmed purchase owns a seat.
• Experiments: two simultaneous holds on one seat, then a worker restart during expiration.
• Evidence: transaction outcomes, seat state history, and recovery without double allocation.
• Extension: an asynchronous waiting list with an explicit fairness policy.

3. Supplier Integration Hub

Problem: Small retailers receive supplier reports with inconsistent units and update schedules. Help staff distinguish validated sellable stock from stale or incompatible information.

Connect two mock suppliers with different schemas, units, and error conventions to one inventory view. Use an adapter to protect internal meaning.

• Invariant: only validated unit conversions update sellable stock.
• Experiments: change a supplier’s pack size, then take that supplier offline.
• Evidence: rejected updates, last-successful synchronization time, and continued processing for the other supplier.
• Extension: compatible schema evolution and replay of corrected records.

4. AI Support Triage Workflow

Problem: Community support staff need to sort incoming requests and prepare responses promptly without losing control of decisions. Help them review suggestions and approve any resulting action. AI is one possible approach, not the problem itself.

Accept a support request, queue a suggestion job, and present the result for human approval before any action. Use a deterministic mock model as the baseline. An external model is optional.

• Invariant: generated output cannot execute an unapproved action.
• Experiments: delay the model beyond its deadline, then deliver a suggestion job twice.
• Evidence: one logical job result, bounded queue behavior, and an audit trail linking approval to the specific proposed action.
• Extension: per-tenant capacity limits so one burst cannot block all requests.

Proposal Outline

State the problem, actors, one complete workflow, and an invariant. Draw components, data ownership, and contracts. Compare your selected style with one credible alternative. Mark failure domains and explain two experiments with expected outcomes and observable evidence. Divide work around demonstrable responsibilities while ensuring every teammate can explain the full workflow.