Learning outcome
By the end of this lesson, you can explain consistency and availability as workload-specific design choices rather than slogans. You can identify invariants, define behavior during failures, distinguish normal operation from network partitions, and defend different choices for different operations in the same system.
Intuition
Start with what users must never observe. A consistency model is useful only when connected to a concrete invariant such as “a coupon cannot be redeemed more than once” or “a user can read their own profile update.” Availability is also more precise than “the service stays up”: ask which operations must accept requests, at what latency, and whether a provisional or stale response is acceptable.
CAP does not force a system to choose one permanent identity as “CP” or “AP.” During a network partition, a replicated operation cannot simultaneously guarantee linearizable behavior and guarantee a successful response from every non-failing node. Real systems make choices per operation, data item, region, and failure mode. Outside partitions, latency, consistency, durability, and cost still involve trade-offs, but CAP alone does not describe them.
Use this decision sequence:
- State the user-visible operation and its invariants.
- Define the required consistency: linearizability, serializable transactions, read-your-writes, monotonic reads, bounded staleness, or eventual convergence.
- Define availability quantitatively: success rate, timeout budget, and permitted degraded responses.
- Name failures: packet loss, region isolation, leader loss, replica lag, retries, and client timeouts.
- Choose partition behavior: reject, wait, serve stale data, accept tentative writes, or route to an authority.
- Explain reconciliation, recovery time, observability, and operational cost.
Deep dive
Strong consistency is justified when concurrent decisions must observe one authoritative order. Typical mechanisms include a consensus-backed leader, quorum protocols, fencing tokens, uniqueness constraints, and idempotency records. These mechanisms can increase latency and may reject or delay operations when a quorum is unavailable. “Use a quorum” is incomplete unless you explain quorum membership, overlapping reads and writes, leader leases, stale replicas, and what happens during reconfiguration.
Eventual consistency is appropriate when temporary divergence is tolerable and updates have a safe merge rule. Availability then depends on accepting operations locally and reconciling later. You must still define conflict semantics: last-write-wins can lose valid updates, clocks can be skewed, and duplicate delivery is normal. Alternatives include operation-based updates, version vectors, conflict-free replicated data types, or domain-specific resolution.
Consistency can be asymmetric. A product page may serve stale descriptions while inventory reservation uses an authoritative conditional write. A social application may accept a post locally but require stronger consistency for privacy-policy changes. Session guarantees such as read-your-writes can provide a good user experience without making every read globally linearizable.
Worked scenario 1: limited inventory checkout
Invariant: confirmed reservations must not exceed stock. During a region partition, allowing every region to confirm against the same remaining units risks overselling. A defensible design routes reservations to a consensus-backed owner for each stock item or pre-allocates bounded inventory tokens to regions. Without authority or an available token, checkout returns a retryable failure or “pending” result rather than false confirmation.
Catalog browsing remains available from local replicas and may be slightly stale. Reservation requests use idempotency keys so retries do not consume inventory twice. Recovery includes reconciling pending attempts, expiring abandoned holds, and monitoring quorum health, token exhaustion, and reservation latency. The trade-off is reduced write availability for a strict business invariant, not reduced availability for the entire storefront.
Worked scenario 2: social timeline and privacy
Timeline reads prioritize low latency and can use cached, eventually consistent fan-out results. A newly published post may appear late or temporarily out of order. Posting can acknowledge after durable local acceptance, with background replication and idempotent fan-out.
Privacy changes require different treatment. Once a block is confirmed, the design requires immediate global enforcement. The authoritative policy write is not reported as confirmed until the revocation mechanism has acknowledged it. Every protected read revalidates against that authority or a cache that can prove it has the current policy version; if neither proof nor the authority is reachable during a partition, access fails closed. Recovery removes previously materialized timeline entries and audits any exposure. This hybrid answer shows that availability decisions follow the consequence of staleness, not the broad product category.
Failure modes
Common weak answers include:
- “Choose CP for money and AP for social media.” This labels domains without identifying operations or invariants.
- “CAP means pick any two.” Partition tolerance is not usually optional across an unreliable network, and the consistency-versus-availability tension becomes unavoidable specifically during a partition.
- “Eventual consistency means no consistency.” Eventual convergence is a guarantee under stated assumptions, but it does not define conflict resolution or user session behavior.
- “Quorums guarantee consistency.” Poorly configured quorums, stale leases, non-overlapping replica sets, or unsafe membership changes can violate that claim.
- “Return success and reconcile later.” Compensation may be impossible for leaked private data, duplicate external payments, or legally binding confirmation.
- Ignoring ambiguous outcomes. A client timeout does not prove that a write failed; retries need idempotency and status lookup.
- Ignoring recovery. A design must explain backlog draining, conflict repair, cache invalidation, and how operators know convergence has completed.
Interview drill
When given a distributed design prompt, say: “I will choose guarantees per operation. First I want to identify irreversible outcomes and stale-read consequences.” Then ask about regional topology, latency targets, acceptable staleness, outage expectations, data ownership, and whether degraded or pending responses count as available.
For each critical operation, present a compact decision record: invariant, consistency level, normal path, partition path, client-visible response, retry semantics, reconciliation, and metrics. Compare at least one alternative. For example, centralized inventory offers simpler correctness but higher cross-region latency; regional token allocation lowers latency and preserves a global bound but adds allocation, stranded-capacity, and rebalancing complexity.
A senior answer also discusses operational cost: consensus maintenance, hot partitions, capacity needed during failover, clock assumptions, disaster recovery, repair tooling, and testing with delayed, duplicated, reordered, or dropped messages.
Revision checklist
- Did I define user-visible invariants before naming a consistency model?
- Did I separate normal latency trade-offs from partition behavior?
- Did I specify availability for individual operations and response types?
- Did I explain what stale reads, duplicate writes, and client timeouts do?
- Did I choose safe retry and idempotency semantics?
- Did I describe conflict resolution or justify rejecting writes?
- Did I cover recovery, convergence, observability, and manual repair?
- Did I distinguish confirmation from tentative acceptance?
- Did I compare an alternative and its latency and operational costs?
- Did I avoid treating the whole system as permanently “CP” or “AP”?