Designing Retry Budgets and Backpressure for Resilient HTTP APIs

Learning outcome

After this lesson, you can design a distributed HTTP call path that tolerates transient faults without turning retries into an outage multiplier. You will be able to distinguish attempt timeouts, total deadlines, per-request retry limits, fleet-wide retry budgets, circuit breakers, concurrency limits, rate limits, queues, and backpressure; explain their interactions; and describe safe failure recovery in a system-design interview.

Intuition

A retry is not free fault tolerance. It is additional offered load sent precisely when a dependency may already be unhealthy. If 20,000 original requests per second each permit two retries, the dependency can receive as many as 60,000 attempts per second. Longer latency also keeps connections, memory, threads, and concurrency permits occupied, so collapse can occur before CPU reaches 100%.

A per-request retry limit bounds one caller's amplification but does not coordinate the fleet. A retry budget places a shared bound on aggregate retry traffic. For example, a service might allow retry attempts equivalent to at most 10% of eligible original attempts over a rolling interval, with a small burst allowance. When the budget is exhausted, callers fail, degrade, or use cached data rather than adding more load.

Backpressure is the receiving side's ability to make upstream demand slow down or stop. In HTTP systems it is usually implemented through bounded admission, fast overload responses, concurrency limits, rate limits, bounded queues, and client behavior that respects those signals. It is a control loop, not merely an error status.

Deep dive

Distinct controls

Timeouts should derive from the remaining end-to-end deadline. Starting every retry with a fresh full timeout can exceed the user's latency objective and leave obsolete work running downstream. Retries should normally be restricted to failures likely to be transient, use exponential backoff with jitter, and require enough remaining deadline to make another attempt useful.

HTTP method labels alone are insufficient for retry safety. Reads are often retryable, but a nominally idempotent operation can still trigger metering or audit side effects. Mutations require application-level guarantees such as a durable idempotency key, deduplication scope, and stable replayed result. A connection failure after sending a request is ambiguous: the server may have committed even though the client saw no response.

A production retry budget needs a defined scope. A budget local to each process is simple but expands when the fleet scales and can be unfair during skewed traffic. A centrally coordinated budget is more accurate but adds latency and another dependency. A common compromise uses local token buckets replenished in proportion to admitted original traffic, conservative per-instance allocations, and bounded bursts. Critical and best-effort traffic should not silently consume the same pool.

Backpressure should occur near the constrained resource. An adaptive concurrency limiter can reduce admissions as latency or queueing grows. A small bounded queue can handle brief bursts; once full, rejection is safer than unbounded waiting. Overload responses should be cheap to produce and may include a retry delay when the server can estimate one. Clients still need jitter and budgets because thousands of callers obeying the same delay can synchronize.

Cascading-failure incident

Suppose an inventory service normally handles 8,000 attempts per second. A database failover raises its latency from 40 ms to 1.5 seconds. Gateway clients have a two-second attempt timeout and permit two immediate retries. In-flight work rises sharply, connection pools saturate, and requests queue. Some original attempts eventually succeed, but their callers have already retried. The inventory service now performs duplicate reads while gateways retain more sockets and memory. Health checks begin timing out, instances are removed, and capacity falls further. A circuit breaker opens on many gateway instances at once; thirty seconds later they all probe simultaneously, producing another spike.

A safer design propagates a total deadline, uses one short jittered retry only for eligible failures, and spends it from a 10% retry budget. Inventory enforces an adaptive concurrency limit and a bounded queue, rejecting excess work quickly. Gateways convert overload into controlled degradation, such as serving slightly stale inventory where product policy permits. Circuit-breaker probes are sparse and randomized. This does not make the database failover invisible; it prevents a partial impairment from becoming a fleet-wide collapse.

Observability

Track logical requests separately from physical attempts. Useful signals include attempt-to-original ratio, retry-budget utilization and exhaustion, retry outcomes, deadline remaining at each hop, timeout phase, breaker state and probe results, admitted and rejected requests, concurrency, queue depth and age, dependency latency distributions, connection-pool saturation, and overload-response rates. Correlate attempts with one logical-request identifier while avoiding unbounded metric labels such as raw URLs, customer IDs, or idempotency keys.

Alert on symptoms and control-loop behavior together. Rising retry ratio with falling success rate indicates harmful amplification. High queue age with moderate CPU can reveal a downstream pool or lock bottleneck. A permanently full retry budget may mean the dependency is unhealthy, but it can also expose a retry policy targeting non-transient failures.

Failure modes

• Retrying every error, including authentication failures, validation errors, deterministic conflicts, and durable not-found results.
• Layering retries at the SDK, service, proxy, and gateway so multiplicative attempts are hidden from each layer.
• Using fixed delays or identical breaker reset intervals, creating retry and recovery herds.
• Treating a circuit breaker as rate limiting; a closed breaker can still overload a merely slow dependency.
• Allowing an unbounded queue, which converts overload into memory growth, stale work, and deadline violations.
• Retrying after the caller has disconnected or after the end-to-end deadline has expired.
• Sharing one budget across unrelated dependencies or priorities, allowing one failure domain to starve another.
• Returning overload responses that are expensive to construct, or advertising retry delays unsupported by actual recovery capacity.
• Recovering all traffic at once. After an incident, use gradual admission, randomized half-open probes, conservative initial concurrency, and rollback criteria. Drain or discard expired queued work before increasing load.

Interview drill

1. Why is a two-retry limit insufficient? State that it bounds one logical request but can still triple fleet load; add an aggregate retry budget.
2. Where should the budget live? Discuss local efficiency versus global accuracy, then propose scoped local allocations with telemetry or coordination appropriate to the risk.
3. When should an HTTP mutation be retried? Only when the failure is transient, the deadline permits it, and application semantics make ambiguous replay safe through durable idempotency or equivalent guarantees.
4. Why use concurrency limiting as well as rate limiting? Because latency inflation increases in-flight resource consumption even at a constant arrival rate.
5. How do you recover safely? Keep backpressure active, discard expired work, probe with jitter, ramp admission gradually, watch latency and rejection signals, and stop or reverse the ramp when saturation returns.

A strong interview answer begins with the overload feedback loop, assigns each mechanism a distinct responsibility, defines retry eligibility and budget scope, and closes with observability plus controlled recovery.

Revision checklist

• Separate logical requests from physical attempts.
• Bound each attempt and propagate one end-to-end deadline.
• Retry only transient, semantically safe operations with jittered backoff.
• Enforce both a per-request attempt cap and a fleet-aware retry budget.
• Protect constrained resources with concurrency limits and bounded queues.
• Use rate limits for throughput policy, not as a substitute for concurrency control.
• Use circuit breakers to avoid repeated doomed calls, with randomized recovery probes.
• Make overload rejection fast and ensure callers honor backpressure.
• Measure retry amplification, queue age, rejections, deadlines, and recovery behavior.
• Restore traffic gradually rather than reopening the entire fleet at once.