Designing Retry Budgets and Backpressure for Resilient HTTP APIs

Learning outcome

After this lesson, you can design a distributed HTTP call path that tolerates transient faults without turning retries into an outage multiplier. You will be able to distinguish attempt timeouts, total deadlines, per-request retry limits, fleet-wide retry budgets, circuit breakers, concurrency limits, rate limits, queues, and backpressure; explain their interactions; and describe safe failure recovery in a system-design interview.

Intuition

A retry is not free fault tolerance. It is additional offered load sent precisely when a dependency may already be unhealthy. If 20,000 original requests per second each permit two retries, the dependency can receive as many as 60,000 attempts per second. Longer latency also keeps connection capacity, memory, and concurrency permits occupied, so collapse can occur before CPU reaches 100%.

A per-request retry limit bounds one caller's amplification but does not coordinate the fleet. A retry budget places a shared bound on aggregate retry traffic. For example, a service might allow retry attempts equivalent to at most 10% of eligible original attempts over a rolling interval, with a small burst allowance. When the budget is exhausted, callers fail, degrade, or use cached data rather than adding more load.

Backpressure is the receiving side's ability to make upstream demand slow down or stop. In HTTP systems it is usually implemented through bounded admission, fast overload responses, concurrency limits, rate limits, bounded queues, and client behavior that respects those signals. It is a control loop, not merely an error status.

Deep dive

Distinct controls

Control	What it bounds	Primary purpose	What it does not solve
Attempt timeout	Target time for one attempt	Requests cancellation or ends waiting, depending on the mechanism	Does not prove underlying work ended or bound the whole retry sequence
End-to-end deadline	Total time available across queueing, attempts, and processing	Preserves the caller's latency objective	Does not regulate fleet-wide load by itself
Per-request retry limit	Attempts generated by one logical request	Prevents an individual retry loop	Does not stop synchronized fleet amplification
Retry budget	Aggregate retry traffic over a population and interval	Keeps retries proportional to healthy demand	Does not reject excessive original traffic
Circuit breaker	Calls sent while a dependency appears unhealthy	Fails fast and limits repeated doomed work	Is not a capacity limiter and can reopen in a herd
Concurrency limit	Simultaneous admitted operations	Protects finite in-flight resources	Does not directly control request rate
Rate limit	Admissions per unit time	Enforces quotas or protects throughput	Can admit too much concurrency when latency rises
Bounded queue	Waiting work	Absorbs short bursts and smooths scheduling	Adds latency and cannot safely absorb sustained overload
Backpressure	Propagation of capacity scarcity upstream	Aligns offered load with available capacity	Fails if clients ignore or defeat the signal

Timeouts should derive from the remaining end-to-end deadline. Starting every retry with a fresh full timeout can exceed the user's latency objective and leave obsolete work running downstream. Retries should normally be restricted to failures likely to be transient, use exponential backoff with jitter, and require enough remaining deadline to make another attempt useful.

Idempotency describes the intended effect of repetition, not identical responses or absence of audit logs. HTTP defines PUT and DELETE as idempotent even though they are unsafe methods. Incidental logging does not negate idempotency. Check that the endpoint honors its semantics. For a non-idempotent mutation with an unknown outcome, require a replay-safety guarantee, such as durable deduplication, before automatic retry. A connection failure after sending a request is ambiguous: the server may have committed even though the client saw no response. See HTTP idempotency.

A production retry budget needs a defined scope. Fixed per-process retry quotas and burst allowances grow with the instance count. A proportional per-process budget does not automatically increase the aggregate retry fraction, but its denominator, burst allowance, and traffic skew still matter. A centrally coordinated budget is more accurate but adds latency and another dependency. A common compromise uses local token buckets replenished in proportion to admitted original traffic, conservative per-instance allocations, and bounded bursts. Critical and best-effort traffic should not silently consume the same pool.

Backpressure should occur near the constrained resource. An adaptive concurrency limiter can reduce admissions as latency or queueing grows. A small bounded queue can handle brief bursts; once full, rejection is safer than unbounded waiting. Overload responses should be cheap to produce and may include a retry delay when the server can estimate one. Clients still need jitter and budgets because thousands of callers obeying the same delay can synchronize.

Cascading-failure incident

Consider an illustrative incident, not benchmark data: an inventory service normally handles 8,000 attempts per second. A database failover raises processing time from 40 ms to 1.5 seconds. Gateway clients have a two-second attempt timeout and permit two immediate retries. In-flight work rises sharply, connection pools saturate, and requests queue. For an attempt that spends 800 ms in a queue before 1.5 seconds of processing, the 2.3-second total exceeds the caller’s two-second timeout. If the original work continues, it may succeed after the caller has started a retry. The inventory service now performs duplicate reads while gateways retain more sockets and memory. Health checks begin timing out, instances are removed, and capacity falls further. A circuit breaker opens on many gateway instances at once; thirty seconds later they all probe simultaneously, producing another spike.

A safer design propagates a total deadline, uses one short jittered retry only for eligible failures, and spends it from a 10% retry budget. Inventory enforces an adaptive concurrency limit and a bounded queue, rejecting excess work quickly. Gateways convert overload into controlled degradation, such as serving slightly stale inventory where product policy permits. Circuit-breaker probes are sparse and randomized. These controls aim to contain the impairment; they must be tested together against the dependency’s actual recovery capacity.

Observability

Track logical requests separately from physical attempts. Useful signals include attempt-to-original ratio, retry-budget utilization and exhaustion, retry outcomes, deadline remaining at each hop, timeout phase, breaker state and probe results, admitted and rejected requests, concurrency, queue depth and age, dependency latency distributions, connection-pool saturation, and overload-response rates. Correlate attempts with one logical-request identifier while avoiding unbounded metric labels such as raw URLs, customer IDs, or idempotency keys.

Alert on symptoms and control-loop behavior together. Rising retry ratio with falling success rate indicates harmful amplification. High queue age with moderate CPU can reveal a downstream pool or lock bottleneck. Persistent retry-budget exhaustion may mean the dependency is unhealthy, but it can also expose a retry policy targeting non-transient failures.

Failure modes

• Retrying every error, including authentication failures, validation errors, deterministic conflicts, and durable not-found results.
• Layering retries at the SDK, service, proxy, and gateway so multiplicative attempts are hidden from each layer.
• Using fixed delays or identical breaker reset intervals, creating retry and recovery herds.
• Treating a circuit breaker as rate limiting; a closed breaker can still overload a merely slow dependency.
• Allowing an unbounded queue, which converts overload into memory growth, stale work, and deadline violations.
• Retrying after the caller has disconnected or after the end-to-end deadline has expired.
• Sharing one budget across unrelated dependencies or priorities, allowing one failure domain to starve another.
• Returning overload responses that are expensive to construct, or advertising retry delays unsupported by actual recovery capacity.
• Recovering all traffic at once. After an incident, use gradual admission, randomized half-open probes, conservative initial concurrency, and rollback criteria. Drain or discard expired queued work before increasing load.

Worked budget check

Suppose one fixed observation window contains 1,000 eligible original attempts. Define the retry allowance as 10% of those originals, with no initial tokens, burst, or carryover. At most 100 retries are permitted, so the total is at most 1,100 attempts. If 90 retries have already been sent, 10 remain; another 50 eligible failures cannot all be retried. Permission also depends on the individual request’s remaining deadline and retry safety.

The denominator matters: retries/originals = 10% is different from retries/all attempts = 10%. The latter allows approximately 111 retries for 1,000 originals. State the definition before quoting a percentage. These figures illustrate the policy, not a recommended production threshold.

Interview drill

1. Why is a two-retry limit insufficient? State that it bounds one logical request but can still triple fleet load; add an aggregate retry budget.
2. Where should the budget live? Discuss local efficiency versus global accuracy, then propose scoped local allocations with telemetry or coordination appropriate to the risk.
3. When should an HTTP mutation be retried? Only when the failure is transient, the deadline permits it, and application semantics make ambiguous replay safe through durable idempotency or equivalent guarantees.
4. Why use concurrency limiting as well as rate limiting? Because latency inflation increases in-flight resource consumption even at a constant arrival rate.
5. How do you recover safely? Keep backpressure active, discard expired work, probe with jitter, ramp admission gradually, watch latency and rejection signals, and stop or reverse the ramp when saturation returns.

A strong interview answer begins with the overload feedback loop, assigns each mechanism a distinct responsibility, defines retry eligibility and budget scope, and closes with observability plus controlled recovery.

Revision checklist

• Separate logical requests from physical attempts.
• Bound each attempt and propagate one end-to-end deadline.
• Retry only transient, semantically safe operations with jittered backoff.
• Enforce both a per-request attempt cap and a fleet-aware retry budget.
• Protect constrained resources with concurrency limits and bounded queues.
• Use rate limits for throughput policy, not as a substitute for concurrency control.
• Use circuit breakers to avoid repeated doomed calls, with randomized recovery probes.
• Make overload rejection fast and ensure callers honor backpressure.
• Measure retry amplification, queue age, rejections, deadlines, and recovery behavior.
• Restore traffic gradually rather than reopening the entire fleet at once.

Sources

• Google SRE: handling overload and retry budgets
• Polly: cooperative timeout behavior