A model-serving client that retries immediately on failure turns a blip into an outage: every caller hammers the recovering service at once. The fix is capped exponential backoff. Simulate the policy and report exactly how long a caller waits.
The client makes attempts against a sequence of responses. After a retryable failure, and only when another attempt is left, it sleeps before trying again:
delay_i = min(base_delay * factor ** i, max_delay) # i = 0-based attempt index
The cap matters — without max_delay, doubling from 1 second reaches over an hour by the 12th retry.
Which statuses are retryable:
Stop after max_attempts attempts with outcome "exhausted". There is no sleep after the final attempt — waiting when you will never retry is pure added latency, and it is the classic off-by-one here.
If the caller makes more attempts than statuses has entries, the last status repeats.
Implement:
def retry_schedule(statuses, base_delay, factor, max_delay, max_attempts):
Return a dict with keys "attempts" (int), "delays" (list of the sleeps actually taken, each rounded to 3 decimals), "total_wait" (their sum, rounded to 3 decimals, as a float) and "outcome" ("success", "failed" or "exhausted"), in that order.
print(retry_schedule([503, 503, 200], 0.5, 2.0, 10.0, 5))
Output:
{'attempts': 3, 'delays': [0.5, 1.0], 'total_wait': 1.5, 'outcome': 'success'}
Two 503s cost 0.5 s and 1.0 s of backoff; the third attempt succeeds, so no further delay is paid.
print(retry_schedule([503, 503, 200], 0.5, 2.0, 10.0, 5))
{'attempts': 3, 'delays': [0.5, 1.0], 'total_wait': 1.5, 'outcome': 'success'}Attempt 0 gets a 503, a retryable status, and one attempt remains, so the client sleeps min(0.5 * 20, 10) = 0.5 s. Attempt 1 gets another 503 and sleeps min(0.5 * 21, 10) = 1.0 s. Attempt 2 returns 200, so the loop stops with outcome success and no trailing sleep — 3 attempts and 1.5 s of total waiting.
statuses is non-empty; if attempts outrun it, the last status repeatsmin(base_delay * factor ** i, max_delay)delays entries and total_wait are floats rounded to 3 decimals