A production agent never has one way to answer a question - it has a chain: the fast cheap model, then the strong expensive one, then the cached stale answer, then a canned apology. What makes the chain safe is that it runs under two hard limits at once: a wall-clock deadline and a step budget. Without them, a chain of three flaky providers is just three ways to blow your latency SLO.
The chain is tried in order. Before each attempt, both guards are checked:
If both guards pass, the provider is attempted: it costs its full latency_ms and one step whether it succeeds or fails. A success ends the run with status ok. A failure moves to the next provider. Running off the end of the chain is status exhausted.
Implement:
def execute_with_fallback(chain, outcomes, deadline_ms, max_steps):
Return a dict with:
Returns the five-key dict above. All the numbers are ints - nothing here is floating point.
chain = [{"name": "fast", "latency_ms": 100}, {"name": "strong", "latency_ms": 400}]
print(execute_with_fallback(chain, {"strong": "ok"}, 1000, 5)["provider"]) # strong
print(execute_with_fallback(chain, {"strong": "ok"}, 400, 5)["status"]) # deadline_exceeded
In the second call fast fails after 100ms, and 100 + 400 > 400, so strong is never started.
chain = [{'name':'fast','latency_ms':100},{'name':'strong','latency_ms':400}]
print(execute_with_fallback(chain, {'strong':'ok'}, 1000, 5)['provider'])
print(execute_with_fallback(chain, {'strong':'ok'}, 400, 5)['status'])strong deadline_exceeded
With a 1000ms deadline, fast is attempted (100ms, fails since it is absent from outcomes), then strong is attempted and succeeds at 500ms total. With a 400ms deadline, fast still costs 100ms, but 100 + 400 = 500 > 400, so the predictive guard blocks strong before it starts.
0 <= len(chain) <= 50; latency_ms >= 0; deadline_ms >= 0; max_steps >= 0elapsed + latency_ms > deadline_ms blocks the attemptoutcomes is treated as "error"provider is "" for every non-ok status