Token Accounting Across a Multi-Model Agent Run
Problem Statement
The OpenTelemetry GenAI conventions exist so that every span an agent emits carries the same usage attributes - input tokens, output tokens, cached input tokens - regardless of which provider served the call. Once you have that, the cost of a run is arithmetic. Getting the arithmetic wrong by a factor of ten is the standard way to be surprised by a bill.
Background
Providers quote prices per million tokens. A usage record looks like:
{"model": "big", "input_tokens": 1000, "cached_input_tokens": 800, "output_tokens": 200}
The trap: input_tokens is the total input, and cached_input_tokens is the subset of it that was served from the prompt cache. The cached portion is billed at the (much lower) cached rate, and only the remainder at the full input rate:
cost=106(input−cached)⋅pin+cached⋅pcached+output⋅pout
Double-counting the cached tokens - charging them at both rates - is the single most common bug in agent cost dashboards.
Your Task
Implement:
def compute_cost(usage, prices):
- usage: list of usage records as above; cached_input_tokens may be absent, meaning 0.
- prices: dict of model -> {"input": float, "cached_input": float, "output": float}, all per million tokens.
Return:
- total_cost: total across every record, rounded to 6 decimal places
- by_model: dict of model -> that model's total cost, rounded to 6 dp, built by iterating the model names in sorted order
- total_tokens: sum(input_tokens + output_tokens) - cached tokens are already inside input_tokens, so do not add them again
Input/Output Format
Returns the three-key dict. All money values are rounded to 6 dp; total_tokens is an int.
Sample
usage = [{"model": "big", "input_tokens": 1000, "cached_input_tokens": 800, "output_tokens": 200}]
prices = {"big": {"input": 3.0, "cached_input": 0.3, "output": 15.0}}
print(compute_cost(usage, prices)["total_cost"])
# 0.00384
200 fresh input tokens at $3/M = $0.0006, 800 cached at $0.30/M = $0.00024, 200 output at $15/M = $0.003. Total $0.00384.
Example:
usage = [{'model':'big','input_tokens':1000,'cached_input_tokens':800,'output_tokens':200}]
prices = {'big': {'input':3.0,'cached_input':0.3,'output':15.0}}
print(compute_cost(usage, prices)['total_cost'])0.00384
Only 1000 - 800 = 200 input tokens are billed at the full rate: 200 * 3.0 / 1e6 = 0.0006. The 800 cached tokens cost 800 * 0.3 / 1e6 = 0.00024. Output is 200 * 15.0 / 1e6 = 0.003. Summing gives 0.00384.
Constraints:
0 <= len(usage) <= 5000; every model inusagealso appears inpricescached_input_tokens <= input_tokens; a missingcached_input_tokensmeans 0- Prices are quoted per million tokens - divide by
10**6 - Cached tokens are a subset of input tokens: bill them once, at the cached rate
- All costs are rounded to exactly 6 decimal places
- Build
by_modelby iterating sorted model names so the dict order is deterministic
1. Background Knowledge
In production AI systems, OpenTelemetry provides standardized telemetry data, including GenAI spans that record token usage for every model call. Understanding how to aggregate this data is critical for cost management. The core concept here is token accounting, where you must distinguish between different types of tokens to calculate accurate costs. Specifically, input tokens represent the total context sent to the model, while cached input tokens represent a subset of those inputs that were retrieved from a prompt cache rather than processed anew.
Pricing models for Large Language Models (LLMs) typically quote rates per million tokens. Crucially, cached tokens are billed at a significantly lower rate than fresh input tokens. This creates a tiered pricing structure within the input category. If you treat all input tokens as "fresh," you will overestimate costs. If you double-count cached tokens (charging them at both the cached rate and the fresh rate), you will drastically overestimate costs. The formula for cost relies on subtracting cached tokens from the total input to find the "fresh" input count.
Finally, aggregation is required because an agent run may involve multiple models and multiple calls. You need to sum costs across all spans, grouping them by model for detailed reporting. The output requires precise floating-point arithmetic rounded to six decimal places to ensure financial accuracy, as small errors can compound over thousands of requests.
2. Algorithm Approach
The problem is a straightforward aggregation and arithmetic task. The general approach involves iterating through a list of usage records and applying a specific cost formula to each one.
- Iterate: Loop through each record in the usage list.
- Extract: Pull out input_tokens, cached_input_tokens (defaulting to 0 if missing), and output_tokens.
- Calculate: Apply the cost formula using the corresponding prices from the prices dictionary.
- Accumulate: Add the calculated cost to a running total and to a per-model accumulator.
- Format: Sort the model keys and round the final values to 6 decimal places.
This is a linear scan algorithm with constant-time lookups for prices. No complex data structures or sorting algorithms are needed beyond the final sorting of model keys.
3. Step-by-Step Strategy
- Initialize Accumulators: Create a variable total_cost initialized to 0.0 and a dictionary by_model to store costs per model.
- Loop Through Usage: For each record in the usage list:
- Extract model, input_tokens, and output_tokens.
- Extract cached_input_tokens, using .get('cached_input_tokens', 0) to handle missing keys safely.
- Retrieve the price dictionary for the current model from the prices input.
- Compute Fresh Input: Calculate fresh_input = input_tokens - cached_input_tokens. Ensure this value is not negative (though logically it shouldn't be).
- Calculate Record Cost:
- Compute the cost for fresh input: (fresh_input * price['input']) / 1_000_000
- Compute the cost for cached input: (cached_input_tokens * price['cached_input']) / 1_000_000
- Compute the cost for output: (output_tokens * price['output']) / 1_000_000
- Sum these three components to get record_cost.
- Update Totals:
- Add record_cost to total_cost.
- Add record_cost to by_model[model] (initializing the key if it doesn't exist).
- Add input_tokens + output_tokens to a total_tokens counter.
- Finalize Output:
- Round total_cost to 6 decimal places.
- Create the by_model output dictionary by iterating through the keys of the accumulated by_model dict in sorted order, rounding each value to 6 decimal places.
- Return the dictionary containing total_cost, by_model, and total_tokens.
4. Common Pitfalls
- Double-Counting Cached Tokens: The most common error is adding cached_input_tokens to the cost calculation twice—once as part of the total input and again as cached. Remember: input_tokens includes cached tokens. You must subtract cached tokens from the total input to get the billable "fresh" input.
- Missing Keys: The cached_input_tokens field may be absent from some records. Always use a default value of 0 when accessing this key to avoid KeyError exceptions.
- Floating-Point Precision: Direct floating-point addition can lead to tiny precision errors. While Python's round() function handles the final output requirement, be aware that intermediate calculations should be done with standard floats. The problem asks for rounding at the end, not at every step.
- Sorting Order: The by_model dictionary must be built by iterating model names in sorted order. In Python 3.7+, dictionaries maintain insertion order, so you must explicitly sort the keys before creating the final output dictionary.
- Total Tokens Calculation: The problem states total_tokens is the sum of input_tokens and output_tokens. Do not add cached_input_tokens again, as they are already included in input_tokens.
5. Time & Space Complexity
- Time Complexity: O(NlogN), where N is the number of usage records. The iteration through the usage list is O(N). The final step of sorting the unique model keys takes O(MlogM), where M is the number of unique models. Since M≤N, the overall complexity is dominated by the sorting step in the worst case, but typically M is small, making it effectively O(N).
- Space Complexity: O(M), where M is the number of unique models. We store a cost accumulator for each unique model in the by_model dictionary. The input and output sizes do not scale with N beyond this dictionary.