Token Accounting Across a Multi-Model Agent Run

1. Background Knowledge

In production AI systems, OpenTelemetry provides standardized telemetry data, including GenAI spans that record token usage for every model call. Understanding how to aggregate this data is critical for cost management. The core concept here is token accounting, where you must distinguish between different types of tokens to calculate accurate costs. Specifically, input tokens represent the total context sent to the model, while cached input tokens represent a subset of those inputs that were retrieved from a prompt cache rather than processed anew.

Pricing models for Large Language Models (LLMs) typically quote rates per million tokens. Crucially, cached tokens are billed at a significantly lower rate than fresh input tokens. This creates a tiered pricing structure within the input category. If you treat all input tokens as "fresh," you will overestimate costs. If you double-count cached tokens (charging them at both the cached rate and the fresh rate), you will drastically overestimate costs. The formula for cost relies on subtracting cached tokens from the total input to find the "fresh" input count.

Finally, aggregation is required because an agent run may involve multiple models and multiple calls. You need to sum costs across all spans, grouping them by model for detailed reporting. The output requires precise floating-point arithmetic rounded to six decimal places to ensure financial accuracy, as small errors can compound over thousands of requests.

2. Algorithm Approach

The problem is a straightforward aggregation and arithmetic task. The general approach involves iterating through a list of usage records and applying a specific cost formula to each one.

1. Iterate: Loop through each record in the usage list.
2. Extract: Pull out input_tokens, cached_input_tokens (defaulting to 0 if missing), and output_tokens.
3. Calculate: Apply the cost formula using the corresponding prices from the prices dictionary.
4. Accumulate: Add the calculated cost to a running total and to a per-model accumulator.
5. Format: Sort the model keys and round the final values to 6 decimal places.

This is a linear scan algorithm with constant-time lookups for prices. No complex data structures or sorting algorithms are needed beyond the final sorting of model keys.

3. Step-by-Step Strategy

1. Initialize Accumulators: Create a variable total_cost initialized to 0.0 and a dictionary by_model to store costs per model.
2. Loop Through Usage: For each record in the usage list:

• Extract model, input_tokens, and output_tokens.
• Extract cached_input_tokens, using .get('cached_input_tokens', 0) to handle missing keys safely.
• Retrieve the price dictionary for the current model from the prices input.

3. Compute Fresh Input: Calculate fresh_input = input_tokens - cached_input_tokens. Ensure this value is not negative (though logically it shouldn't be).
4. Calculate Record Cost:

• Compute the cost for fresh input: (fresh_input * price['input']) / 1_000_000
• Compute the cost for cached input: (cached_input_tokens * price['cached_input']) / 1_000_000
• Compute the cost for output: (output_tokens * price['output']) / 1_000_000
• Sum these three components to get record_cost.

5. Update Totals:

• Add record_cost to total_cost.
• Add record_cost to by_model[model] (initializing the key if it doesn't exist).
• Add input_tokens + output_tokens to a total_tokens counter.

6. Finalize Output:

• Round total_cost to 6 decimal places.
• Create the by_model output dictionary by iterating through the keys of the accumulated by_model dict in sorted order, rounding each value to 6 decimal places.
• Return the dictionary containing total_cost, by_model, and total_tokens.

4. Common Pitfalls

• Double-Counting Cached Tokens: The most common error is adding cached_input_tokens to the cost calculation twice—once as part of the total input and again as cached. Remember: input_tokens includes cached tokens. You must subtract cached tokens from the total input to get the billable "fresh" input.
• Missing Keys: The cached_input_tokens field may be absent from some records. Always use a default value of 0 when accessing this key to avoid KeyError exceptions.
• Floating-Point Precision: Direct floating-point addition can lead to tiny precision errors. While Python's round() function handles the final output requirement, be aware that intermediate calculations should be done with standard floats. The problem asks for rounding at the end, not at every step.
• Sorting Order: The by_model dictionary must be built by iterating model names in sorted order. In Python 3.7+, dictionaries maintain insertion order, so you must explicitly sort the keys before creating the final output dictionary.
• Total Tokens Calculation: The problem states total_tokens is the sum of input_tokens and output_tokens. Do not add cached_input_tokens again, as they are already included in input_tokens.

5. Time & Space Complexity

• Time Complexity: $O(N \log N)$, where $N$ is the number of usage records. The iteration through the usage list is $O(N)$. The final step of sorting the unique model keys takes $O(M \log M)$, where $M$ is the number of unique models. Since $M \le N$, the overall complexity is dominated by the sorting step in the worst case, but typically $M$ is small, making it effectively $O(N)$.
• Space Complexity: $O(M)$, where $M$ is the number of unique models. We store a cost accumulator for each unique model in the by_model dictionary. The input and output sizes do not scale with $N$ beyond this dictionary.