Estimate Token Count of a Message List
Problem Statement
Before sending messages to an LLM you need a cheap token estimate. Use the common heuristic that one token is roughly four characters of text, rounded up per message, then summed.
Background
Exact tokenization needs the model's tokenizer, but agents often gate context with a fast approximation: ceil(len(content) / 4) tokens per message, plus a small fixed per-message overhead for role/formatting markers. Summing these gives a conservative estimate used to decide whether to trim.
Your Task
Implement:
def estimate_tokens(messages, per_message_overhead=3):
- messages: list of dicts each with a "content" string.
- Each message costs ceil(len(content)/4) + per_message_overhead tokens.
- Return the total as an int.
Input Format
- messages (list of dicts), per_message_overhead (int).
Output Format
- A single int.
Sample
print(estimate_tokens([{"content": "hello"}, {"content": "world!!"}]))
Output:
10
Example:
print(estimate_tokens([{"content": "hello"}, {"content": "world!!"}]))10
- Process the first message with content "hello" (length 5): estimate tokens as ⌈5/4⌉+3=2+3=5, adding the character-based estimate plus the fixed overhead.
- Process the second message with content "world!!" (length 7): estimate tokens as ⌈7/4⌉+3=2+3=5, applying the same ceiling division and overhead logic.
- Sum the individual message costs to get the total context estimate: 5+5=10.
- The final output is 10
Constraints:
0 <= len(messages) <= 10000.- Per-message cost is
ceil(len(content)/4) + per_message_overhead. - Return an int.
1. Background Knowledge
Large Language Models (LLMs) do not process raw text directly; they operate on discrete units called tokens. A token is a sub-word fragment (e.g., "un", "happy", "##ness") that the model's tokenizer maps to a vector. Because exact tokenization requires loading the specific model's tokenizer (which is computationally expensive and model-specific), AI agents often use a heuristic approximation for quick context-window checks.
The most common heuristic is that 1 token ≈ 4 characters for English text. This ratio holds reasonably well for general prose but can vary for code, non-Latin scripts, or highly technical content. For budgeting purposes, agents use this approximation to decide whether to trim older messages from the context window before sending a request.
In addition to the content itself, each message in a conversation has structural overhead: role markers (e.g., system, user, assistant), delimiters, and formatting tokens. These are not part of the user's text but still consume tokens in the final prompt. A small fixed integer (commonly 3–4 tokens) is added per message to account for this overhead.
2. Algorithm Approach
This is a straightforward linear scan with accumulation problem. The approach is:
- Iterate over each message in the list.
- For each message, compute its token cost using the ceiling division heuristic plus the fixed overhead.
- Sum all individual costs into a running total.
- Return the total as an integer.
No sorting, recursion, or data structures beyond a simple accumulator are needed. The core operation is ceiling division: ⌈ba​⌉, which can be computed in Python as (a + b - 1) // b or via math.ceil(a / b).
3. Step-by-Step Strategy
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.