Estimate Token Count of a Message List
Problem Statement
Before sending messages to an LLM you need a cheap token estimate. Use the common heuristic that one token is roughly four characters of text, rounded up per message, then summed.
Background
Exact tokenization needs the model's tokenizer, but agents often gate context with a fast approximation: ceil(len(content) / 4) tokens per message, plus a small fixed per-message overhead for role/formatting markers. Summing these gives a conservative estimate used to decide whether to trim.
Your Task
Implement:
def estimate_tokens(messages, per_message_overhead=3):
- messages: list of dicts each with a "content" string.
- Each message costs ceil(len(content)/4) + per_message_overhead tokens.
- Return the total as an int.
Input Format
- messages (list of dicts), per_message_overhead (int).
Output Format
- A single int.
Sample
print(estimate_tokens([{"content": "hello"}, {"content": "world!!"}]))
Output:
10
Example
Input
print(estimate_tokens([{"content": "hello"}, {"content": "world!!"}]))Output
10
Reasoning
- Input: a list of two messages whose
"content"strings are"hello"(5 characters) and"world!!"(7 characters), withper_message_overheadleft at its default of 3. - Text cost: each message's character count is divided by 4 and rounded up to a whole token. For
"hello", becomes ; for"world!!", also becomes . - Overhead: every message adds a flat 3 tokens for role and formatting markers, so each costs .
- Output: the total over all messages, , a single integer.
- Notice that rounding up happens per message, before summing. Pooling the text first would give text tokens instead of 4, so the per-message rule is the more conservative estimate. Notice too that the overhead dominates short messages: with an overhead of 0 the answer would be just 4, and an empty list costs 0.
Constraints
0 <= len(messages) <= 10000.- Per-message cost is
ceil(len(content)/4) + per_message_overhead. - Return an int.