Token Budget Calculator
Calculate token budgets for a prompt with system, examples, and query.
Given a maximum context length and the token counts for each part, determine how many few-shot examples can fit while reserving space for the query and a completion buffer.
Input:
- Line 1: max_tokens completion_buffer
- Line 2: system_tokens query_tokens
- Line 3: N (number of available examples)
- Next N lines: example_tokens (tokens per example)
Output:
- Line 1: Number of examples that fit
- Line 2: Total tokens used (system + selected examples + query)
- Line 3: Tokens remaining for completion
Example:
4096 512 200 100 5 150 150 150 150 150
3 650 2934
- The maximum available tokens for examples and the query are calculated by subtracting the completion buffer from the max tokens: 4096−512=3584.
- We reserve space for the system and query tokens: 200+100=300 tokens.
- The remaining tokens for examples are 3584−300=3284, and each example takes 150 tokens, so we can fit 1503284≈21.9 examples, but since we have 5 examples and the calculator can only use whole examples, we calculate how many examples fit: 3284÷150=21.9, so 21 examples would take 21⋅150=3150 tokens, exceeding the available tokens when adding system and query tokens, thus we try with fewer examples.
- We try with 3 examples: 3⋅150=450 tokens for examples, 450+300=750 total tokens used, leaving 4096−750−512=2934 tokens for completion, which fits within the completion buffer, so 3 examples fit.
Constraints:
- Select examples in order (first N that fit)
- Must reserve completion_buffer tokens
- Total used = system + examples + query
- Available = max_tokens - completion_buffer
More from LLM 3: Applications & Evaluation
Background Knowledge
The problem of calculating token budgets for a prompt is closely related to the concept of text generation and language modeling. In the context of large language models (LLMs), a token is a basic unit of text, such as a word or a character. The maximum context length, also known as the sequence length, is a critical parameter that determines how much text the model can process at once. The completion buffer is a reserved space for the model to generate text, and it's essential to ensure that there's enough space for the model to produce a meaningful response.
The problem also involves few-shot learning, which is a technique used to fine-tune LLMs on a small number of examples. The goal is to determine how many examples can fit within the given token budget while reserving space for the query and the completion buffer. This requires careful resource allocation, as the model needs to balance the number of examples with the available tokens. The token counts for each part of the prompt, including the system, examples, and query, are crucial inputs that will help determine the optimal allocation of tokens.
Understanding the trade-offs between the number of examples, token counts, and the completion buffer is essential to solving this problem. As the number of examples increases, the total token count will also increase, which may leave less space for the completion buffer. Conversely, reserving too much space for the completion buffer may limit the number of examples that can be included. The key is to find a balance that maximizes the number of examples while ensuring that there's enough space for the model to generate a meaningful response.
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.