Temperature Sampler

easyFREEText Generation & AdvancedNLP 3: Advanced

Given a list of logits (raw scores) for vocabulary words and a temperature value, apply temperature scaling and compute the resulting probability distribution using softmax.

Temperature scaling divides each logit by the temperature before applying softmax: $p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}$

Input: temperature_sample(pairs, T)

pairs: a list of (word, logit) tuples, e.g. [("cat", 2.0), ("dog", 1.0), ("fish", 0.5)]
T: the temperature (float, $T > 0$ )

Output: A list of strings, one per word, each formatted as "word probability", sorted by probability descending (ties: alphabetical). The probability is rounded to 4 decimal places and written as that rounded number, without padding trailing zeros (e.g. 0.5, 0.982, 0.6285).

Example

Input

cat:2.0,dog:1.0,fish:0.5
1.0

Output

cat 0.6285
dog 0.2312
fish 0.1402

Reasoning

Input: pairs holds each word with its logit, a raw unnormalised score: cat $2.0$ , dog $1.0$ , fish $0.5$ . T $= 1.0$ is the temperature that every logit is divided by.
Scale: with $T = 1$ the scaled logits stay $2.0,\ 1.0,\ 0.5$ .
Exponentiate: $e^{2.0} \approx 7.3891$ , $e^{1.0} \approx 2.7183$ , $e^{0.5} \approx 1.6487$ , and their total is $\approx 11.7561$ .
Normalise: $p_{cat} = 7.3891 / 11.7561 \approx 0.6285$ , $p_{dog} \approx 0.2312$ , $p_{fish} \approx 0.1402$ . The three sum to $1$ , and the words are listed from most to least likely.
Notice: $T$ controls sharpness. At $T = 2$ the scaled logits become $1.0,\ 0.5,\ 0.25$ and the distribution flattens to roughly $0.481,\ 0.2918,\ 0.2272$ ; at $T = 0.5$ the scaled logits become $4.0,\ 2.0,\ 1.0$ and the mass shifts sharply toward cat, to roughly $0.8438,\ 0.1142,\ 0.042$ .

Constraints

Temperature T > 0
Apply softmax after dividing logits by T
Round probabilities to 4 decimal places
Output sorted by probability descending (ties: alphabetical)

Learn this in

Language ModelingNLP Study Plan

More from NLP 3: Advanced

ROUGE-1 Score Calculatoreasy Context Window Findereasy Keyword Answer Extractoreasy Levenshtein Distanceeasy Word-Level Translatoreasy

Concept primer

Background Knowledge

The problem involves applying temperature scaling and softmax to a list of logits. Logits are raw, unnormalized scores that a model outputs for each class or word in a vocabulary. Softmax is a function that takes these logits and converts them into a probability distribution, where each word has a probability between 0 and 1, and the probabilities sum up to 1. Temperature scaling is a technique used to control the "softness" of the softmax output. By dividing each logit by a temperature value, we can adjust the level of uncertainty in the output probabilities.

The softmax function is defined as $p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$ , where $z_i$ is the logit for the $i^{th}$ word, and $p_i$ is the corresponding probability. When we apply temperature scaling, we replace $z_i$ with $z_i / T$ , where $T$ is the temperature. This has the effect of "flattening" the probability distribution when $T$ is high, and "sharpening" it when $T$ is low. Understanding how softmax and temperature scaling work is crucial to solving this problem.

In the context of Natural Language Processing (NLP), temperature scaling can be used to control the level of randomness in text generation models. For example, a high temperature can result in more diverse and unpredictable text, while a low temperature can result in more repetitive and predictable text. The Temperature Sampler problem requires applying these concepts to a given list of logits and temperature value, and computing the resulting probability distribution.

Algorithm/Approach

The general approach to solving this problem involves the following steps:

Parse the input data, including the word-logit pairs and the temperature value
Apply temperature scaling to the logits
Compute the softmax of the scaled logits
Sort the resulting probabilities in descending order
Output each word and its corresponding probability

This approach involves a combination of data parsing, mathematical computations, and sorting.

Step-by-Step Strategy

To implement the solution, follow these steps:

Parse the input string into a dictionary or list of word-logit pairs
Extract the temperature value from the input
Apply temperature scaling to each logit
Compute the softmax of the scaled logits using the formula: $p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}$
Sort the resulting probabilities in descending order
Output each word and its corresponding probability, rounded to 4 decimal places

Common Pitfalls

When implementing the solution, watch out for the following:

Make sure to handle the input data correctly, including parsing the word-logit pairs and extracting the temperature value
Be careful when applying temperature scaling, as dividing by zero can result in errors
Use a stable implementation of the softmax function to avoid numerical issues
Make sure to sort the probabilities in descending order, as required by the problem statement

Time & Space Complexity

The time complexity of the solution is expected to be O(n log n) due to the sorting step, where n is the number of word-logit pairs. The space complexity is expected to be O(n) for storing the word-logit pairs and the resulting probabilities. However, the exact complexity may depend on the specific implementation and the size of the input data.

solution.py

Python 3.13

Test results