Temperature Parameter Control

Why It Matters

How it Works

Intuition: The Creative Thermostat

Imagine you are asking a chef to suggest a recipe. If the chef is "cold," they will always give you the most standard, safe, and predictable recipe (e.g., plain scrambled eggs). If the chef is "hot," they might experiment with exotic spices or unusual combinations, leading to surprising and creative results, but occasionally producing something inedible. In Generative AI, the "Temperature" parameter acts exactly like this thermostat. It controls the "randomness" of the model's output by reshaping the probability distribution of the next token. When we generate text, the model doesn't just pick one word; it assigns a probability to every word in its vocabulary. Temperature allows us to decide how strictly we want to follow those probabilities.

The Mechanism of Distribution Shaping

At its core, temperature control is a transformation applied to the logits before they are passed through the softmax function. By dividing the logits by the temperature value (), we alter the relative "distance" between the probabilities of different tokens. When , the differences between the logits are amplified. The high-probability tokens become even more dominant, and the low-probability tokens are pushed closer to zero. This results in a "sharpened" distribution, making the model behave more greedily and deterministically.

Conversely, when , the differences between logits are compressed. The probability mass is spread more evenly across the vocabulary. The most likely token loses some of its dominance, and the less likely tokens gain a higher chance of being selected. This "flattens" the distribution, allowing the model to explore more diverse paths. If becomes extremely high, the distribution approaches a uniform distribution, where every token has an almost equal chance of being chosen, resulting in incoherent, random gibberish.

Edge Cases and Practical Constraints

While temperature is a powerful tool, it is not a silver bullet for quality. At very low temperatures (e.g., ), the model may fall into "repetition loops," where it gets stuck repeating the same phrase because it is mathematically locked into the most probable path. This is a common failure mode in greedy decoding. On the other hand, very high temperatures can lead to "hallucination amplification," where the model ignores its internal knowledge base in favor of statistically improbable sequences.

Furthermore, temperature does not operate in a vacuum. It interacts heavily with other decoding strategies like Top-k and Top-p (Nucleus) sampling. If you set a high temperature but use a very restrictive Top-k value (e.g., ), the temperature will have no effect because the model is forced to choose from only the single most likely token. Practitioners must balance these parameters to achieve the desired balance between "creativity" and "coherence."

Common Pitfalls

Sample Code

import torch
import torch.nn.functional as F

def get_next_token_probs(logits, temperature=1.0):
    """
    Applies temperature scaling to logits and returns probabilities.
    """
    # Ensure temperature is not zero to avoid division by zero
    temp = max(temperature, 1e-6)
    
    # Apply temperature scaling
    scaled_logits = logits / temp
    
    # Calculate softmax to get probabilities
    probs = F.softmax(scaled_logits, dim=-1)
    return probs

# Example usage:
logits = torch.tensor([2.0, 1.0, 0.1])
# Low temp: High confidence in the first token
print(f"Low Temp (0.1): {get_next_token_probs(logits, 0.1)}")
# High temp: More uniform distribution
print(f"High Temp (2.0): {get_next_token_probs(logits, 2.0)}")

# Sample Output:
# Low Temp (0.1): tensor([0.7311, 0.2689, 0.0000])
# High Temp (2.0): tensor([0.4683, 0.2840, 0.2477])

Key Terms

Related Concepts