If you’ve played with the APIs or playgrounds of ChatGPT, Claude, or other AIs, there are settings you’ll almost always run into: temperature and top-p. You can tell the feel of the output changes when you move the sliders, but it’s hard to know exactly what you’re adjusting. Many people feel this way.
These two are parameters that control how the AI chooses the next word when generating text. With the same question, raise the temperature and you get free-wheeling, varied answers; lower it and you get safe, predictable ones. This difference in behavior clicks into place once you understand how the AI picks words one at a time—the mechanism technically called sampling (decoding).
In this article, we dig into how an AI chooses the next word, in this order: probability distribution, greedy decoding, temperature, top-k, top-p. Drawing on Holtzman et al.’s now-classic 2019 paper “The Curious Case of Neural Text Degeneration”1, we go as far as a surprising fact: why “always picking the most plausible word” doesn’t work well. Knowing the term token makes this easier to read, but it’s written so you can follow without it.
The Premise: AI Builds a “Probability Table” Before Choosing a Word
Boiled down, the way a large language model (LLM) produces text is a repetition of “predicting the next word (token).” For the input “Today’s weather is,” the AI internally computes, for each of the tens of thousands of words in its vocabulary, “what’s the percentage chance this comes next.”
The computation used here is the softmax, which converts the raw scores (logits) the AI produces into a probability distribution that sums to 100%3. Picture a “probability table” like this:
Input: "Today's weather is"
↓ AI computes the next word's probability distribution (softmax)
sunny ████████████ 60%
cloudy █████ 25%
rain ██ 8%
snow █ 4%
……(thinly spread across the remaining tens of thousands of words) 3%The crucial point here is that the AI has only gotten as far as building this table—it hasn’t yet settled on a single word. Which word it actually picks out of this probability distribution—that selection rule is precisely sampling, and it’s where temperature and top-p come into play.
The Safest Choice, “Greedy Decoding,” and Its Surprising Pitfall
The most naive way to choose is “always pick the word with the highest probability.” This is called greedy decoding. In the earlier example, it means picking “sunny (60%)” every time.
At first glance this seems best. Since you always pick the most plausible word, you’d think you’d get the most correct text. But here lies a fascinating paradox of generative AI. Holtzman et al.’s paper “The Curious Case of Neural Text Degeneration”1 overturned exactly this intuition.
Their observation runs like this: “using likelihood as a training objective leads to high quality models for a broad range of tasks. Yet using that same likelihood as a decoding objective (word selection) leads to text that is bland and strangely repetitive”1. Keep choosing the safest word every time, and the text becomes too bland and dull—sometimes falling into loops of the same phrasing, “very, very, very…”. They named this phenomenon degeneration.
Why does this happen? The paper points out that “there are surprising distributional differences between human text and machine text”1. The natural text humans write isn’t actually picking the highest-probability word every time; it occasionally mixes in unexpected words. That very “fluctuation” is what makes writing feel alive. A machine that always picks the maximum probability loses this human-like fluctuation and, paradoxically, becomes unnatural. The paper concludes that “decoding strategies alone can dramatically affect the quality of machine text”1. In other words, an AI’s smarts depend not only on its contents (the model) but heavily on that final “way of choosing words.”
Temperature: Changing How “Sharp” the Probability Distribution Is
This is where sampling—choosing randomly according to the probabilities—comes in, and what adjusts that randomness is temperature. Think of temperature as a dial that changes the “sharpness” of that earlier probability distribution.
Technically, temperature is the number (τ) by which the scores are divided in the softmax computation3. Set the hard formula aside; grasping the effect is enough. In Wikipedia’s framing, raising the temperature makes “the output distribution more uniform (higher entropy); it becomes ‘more random,’” while conversely lowering the temperature makes “the distribution sharper, with one value dominating”3. Bring the temperature infinitely close to 0 and it ends up always picking the maximum-probability word—that is, it coincides with greedy decoding3.
Low temperature (e.g. 0.2) High temperature (e.g. 1.2)
sunny ██████████████ 88% sunny ███████ 38%
cloudy ██ 9% cloudy █████ 28%
rain ▏ 2% rain ███ 18%
snow ▏ 1% snow ██ 16%
→ sharper. safe, predictable → flattened. varied, surprisingAt low temperature, the already-high “sunny” stands out even more, and the AI makes almost only safe choices. At high temperature, the gaps in probability are smoothed out, and even normally-unlikely “rain” or “snow” get a chance. As a result, low-temperature output tends to be accurate and consistent but dull, while high-temperature output tends to be varied and creative but sometimes off-track or incoherent.
As a general rule of thumb, for tasks where accuracy is everything—fact-checking, code generation, summarizing, classification—a lower setting (around 0–0.3) is standard, while for tasks that want breadth of imagination—stories, poetry, brainstorming—a higher setting is standard. But this is a heuristic; the optimal value varies by task and model. Note that pushing the temperature too high also raises the risk of plausible-sounding falsehoods—hallucination.
Top-k and Top-p: Where to Cut the Suspicious “Tail”
Whereas temperature changes “the sharpness of the whole distribution,” there’s also an approach that narrows the candidates themselves: top-k and top-p. The aim is shared: to cut off the long “tail” of the probability distribution—the part where improbable words are thinly spread—and prevent the accident of a strange word being chosen from there.
Top-k is simple; in Wikipedia’s definition, it “restricts the pool of candidate tokens to the k most likely tokens”2. With top-k = 50, only the top 50 words are candidates and the rest are all ignored.
Top-p (also called nucleus sampling) is the method Holtzman et al. proposed in that paper1. Wikipedia defines it as “a stochastic decoding strategy for generating sequences from autoregressive probabilistic models”2, and explains its core like this: the next token is “sampled only from the smallest possible set of high-probability candidates whose cumulative probability exceeds p”2. Concretely, you line up the words from highest probability, add them up from the top, draw a line where the total exceeds, say, 0.9 (90%), and make only the inside (the “nucleus” of probability) candidates.
The decisive difference between the two is whether the number of candidates is fixed or variable. Wikipedia notes that “the main advantage of top-p is its adaptability. Top-k always samples from a fixed number of tokens, which may be too restrictive or too broad depending on the context”2.
top-k (e.g. k=2): fixed at the top 2
Only sunny / cloudy become candidates (rain / snow always excluded)
top-p (e.g. p=0.9): variable until cumulative reaches 90%
"Today's weather is" → sunny 60% + cloudy 25% + rain 8% = 93%, draw line → 3 candidates
"His name is" → if candidates are spread out, it could be 10+For a scene where the answer is narrow, like “Today’s weather is,” top-p leaves only a few candidates. Conversely, for a scene where anything goes, like “His name is,” it automatically widens the pool. Stretching and shrinking the net’s size according to context—this is the advantage of nucleus sampling, which Holtzman et al. described as “effectively truncating the less reliable tail of the distribution while allowing for diversity”1.
| Method | What it does | Number of candidates |
|---|---|---|
| Greedy | Always pick the single highest-probability word | 1 (fixed, deterministic) |
| Temperature | Change the sharpness of the whole distribution | Doesn’t narrow (reshapes the whole) |
| top-k | Restrict candidates to the top k | Fixed2 |
| top-p (nucleus) | Restrict candidates up to cumulative probability p | Variable, adaptive2 |
Combining Them in Practice, and a Useful Perspective to Know
In real APIs, these are often used together. Typically it’s a two-stage setup: first cut off the “suspicious tail” with top-p (or top-k), then adjust how much the remaining candidates vary with temperature. Many providers recommend moving either temperature or top-p, but not pushing both to extremes at once. Strongly applying both can cancel out or overdo each other’s effects, making behavior hard to predict.
One distinction worth adding, since it’s easy to confuse. Temperature and top-p only change “how you choose from an already-built probability distribution”—they don’t change the AI’s knowledge or reasoning ability itself. This is a different layer from how recent reasoning models get smarter by “thinking before answering.” If cooking is the analogy, sampling is “the plating and the intensity of the seasoning,” while reasoning is the “cooking process” itself—organizing it that way helps.
And one more thing. Output varying slightly each time is, in most cases, due to this randomness of temperature and sampling. The puzzlement of “same prompt, but a different answer than yesterday” usually traces back to here. Lower the temperature toward 0 for test uses that need reproducibility; raise it for uses that want creativity—knowing what the parameters mean lets you tame an AI’s output to your purpose.
If you’re interested in how to guide an AI’s output through prompt craft, read What Is Prompt Engineering?; for the mechanism by which AI produces plausible errors, read What Is Hallucination?. Together they let you grasp the same question—“how to control an AI’s answers”—from three angles in three dimensions: the entrance (prompt), the contents (model), and the exit (sampling).
Sources
- The Curious Case of Neural Text Degeneration - Ari Holtzman et al., 2019 (ICLR 2020). The original paper for nucleus sampling (top-p) and the observation of text degeneration
- Top-p sampling - Wikipedia (definitions and difference of top-p / nucleus sampling and top-k)
- Softmax function - Wikipedia (softmax with temperature; the effects of high vs. low temperature)