Temperature, top-k, top-p, and why T = 0 is not deterministic

01The pipeline, in order

At each step of generation the model's final layer produces a vector of logits, one real number per vocabulary entry, typically 32,000 to 200,000 of them. Nothing about that vector is a probability yet. The decoding parameters are a sequence of transformations on it, and the order matters.

logits z = [z1, z2, …, zV]
   │
   ▼
1. temperature       zi / T                    reshapes the distribution
   │
   ▼
2. top-k             keep the k largest, rest → −∞    hard cap by rank
   │
   ▼
3. top-p             keep the smallest prefix with     cap by probability mass
                     cumulative mass ≥ p, rest → −∞
   │
   ▼
4. softmax over survivors, then one random draw

This is the order in the Hugging Face generation code and in most serving engines. Some stacks apply top-p before top-k, or add other filters; the order is a property of the implementation, not of the model.

02Temperature

Temperature T divides every logit before the softmax:

q_i = z_i / T,    p_i = e^q_i / ∑_j e^q_j

Because the softmax is exponential, scaling the logits changes the ratios between probabilities, not just their spread. Two tokens whose logits differ by 2 have a probability ratio of e², about 7.4, at T = 1. At T = 2 the gap becomes 1 and the ratio 2.7. At T = 0.5 the gap becomes 4 and the ratio 55.

• T > 1 flattens. Gaps shrink, the distribution moves toward uniform, low-probability tokens get drawn more often. Output is more varied and, past a point, less coherent.
• T < 1 sharpens. Gaps grow, the leading tokens take more of the mass, output is more predictable.
• T = 1 is the model's own distribution, the one it was trained to produce.

Temperature never changes the ranking. The most likely token at T = 1 is the most likely token at any T. It changes how much the ranking matters.

03Top-k

Top-k is a hard cut by rank. Sort the logits, keep the k largest, set every other logit to −∞ so it gets zero probability after the softmax. With k = 50, exactly 50 candidates survive, whether the 50th has a probability of 1 percent or one in a million.

That is both the point and the problem. It guarantees the tail is gone, which stops the model drawing a nonsense token on the one-in-ten-thousand draw. But a fixed k is wrong in both directions: when the model is confident and two tokens hold 99 percent of the mass, 48 of the survivors are junk; when the model is genuinely uncertain across 200 plausible continuations, 150 reasonable ones are cut.

04Top-p, or nucleus sampling

Top-p cuts by cumulative mass rather than by rank. Sort the tokens by probability, descending; walk down adding probabilities until the running sum reaches p; keep that set, the nucleus, and set the rest to −∞.

The nucleus adapts to the model's confidence, which is why top-p is the usual default and top-k the backstop. The failure mode is the opposite of top-k's: when the distribution is very flat, a nucleus of 0.95 admits a long tail of individually improbable tokens.

05How the three interact

• Temperature runs first, so it changes what top-p sees. A higher temperature flattens the distribution, so the same p admits more tokens. Setting T = 1.5 and p = 0.9 is a much wider nucleus than T = 0.7 and p = 0.9. The two are not independent knobs.
• Top-k runs before top-p, so it is a ceiling. Top-p can only shrink the set top-k left. With k = 50 and p = 0.9, the nucleus is the smallest prefix of those 50 reaching 0.9 of their renormalised mass, never more than 50.
• Softmax runs again at the end. After the cuts, the survivors are renormalised to sum to 1 and one is drawn. The mass that was cut is redistributed proportionally, so the leading token's probability goes up slightly whenever anything is removed.

Most providers recommend changing temperature or top-p, not both, for exactly the first reason. Other filters exist, such as min-p, which keeps tokens above a fraction of the top token's probability, and repetition penalties, which adjust logits for tokens already in the text; they slot into the same pipeline before the final softmax.

06What temperature zero actually does

Dividing by zero is undefined, so T = 0 is a convention rather than a value. What it means is the limit. As T shrinks, the gaps between scaled logits grow without bound, the largest one takes all of the mass, and the softmax becomes a one-hot vector:

lim_T→0 softmax(z / T)_i = 1 if i = argmax z, 0 otherwise

So temperature zero is greedy decoding: at every step, take the token with the largest logit. Serving code special-cases it and calls argmax directly rather than dividing. Two consequences follow.

• Top-k and top-p become irrelevant. Both keep the top token, and only the top token has any probability. Whatever you set them to, the answer is the argmax.
• Greedy is not "the best answer". It is the locally most likely token at each step, which is not the globally most likely sequence and is often a worse sequence than a sampled one. Greedy decoding repeats itself and gets stuck in loops more than sampling at a moderate temperature does.

07Why greedy is not deterministic in practice

Argmax of a fixed vector is deterministic. The vector is not fixed. The logits are the end of billions of floating-point operations, and small differences in how those operations are carried out produce logits that differ in the last few bits. Usually that changes nothing. Sometimes it changes the argmax, and then it changes everything after it.

Floating-point addition is not associative

In real arithmetic (a + b) + c = a + (b + c). In 16- or 32-bit floating point each addition rounds, and the rounding depends on the intermediate value, so the two orders can give different results. A matrix multiply, a layer norm or an attention softmax is a large reduction, a sum over thousands of terms, and a GPU splits that sum across many threads and combines the partial sums. Any change in how the sum is split changes the rounding, and the final logit comes out as 12.400001 in one run and 12.400000 in another.

Near-ties at the top flip

A difference in the sixth decimal place matters only when two logits are that close. But in a vocabulary of 100,000, over a response of 500 tokens, there are steps where the top two candidates are close. At such a step a rounding difference reorders them:

run 1   logit(" Paris") = 15.000001   logit(" France") = 15.000000   →  " Paris"
run 2   logit(" Paris") = 15.000000   logit(" France") = 15.000001   →  " France"

Generation is autoregressive. A different token at step n is a different prompt for step n + 1, so the two runs are now computing different things, and the rest of the sequence diverges completely. The divergence is not a small perturbation; it is a different answer.

Kernels change strategy with batch size

This is the cause that explains most of what people see on a production server, and it is often missed. The individual GPU kernels in an inference engine are, for a fixed input shape, usually deterministic: the same split, the same order, the same rounding every time. What is not fixed is the input shape. A server batches your request with whatever else arrived, and a matrix multiply over 8 sequences chooses a different tiling and reduction order from one over 64 sequences. Your request's logits therefore depend on how many other requests were in flight, which you do not control and cannot see. The kernels are not batch-invariant.

The practical upshot is that the same prompt, at temperature zero, on the same hardware and the same weights, returns different completions depending on server load. A 2025 study from Thinking Machines measured a popular model producing 80 distinct completions over 1,000 greedy runs of one prompt, and traced it to this effect rather than to in-kernel randomness; rewriting the reduction kernels to be batch-invariant made the output identical across all 1,000 runs, at some cost in speed.

Mixture-of-experts routing and multi-GPU execution

Mixture-of-experts models such as Mixtral and DeepSeek route each token to a few of many expert sub-networks, and the experts are executed in batches whose composition depends on every token in the batch. Expert capacity limits can even drop or reroute tokens under load. Tensor and pipeline parallelism across GPUs add communication steps whose combining order can vary. All of these change numerics with the batch, which compounds the previous cause.

08What to do about it

• Do not treat temperature zero as a reproducibility guarantee. It removes the sampling randomness, not the numerical kind. A seed parameter helps only with the former.
• Evaluate with repeats. Run each evaluation prompt several times and report the spread, or the majority answer. A single greedy run is one draw from a distribution you cannot see.
• Compare models on the same batch conditions. Two runs of an evaluation at different times of day can differ for reasons that have nothing to do with the prompt or the model version.
• If you need bit-exact outputs, you need batch-invariant kernels or a batch size of one on dedicated hardware. Hosted APIs generally offer neither, and say so.
• Build for variation. Parse outputs defensively, use structured output modes where exactness of format matters, and do not write a test that asserts a specific string from a model.

09Takeaways

1. Temperature, then top-k, then top-p, then softmax and a draw. The order is the implementation's, and it matters.
2. Temperature changes ratios, not rankings. It makes the ranking matter more or less.
3. Top-k caps by rank, top-p by mass. Top-p adapts to confidence; top-k is the backstop against a flat tail.
4. Temperature and top-p are coupled. Raising temperature widens the nucleus at the same p.
5. Temperature zero is argmax. The other knobs do nothing, and greedy is not the best sequence.
6. Greedy is deterministic only in mathematics. Floating-point sums depend on order; order depends on batch size and scheduling; a near-tie turns a rounding difference into a different answer.
7. Repeat your evaluations. One greedy run is one sample.