What LLM Temperature Does to an Answer, Worked Out by Hand

By Pooja Goenka · Published 2026-10-21

Temperature is a number that reshapes the probabilities a language model uses to pick its next token. It divides the model's raw scores before they are turned into probabilities, so a low temperature makes the likeliest token win almost every time, and a high temperature gives unlikely tokens a real chance. It does not make the model smarter or more creative on its own. It changes how often the model takes its second-best idea.

Here is the short version:

• The model scores every possible next token. Those scores are called logits.
• Softmax turns the scores into probabilities that add up to 1.
• Temperature divides the scores first. Below 1 sharpens the distribution, above 1 flattens it.
• At 0 most APIs just take the top token, so repeat calls are close to identical, though not always guaranteed to be.
• Use low values when another system depends on the answer, and higher ones when you want variety.

I teach this in a course, and the explanation that works best is five numbers and a calculator, so that is what this post is.

The setup: one next-token choice

Take the sentence "I keep my socks in the kitchen ..." and ask a model for the next word. Under the hood it does not hold one answer. It holds a score for every token in its vocabulary, which is tens of thousands of entries. To keep the arithmetic readable I will use five candidates and invent the scores. They are made up for this post, not read from any real model.

The scores themselves mean little. Only the gaps between them matter, because the next step compares them.

Softmax, at temperature 1

Softmax raises e to the power of each score, then divides each result by the total. That turns any list of scores into a list of probabilities.

• e^3.0 = 20.09
• e^2.2 = 9.03
• e^1.6 = 4.95
• e^1.0 = 2.72
• e^-0.5 = 0.61

The total is 37.39. So the probability of "drawer" is 20.09 / 37.39, which is 53.7%. "Cupboard" gets 24.1%, "cabinet" 13.2%, "shelf" 7.3% and "bed" 1.6%.

At temperature 1, then, the model says "drawer" a little more than half the time, and something else the rest of the time. Ask the same question ten times and you should expect "drawer" five or six times.

What temperature changes

Temperature enters before softmax. You divide every score by T, then do exactly the same calculation.

At T = 0.5 the scores become 6.0, 4.4, 3.2, 2.0 and -1.0. The gaps have doubled. Run softmax and "drawer" now gets 78.0%, because e^6.0 is 403 and everything else is far smaller.

At T = 2 the scores become 1.5, 1.1, 0.8, 0.5 and -0.25. The gaps have halved, and "drawer" falls to 36.9%.

Here are all four settings from the code in the next section:

The ranking never changes. "Drawer" is first in every row. What changes is the distance between first place and the rest, and everything people say about temperature, including "creativity" and "randomness", describes that table.

The same thing in Python

A short script reproduces every number above. You can paste it into any Python 3 environment, with no model and no key.

import math

tokens = ["drawer", "cupboard", "cabinet", "shelf", "bed"]
logits = [3.0, 2.2, 1.6, 1.0, -0.5]   # invented scores, for illustration

def probabilities(logits, temperature):
    scaled = [x / temperature for x in logits]
    biggest = max(scaled)                    # subtract the max so exp() cannot overflow
    exps = [math.exp(x - biggest) for x in scaled]
    total = sum(exps)
    return [e / total for e in exps]

for t in (0.2, 0.5, 1.0, 2.0):
    row = probabilities(logits, t)
    print(f"T={t}: " + "  ".join(f"{w} {p:.0%}" for w, p in zip(tokens, row)))

The line that subtracts the largest value looks like a detail. It is there because e raised to a large number overflows, and subtracting the same constant from every score does not change the probabilities. Real implementations do this too.

Try T = 0.01 and you will see "drawer" at 100% to every decimal place you print. Try T = 10 and the five probabilities drift towards 20% each. The two ends of the dial are "always the top token" and "any token, almost equally".

Why 0 is a special case

You cannot divide by zero, so APIs treat a temperature of 0 as "pick the highest-scoring token". That is called greedy decoding. It makes repeat calls close to identical, but I would not promise byte-for-byte identical output from a hosted model. Anthropic's documentation, for one, says results are not fully deterministic even at 0.0. If you need exact repeatability, test it on the provider you use rather than assuming.

Ranges also differ by provider. OpenAI's API accepts values from 0 to 2, and Anthropic's Messages API accepts 0 to 1. Check the reference for the model you call.

Where temperature misleads people

The popular picture is a creativity slider. I think that picture causes two mistakes.

The first mistake is turning it up to get better writing. A higher temperature does not add ideas the model lacks. It only lets lower-ranked tokens through more often. Sometimes the lower-ranked token is the interesting one. Often it is just the wrong word, and because each token feeds into the next, one odd choice early on can push the rest of the answer somewhere odd. If a draft is dull, a better prompt usually helps more than a hotter dial.

The second is turning it to 0 and assuming the answer is now correct. Greedy decoding gives you the model's most likely continuation, and the most likely continuation can be confidently wrong. I wrote about that failure in why language models invent sources. Temperature controls how much the answer varies. It has no say in whether the answer is true.

The slider picture has a fair case. For brainstorming ten headlines, a higher setting does give you more spread, which is what you want. The slider is a useful shortcut when you remember that "creative" here means "more varied".

Temperature and top-p

Many APIs offer a second control, top-p, which cuts off the unlikely tail instead of reshaping the whole distribution. With a top-p of 0.9 the model keeps the most likely tokens until their probabilities add up to 90% and ignores the rest. OpenAI's reference suggests changing temperature or top-p but not both, since they interact, and that is a sensible default to follow until you have a reason to do otherwise.

What to set in practice

• Extracting a date, a category label or a JSON field: use a low value, often 0. Another program will read the result, and variation is a cost.
• Answers that rely on retrieved passages: keep it low, so the model stays close to the evidence you gave it. Building a retrieval pipeline shows where that setting goes.
• Agents that call tools: low, because a tool name or argument that varies from run to run is a bug. The from-scratch agent loop is a good place to see why.
• Brainstorming, naming, alternative phrasings: try 0.8 to 1.2 and compare several runs side by side.

Treat these as starting points to test, since a model's output at a given setting depends on the model.

Where LogicWiz fits

The LogicWiz GenAI course covers this in the lesson "Nova Finds Her Voice". The lesson has a walkthrough animation of the setting, and a lab cell that sends one prompt at temperature 0 and again at 1.2 and prints both replies from a live model. That lesson sits in the later chapters, so it needs a free account (the early chapters do not). The lab calls a hosted model, so it asks for your own key, as this post on API keys explains. The course is completely free, with no card. The softmax code above needs none of that.

Common questions

What does temperature do in an LLM?

It divides the model's raw token scores before they are turned into probabilities. Below 1 the likeliest token becomes even more likely, and above 1 the choices flatten out so less likely tokens are picked more often.

What is the best temperature for ChatGPT or an API call?

There is no single best value. Use 0 or close to it when the output feeds another system, and something around 0.7 to 1 when you want variety. Test a few values on your own prompt, since results depend on the model.

Does temperature 0 give the same answer every time?

Usually very close, and often identical, but a hosted model may still vary slightly. Anthropic's documentation says results are not fully deterministic even at 0.0.

Is temperature the same as top-p?

No. Temperature reshapes all the probabilities. Top-p drops the unlikely tail and keeps the rest. Most guidance is to adjust one of them at a time.

Can a higher temperature make a model hallucinate more?

It can make odd word choices more frequent, and an early odd choice can steer the rest of the answer. But a model can state falsehoods at temperature 0 too, because the most likely continuation is not always the true one.