One Word at a Time

Part of the free Generative AI course on LogicWiz, module: Nova Finds Her Voice.

Episode 25: One Word at a Time

"Watch Nova answer a question and you'll notice something odd: the words appear one after another, like she's typing. She isn't slow — she literally can't say the whole thing at once. Every model you've ever used works this way, and once you see why, half of AI's quirks suddenly make sense."

The Magician's Trick

You've used Nova for a whole course now. Time to look under the hood at the single most important thing about how she — and every LLM — actually works.

Here's the trick: a model doesn't write a sentence. It writes the next word, over and over. When you watch ChatGPT or Nova "type" a reply, you're literally watching it produce one small piece, add it on, and go again. It's not a stylistic choice; it's the only thing the model can do.

That one fact explains a surprising amount: why replies stream in, why longer answers cost more, and even how the same idea lets a model generate music and images. Let's build the intuition from the ground up.

One Token at a Time

Say Nova starts with the words "The cat" and needs to continue. She doesn't plan the whole sentence. She asks one question — "what word most likely comes next?" — answers it ("sat"), tacks it on, and then asks the exact same question again about the now-longer text. Round and round:

That cycle has a name: autoregressive generation. Break the word apart and it explains itself — auto (self) + regressive (feeding its own output back in). The model's newest word becomes part of the input for its next guess. Predict → append → feed back → repeat, until it decides it's done.

This is why answers stream onto your screen word by word: you're watching the loop run in real time. There's no finished paragraph hidden somewhere that's being revealed slowly — each word is being computed the moment before you see it.

What's a Token, Anyway?

One correction to the picture above: models don't actually work in whole words — they work in tokens. A token is a chunk of text, usually a word or a piece of a word. "cat" might be one token; "unbelievable" might split into "un", "believ", "able". Roughly, 1 token ≈ ¾ of a word in English.

Why care? Because everything — how much you pay, how much a model can remember, how fast it replies — is counted in tokens, not words. We'll lean on that in the next episode.

💡 Here's a preview that pays off later in this chapter: a token doesn't have to be text. The same next-token loop can predict the next chunk of audio or the next piece of an image. "Predict the next token" is a shockingly general idea — it's how one model can write, sing, and draw.

Two Ways to Learn Language

Autoregressive isn't the only way a model can learn language — it's just the one that won for chat. There's a rival worth knowing, because it explains why some models are built for writing and others for understanding.

Causal (autoregressive) modeling looks only at the words before a spot and predicts the next one — like finishing someone's sentence. Because it writes forward, it's perfect for generating replies. GPT, Claude, and Gemini all work this way.
Masked modeling hides a word in the middle of a sentence and uses the words on both sides to guess it — like a fill-in-the-blank puzzle. BERT (from Google, 2018) is the famous example. It's brilliant at understanding text (search, classification) but it isn't built to write new text left-to-right.

Chat needs to write forward, one word after another — so the causal/autoregressive approach became the standard, and nearly every modern chat model uses it.

See It in Code

Enough theory — let's run the real loop against a real model. The trick is one parameter: max_tokens=1 forces the model to emit exactly one token and stop. So we call it, take that single token, glue it onto our text, and call again with the longer text — over and over. That's the autoregressive loop, by hand, in slow motion. Step through the walkthrough first, then run it:

Watch the sentence grow one token per pass. Notice two things: each call resends the entire sequence (the model has no memory of the previous call), and a "token" sometimes arrives as a fragment of a word — the exact predict → append → feed-back cycle you watched animate above, now running on a real model you called yourself.

Making It Faster

One token at a time is simple, but it can be slow — a long answer means running the whole model once for every single word. So researchers cheat (cleverly):

Speculative decoding — a small, fast model drafts several tokens ahead, and the big model just checks them in one pass. Right guesses are kept; the big model only redoes the wrong ones. Same output, fewer slow steps.
Multi-token prediction — train the model to propose more than one token per pass.

You don't need the details — just know that "one token at a time" is the default, and these tricks speed it up without changing the core idea.

What Nova Learns Next

Now you know Nova writes one token at a time, feeding each word back into herself. That single mechanic has a direct consequence for your wallet: because output is produced token-by-token but input can be read all at once, they cost very different amounts. Next episode follows the money — why every word has a price, and why output costs more than input.