The Price of Every Word

Part of the free Generative AI course on LogicWiz, module: Nova Finds Her Voice.

Episode 26: The Price of Every Word

"Arjun ships Nova to a thousand users and the first invoice lands. It's not a flat fee — he's charged for every word going in and every word coming out, and the two prices aren't even close. To keep Nova affordable, he needs to understand exactly what he's paying for."

Nova Sends a Bill

Last episode you saw that Nova writes one token at a time. That single fact is why Nova costs money the way she does — and why a careless prompt can quietly run up a bill.

LLM providers don't charge per question or per month of "Nova." They charge per token — a fraction of a cent for each chunk of text that goes in and each chunk that comes out. Understand the two halves of that bill and you can make Nova dramatically cheaper without making her worse.

You Pay by the Token

Remember: a token is a chunk of text (~¾ of a word). Every request has two token counts:

• Input tokens — everything you send: your prompt, the conversation history, any retrieved documents.
• Output tokens — everything the model generates back.

Pricing is usually quoted as dollars per million tokens. A tiny number per token, but it adds up fast when you're serving thousands of users or stuffing long documents into every prompt.

Here's the twist that surprises everyone: input and output are priced differently — output is much more expensive. Same tokens, very different cost. Why?

Why Output Costs More

Here's a fact that surprises everyone: the very same words cost more when Nova writes them than when you send them. Word for word, her reply is several times pricier than your prompt. Why should identical text carry two different prices?

The reason is the difference between reading and writing.

Reading you can do all at once. Hand someone a finished page and their eyes take in the whole thing — you could even tear it into five pieces and give one to each of five friends to read at the same time. The words already exist, so nothing stops you from looking at them all together.

Writing is stubbornly one-word-at-a-time. You can't write the fifth word of a sentence before the first four exist — each word depends on the ones before it. You can't hand "write word 5" to a friend while you're still on word 2. Writing has to march forward, in order.

That is exactly the difference between input and output:

{{visual:input-vs-output-cost}}

• Input = reading. Your whole prompt already exists as text, so the model reads all of its tokens in one pass, together (in parallel). A 1,000-token prompt is still basically one big read. Cheap per token.
• Output = writing. Because the model is autoregressive (Episode 25), it writes one token, feeds it back in, and writes the next — it literally cannot produce token 5 before token 4 exists. So every single output token means running the entire model again. Want 100 output tokens? That's ~100 full runs of the model.

So the model works far harder to write than to read: your whole 1,000-token prompt is one quick read, but a 200-token answer is ~200 separate runs of the model. That extra work is exactly what lands on the bill — providers charge several times more per output token than per input token. Let's put real money on that.

Zoom right in on a single token and the reason is sharp: one input token is cheap because it shared that one parallel pass with the entire prompt — thousands of tokens splitting the same ride. One output token is expensive because it monopolized a whole pass by itself: the entire model ran, start to finish, just to produce that one word. Same-sized token, very different bill.

Here's the picture that sticks: input tokens carpool. They all pile into one pass and split the fare, so each rider pays almost nothing. Each output token takes a solo taxi — one passenger per trip, the whole model running just for it, full fare every time. Whether the answer is 10 tokens or 10,000, every output token rides alone. That's why, token for token, output is the pricey seat.

Now put real money on it

Providers quote two prices per model — one for input, a higher one for output. Take a typical pair: $5 per million input tokens and $25 per million output tokens (a 5× gap). Shrink that down to a single token:

• one input token → $5 ÷ 1,000,000 = $0.000005
• one output token → $25 ÷ 1,000,000 = $0.000025

Same-sized token, 5× the price. Now price a real turn — a 1,000-token prompt and a 1,000-token reply, the same length on each side:

Identical length, yet the reply costs 5× more — 2.5 cents versus half a cent. It isn't that Nova wrote more; it's that every word she writes is 5× the price of a word you send.

So is a short answer to a long prompt basically free? Less than you'd hope. Price a 1,000-token prompt with just a 200-token reply:

• input: 1,000 × $0.000005 = $0.005
• output: 200 × $0.000025 = $0.005

The little 200-token reply already ties the giant 1,000-token prompt — because its tokens are 5× pricier. Let the reply grow to 1,000 tokens and it becomes the bulk of your bill. That's the real takeaway: output length is the expensive knob.

💡 "But if my prompt is huge, isn't it re-read for every output token?" Great question — and the answer is the trick that makes all this work: the model processes your prompt once (the prefill pass) and stashes the result in memory (a KV cache). For each new token it just consults that memory instead of re-running the whole prompt through the network — so every input token is paid for a single time. A long prompt still isn't free (each output token must attend to that whole memory, and it takes up space — which is exactly why context is limited), but because the input is computed once, token-for-token it stays much cheaper than output, where every token demands its own fresh pass. (The expensive all-pairs step — relating every prompt token to every other — runs once, in prefill. Each output token still glances at that whole memory once, which is why long prompts slow every token; it just never rebuilds that pairwise work.)

The Price List

Zoom out to a few real models and the same pattern is everywhere:

{{visual:token-price-scorecard}}

Two rules worth memorizing:

1. Output > input — typically 3–5× more per token (sometimes higher). Verbose answers are where your money goes.
2. Audio (and other media) tokens ≫ text tokens — a realtime audio model can cost far more per token than a text one. That's a big reason multimodal features (next episode) are expensive.

💡 The cheapest lever you have is asking for less output. "Answer in one sentence" or "reply with just the number" can cut your bill by an order of magnitude — because you're buying fewer of the expensive tokens.

Capping the Cost with max_tokens

You don't have to hope the model is brief — you can set a hard ceiling. The max_tokens parameter caps how many tokens the model is allowed to generate. Generation stops when either the model finishes naturally or it hits your max_tokens limit — whichever comes first. Walk through it, then run it:

{{visual:max-tokens-walkthrough}}

{{cell:l26-maxtokens}}

Notice the resp.usage numbers at the end — that's the model handing you an itemized receipt of exactly how many input and output tokens you spent. (Running this needs your OpenAI key in a real browser tab.)

It's Really About Energy

Zoom out and ask: why does a token cost anything at all? It's not the software — it's the physics.

• Electricity. Models run in data centers on GPUs that draw huge power, 24/7. That's the biggest running cost.
• Cooling. Those GPUs get blisteringly hot, so data centers burn even more energy (and water) on cooling.
• Hardware. The GPUs themselves are a massive up-front (capital) cost.

The slogan to remember: it doesn't depend on the model, it depends on the energy. This is also why a cascaded setup (small specialized models) can be far cheaper than one giant multimodal model — small models sip power; giant ones guzzle it. (More on that in the voice episode.)

And the final price? That's partly business strategy, not just cost. Like a grocery store that slashes prices to win volume versus a boutique that charges more per item, providers pick margins to fit their goals — so two models with similar running costs can still be priced very differently.

What Nova Learns Next

You now know Nova is billed by the token, that output is the expensive half, and how to cap it. Those "audio tokens cost more" hints weren't idle — next episode Nova stops being text-only. She learns to see and hear: understanding images, and how the same token loop generates music, audio, and pictures.