The Goldfish Problem

Part of the free Generative AI course on LogicWiz, module: Nova Goes to Production.

Episode 19: The Goldfish Problem

"Nova answers every question perfectly — and forgets it the instant she's done. Ask a follow-up and she stares at you blankly."

"What About the Annual Plan?"

Priya is chatting with Nova. She asks: "How much is the Pro plan?" Nova nails it — retrieves the pricing ticket, answers cleanly. Then Priya types the most natural follow-up in the world:

Priya: "And what about the annual one?"

Nova: "I'm not sure what you're referring to — could you clarify?"

Look at what Priya actually sent this turn: just "And what about the annual one?" The annual what? You can answer instantly — one message ago she asked about the Pro plan, so "the annual one" obviously means the annual Pro plan. Nova can't, and here's exactly why: an LLM is stateless — it sees only the text in this one call, nothing before it. Nova's app never saved that first "How much is the Pro plan?" exchange and never fed it back in, so all the model received this time was the bare sentence "And what about the annual one?" — no plan, no earlier question, nothing to anchor "the annual one" to. Annual what? It genuinely has no idea. Every turn is a cold start; Nova has the memory of a goldfish.

The missing piece is now obvious: conversation history. If that earlier turn ("Pro plan → $20/month") had ridden along with the new question, Nova would have resolved "the annual one" to the annual Pro plan without blinking — which is exactly what we build next.

{{visual:conversation-memory}}

Every stateless assistant fails the same way: the user is forced to repeat context in every single message — "what about the annual Pro plan?" — which is exactly the robotic, un-human experience good products avoid.

The Fix: Hand Her the Whole Conversation

The model has no memory, so you give it one. Before each answer, you bundle the entire conversation so far into the prompt — turn 1, turn 2, turn 3 — so the model reads the whole thread and can resolve "the annual one" against what came before.

It's less an architecture change than a prompt change: keep a running list of messages, and prepend it to every call.

{{visual:history-replay}}

The prompt for a multi-turn RAG assistant has four functional blocks, in order:

1. System message — Nova's persona and rules ("You are SupportDesk AI. Answer only from the context...").
2. Conversation history — the previous turns, as a Human → AI → Human → AI sequence. (In LangChain this slot is a MessagesPlaceholder.)
3. Retrieved context — the chunks fetched from Pinecone for this question.
4. Current question — the reader's latest message, which is now allowed to say "the annual one."

flowchart TD
    S["1 · System message<br/>(persona + rules)"] --> H["2 · Conversation history<br/>(Human → AI → Human → AI)"]
    H --> C["3 · Retrieved context<br/>(from Pinecone)"]
    C --> Q["4 · Current question<br/>('what about the annual one?')"]
    Q --> LLM["LLM → grounded, context-aware answer"]

Feed the model those four parts and "the annual one" suddenly has an antecedent. Watch a follow-up land with history vs. without it:

{{visual:history-walkthrough}}

{{cell:l19-history}}

💡 Tip: History and retrieval are different memories. Retrieved context is what LogicWizNews knows (the tickets). Conversation history is what you and Nova just said. A good multi-turn prompt carries both — the facts and the thread.

The Catch: History Grows Forever

There's a bill attached to that memory. Every turn you add makes the prompt longer, and you re-send the whole history on every call. A 40-message conversation means each new answer pays for 40 messages of tokens. History that grows unbounded eventually blows past the model's context window — and your budget.

So engineers choose between two strategies:

Both are a few lines of LCEL — carry the raw turns, or run one LLM call to compress the thread into a running summary. (Skip the old ConversationSummaryMemory helper — it's deprecated; doing the compression yourself is clearer and current.) Walk the code, then run it:

{{visual:summary-walkthrough}}

{{cell:l19-summary}}

There's no universally right answer; it's a per-product call. And it's a genuinely economic one.

⚠️ Warning: Watch the unit economics. If a reader pays $1 for a query but replaying six months of history costs $5 of tokens to answer it, the feature is losing money on every use. "Remember everything forever" is a great demo and a terrible default — decide how much history is worth it.

📌 Summary: LLMs are stateless, so you feed them the conversation. A multi-turn prompt = system + history + retrieved context + question. As history grows, choose full (lossless, pricier) or summarized (cheap, lossy) — tuned to your product and its token budget.

What Nova Learns Next

Nova now retrieves well, orchestrates cleanly, and holds a conversation. But Arjun has no idea if she's actually good. Is her retrieval finding the right tickets? Are her answers faithful to them, or is she quietly making things up? "It looks fine" is not a number.

Next episode, Arjun stops guessing and starts measuring — precision, recall, groundedness — and builds the evaluation harness that tells him, with data, whether a change made Nova better or worse.