One Chain to Rule Them All

Part of the free Generative AI course on LogicWiz, module: Nova Goes to Production.

Episode 18: One Chain to Rule Them All

"Nova retrieves. Nova answers. But her pipeline is a pile of glue code — and half her context is the same ticket five times over."

Two Cracks in a Working System

Chapter V got Nova working: she retrieves from Pinecone and answers from real tickets. Then Arjun opens the code on Monday and spots two cracks.

Crack one — the glue. Every feature is the same hand-wired dance: embed the question, index.query, join the matches into a string, build a prompt, call the model, dig out .choices[0].message.content. Copy-pasted across a dozen endpoints. Change one step and you go hunting through all twelve.

Crack two — the echo. Arjun asks about a checkout timeout. Retrieval returns the top 5 tickets... and all 5 are the same fix, worded five slightly different ways. Nova reads five copies of "retry after 30s" and learns nothing the first copy didn't already say. The other four slots — which could have held the related billing quirk or the CDN note — are wasted.

This episode fixes both. First we make retrieval smarter (stop the echo), then we make the whole pipeline one line (kill the glue).

The Echo Problem: Similarity Is Too Similar

Plain similarity search has exactly one job: return the k nearest vectors. That is the problem. If your archive holds forty near-duplicate "timeout" tickets, the five nearest are... five timeouts. Sky-high relevance, zero diversity. Nova's context window fills up with one idea.

What you actually want: the first result is the best match, but each next result should be relevant and different from what you already picked — cover the topic broadly instead of stacking near-duplicates.

That is Maximal Marginal Relevance (MMR). It's a re-ranking step: fetch a wider pool (say 20 candidates by similarity), then greedily build the final k by scoring each candidate on two things at once — its relevance to the query and its dissimilarity from the results already chosen. A single knob (often called lambda) sets the balance: turn it all the way to relevance and MMR collapses back into plain similarity; dial in diversity and it spreads the net across the topic.

💡 Tip: MMR doesn't replace similarity search — it refines it. You still fetch by similarity (fast, done by Pinecone), then re-rank a small pool for diversity (cheap, done in a few lines). Reach for it when your data clusters into near-duplicates; skip it when every chunk is already distinct.

Watch it work on Nova's archive — pull a wide pool from Pinecone, then re-rank for a diverse few. Walk the code first, then run it:

Same query, same real index. Plain similarity handed back the same 504 timeout three times; the MMR re-rank kept one timeout and added two different causes — an expired coupon and a failed address check. Three angles Nova can actually reason across, instead of one idea in triplicate. (In a server-side LangChain pipeline that re-rank is a single flag, search_type="mmr" — see the walkthrough above.)

Choosing k: The Goldilocks Number

MMR decides which chunks. k decides how many. And k is a real dial, not a guess:

Too few (k = 1) — one chunk, and if the answer spans two tickets, Nova is missing half the story.
Too many (k = 20) — twenty chunks, and the one relevant line drowns in noise. The model gets distracted, answers get vaguer, and every extra chunk is tokens you pay for.

The trap is judging k by vibes — "looks about right." The professional move is data-driven: try k = 2, 3, 5, measure answer accuracy on a set of real questions, and keep the k where accuracy peaks. (We build exactly that measurement in Lesson 20.)

⚠️ Warning: There's no universal best k. A FAQ bot might peak at k = 2; a legal-research tool might need k = 8. Tune it per use case, against your questions — never copy a number from a blog post.

Two More Retrieval Upgrades

MMR and k tune how you pick from one index. Two more upgrades change what you search — and both come straight from real production pain.

Hybrid search. Not every query is semantic. A reader typing "error 504" wants an exact match on that code, not the nearest meaning — and pure vector search is genuinely weak at exact tokens (it might rank a "503 CDN" ticket right beside the 504 one). The fix: run a keyword index (exact terms, error codes, order IDs) alongside the semantic index, and merge the two result sets. You keep the paraphrase-matching of embeddings and the precision of exact-term lookup.

Hierarchical indexing. Picture Nova's 200-page refund & billing manual, chunked into ~800 paragraphs. A reader asks: "Refund on an annual plan after 40 days?" A flat index compares that question against all 800 chunks at once — and the answer drowns in noise: "40 days" also appears in the enterprise SLA section, "refund" shows up in a dozen unrelated places. Semantically close, contextually wrong.

Hierarchical indexing keeps the document's structure by searching in layers. Instead of one flat pile, you build two indexes:

A coarse index — one embedding per section, built from a short summary of that section (≈15 entries, one per chapter).
A fine index — the actual paragraph chunks, each tagged with the section it came from.

Retrieval then drills down a tree instead of scanning everything:

Search the coarse index first — match the question against the ~15 section summaries. Winner: §2 Annual plan refunds (the enterprise "40-day" section loses at the summary level).
Search the fine index, but only inside §2 — now run the normal similarity search over just that section's ~40 paragraphs, not all 800. Out comes the one paragraph that answers it.

Two small searches instead of one big one: first pick the right room, then find the right page in that room. It's the library trick — floor → shelf → book → index — and in Pinecone, step 2 is just an ordinary index.query(...) with a metadata filter (section == "annual-refunds"), exactly the filtering you met in Chapter V.

Here it is on real Pinecone, in the two stages production actually uses — index once (offline), retrieve many times.

1 · Index the two levels. Embed each section summary (tagged level="section") and each paragraph (tagged level="para" plus its section), and upsert both into the same index — coarse and fine, told apart only by metadata:

2 · Retrieve, coarse → fine. Now a query drills down: match the summaries to pick the section, then search only that section's paragraphs with a metadata filter:

Reach for it when documents are large and well-structured (manuals, contracts, API docs). Skip it for a flat pile of short, unrelated docs — like independent support tickets — where there's no hierarchy to exploit.

📌 Summary: MMR kills redundancy, k balances context vs. noise, hybrid search catches exact terms, hierarchical indexing drills into structure. These are the knobs that turn "retrieval works" into "retrieval is good."

Killing the Glue: One Chain

Now crack two — the copy-pasted plumbing. Every RAG feature runs the same five stages:

flowchart LR
    R["Retrieve<br/>(Pinecone)"] --> P["Augment<br/>(prompt)"]
    P --> L["Generate<br/>(LLM)"]
    L --> O["Post-process<br/>(parser)"]

Writing that by hand every time is the glue. LangChain turns each stage into a snap-together Lego block, and LCEL (LangChain Expression Language) snaps them with a single | — the same pipe idea as a Unix command line, where each block's output flows into the next:

chain = retriever | prompt | llm | parser

Four blocks, one chain, and the whole RAG pipeline is one object you can invoke, swap pieces in, and reuse everywhere.

Here's each block:

retriever — a small runnable that embeds the question, queries your real Pinecone index, and returns the matching text. (A retriever is just a question → context step; wrapped in RunnableLambda, it slots straight into the chain.)
prompt — a ChatPromptTemplate with the anti-hallucination rules baked in: "You are SupportDesk AI. Answer only from the context. If it's not there, say you don't know."
llm — a ChatOpenAI model.
parser — a StrOutputParser that turns the model's raw message object into a clean string (and is where you'd later add citations).

One wrinkle: the retriever returns a list of Document objects, but the prompt wants a plain string. A tiny format_docs helper joins them — and joining with clear separators makes document boundaries visible, so the model can attribute each fact to its chunk.

Assemble it once, and Nova's entire answer path is a single chain.invoke("..."):

Change the model? Swap the llm block. Add re-ranking? Slot it after retriever. Want MMR? Configure the retriever. The glue is gone — the pipeline is a chain you edit like Lego.

The Latency Bill

A clean pipeline can still be slow — and in production, slow loses users. So it pays to know where the seconds go.

Retrieval and embedding are nearly instant; the LLM generation step is the bottleneck (~80% of the bill), and post-processing adds a little on top. So the optimisation budget belongs on the model, not the lookup. Three forces move the needle:

Context size — the more chunks you stuff in, the longer the model takes to read them. A big context can add a second or two on its own. (One more reason k and MMR matter.)
Depth & temperature — a model set to think hard is slower. temperature=0 is faster and cheaper, trading away some nuance.
The model itself — even on fast hardware, a big model just takes time to generate. A smaller model answers in a fraction of it.

And a hard truth about third-party models: their latency is a black box you don't control. An OpenAI call is usually 1–2s, but under load (say, a new-model launch) it can spike toward 10s. You can't fix their internals — only optimise your half (retrieval, caching) and pick a model that fits your speed budget.

⚠️ Warning: Millisecond latency and deep, accurate reasoning are usually incompatible — you pick a point on the curve. Latency first (a live chat)? Reach for a smaller, faster model and accept slightly shallower answers. Correctness first (a legal lookup)? Pay the seconds. Then set timeouts that match reality, not wishful thinking.

Keeping the Brain Current

A RAG index is not a one-time build. Tickets get resolved, policies get rewritten, prices change — and the moment the source of truth moves, your vector store is stale. A stale index means Nova confidently serves last month's policy.

So the offline pipeline has to re-run as data changes. Two strategies:

Scheduled (cron) — re-run the whole ingest on a fixed cadence (nightly, weekly). Dead simple, and fine when data drifts slowly.
Change detection — figure out which documents changed and re-index only those: pull the new version, delete the old embeddings, embed and upsert the new ones in their place. Efficient at scale. (Detecting what changed — file timestamps, source webhooks — is a plain software problem, not an AI one.)

How fresh is fresh enough? "Live" isn't universal — it's defined by your refresh frequency. Picture a bank that changes a withdrawal limit at 00:00. If your cron job re-indexes at 00:00, a query at 04:00 gets the right answer; if it only runs weekly, a customer at the ATM gets last week's limit. An index's liveness is exactly how quickly it mirrors the source of truth — and note it's the RAG system, not the LLM, that "knows" the new fact: the model only sees what the freshly-indexed store hands it.

flowchart LR
    S["Source changes<br/>(policy updated 00:00)"] --> D{"Detect change"}
    D -->|cron: nightly| I["Re-index changed docs<br/>delete old · upsert new"]
    D -->|event: webhook| I
    I --> V[("Vector store<br/>now current")]
    V --> A["Query at 04:00 → correct"]

💡 Tip: For high-urgency data, an event-based trigger (re-index the instant the source changes) beats a schedule. For everything else, a cron on a sensible interval — matched to how fast your data really moves and what re-indexing costs — is the pragmatic default. Same trade-off as everywhere in RAG: freshness vs. cost.

📌 Summary: This episode made retrieval sharper — MMR for diversity, a data-tuned k, and hybrid + hierarchical indexing — and the pipeline simpler: LangChain's Lego blocks snap into one retriever | prompt | llm | parser chain you invoke once. Then it made both production-ready: budget your latency (~80% of it is the model), and keep the index fresh with scheduled or event-driven re-indexing.

What Nova Learns Next

Nova's pipeline is clean and her retrieval is sharp. But she still has the memory of a goldfish: every question starts from zero. Ask "what about the annual plan?" right after a billing question and she has no idea what "that" refers to.

Next episode, Arjun gives Nova a memory — conversation history — so she can hold a real multi-turn conversation without forgetting what you said two messages ago.