Serving answers from your own data (RAG)

Part of the free System Design course on LogicWiz, in the chapter: Designing GenAI systems. The interactive whiteboard exercises and simulations run in the app.

Week 11: The bot that had never read the policy

"The demo bot promised a customer a full refund on an opened jar of ghee. Our policy says no such thing, and the bot had never read our policy."

Farah, SwiftKart's support lead

The model has never read your help centre

This week SwiftKart launches two things built on a large language model (LLM), the kind of AI model that writes text. The first is a shopping assistant in the app ("what do I need for paneer butter masala for four?"). The second is a support bot that answers questions about orders, refunds and delivery. Farah, who runs support, found the problem in five minutes. Asked about returns, the bot gave a confident, friendly, wrong answer.

The model was not broken. It had never seen SwiftKart's return policy. A general model knows what return policies usually say, so it wrote a plausible one.

The fix has a long name and a simple idea. Retrieval-augmented generation (RAG) works like this. Before asking the model, look up the relevant passages in your own documents. Paste them into the prompt (the text you send the model), with an instruction to answer only from them. It is an open-book exam.

Meera came to the meeting with a different plan: "Let's fine-tune the model on our help centre, and it will know the answers." Fine-tuning means further training a model on your own examples. It is good at teaching a style, a format or a narrow task. It is a poor way to store facts that change. Say the return window goes from 7 to 10 days for the Diwali sale. With fine-tuning you retrain and redeploy. Even then, the model may blend the old and new numbers. It cannot cite its source. You also cannot delete one leaked page from its weights (the numbers the model learned in training). With RAG you edit one document, and the next answer is right.

💡 Tip: The one-line interview answer: fine-tune for behaviour, retrieve for facts. Real systems often do both, but fresh facts always come from retrieval.

Ingestion: chunk, embed, store

RAG has two paths. The ingestion path runs whenever documents change and prepares them for search.

  1. Split into chunks. A three-page refund policy is too big to paste whole and too broad to match one question. Split it into chunks of about 500 tokens. A token is the unit a model reads and charges for, roughly three-quarters of a word. Let chunks overlap a little so no sentence is cut in half. Keep headings so each chunk makes sense alone.
  2. Embed. An embedding model turns each chunk into a vector: a list of, say, 768 numbers. Text with similar meaning gets vectors that lie close together. "My money is stuck" and "refund pending" end up near each other though they share no words.
  3. Store with metadata. Write each vector into a vector store, a database that finds the vectors nearest to a given one. Store it with its chunk text, document id, version, audience (customer, agent or internal) and region.
  4. Re-index on change. When an article is edited, re-chunk it, embed the new chunks and delete the old ones by document id.

Now the worked numbers. SwiftKart's help centre, policies and FAQs come to about 40,000 pages of roughly 5 chunks each. That is 200,000 chunks of about 500 tokens, about 100 million tokens in all.

Index size ≈ chunks × dimensions × 4 bytes per number

  • 768 dimensions: 768 × 4 = 3,072 bytes, about 3 KB per vector. 200,000 × 3 KB is about 600 MB.
  • 1,024 dimensions: about 4 KB per vector, so about 800 MB.

Add the search structures (often another half again) and the chunk text (about 2 KB each, roughly 400 MB more). The whole thing is a couple of gigabytes. It fits in the memory of one machine; run two copies for availability. The hard parts are elsewhere.

The query path, step by step

When a customer asks "can I return opened ghee?", the service does this:

  1. Embed the question with the same embedding model used at ingestion. A different model puts vectors in a different space, and nothing matches.
  2. Retrieve the top-k chunks: the k most similar ones, for example the 50 nearest.
  3. Optionally go hybrid. Run a keyword search alongside. Vectors are good at paraphrase but weak at exact strings: an error code like "UPI-U30", an order id, a brand name. Merge both result lists.
  4. Rerank. A reranker is a slower, more accurate model. It reads the question and each candidate together and scores the pair. It cuts 50 candidates to the best 5.
  5. Build the prompt: instructions, the 5 chunks labelled [1] to [5], and the question, with "answer only from these sources and cite them".
  6. Call the model through the LLM gateway (next lesson) and stream the answer back.
flowchart LR
  Q["Question"] --> E["Embed"]
  E --> V["Vector + keyword search"]
  V --> R["Rerank to top 5"]
  R --> P["Prompt with citations"]
  P --> L["LLM"]

A latency budget makes the steps concrete. Roughly:

  • embedding the question: about 30 ms;
  • the search: about 30 ms;
  • reranking 50 candidates: about 100 ms;
  • the model's first token: about 600 ms more.

A full 250-token answer at about 50 tokens a second takes another 5 seconds. That is why the answer is streamed (sent word by word as it is written), not delivered in one piece. Retrieval is cheap; generation takes most of the time.

Permissions and freshness

SwiftKart's index holds customer articles, agent playbooks and an internal fraud runbook. A customer's question may happen to sit close to the fraud runbook. Then top-k similarity will return it.

Permission-aware retrieval means filtering by the user's access inside the retrieval query, before any chunk reaches the prompt. Only chunks whose audience and region match this user come back. Telling the model "do not reveal internal documents" is not a control. Anything in the prompt can end up in the answer. The same goes for personal data. The service fetches a customer's own orders from the orders database with their user id. They never go into a shared index.

⚠️ Warning: Filter before the model, never after. Once a chunk is in the prompt, assume the user can see it.

Freshness comes from the ingestion path. The content team publishes a change, and the CMS (the tool they edit articles in) puts an event on a queue. A worker re-chunks, embeds and replaces that document's chunks, typically within a minute or two. Keep the document version in the metadata so you can tell which version an answer cited. One trap: switching to a new embedding model means re-embedding every chunk (all 100 million tokens). Old and new vectors cannot be compared. Build the new index in the background, then switch over.

Knowing whether it works

Farah's team wrote a small labelled question set: 200 real customer questions, each tagged with the document that answers it. It gives two numbers:

  • Retrieval hit rate: how often the right chunk is in the top 5. If 170 of 200 questions retrieve it, the hit rate is 85%. When retrieval misses, no prompt can save the answer. So fix chunking, hybrid search or reranking first.
  • Answer faithfulness: how many answers make only claims the cited chunks support. A reviewer marks each answer, or a second model does with human spot-checks. A bot that retrieves the right policy and still says "full refund" fails here.

Run the set on every change to chunking, the embedding model, the prompt or the LLM, and compare with the last run.

📌 Summary: RAG pastes your own retrieved documents into the prompt: fine-tune for behaviour, retrieve for facts. Filter by permission before the prompt, and measure retrieval hit rate and faithfulness on a labelled set.

What Meera learns next

The assistant and the support bot now call the model hundreds of thousands of times a day. Next: the LLM gateway that keeps those calls affordable, fast and safe.

Check your understanding

  1. SwiftKart extends its return window from 7 to 10 days for the Diwali sale, starting tomorrow. The support bot must give the new number from day one. What is the right change?
  2. The help centre is about 200,000 chunks, each embedded as a 1,024-dimension vector of 4-byte floats. Roughly how big are the vectors alone?
  3. A customer asks about a delayed refund and the bot quotes a line from the internal fraud runbook. The team proposes fixes. Which one actually closes the hole?
  4. On a 200-question labelled set, the right chunk appears in the top 5 for only 110 questions, and most wrong answers come from those misses. Where should the team work first?