Design an AI support assistant (RAG)

A help bot that answers from SwiftKart's help articles and the customer's own orders, with citations, and hands off to a human when unsure. The core is ingestion, retrieval that respects permissions, an LLM gateway with fallbacks, and evals that every change must pass.

SwiftKart wants an AI help assistant in the app, and for support agents. It answers questions like "where is my order?", "can I return opened atta?" and "why was I charged twice?". It uses help-centre articles, policies and the customer's own orders. It must never show one customer another customer's data, and must hand off to a human when unsure.

Requirements

Functional

Ingest help articles and policies, and keep the index up to date when they are edited.
Answer questions grounded in retrieved content (based only on it), with citations to the source articles.
Look up the signed-in customer's own orders and refunds through read-only tools.
Hand off to a human agent, with a summary of the conversation, when confidence is low, the user asks, or the topic is sensitive.
Agents see internal procedures that customers must not see.

Non-functional

Peak of about 20 turns per second. First token under 2 seconds for 95% of turns (p95); full answer under 6 seconds.
Zero cross-customer data exposure, enforced in code rather than by the prompt.
Cost budget of about ₹5 per conversation. Answers get simpler, but do not stop, when the model provider is slow or down.
Grounded answers: no release ships unless it passes the eval set (a fixed set of test questions). For example, at least 95% of answers cite a correct source.

Out of scope

Training or fine-tuning models.
Voice support.
Refunds without approval: the assistant can propose one; a rule engine or a human approves it.

Estimate

Conversations: 2M daily users x ~2% who open help = 40,000 a day; x ~4 turns = 160,000 turns a day = ~2 turns/s average; festival evening peak ~10x = ~20 turns/s.
Tokens per turn: ~1,000 instructions + 5 retrieved chunks x 400 = 2,000 + ~1,000 history + ~50 question = ~4,000 input; ~250 output.
Daily tokens: 160,000 x 4,000 = 640M input; 160,000 x 250 = 40M output.
Cost at illustrative prices of ₹200 per million input tokens and ₹800 per million output (the same hypothetical prices as Week 11). Daily cost: 640 x 200 = ₹1,28,000 + 40 x 800 = ₹32,000 = ~₹1,60,000 a day = ~₹4 per conversation. Within budget, and routing simple intents to a smaller model brings it down further.
Corpus: ~50,000 articles, policies and FAQs x ~2,000 tokens = 100M tokens; in 400-token chunks = ~250,000 chunks; x 1,024 dimensions x 4 bytes = ~1 GB of vectors. Small: it fits in one machine's memory, with a replica.
Concurrency: 20 turns/s x ~5 s per streamed answer = ~100 generations in flight; that, not storage, sets the provider quota to reserve.

API

POST /assist/conversations {channel: "app" | "agent"} -> 201 {conversationId}
POST /assist/conversations/{id}/messages {text} -> 200 (streamed) {answer, citations[], suggestedActions[]}
POST /assist/conversations/{id}/handoff {reason} -> 202 {ticketId}
POST /assist/messages/{id}/feedback {helpful, reason} -> 204
POST /kb/webhooks/document-changed {docId, version} -> 202

Data model

documents(doc_id PK, version, audience customer | agent, locale, updated_at, source_url)
chunks(chunk_id PK, doc_id, version, text, embedding vector(1024), audience, locale): vector index with metadata filters
conversations(conv_id PK, user_id, channel, status, created_at)
messages(conv_id, seq, role, text, citations[], model, tokens_in, tokens_out, latency_ms)
eval_cases(case_id PK, question, expected_facts[], must_cite[], must_hand_off bool)

A request, step by step

Ingestion: when an article is published or edited, the CMS sends an event. A worker splits the article into chunks of about 400 tokens along its headings. It embeds them through the LLM gateway (turns each chunk into a vector of numbers that captures its meaning). It upserts them (inserts or updates) tagged with the doc version, the audience (customer or agent) and the locale. Old versions are deleted after the new one is in place.
A customer asks "can I return opened atta?". The gateway authenticates them. The assistant service takes user_id and role from the auth token, never from the conversation text.
The service embeds the question and retrieves the top chunks with a filter that is always applied: audience = customer, plus the locale. It may rerank them (sort them again with a more careful model). If the question is about the customer's order, it calls the order API, which uses the caller's user_id from the token.
It builds the prompt: instructions, retrieved chunks with their ids, tool results and a short history. It calls the LLM gateway. The gateway picks a model, enforces quotas and timeouts, and switches to a second model if the first fails. The answer streams back with citations.
Guardrails check the output. Every factual claim must cite a retrieved chunk. No personal data other than the caller's may appear. Refund or legal topics trigger a handoff. Low confidence, or a request from the user, creates a support ticket with a summary. Everything is logged for evals.

Deep dives

Permission-aware retrieval: never another customer's data

Weak: put everything in one index, and tell the model not to reveal it. A prompt is not an access-control system.

A cleverly worded question, or plain bad luck in retrieval, can put another customer's order or an internal fraud procedure into the context. The model may then repeat it.

Good: filter at retrieval time. Tag every chunk with its audience, region and language. Apply the filter inside the vector query, using the caller's role from the auth token.

Agents see procedures meant only for agents. Customers never retrieve them, so the model never sees them in a customer chat.

Great: keep personal data out of the shared index completely. Help articles are shared and indexed. A customer's orders are not.

Personal data arrives only through tools, such as the order API. These tools take identity from the logged-in session, not from parameters the model chooses. So "show me order 123 of user 456" returns nothing.

Add an output check that hides phone numbers and addresses that are not the caller's. Run a test suite of leak attempts on every release.

Trade-off: tools add a network call and some latency, compared with putting everything into one index. But the security boundary becomes the same one the rest of the app already enforces.

The LLM gateway: routing, fallbacks and cost

Weak: every service calls a model provider directly, with its own API key. There are no shared timeouts. Nothing switches to another model when the provider is slow during a festival peak. Nobody knows which feature spent the budget. And a leaked key means changing secrets in ten places.

Good: one gateway in front of all model calls. It holds the keys and enforces quotas and timeouts per feature. It retries once, with backoff, on temporary errors. It logs tokens and latency for every call. When the main model is down, it routes to a fallback model.

Great: the gateway also makes cost and quality decisions. Send simple intents, such as order status or a policy question, to a smaller and cheaper model. Send hard ones to a larger model.

Cache answers to general questions that contain no personal data. Key the cache by the normalised question and the doc version. Limit the context size. Stream tokens so the user sees the answer start within 2 seconds.

When every model is unavailable, the assistant falls back to normal search over the help centre plus a "talk to an agent" button. Customers are never stuck.

Trade-off: every AI call now depends on the gateway. So it needs replicas and its own SLO (service level objective: a reliability target, such as 99.9% of calls succeeding). In return, provider outages and cost spikes are handled in one place.

Evals and guardrails: knowing it is right

Weak: try ten questions by hand before launch. Later, a prompt change or a model upgrade breaks refund answers without anyone noticing. You find out from angry customers.

Good: an offline eval set that runs on every change. Collect a few hundred real questions, with personal details removed. For each, record the expected facts, the required citations, and whether it must be handed off.

Score groundedness: is each claim supported by a retrieved chunk? Also score correctness against the expected facts, and handoff behaviour. Treat it like a test suite: a change that lowers the score does not ship.

Great: offline evals plus live signals and runtime guardrails. For each release, track the thumbs-down rate, the handoff rate, how often customers contact support again within 24 hours, and the cost per conversation.

Every week, have humans review a sample of conversations, and turn failures into new eval cases. At runtime, check that citations exist and match. Block answers that promise refunds the policy does not allow. Send sensitive intents (fraud, legal, safety) straight to humans.

Trade-off: evals cost time and some model spend on every change. Graders that are themselves LLMs need their own spot checks. Without evals, though, you cannot tell an improvement from a regression.

What breaks

The main model provider is slow or down. The gateway times out and switches to a second model. If all models fail, the app shows help-centre search and a button to reach a human.
The vector index is unavailable. Answer only questions that tools can answer, such as order status. Hand off policy questions instead of answering without sources.
A policy changes but old chunks remain. Upserts are tagged by version, and old versions are deleted when the new one is published. The eval set includes cases about recently changed policies.
Costs spike because of a prompt bug or abuse. The gateway enforces quotas per user and per feature, and an alert fires on cost per conversation.
A prompt injection (text written to trick the model) tries to make tools fetch another user's data. Tools ignore any identity the model supplies, so the attempt fails safely and is logged.

What each level is expected to cover

Mid level: Describes ingestion (chunk, embed, index), retrieving the top chunks into the prompt, citations, a handoff path and basic filtering by audience. Estimates tokens per turn and the daily cost.

Senior: Makes the security boundary explicit: personal data only through tools tied to the logged-in identity, and filters inside retrieval. Designs the LLM gateway with routing, fallbacks, quotas and caching. Sets up offline evals that every release must pass, plus live metrics. Plans how the assistant keeps working when models or the index fail.

Common mistakes

Relying on the system prompt to keep other customers' data secret.
Putting personal order data in the shared vector store.
No eval set, so model or prompt changes ship based on a feeling that they look fine.
Calling a provider directly from each service, with no timeouts, fallback or cost tracking.
Letting the assistant take actions that cannot be undone, such as refunds, without a rule check or a human.

Follow-up questions

How would you support Hindi and Hinglish questions over English articles?
How do you decide when the assistant is confident enough to answer instead of handing off?
A new model is cheaper. How do you decide whether to switch?
How would you let the assistant cancel an order safely?
What would you log, and what would you deliberately not log, for privacy?