Indexing & Retrieval

Part of the free Generative AI course on LogicWiz, module: Nova Reads the Archives.

Episode 17: Finding the Needle in Ten Million

"Nova can rank ten tickets by meaning in a blink. The archive has ten million. Same idea, and it grinds to a halt."

The Loop That Doesn't Scale

Back in Episode 15, semantic search was a tidy little loop: embed the query, then compare it against every ticket vector and keep the closest. Beautiful for ten tickets. Arjun points it at the full archive — ten million chunks — and Nova takes thirty seconds to answer one question.

The problem is the word every. Comparing the query against all ten million vectors is a full scan — O(N) work that grows with the archive. It's the same reason you don't find a word in a dictionary by reading it cover to cover.

Databases solved this decades ago with an index: a structure that lets you jump to the relevant rows instead of scanning all of them. Vector databases borrow the idea. Indexing organises the millions of vectors so that, given a query, the store leaps straight to the right neighbourhood and searches only there — turning a thirty-second scan into a few milliseconds.

💡 Tip: Indexing is an offline step — it happens during ingestion, right as vectors go into the store, before any reader asks a question. At query time you use the index; you never rebuild it. Sequence: chunk → embed → index → store.

You don't build the index yourself — vector databases like Pinecone, Chroma, and FAISS do it for you. Your job is just to write your vectors in. And that's a piece of code we haven't shown yet, so let's fix that.

Writing Vectors Into the Store

The offline job is short: for each chunk, you hand the store three things — an id, the vector (from the embedding model), and a bit of metadata (which topic, which ticket, which date). The store indexes it and it's ready to search.

The cell below writes to a real Pinecone index, so it needs a one-time setup (about 3 minutes, free — no card required):

🔑 Step 1 — Get your Pinecone API key

Create a free account at app.pinecone.io — the Starter plan is free.

In the console's left sidebar, click API Keys.

Copy your default key (it looks like pcsk_...). Keep it handy for Step 3.

🗄️ Step 2 — Create the index (this is the "store" your vectors go into) 4. In the left sidebar, click Database, then Create index. 5. Name it exactly logicwiznews-articles. 6. Set Dimensions to 1536 and Metric to cosine — these must match the text-embedding-3-small model. 7. Pick Serverless, cloud AWS, region us-east-1, then click Create index and wait a few seconds for it to go green.

🔌 Step 3 — Plug it in 8. Click Set Pinecone Key at the top of this lesson and paste the key from Step 1.

That's it — the cell below now reads and writes your live Pinecone index. (Pinecone occasionally reshuffles its console wording, but those exact values — name logicwiznews-articles, dimension 1536, metric cosine, serverless AWS us-east-1 — are what matter.)

With that in place, here's the write end to end. It has a few moving parts, so step through them one block at a time first:

Now the whole thing, runnable — this is exactly the code you just toured:

That's the entire offline half of RAG: text in, vectors written, ready to query. This writes to a real Pinecone index — tap Set Pinecone Key above the lesson, and make sure a serverless index named logicwiznews-articles exists (dimension 1536, metric cosine). It's the very same store Nova runs on in Episode 19; here you keep the runbook tidy in its own runbook namespace, separate from everything else.

And in production, one index is rarely enough.

Why One Index Isn't Enough

Think of a big supermarket. You don't find milk by walking every aisle — you head straight to Dairy. Multiple indexes are those aisles.

Here's what that means concretely. An index is a single searchable store of vectors. You could pour everything into one — every FAQ, every ticket, every runbook, HR records, all mixed together. Or you can keep several separate stores, each holding just one kind of data, and send each query to only the store it belongs to. LogicWizNews goes with the second option and keeps five:

FAQ & docs — how-to answers and policies ("what's the refund window?")
Incident runbooks — error codes and outage fixes ("error 504 at checkout")
Billing tickets — past payment and subscription cases
Product specs — feature details and limits
HR (restricted) — salaries and staff data, locked to HR users only

Watch what that split does to a single query — one mixed pile vs. five labelled stores:

So why five and not one? Four forces decide where the lines get drawn:

1. Different data types. PDFs, FAQs, code snippets, tables, and tickets are different animals. Put each in its own index and a code query never has to wade through HR PDFs.

2. Different search strategies. Not everything is semantic. A reader typing "error 504" wants an exact match on that code — not the "nearest meaning." That's keyword search. Semantic questions ("why can't I log in") want dense-vector search. Systems that need both run hybrid search — a semantic index and a keyword index, combined.

3. Different update frequencies. Live metrics change by the second; old resolved tickets change never. Splitting them lets you re-index the fast-moving data often and the slow stuff weekly — cheaply.

4. Security. Salaries belong in an HR-only index; product docs are open to everyone. Separate indexes become access-control boundaries — a query from a non-HR user simply never touches the salary index. (These separate buckets are often called namespaces.)

flowchart TD
    Q["Reader query"] --> R{"Which index?"}
    R -->|policy question| A[("FAQ / Docs")]
    R -->|error code| B[("Incidents / Runbooks")]
    R -->|payment issue| C[("Billing tickets")]
    R -->|HR + authorized| D[("HR — restricted")]

Routing: Sending the Query to the Right Pile

Multiple indexes raise an obvious question: when a query comes in, which index do you search? You don't want to search all five — that defeats the point.

The answer is intent classification (a.k.a. routing): look at the query, decide what kind of question it is, and send it to the matching index.

"What's the PTO carryover limit?" → a policy question → route to the FAQ/docs index.
"Error 504 on checkout" → a troubleshooting question → route to the incidents index.

How you classify is up to you — a few keyword rules, a trained classifier, or (increasingly) just asking an LLM "which category is this?" The architecture is what matters: a routing step exists between the query and the indexes. Watch one route three different questions:

Here's a router in action — an LLM deciding which index a query belongs to:

📌 Summary: Production RAG runs multiple indexes, split by data type, search strategy, update frequency, and security. A router classifies each query's intent and sends it to the right index — so retrieval searches a small, relevant slice instead of the whole world.

A Quick Contrast: Database Index vs. Vector Index

If you've used SQL, this all sounds familiar — and it should. Both kinds of index exist to avoid a full scan by narrowing the search space. But the lookup is fundamentally different:

A transactional index (Postgres, etc.) maps a query to an exact value through a pointer — "give me the row where id = 42." Precise, binary, match-or-no-match.
A vector index uses geometry — "give me the points nearest this one." Fuzzy, ranked, meaning-based.

Same goal (skip the full scan), different machinery (pointers vs. proximity). And a vector index does one extra job a SQL index doesn't: it segregates data into namespaces for the routing and security we just covered.

The Retrieval Pipeline, Start to Finish

We now have every piece. Let's assemble the online path a question travels — this is the whole RAG retrieval pipeline:

sequenceDiagram
    participant User
    participant Router
    participant Retriever
    participant VectorDB
    participant Reranker
    participant LLM
    User->>Router: question
    Router->>Retriever: routed to the right index
    Retriever->>VectorDB: query embedding + "give me top-k"
    VectorDB-->>Retriever: k candidate chunks + metadata
    Retriever->>Reranker: question + candidates
    Reranker-->>Retriever: top-n, best first
    Retriever->>LLM: question + ranked context
    LLM-->>User: grounded answer + citations

Two steps there deserve a closer look. Think of it like hiring: a quick résumé screen pulls ~20 plausible candidates fast, then a careful interview ranks the best 3. Retrieval does the same in two passes.

Similarity search → top-k (the résumé screen). The retriever embeds the query, hits the right index, and asks for the k nearest chunks — say the top 20 — each with its metadata (which ticket, which date, which priority). It's fast and approximate: it casts a slightly wide net so the good stuff is in the net, even if the order isn't perfect.

Re-ranking → top-n (the interview). A re-ranker takes the question and those 20 candidates and scores each one specifically for how well it answers this question, then keeps the best n (say, 3) in strict order.

Why bother? Because the order matters to the LLM. Telling the model "this chunk is rank 1, this is rank 2" helps it weight the most relevant context first — sharper answers, fewer distractions. It costs an extra pass, but it buys precision.

Watch a re-ranker reorder raw similarity hits into what actually answers the question:

💡 Tip: Retrieval is a funnel. Millions of chunks → route to one index → top-k by fast vector search (~~20) → re-rank to top-n (~~3) → into the prompt. Each stage is cheaper and coarser than the next is precise. You don't feed the LLM the archive; you feed it three perfect paragraphs.

Bringing It All Home

That funnel is the entire RAG engine you've built across this chapter. Put the pieces in a line:

Episode 14 — why: the LLM doesn't know your data and can't be handed all of it, so retrieve and augment.
Episode 15 — embeddings: turn text into meaning-vectors; cosine similarity is the ruler that measures how close two of them are.
Episode 16 — chunking: slice documents so each vector is one clean idea.
Episode 17 — indexing & retrieval: organise millions of vectors, route to the right index, and re-rank down to the perfect few.

Time to run the whole thing end to end — this time on real Pinecone. And notice what you don't do: you never compute cosine similarity by hand. You write the tickets in, then call index.query(...) and the database runs the nearest-neighbour search for you — over millions of vectors, in milliseconds. (Those earlier cells that looped over an in-memory list computing cosine were showing you the mechanism the vector DB runs internally; in production you let the index do it.)

📌 Summary: The retrieval pipeline is a funnel — route → top-k similarity search → re-rank to top-n → grounded generation with citations. Indexing makes it fast; routing makes it targeted; re-ranking makes it precise. That's production RAG.

What Nova Becomes Next

Nova now has a searchable memory of everything LogicWizNews knows. She retrieves the right facts, grounds every answer, and refuses to guess. Readers get correct answers in seconds instead of hours.

But so far Nova only ever does one retrieval per question. What about a question that needs two lookups — "compare our refund policy to what we told this specific customer last week"? Or a question where she should decide whether to search at all? That's the leap from a RAG pipeline to a RAG agent — Nova choosing her own retrieval steps, checking her work, and looping until the answer is solid. That's where the story goes next.