Embeddings & Semantic Search

Part of the free Generative AI course on LogicWiz, module: Nova Reads the Archives.

Episode 15: The Meaning Machine

"Last week Nova learned she needs to search by meaning. This week we find out what 'meaning' even looks like to a machine — it looks like a list of numbers."

The Black Box We Left Open

Last episode ended on a cliffhanger. RAG needs to find the tickets that mean the same thing as a reader's question — even when they share no words. "Can't log in" has to find "authentication fails after credential change."

But a computer doesn't understand meaning. It understands numbers. So the whole trick comes down to one question: how do you turn a sentence into numbers that capture what it means?

That conversion is called an embedding, and it's the single most important idea in this entire chapter. Get embeddings, and semantic search, chunking, and retrieval all fall out of it.

Text → Numbers That Mean Something

An embedding is a list of numbers that represents the meaning of a piece of text as a point in space.

Not a random list. A carefully learned one, produced by an embedding model — a pre-trained neural network whose entire job is to read text and output a fixed-length list of numbers (a vector) such that things that mean similar things get similar numbers.

The model we'll use is text-embedding-3-small — OpenAI's small, cheap embedding model. It turns any text into a vector of exactly 1536 numbers. (Other providers make embedding models too — Cohere, Voyage, open-source ones from Hugging Face. Different model, different length, same idea.)

"1536 numbers" sounds abstract. Let's just look at one:

That's it. That wall of decimals is the sentence, as far as the machine is concerned. Every ticket, every query, every article becomes one of these 1536-number lists.

💡 Tip: The number 1536 is the model's dimensionality — how many coordinates each point has. You don't get to pick it; it's baked into the model. Bigger models use more dimensions (more nuance, more cost); smaller ones use fewer (cheaper, slightly coarser). It's a classic interview question, so remember: text-embedding-3-small → 1536.

A Map Where Distance Means Similarity

Here's why turning text into a point is so powerful. If every sentence is a point in space, then sentences with similar meaning land near each other, and unrelated ones land far apart.

Picture a giant map. "Paris" and "London" sit close together in one neighbourhood (European capitals). "Dog" and "cat" cluster somewhere else (pets). And crucially, "can't log in" lands right next to "authentication failure" — even though they share no letters — because the model learned they mean the same thing.

The model is smart enough to use context, too. The word "Paris" alone is ambiguous — the city, or the person? The surrounding words push the vector toward the right neighbourhood: "Paris in springtime" lands near travel; "Paris wore a gown" lands near celebrities. A single word can't carry its full meaning; the vector is shaped by everything around it.

So the plan is set: convert everything to points, and to find relevant tickets, look for the points nearest the query point. But "nearest" needs a precise definition. What does distance between two 1536-dimensional points actually mean?

Cosine Similarity: Measuring "Same Direction"

You have two vectors and you want a single number for how alike they are. The trick the whole industry uses is cosine similarity.

Instead of measuring how far apart two points are, cosine similarity measures the angle between them — are they pointing the same way?

Vectors pointing in nearly the same direction → angle near 0° → cosine ≈ 1.0 (very similar).
Vectors at a right angle → cosine ≈ 0 (unrelated).
Pointing opposite ways → cosine ≈ -1 (opposite meaning).

Why the angle and not plain straight-line distance? Because direction captures meaning while ignoring magnitude — a short ticket and a long ticket about the same problem point the same way, even if one vector is "bigger." Direction is the meaning; length is mostly noise.

The formula is just the dot product divided by the two lengths — but you'll never hand-write it in production, you'll call a library:

cosine(A, B) = (A · B) / (‖A‖ · ‖B‖)

Let's compute it for real. Three sentences: two that mean the same thing in different words, and one about something else entirely. Watch the scores:

The pair that means the same thing scores high; the unrelated one scores low. No shared keywords required. That single number is the entire basis of semantic search.

⚠️ Warning: A high cosine score means "similar direction in the model's space," not "true." The model can be confidently wrong about two things being related. That's why RAG still hands the retrieved text to an LLM to reason over — retrieval narrows the field, it doesn't decide the answer.

Semantic Search: Retrieval, Finally

Now we have every piece. Semantic search is embarrassingly simple once embeddings and cosine similarity exist:

Offline: embed every ticket, store the vectors.
Online: embed the reader's query into a vector.
Compute cosine similarity between the query vector and every ticket vector.
Return the top-k highest-scoring tickets.

That's retrieval. Watch it solve the exact problem that broke keyword search last episode — "can't log in" finding the "authentication" ticket:

The query shares zero words with the winning ticket, yet it ranks first. That's the payoff of the whole chapter so far.

📌 Summary: Embed text → points in space. Nearby points mean similar things. Cosine similarity scores how aligned two points are. Semantic search = embed the query, rank everything by cosine, take the top-k. Meaning, not words.

Where Do the Vectors Live? The Vector Store

In the demo above we compared the query against a handful of tickets with a quick loop. That's fine for ten tickets. It falls apart at ten million.

A vector store (or vector database) is a specialised database built for exactly one job: store millions of embeddings and, given a query vector, return the nearest ones fast. Names you'll hear: Chroma, FAISS, Pinecone. They differ in features and cost, but all answer the same question — "which stored vectors are closest to this one?"

Here's the mental model of the whole system, split across the two clocks from last episode:

flowchart TD
    subgraph Offline["🌙 Ingestion"]
        A["Tickets"] --> B["Embedding model"]
        B --> C["Vectors"]
        C --> D[("Vector store<br/>Chroma · FAISS · Pinecone")]
    end
    subgraph Online["⚡ Query"]
        E["Reader query"] --> F["Embedding model"]
        F --> G["Query vector"]
        G --> H{"Nearest-neighbour<br/>search"}
        D -.-> H
        H --> I["Top-k tickets (as text)"]
    end

One detail worth pausing on: the store returns the original text, not the raw vectors. Vectors are how it searches; but a wall of 1536 numbers is useless to an LLM (or a human). So each vector is stored alongside the text it came from, and that text is what gets handed to Nova.

💡 Tip: You can't SELECT * WHERE text LIKE '%login%' a vector store — there are no words in there to match, only geometry. The only way to search it is by similarity: embed your query and ask for the nearest points. That constraint is the whole reason embeddings exist.

One Space to Hold Them All: Multilingual & Multimodal

Two facts about embedding space that feel like magic but fall straight out of "meaning becomes location."

Multilingual. A reader writes in Hindi; your archive is in English. Does search break? No — if the meaning is the same, a good multilingual embedding model maps both languages to the same neighbourhood. The Hindi query for "payment failed" lands next to the English "card declined" ticket. The vectors don't care what language made them; they encode meaning, and meaning is language-agnostic. (The LLM then answers back in the reader's language — that's just generation.)

Multimodal. The same idea stretches across formats. Text goes through a text-embedding model, images through an image-embedding model, audio through an audio-embedding model — different doors, but they all open into the same vector space. A photo of a golden retriever and the phrase "a happy dog" can land near each other. That's how image search by text prompt works.

flowchart LR
    T["English text"] --> ME["Embedding models"]
    H["Hindi text"] --> ME
    I["Image"] --> ME
    A["Audio"] --> ME
    ME --> V[("One shared<br/>vector space")]

The rule: the input decides which model you use; the meaning decides where it lands. Different doors, one room.

A Word on Cost

Embeddings aren't free, but they're cheap. text-embedding-3-small runs about $0.02 per one million tokens — pennies for a whole ticket archive.

And you pay in both directions: once offline to embed the entire archive (ingestion), and a tiny bit online every time you embed a reader's query to search. The query cost is negligible; the ingestion cost is a one-time-per-update batch. Budget for the archive, forget about the queries.

📌 Summary: A vector store keeps millions of embeddings and finds nearest neighbours fast. It returns text, not numbers. Meaning is language- and format-agnostic, so one space holds English, Hindi, images, and audio together. Embedding costs pennies, paid on ingest and on each query.

What Nova Learns Next

We've been embedding whole tickets as if each one were bite-sized. But real documents aren't — a runbook can be forty pages, an article thousands of words. Embed all of that as one giant vector and the meaning smears into mush; retrieval gets vague and the LLM drowns in irrelevant text.

Next episode: chunking — the art of slicing big documents into pieces that are small enough to be precise but whole enough to still make sense. Get it wrong and RAG falls apart no matter how good your embeddings are. Get it right and Nova retrieves exactly the paragraph she needs.