The Best of Both Worlds: Hybrid Search & RRF

Part of the free Generative AI course on LogicWiz, module: Finding the Needle: Hybrid Search & Retrieval.

Episode 44: The Best of Both Worlds — Hybrid Search & RRF

Dense found the "LumaGlow" fan; sparse nailed the "1100 lumen" spec. "So which list do I trust?" Arjun asked. "Neither," said Anjali. "You fuse them."

Hybrid Search: Run Both, Then Merge

Quick refresher, in case it's been a few episodes — here's why one retriever was never enough:

Dense (semantic) search matches meaning, not exact text. It's brilliant at paraphrases — "bright lamp" finds "luminous light." But ask it for a precise spec like "1100 lumen" and it stumbles. To an embedding model, "1100 lumen," "1000 lumen," and "900 lumen" all mean roughly the same thing — "a lamp's brightness rating" — so their vectors land almost on top of each other. Dense search will happily hand back a 900-lumen lamp as a "close" match, because it never actually reads the number — it only feels the overall meaning.
Sparse (lexical) search matches exact words. It hunts for the literal token "1100," so it pulls the exact-spec product and leaves the 900-lumen one behind. But on its own it's blind to synonyms — search "bright" and it would miss "luminous" entirely.

See the trap? Dense nails "bright" ≈ "luminous" but fumbles "1100 lumen"; sparse nails "1100 lumen" but fumbles "bright" ≈ "luminous." Each is strongest exactly where the other is weakest — so the smart move is to stop choosing between them.

Hybrid search does exactly that: run the dense (semantic) retriever and the sparse (lexical) retriever in parallel, then blend their results into one ranked list. You stop trading precision for recall — you get both.

The setup is a dual pipeline. Every document is embedded twice and stored together:

a dense vector (semantic meaning), indexed with HNSW, and
a sparse vector (expanded keywords via SPLADE), indexed with an inverted index.

At query time, the query is also embedded both ways, each side searches its own index, and you get two ranked lists back.

The Problem: Two Lists, Two Score Scales

You can't just add the scores together. The two pipelines speak different mathematical languages:

Dense returns cosine similarity — typically 0 to 1 (or −1 to 1).
Sparse returns a dot-product score — an unbounded number that can run from 0 to the hundreds.

A sparse score of 98 next to a dense score of 0.82 is meaningless to compare directly — the sparse side would steamroll the dense side. You need a way to merge that ignores the raw magnitudes.

Reciprocal Rank Fusion: Fuse by Rank, Not Score

RRF — Reciprocal Rank Fusion — is the elegant fix. Its one big idea: throw away the raw scores entirely and use only each item's rank position (1st, 2nd, 3rd…) in each list.

Because ranks are just 1, 2, 3, 4… in every list, the scale problem vanishes automatically — it's a built-in normaliser. Each item gets a fused score by summing a small value for each list it appears in:

score(item) = Σ 1 / (k + rank) — summed over every list the item appears in

rank is the item's position in that list (1-indexed).
k is a smoothing constant, conventionally 60.

Then you sort every item by its fused score, descending, and that sorted list is your hybrid result.

Implement the real thing — it's about ten lines of pure Python, and it's exactly what production systems run:

Why RRF Just Works

Two properties fall out of that tiny formula:

It rewards consensus. An item that appears in both lists sums two reciprocals, so it naturally floats to the top — results confirmed by both keyword and meaning are exactly what you want first. An item in only one list contributes just once (its missing side counts as zero).
The constant k tames the top. Without k, rank 1 would score 1.0 while rank 2 scores 0.5 — a huge, top-heavy cliff. Adding k = 60 makes rank 1 score 1/61 and rank 2 score 1/62 — close together, so a single #1 can't dominate. It smooths the gradient into a fair, gradual decay. (61, 200, 300 all work; 60 is the balanced default.)

📌 RRF is modality-agnostic and treats both pipelines equally — no built-in bias toward dense or sparse. It's a rank-aggregation method, so it can fuse three or four retrievers just as easily as two.

When RRF Isn't Enough: Cross-Encoders

RRF handles the large majority of cases (roughly 60–80%). But when a query returns many documents that are extremely similar to each other, a purely mechanical rank fusion can't tell which is truly best — it doesn't understand the text.

That's when you add a cross-encoder re-ranker as a final stage. Unlike the retrievers (which embed query and document separately), a cross-encoder reads the query and a candidate together and uses attention to judge fine-grained relevance — powerful but expensive, so you only run it on the top handful RRF hands you.

What Nova Learns Next

Nova now has the complete retrieval brain: dual-pipeline hybrid search, fused with RRF, optionally sharpened by a cross-encoder. In Episode 45 we stop drawing diagrams and build it for real — Qdrant, FastEmbed, SPLADE, a real product dataset, and finally wrapping the whole thing as a tool an agent can call.