Is Nova Actually Good?

Part of the free Generative AI course on LogicWiz, module: Nova Goes to Production.

Episode 20: Is Nova Actually Good?

"Arjun shrank the chunk size, the demo felt snappier, and he shipped it on Friday. Monday morning a customer forwarded Nova's reply — it promised a 30-day refund policy LogicWizNews has never offered. Did his change help or hurt? Arjun honestly can't say."

The Trap of "Looks Fine"

Every time Arjun tweaks something — chunk size, k, the prompt, the model — he does the same thing: he tries three questions, the answers look fine, he ships. It's the software version of tasting one spoonful and declaring the whole pot seasoned.

The trouble is that "looks fine" isn't a number. It can't tell Friday's Nova apart from Monday's. It can't catch the confident hallucination hiding in the fourth question he never tried. You cannot improve what you cannot measure — so this episode turns "looks fine" into hard numbers, the kind that let you say "v2 is 12% better" instead of "feels snappier."

Start by noticing that Nova answers in two steps, like a librarian handling a request:

Fetch the right pages from the archive — retrieval.
Write an answer using only those pages — generation.

A wrong final answer can come from either step: she grabbed the wrong pages, or she had the right pages and still wrote nonsense. Judge only the final answer and you can't tell which half broke — so you measure each step separately.

flowchart TD
    E["RAG Evaluation"] --> R["① Retrieval metrics<br/>Did we fetch the right tickets?"]
    E --> G["② Generation metrics<br/>Is the answer good, given those tickets?"]

🧭 Keep the two layers apart and every failure has an address. A bad answer from good context is a generation bug — fix the prompt or model. A good model fed garbage context is a retrieval bug — fix chunking or k. Blend them together and the failure hides.

Part 1 — Grade the Search

Did Nova fetch the right tickets in the first place?

First, you need an answer key

Think about grading a 100-question test that a hundred learners just sat. You can't mark a single paper until you have an answer key — the sheet that says what the correct answer to each question is. Grading is then just matching: line up each learner's answer against the key and count the matches. No key, no score — you'd just be guessing whether an answer is right.

Grading retrieval works the exact same way. The answer key is called the ground truth: for a batch of real questions, a human writes down which tickets actually contain the answer. Then you check what Nova retrieved against that key — the tickets she got right are the matches. Build the key in three steps:

Collect real questions — pull them from production logs, so you grade Nova on how people actually ask, not how you imagine they do.
Mark the relevant tickets — for each question, tag the ticket ids that hold the answer. ("I can't log in" → TICK-014.)
Have a couple of people cross-check — deciding a ticket is "relevant" is a judgment call, not a hard fact, so anyone labelling alone quietly turns their own opinion into "the truth." Picture the question "I can't log in after resetting my password." One labeller marks only TICK-014 (password reset) as relevant. Another also counts TICK-022 (account locked after too many attempts), because it genuinely helps with that situation. If only the first person builds the key, Nova gets marked wrong every time she retrieves TICK-022 — even though it's a perfectly useful ticket. That's not Nova failing; that's one person's bias baked into the answer key. A second reviewer surfaces the disagreement so you can settle it instead of shipping it as gospel.

💡 If two reviewers keep disagreeing on what counts as "relevant," don't throw more people at it — your definition is fuzzy. Sharpen the written rule, then re-tag. And start small: 20–30 questions is plenty to catch the big problems.

Two ways the search can fail

Picture Nova casting a net into the ticket archive to haul back the ones that matter.

Start with the idea that trips everyone up: a single question usually has more than one relevant ticket. Ask "I can't log in after resetting my password" and the real answer is spread across, say, four tickets — the password-reset steps, the "account locked after too many tries" note, a clear-your-cookies tip, and the session-token explainer. Nova should catch all four. So "did retrieval work?" is never a plain yes/no — it's "how many of the relevant tickets did she catch, and how much junk did she drag in with them?" Two numbers, because there are two ways to fail:

She misses relevant tickets that were down there. → poor recall
She hauls up junk — tickets that have nothing to do with the question. → poor precision

Now let's put numbers on that login question. Those 4 tickets are the complete set of relevant ones — the answer key for this question. Nova's net brings back 5 tickets, of which 3 are actually relevant (and 2 are junk):

Recall — "Did I miss anything?" Of all the relevant tickets, how many did Nova catch? → caught relevant ÷ all relevant = 3 ÷ 4 = 0.75. She reeled in 3 of the 4 that existed, so one relevant ticket got away. (That denominator of 4 is the whole point of recall: several tickets were relevant, and recall asks how many of them she caught.)
Precision — "Is my catch clean?" Of everything Nova hauled up, how much is actually relevant? → caught relevant ÷ everything retrieved = 3 ÷ 5 = 0.60. Two of the five are junk.

Same three matches (3), but each metric divides by a different total: recall divides by what existed (4), precision by what she fetched (5).

The tug-of-war (and how you tune k)

Now the part that trips everyone up: precision and recall pull against each other. Widen the net — raise k, retrieve more — and recall climbs (you catch more relevant tickets) but precision drops (you also scoop up more junk). Tighten the net and it flips. You can't max both at once — so drag k through the visualizer and watch them fight:

Since chasing one wrecks the other, you fold them into one balanced score:

F1 = 2·P·R / (P+R) — a blend that only stays high when both precision and recall are high.

Why not just average them? Because an average is easy to cheat. Return every ticket in the archive and recall hits a perfect 1.0 — but precision is nearly 0. A plain average shrugs "0.5, not bad!"; F1 says ≈0 — useless. That's the whole point of F1: you can't game it by acing one metric and trashing the other.

💡 Tip: this is exactly how you tune k from Lesson 18 with data instead of a hunch. Sweep k = 2, 3, 5, compute F1 on your answer-key questions each time, and keep the k where F1 peaks — the top of the hump in the visualizer.

Part 2 — Grade the Answer

Nova fetched the right tickets. Is her answer actually any good?

Good retrieval doesn't guarantee a good answer — hand Nova the perfect ticket and she can still embellish it. Three metrics grade the response itself (each scored 0.0–1.0):

Groundedness (faithfulness) — is every claim in the answer backed by the retrieved context, or is Nova reaching into her own memory and inventing things? The single most important production metric. Target > 0.75. Ungrounded RAG is just a chatbot with extra steps.
Completeness — does the answer cover all the key points in the context, or stop half-way? Target > 0.75.
Answer relevance — does it address the question that was asked, or wander off-topic?

Groundedness is the one to burn into memory — here's what "check every claim" actually looks like:

Grading answers at scale: the LLM-as-Judge

Groundedness is exactly the check you just stepped through — list each claim, trace it back to the context, score. Doing that by hand across ten thousand answers is hopeless. So you automate the reviewer: hand the very same job to a second, stronger LLM.

"Hold on — an AI marking another AI's homework? Why would that be any more trustworthy?" It feels circular, but grading is far easier than answering. The judge never composes a fix from scratch; for each sentence it answers one narrow, checkable question — "is this claim supported by that context?" — and nothing more. A capable model does that reliably, the same way a fresh reader can catch a wrong citation in a paper they could never have written themselves.

You hand the judge four things — the question, the retrieved context, Nova's answer, and a scoring rubric — and it hands back a score plus a written rationale. Four rules turn that score from a coin-flip into something you can trust:

Rubric with anchors — don't just say "score it." Define what each number means: 1.0 = every claim supported, 0.5 = partially supported, 0.0 = contradicted or invented. Without anchors, one run's 0.8 doesn't mean the same thing as the next run's 0.8.
Chain-of-thought — make the judge list each claim and check it against the context first, and give the number last. A score with no reasoning is a guess; force the work before the verdict.
A stronger model than the student — the judge should be smarter than the model it's grading. Evaluation is a one-off offline pass, so paying more for a sharper judge is cheap insurance.
Temperature 0 — remember temperature is the model's randomness dial: high temperature makes it more creative and varied (great for brainstorming), low temperature makes it more focused and repeatable. At 0 it's as deterministic as it gets — feed it the same input and you get the same output. For a grader that's exactly what you want: the same answer must always earn the same score. You need a consistent ruler, not a creative one that scores 0.7 today and 0.9 tomorrow.

Run it and watch the judge discriminate. The faithful answer scores 1.0. Bolt one invented sentence onto the second answer — "you'll get a full refund automatically" — and its score drops below the 0.75 gate. The judge caught the exact hallucination the visual flagged, with no human in the loop. That is what makes evaluation scale from three eyeballed answers to ten thousand graded ones.

📌 Summary: Two layers, six metrics. Retrieval: precision / recall / F1 (grade with a ground-truth set). Generation: groundedness / completeness / answer relevance (grade at scale with an LLM-as-judge — rubric, anchors, chain-of-thought, stronger model, temp 0). Groundedness > 0.75 is the line you don't cross below.

Part 3 — Turn Numbers into a Decision

A scorecard you never act on is just trivia. The real payoff of all this measuring is that the numbers make the decision for you — ship it or don't — instead of leaving it to Arjun's gut. Let's watch the whole thing play out on the exact change that burned him on Friday.

Arjun wants to shrink the chunk size to make Nova feel snappier. Old Arjun would try three questions and ship. New Arjun runs an A/B test first: he sends every question in his answer-key set through both versions — today's Nova (the baseline) and the new one (the candidate) — and lines their scores up side by side. Step through what he sees:

Two ideas do all the work there:

An A/B test is: same questions, two configs, compared. He changes one thing (chunk size) and keeps the question set fixed — so any difference in the scores was caused by that change and nothing else.
The release gate is a bouncer, not a suggestion. One metric — groundedness — has a hard floor of 0.75. A candidate can be faster, cheaper, whatever; if groundedness drops below the line, it does not ship. That single rule is exactly what would have stopped Friday's phantom-refund answer at the door.

When the gate blocks something, you don't throw the change away — you run the fix loop:

flowchart LR
    A["📊 Analyze<br/>which metric failed?"] --> D["🔍 Diagnose<br/>why did it fail?"]
    D --> F["🔧 Fix<br/>change one thing"]
    F --> R["🔁 Re-evaluate<br/>same question set"]
    R --> A

That's precisely what turns v1 into v2 above. Arjun analyzes (groundedness tanked), diagnoses (chunks so small they chopped the answer in half, so Nova filled the gaps by guessing), fixes (bumps the chunk size back up and adds a little overlap), and re-evaluates on the same set. v2 keeps most of the speed and clears the gate — so that's the version customers get.

⚠️ Always A/B test on the same fixed question set. Change the questions and the config at the same time and you've learned nothing — you can't tell which one actually moved the score.

What Nova Learns Next

Nova is measured, tuned, and gated — no more shipping on vibes. But she's still a straight line: retrieve once, answer once. Some questions need more — "compare our refund policy to what we told this customer last week" is two lookups and a comparison. That needs Nova to decide what to do, step by step.

Next episode is the finale: agentic RAG — Nova reasoning in a loop, choosing her own tools, and deciding whether and what to retrieve before she answers.