How to Prove a Fine-Tune Actually Reduced Hallucination — Not Just Memorized Your Test Set

You fine-tune a small model to stop making things up. You run your eval. The hallucination count drops from three to zero. You feel great for about four minutes — and then the doubt creeps in.

Did the model actually learn not to fabricate? Or did it just memorize the eleven test prompts you've been staring at all week? And — the question almost nobody asks — is your eval even measuring what you think it is?

A "hallucination went to zero" result is the single least trustworthy number in fine-tuning. It's cheap to produce and easy to fake by accident. This post is about the two checks that turn it into something you can actually believe, using a local run that went exactly this way — including the part where half the remaining "failures" turned out to be my own scoring code lying to me.

The setup, briefly

Real example. The task is a grounded report-writer: given a block of evidence and a block of pre-computed numbers, produce a fixed-structure Korean report where every number in the body must trace back to a source, and every citation must be one of the provided documents. Inventing a number, or citing a document that wasn't supplied, is the failure mode. All the data here is synthetic — a dummy vault built for exactly this kind of testing.

The whole thing ran locally on a Mac Mini M4 with 16GB of RAM, fully offline, using Apple's MLX and a LoRA adapter on top of a 4-bit Qwen2.5-7B-Instruct. The adapter itself is tiny: rank 8, 8 layers, learning rate 1e-4, 120 iterations, batch size 1. Training set: 30 examples, 11 held out for validation. No cloud GPU, no pod, no overnight rental bill. If you have an Apple Silicon machine, this is a weeknight, not a project. (If you want the actual install-to-inference commands, they're in the companion how-to: How to Fine-Tune a Local LLM on an Apple Silicon Mac.)

Here's the honest bias I brought in: I didn't expect fine-tuning to be the lever. On this task the report structure already saturates at 7B with prompting alone — a 3B model gets the format right about 69% of the time, a 7B nails it 100% with nothing but a good system prompt. My working assumption was that grounding (not inventing numbers) would come from retrieval and a verification layer, not from touching the weights. So when a LoRA run moved the grounding number, my first instinct wasn't celebration. It was suspicion. That instinct is the whole point.

Check 1: the before/after (necessary, nowhere near sufficient)

The scoring is mechanical, which is what you want. A verifier reads the generated report and checks three things: required sections present; every number in the body matches (within rounding) a number that was actually provided; every cited source appears in the supplied set. Any ungrounded number or out-of-set citation flags the report as "needs human review."

On the 11 held-out validation prompts:

Three-to-zero. Clean. And on its own, close to meaningless.

The problem is that the validation prompts cover the same report topics the model trained on — different phrasings, but the same underlying document types. A drop to zero here is consistent with "the model learned to ground its numbers," but it's equally consistent with "the model memorized what a correct report for these topics looks like." You cannot tell those two apart from this table. Anyone who stops here and ships is shipping a vibe, not a result.

Check 2: hold out whole topics, not just phrasings

The fix for the memorization question is to make the test set genuinely foreign. So I retrained on a split that excluded four report types entirely — active-ingredient quality, Phase 3 efficacy/safety, safety pharmacology, and GLP toxicology — and then evaluated on 14 prompts drawn only from those unseen types. If the adapter memorized templates, it should fall apart here. If it learned the actual behavior — "transcribe the numbers you're given, don't invent, don't cite what you weren't handed" — the gain should carry over to topics it never saw.

Eight down to one, on topics the model had never been trained to write. That's the check that earns the result. A behavior that transfers to unseen categories isn't a memorized template — it's a learned disposition. The base model fabricated on more than half the unfamiliar prompts; the adapter carried its grounding habit into territory it had never seen.

This is the single most useful habit I can pass on: your real test set is the one made of things your training data doesn't contain. Held-out phrasings of trained topics measure recall. Held-out topics measure whether anything generalized. Only the second one tells you if you built something.

Check 3: audit the scoreboard before you trust the score

That leaves the one remaining "failure": 1 out of 14. Before writing it down as a real residual hallucination rate, I did the thing that's easy to skip when the number is already good enough to be happy about — I opened up the one failing case to see what the model actually wrote.

It hadn't hallucinated.

Two harness artifacts were manufacturing the failure:

1. Decoding wasn't deterministic. The eval was calling the model without pinning temperature to 0, so the same prompt could produce slightly different text across runs. A grounding eval has no business being stochastic — you want the model's variance measured, not the sampler's. Pin it to 0 and the run becomes reproducible.

2. The scorer was parsing the wrong text. The verifier splits each prompt into its evidence block and its computed-numbers block, then matches the report's numbers and citations against them. But the prompt's instruction sentence — "using only the following evidence and computed results…" — contains the very same section markers the parser was splitting on. So it split on the first occurrence, grabbed the instruction line instead of the real evidence, and then "correctly" flagged perfectly grounded numbers as ungrounded, because it was comparing them against the wrong block.

The fix was one line of intent: split on the last occurrence of the marker, not the first.

before:  user_msg.split("[evidence]", 1)      → grabs the instruction sentence
after:   user_msg.rsplit("[evidence]", 1)[-1]  → grabs the real evidence block

A hallucination metric is only as honest as the parser underneath it. My scorer was confidently reporting a fabrication that never happened — which is, when you think about it, the exact failure I was fine-tuning the model to stop doing. The eval was hallucinating about the model.

What was actually true

Strip out the two artifacts and the picture is boringly good: the adapter grounded the numbers on the held-out phrasings and carried that behavior to entirely unseen report types, and the residual was a scoring bug, not a model failure. But I only get to say that sentence because of the checks — not because of the first table.

The recalibration is worth stating plainly, because it went against my prior: on a small local model, a lightweight LoRA did move grounding, in a task where I'd bet retrieval and verification were the only levers that mattered. That's genuinely useful to know — you can shift a 7B model's tendency to fabricate with an adapter you can train on a laptop over dinner. But the result was one debugging session away from being a lie I'd have believed, and the difference was entirely in how I checked it.

Three questions before you believe a fine-tuning win

Steal these. They cost an afternoon and they're the difference between a result and a rumor:

• Did it generalize, or memorize? Rebuild your test set out of topics or categories that appear nowhere in training. A gain that survives unseen categories is real; a gain that only shows on reworded training topics is recall.
• Is the eval deterministic? Pin temperature to 0 for grounding-style metrics. If your before/after moves when you rerun it, you're measuring the sampler, not the model.
• Is the scorer itself correct? Open your best and your worst cases by hand and read what the model actually produced. Verifiers have bugs, and a buggy verifier will invent failures — or hide them — with total confidence.

The model wasn't the thing most likely to fabricate in this whole setup. My evaluation harness was. Distrust the scoreboard before you distrust the player.

If you liked the grounding angle, the build-time version of this idea — refusing to ship the moment a citation can't be traced to its source — is in A Build-Pipeline Checklist for Catching AI-Fabricated Research Citations.

2026.07.07