How do you know your AI is right?

There's a question I ask every team that has shipped a retrieval-augmented AI feature: how do you know it's right?

The answers are revealing. "The demo worked." "It looks good." "We spot-check a few." "Users haven't complained." What almost nobody has is a number — a real, reproducible measurement of how often the system retrieves the right source, answers faithfully to that source, and admits when it doesn't know.

That gap is not a small thing. It's the difference between a system you can stand behind and a system that happens to have worked the last time you looked at it. Most AI systems fail in production for the same reason: they were designed by people who never felt the consequences of a wrong answer. Evaluation is where you build that consequence back into the loop — before your users do it for you.

This is how I built an evaluation harness for a production RAG system, and the handful of ideas in it that matter far more than the framework you pick.

"Trustworthy" is not one thing

The first move is to stop treating "is it good?" as a single question. A retrieval system that produces an answer can fail at several independent stages, and a real eval measures each one separately. Industry-agnostic, these are the dimensions that matter:

• Retrieval accuracy — did the system surface the right source material at all? If the correct passage never made it into the context, nothing downstream can save the answer. This is the foundation, and it's measurable: for a question whose correct source you already know, is that source in the top-k retrieved chunks?
• Faithfulness — does the generated answer actually follow from the retrieved text, or did the model embroider? This is the hallucination axis. An answer can be fluent, confident, and completely unsupported by the source it claims to cite.
• Answer relevancy — is the answer actually responsive to the question asked, or is it a correct-but-adjacent fact? Retrieval and faithfulness can both pass while the model quietly answers a slightly different question.
• Abstention — when the source material genuinely doesn't address the question, does the system say so, or does it manufacture a plausible answer? I'll come back to this one, because it's the metric almost everyone skips and the one where trust actually breaks.
• Domain-correctness — the hard, system-specific part. In my case the sources form an authority hierarchy: some documents legally override others, and later amendments supersede earlier text. A "correct" answer isn't just supported by a source — it has to be grounded in the controlling one. Your domain has its own version of this: the rule that a generic RAG benchmark will never test for you.
• Calibration — does the system's own confidence signal track reality? If it labels an answer "high confidence," is that answer actually right more often than a "low confidence" one? A confidence score that doesn't correlate with correctness is worse than none, because it launders uncertainty into false assurance.

Six axes, six numbers. The instant you decompose it this way, "how do you know it's right?" stops being a vibe and becomes an instrument.

The architecture: a gold set, honest oracles, and a judge

The harness itself is not exotic. Its credibility comes from discipline, not cleverness.

A gold set of questions with known answers. The core asset is a curated set of questions where I already know the correct answer and the correct source citation. This includes ordinary questions, deliberately hard ones (conflicts between sources, superseding amendments), and — critically — questions the source material does not answer. You build this by hand, with domain expertise, and it is the single most valuable artifact in the whole system. More on why in a moment.

Human-verified oracles. A subset of the gold set is verified by a human expert, not just generated. When a metric is computed against the human-verified slice, I can make a stronger claim than "the model agrees with itself."

An LLM-as-judge for the fuzzy metrics. Faithfulness and relevancy don't reduce to string matching — you need semantic judgment about whether a claim is supported by a passage. A smaller, cheaper model runs as an automated judge for these. This is standard practice now. It's also where most eval harnesses quietly go wrong, which brings me to the first idea that actually matters.

Standard vocabulary on top. I map the harness's metrics onto the vocabulary the broader field already uses — faithfulness, answer relevancy, context precision and recall, noise sensitivity. The internal names don't matter; being legible to an outside evaluator does. When someone technical asks "how do you measure this?", the answer should land in terms they already trust, not a private dialect.

The shape of it

The pipeline is simple; the discipline is in what each stage emits and in one component being held to a higher standard than the rest:

A single evaluation item is the atom of the whole system — a question, the source you already know is correct, and the behavior you expect:

Every item flows through the same scoring pass, and each stage emits an independent, named result — no single blended "score" hiding a failure:

The judge on step 3 is the part under the microscope. Before I trust judge.isSupported, I run it against a public set of human-labeled hallucinations and record its precision and recall. In the harness, the evaluator is a component with a spec — not an oracle above suspicion.

Idea 1: Test your tester

Here is the trap. You build an LLM-as-judge to detect hallucinations. It gives you a number: "94% faithful." You put it in a slide.

But the judge is also a model. It can be wrong. If your hallucination detector has poor precision, it waves through fabrications; if it has poor recall, it flags good answers as hallucinations and you chase ghosts. A number produced by an unvalidated judge is not evidence — it's a second opinion from a witness whose eyesight you never checked.

So I calibrate the judge itself. There are public datasets with human-labeled hallucination annotations. You run your judge against that labeled set and measure its own precision and recall. Now you can say something honest: "our faithfulness judge agrees with human labels X% of the time, and here's its error profile." Only then does the 94% mean anything.

This is the move that separates real evaluation from theater. Everyone measures the system. Almost nobody measures the instrument they're measuring it with. If you take one idea from this piece, take that one.

Idea 2: Abstention is a first-class metric

Most evals measure the quality of answers the system gives. They almost never measure what happens when the system shouldn't answer at all.

This is backwards. In any high-stakes domain, the most dangerous failure isn't a wrong answer to a hard question — it's a confident, fabricated answer to a question the source material never addressed. The user has no way to know the ground fell out from under them, because the output looks exactly like a grounded answer.

So abstention gets its own metrics. For the slice of gold-set questions where the correct behavior is "the documents don't address this," I measure: does the system correctly abstain (a silence pass rate), and when it fails, how often does it fabricate versus hedge (a fabrication rate)? A system that answers everything is not more capable — it's less trustworthy, and the eval should punish it for exactly the behavior that feels impressive in a demo.

Knowing what it doesn't know is not a soft quality. It's a number, and you can move it.

Idea 3: There's no benchmark for your hard part — that's the moat

When I went looking for an off-the-shelf benchmark for the domain-correctness axis — the authority-hierarchy resolution that is the actual point of my system — there wasn't one. There are excellent general RAG benchmarks and several legal-document datasets, but none of them tests "when these two sources conflict, did you pick the one that legally controls?"

The naive reaction is disappointment. The correct reaction is recognition: the absence of a benchmark for your hardest problem is a signal that your hardest problem is your differentiation. If a public leaderboard already measured it, it wouldn't be a moat.

So you build the gold set yourself. It's slow, it requires real domain expertise, and it is precisely the work that can't be commoditized. The synthetic hierarchy-and-conflict corpus I built is now a durable asset — the only yardstick that measures the thing that matters most, owned end to end. General benchmarks tell you if your plumbing works. Only a domain gold set tells you if your system is right about the thing you built it to be right about.

A related refinement: I moved citation scoring from section-level to span-level. "The answer cited the right section" is a weak claim if the section is long — the model can point at the right chapter and still miss the controlling clause. Span-level precision and recall on the exact cited characters catches "right neighborhood, wrong sentence," which is a very common and very quiet failure mode.

Operating it: numbers that change what you do

A harness that produces numbers you look at once is a report. A harness that produces numbers you act on is infrastructure.

• The confidence policy is tuned against calibration data, not intuition. The thresholds that decide when the system shows a "strong" versus "limited" signal — or abstains, or recommends human review — are set where the calibration curve says they should be.
• Every run is versioned — model versions, test-set version, date. An accuracy number without those three is not reproducible, and a number that isn't reproducible isn't a measurement.
• Regressions are visible before release, not after a user finds them. When retrieval weights or a prompt change, the harness re-runs and the deltas are the review.
• The results are legible to outsiders. Because the metrics map to standard vocabulary and the scoring method is open, someone technical can reproduce the claim rather than take it on faith.

That last point is the whole philosophy compressed: explainability and measurement have to travel with the answer. A confidence number the user can't interrogate is decoration. A published accuracy figure with no reproducible method behind it is marketing. The harness exists so that the trust the system asks for is trust it can actually back.

The takeaway

Building the model is the part everyone talks about. Knowing whether to trust it is the part that ships or sinks a product — and it's mostly unglamorous instrument-building: a gold set assembled by hand, a judge you validated against human labels, an abstention metric most people skip, and a domain benchmark that didn't exist until you built it.

None of it requires a novel algorithm. It requires deciding that "the demo worked" is not an acceptable answer to "how do you know it's right?" — and then doing the work to have a better one.

Eric Tetzlaff is an AI systems architect focused on retrieval, evaluation, and making machine uncertainty legible to the people who bear the consequences of it. He writes about building AI systems that hold up under real-world pressure.