How to Evaluate an LLM App: Evals from Scratch

You build an LLM feature. You try it a few times, the answers look good, so you ship it. Then you tweak the prompt to fix one bad case — and quietly break three others you never re-checked.

That’s the trap. With ordinary code you’d never ship without tests, but LLM outputs feel too fuzzy to test, so teams fall back on vibes. The discipline that replaces vibes is evals — and it’s far more approachable than it sounds.

I teach this to engineering teams, and it’s the step almost everyone skips until a confidently wrong answer reaches a real user. Let’s build the habit from nothing.

In one breath: An eval is a fixed set of real questions with known-good answers, plus a way to score what your app produces — run automatically on every change, so you can see improvement and catch regressions instead of guessing.

Why “it looks good” isn’t enough

Trying your app a handful of times tells you almost nothing. You test the cases you happen to think of, in a good mood, and call it working.

The real problem is change. Every prompt tweak, model swap, or retrieval adjustment silently shifts behaviour across every input — not just the one you were fixing. Without a fixed yardstick you can’t tell whether a change helped overall or just moved the failure somewhere you didn’t look.

You can’t improve what you don’t measure. An eval is that measurement: the difference between “this feels better” and “this scored 82% and the last version scored 74%.”

Pause & recall. You change one line of your prompt and the example you were testing now works. What don’t you know yet — and what would you need to know it? Answer before reading on.

Reveal the answer

You don’t know what that change did to every other input. To know, you’d need a fixed set of cases you re-score every time — an eval.

What an eval actually is, from first principles

Strip away the tooling and every eval is just two parts:

The thing being tested — your app, run on one test case (an input, and ideally the answer you’d accept as good).
The grader — a way to score that output. A number, a pass/fail, a rubric.

Run the grader across a whole set of cases, average the scores, and you have a single number that describes how your app behaves right now. Change something, re-run, compare. That’s the entire game.

Everything else — frameworks, dashboards, fancy metrics — is convenience on top of those two ideas. Build the two by hand once and you’ll never be mystified by an eval tool again.

Step 1 — Define “good” for your app

This is the step people rush, and it’s the one that matters most. Generic scores like “helpfulness” or “hallucination rating” rarely capture what your feature actually needs.

Get specific. For a support assistant, “good” might be: answers accurately and keeps a friendly tone and escalates when it can’t help. Three different things, and a single number hides which one failed.

Write down, in plain language, what a great answer looks like for your task — and what an unacceptable one looks like. That sentence is the seed of every metric you’ll pick later. Skip it and you’ll measure what’s easy instead of what matters.

Step 2 — Build a golden set

Your golden set is a fixed collection of test cases — inputs paired with the answer (or the properties of an answer) you’d accept as correct. It’s the yardstick everything else hangs on.

You don’t need thousands. Start with 25–50 cases, and make them earn their place:

Real usage. Pull actual questions people ask (or would ask). Invented cases test an imaginary product.
The hard and ambiguous ones, on purpose. Edge cases, tricky phrasings, questions near your policy boundaries.
Known failures. Every time the app embarrasses itself, that case joins the set. Your golden set should remember your mistakes so they can’t come back.

Then version it. Keep it in the repo like code, so today’s score is comparable to last month’s. A golden set that quietly changes is a ruler made of rubber.

Pause & recall. Why is a case that your app currently fails one of the most valuable things you can add to a golden set? One sentence.

Reveal the answer

Because it turns a one-off embarrassment into a permanent regression test — it can never silently come back.

Step 3 — Choose 3–5 metrics that matter

Resist the urge to measure everything. Cap it at five. A useful split is two or three system metrics plus one or two custom ones tied to your task:

Metric	Question it answers	When to use
Correctness	Does the answer match the known-good answer?	Almost always
Relevance	Does it actually address what was asked?	Open-ended answers
Groundedness / faithfulness	Is the answer supported by the source it was given?	RAG apps
Safety	Does it avoid harmful or off-limits output?	Anything user-facing
A custom metric	Your task’s specific “good” (tone, format, escalation…)	Every serious app

More than five metrics and no one reads them; you end up with a dashboard nobody trusts. Pick the few that map to your Step 1 definition of “good,” and let the rest go.

Step 4 — Pick a grader: code, human, or model

How do you actually turn an answer into a score? Three options, in rough order of when to reach for them:

Code. If correctness is checkable — exact match, a number, valid JSON, contains the right ID — just write the check. Deterministic, free, instant. Use it wherever you can.
A human. The gold standard for nuance (tone, helpfulness, “is this actually good?”). Slow and unscalable, so you use it sparingly — mostly to calibrate the third option.
An LLM as a judge. Hand a second model the answer and a rubric and let it score. Scalable and surprisingly good — but only if you treat it with suspicion.

That suspicion matters. LLM judges have real biases: they favour longer answers, can prefer whichever option comes first, and tend to be lenient. So calibrate before you trust: have a human score a sample, then check the judge agrees with the human. If it doesn’t, fix the rubric before you rely on it. An uncalibrated judge is just vibes wearing a lab coat.

Step 5 — Run it on every change

An eval you run once is a science project. An eval you run automatically is a safety net.

Wire it into your workflow so it runs on every change — the same instinct as tests in CI. Now a prompt tweak that lifts your score from 74% to 82% is a fact, and one that drops it is caught before it ships.

Then close the loop with production. When a real answer goes wrong — and it will — don’t just patch it. Add it to the golden set. Over time your eval grows into a precise map of exactly the ways your app can fail, drawn from reality rather than imagination.

Evals for RAG and agents, specifically

Generic correctness is table stakes. The two most common LLM apps each need their own checks:

RAG. The key question isn’t just “is the answer right” but “is it grounded” — supported by the retrieved passages, not invented. Score faithfulness (does every claim trace to the context?) and retrieval quality (did you fetch the right chunks in the first place?). A RAG app can be wrong two ways — bad retrieval or an ungrounded answer — and you want to know which. This is the exact discipline behind answering from your own words with sources.
Agents. Here the path matters as much as the answer. Did the agent pick the right tool? Did it stop when it had enough, instead of looping? Evaluate the trajectory (the sequence of decisions), not only the final output — a right answer reached by luck will fail next time.

A worked example: evaluating the Writing Bed

The Writing Bed answers questions from my blog posts. Here’s how I actually check it, not in theory:

The golden set is a couple dozen questions I know the posts answer, plus a handful I know they don’t — because the second group is the real test.
Metric one: groundedness. For an in-scope question, is every part of the answer supported by the retrieved posts? A confident answer citing a post that doesn’t say that is a failure, even if it sounds right.
Metric two: honest refusal. For the out-of-scope questions, does it decline and point you back to the writing — instead of improvising? A RAG tool’s willingness to say “I don’t know” is the whole product, so I measure it directly.

When I changed the chunking, the eval — not my gut — told me whether retrieval got sharper or just different. That’s the point: the number decides, so I don’t have to trust a good mood.

This is also the difference between choosing an approach and proving it works: picking RAG over fine-tuning was the design decision; the eval is what tells me the build actually delivers.

What I watch teams get wrong

The same handful of mistakes, over and over:

No golden set. “We test it before releases” means someone tries it a few times. That’s not an eval; it’s a vibe with a straight face.
Too many metrics. Fifteen scores nobody reads. Pick five, tops.
An uncalibrated LLM judge. Trusting a model’s scores without ever checking them against a human. Calibrate first.
Evaluating only before launch. The failures that matter arrive in production. If prod cases never flow back into your golden set, your eval slowly goes blind.
Chasing generic scores. Optimising “helpfulness” while the thing users actually needed — the right format, the honest refusal — goes unmeasured.

Where to start

Don’t build a framework. Build the smallest real thing: twenty questions with known answers, three metrics, one grader — today. Run it, write down the number, and now every change you make has a verdict instead of an opinion.

The tooling is the easy, swappable part. The judgement — knowing what “good” means for your app, and being honest about when it isn’t there yet — is the valuable part, and it’s the same whether you’re shipping a weekend tool or an assistant for thousands of people.

Check yourself.

What are the two parts of any eval, underneath all the tooling?

Why is a case your app currently fails worth adding to the golden set?

Before trusting an LLM-as-judge score, what must you do first — and why?

For a RAG app, name the two different ways an answer can be wrong.

If any answer won’t come, that’s your reread map — the sections above hold all four.

Want to see the systems these evals are for? Go poke the Writing Bed, Sprout and Scout in the Greenhouse — then come back and imagine the golden set you’d write for each. 🌱

How do you know your LLM app works?