LLM Evaluation: How to Test AI Features Before Production

LLM evaluation is the practice of measuring whether a language model feature produces good enough outputs, consistently, across the inputs real users will send. Because model outputs vary and there is rarely one correct string, ordinary assertions do not work on their own. You need a dataset of representative cases, a scoring method for each quality you care about, and a process that runs those scores on every change. This guide shows how to set that up.

Why testing LLM features is different

Traditional tests check that f(x) == y. LLM features break that model in three ways:

• Nondeterminism: the same prompt can produce different outputs across runs, especially at higher temperature.
• Many valid answers: "Your order ships Tuesday" and "It will ship on Tuesday" are both correct.
• Hidden coupling: a small prompt tweak, a new model version, or a changed retrieval index can silently improve one behavior and break another.

So instead of pass or fail on a single case, LLM evaluation measures pass rates and scores across many cases, and watches how they move. You still want regular unit and integration tests around the deterministic code; the testing pyramid still applies. Evals sit on top of it.

Step 1: Build a golden dataset

Your eval set is the most valuable asset in the whole process. Start small and grow it:

1. Collect real inputs. Pull anonymized queries from logs, support tickets, or pilot users. Invented examples tend to be too clean.
2. Cover the distribution. Include common requests, edge cases, ambiguous questions, out-of-scope requests, and adversarial inputs.
3. Write expectations, not exact answers. For each case, record what a good response must contain, must not contain, or must do, such as calling a specific tool.
4. Add every production failure. When a user reports a bad answer, turn it into a test case so it cannot silently return.

Even a few dozen carefully chosen cases catch many regressions. You can grow to hundreds as the feature matures.

Step 2: Choose LLM evaluation metrics

Different qualities need different scorers. Use the cheapest reliable method for each.

For a RAG feature, useful dimensions include retrieval quality (did the right documents come back?), groundedness (is every claim supported by those documents?), and answer relevance (does it address the question?). For agents, add task success and tool-call correctness.

Step 3: Use LLM-as-judge carefully

Using a model to grade another model scales well, but a judge is only useful if it agrees with humans. Make it reliable:

• Give the judge a specific rubric with concrete criteria, not "rate quality 1 to 10".
• Prefer binary or small-scale judgments ("Is every claim supported by the context? yes or no") over fine-grained scores.
• Ask for a short justification before the verdict, which makes results easier to audit.
• Calibrate: have humans label a sample, compare with the judge, and adjust the rubric until agreement is acceptable.
• Watch for known biases such as favoring longer answers or the first option in pairwise comparisons, and randomize order when comparing two outputs.

Step 4: Automate an eval harness

A harness runs every case, applies scorers, and reports aggregate results. Here is a compact Python version you can adapt. The generate and judge functions wrap your application and your grading model.

import json
from dataclasses import dataclass, field


@dataclass
class Case:
    id: str
    input: str
    must_include: list[str] = field(default_factory=list)
    must_not_include: list[str] = field(default_factory=list)
    rubric: str | None = None


def score_case(case: Case, output: str, judge) -> dict:
    text = output.lower()
    checks = {
        "includes": all(s.lower() in text for s in case.must_include),
        "excludes": not any(s.lower() in text for s in case.must_not_include),
    }
    if case.rubric:
        checks["judge"] = judge(case.input, output, case.rubric)  # returns True or False
    return {"id": case.id, "passed": all(checks.values()), "checks": checks}


def run_suite(cases: list[Case], generate, judge, runs: int = 3) -> float:
    results = []
    for case in cases:
        for _ in range(runs):  # repeat to expose nondeterminism
            results.append(score_case(case, generate(case.input), judge))
    pass_rate = sum(r["passed"] for r in results) / len(results)
    failures = [r for r in results if not r["passed"]]
    print(json.dumps({"pass_rate": round(pass_rate, 3), "failures": failures[:10]}, indent=2))
    return pass_rate


if __name__ == "__main__":
    cases = [Case("refund-window", "How long do I have to return a laptop?", must_include=["days"])]
    rate = run_suite(cases, generate=lambda q: "You have 15 days to return it.", judge=lambda *a: True)
    raise SystemExit(0 if rate >= 0.9 else 1)

The exit code lets CI block a merge when the pass rate drops below your threshold.

Step 5: Gate releases and monitor production

Offline evals tell you whether a change is safe to ship; online monitoring tells you whether it stayed safe.

• Run evals in CI on every change to prompts, models, retrieval settings, or tool definitions.
• Compare against a baseline, not an absolute bar. A drop in any category is worth investigating even if the total looks fine.
• Canary new versions to a small slice of traffic and compare online signals before a full rollout.
• Sample production traffic for periodic judge and human review, since real inputs drift over time.
• Trace every request so you can reproduce failures. Good observability is what turns a vague complaint into a new test case.

Hallucination is often the first thing teams want to measure; How to Reduce Hallucinations in LLM Applications covers groundedness checks in more depth.

Common LLM evaluation mistakes

• Testing only happy-path examples written by the team that built the feature.
• Relying on one aggregate score that hides regressions in important categories.
• Trusting an uncalibrated judge model.
• Running each case once and mistaking luck for improvement.
• Leaving safety cases, like jailbreaks and prompt injection, out of the suite.

Key takeaways

• LLM evaluation measures pass rates and scores over a representative dataset, not single exact matches.
• The golden dataset is the core asset, so seed it from real inputs and add every production failure.
• Use deterministic checks wherever possible and LLM-as-judge for open-ended qualities.
• Calibrate judges against human labels before trusting them.
• Run evals in CI, compare with a baseline, and keep monitoring after release.

Frequently asked questions

How many test cases do I need for LLM evaluation?

Start with a few dozen high-quality cases that cover your main use cases and known edge cases. Grow the set as you find failures in production. Coverage of important categories matters more than raw count.

Is LLM-as-judge reliable?

It can be, when the rubric is specific, the judgments are simple, and you have checked agreement against human labels. Uncalibrated judges can be biased toward longer or more confident answers, so audit a sample of their verdicts regularly.

What is the difference between offline and online evaluation?

Offline evaluation runs a fixed dataset before release to catch regressions. Online evaluation measures real user traffic after release through feedback, task completion, and sampled reviews. You need both, because real inputs differ from any test set.

Should I rerun evals when the model provider updates a model?

Yes. Treat a model version change like a dependency upgrade and run the full suite before switching. Pin model versions in production where your provider allows it, so changes happen on your schedule.