Skip to content
MINH VO A working notebook
by an engineer in Vietnam
Foundation8 min read

Choose a baseline that makes an LLM result interpretable

Choose a simple task-specific baseline, freeze the evaluation set, and interpret an LLM result against correctness, abstention, and operating cost.

Open notebook with connected abstract shapes and olive thread linking paper swatches

For an engineer deciding whether to ship an LLM feature, an evaluation needs a baseline that represents a credible alternative for the same task. Without one, a success rate says little about whether the model adds useful capability. This lesson assumes you can read a short Python function and count outcomes in a table. For a small support-routing problem, a keyword rule or existing classifier can be more informative than a score from an unrelated general benchmark.

Suppose a hypothetical product routes messages to billing, account access, or technical support. Messages can also need clarification. The goal is to send a user to the right queue while avoiding confident misrouting when the request is ambiguous. That is a narrower and more useful objective than asking which system is “smarter.”

Define the labels before the systems

Write label instructions that a reviewer can apply without seeing the model output. A billing message concerns charges, invoices, or payment adjustments. Account access concerns signing in or recovering access. Technical support concerns using the product after access is available. A message with too little information receives a clarification label.

Boundary cases matter. “I cannot log in to update my card” mentions both access and billing. Decide whether the immediate blocker controls the label or whether the system may assign multiple queues. That policy belongs in the task definition. If two reviewers apply different policies, the evaluation will measure their disagreement as if it were a model defect.

Create a small development set for writing rules and prompts. Keep a separate test set untouched until the comparison is ready. Include near misses such as “I was charged with organizing the demo,” where the word “charged” is not a billing request. This is a synthetic language example, not a support record from a real customer.

Build an inspectable baseline

A first baseline might use a few explicit rules and abstain when several categories match. Its purpose is to represent a plausible low-cost option, not to perform badly so the LLM looks good. Tune the rule set using development data only.

rules = {
    "billing": {"invoice", "refund", "payment"},
    "access": {"login", "password", "signin"},
    "technical": {"crash", "export", "upload"},
}

def route(tokens):
    if not isinstance(tokens, (list, tuple)):
        raise TypeError("tokens must be a reusable list or tuple")
    if any(not isinstance(token, str) for token in tokens):
        raise TypeError("tokens must contain strings")
    vocabulary = set(tokens)
    matches = [name for name, words in rules.items() if words & vocabulary]
    return matches[0] if len(matches) == 1 else "clarify"

The accepted input is a reusable list or tuple; an empty sequence asks for clarification. A one-shot iterator is rejected rather than consumed once for each category. This illustrative baseline assumes tokens are already normalized. It does not handle phrases, negation, or multilingual text well. Those are limitations to measure, not reasons to omit it. If the existing product has a stronger classifier, include that system as the primary operational baseline.

A frozen support-message set feeds rule, classifier, and LLM alternatives evaluated with the same routing and cost criteriaView full-size image ↗

Each system receives the same message and is judged against the same label policy. The LLM may receive a description of the taxonomy, but it should not see the held-out answer. Freeze the prompt and rule version before running the final comparison.

Report more than aggregate accuracy

Imagine a test set with 80 billing messages and only ten messages in each of two other categories. A system that always predicts billing has 80 percent accuracy while failing every access and technical case. Class-level recall and a confusion matrix reveal that failure immediately.

Include the clarification behavior separately. A system can reduce wrong routing by asking for clarification more often, but excessive clarification burdens users. Report coverage, the fraction of messages routed directly, alongside the error rate among those routed. The acceptable balance depends on the cost of a wrong queue and the user’s tolerance for another question.

The HELM paper argues for evaluating language models across multiple scenarios and metrics rather than a single narrow score. For this application, the relevant metrics might include correct routing, harmful misrouting, clarification frequency, latency, and cost. You do not need every metric in a broad benchmark; choose those that change the product decision.

Work through a coverage tradeoff with fixed outputs

Use twelve manually authored cases in this order. The gold label follows the immediate-blocker policy above; “Invoice?” is deliberately insufficient context. Token lists are supplied as part of this fixture, so this is not a tokenizer comparison. Candidate outputs are invented for arithmetic, not captured from an LLM.

cases = [
    ("b1", "Send my invoice", ["invoice"], "billing"),
    ("a1", "Reset my password", ["password"], "access"),
    ("t1", "Export fails", ["export"], "technical"),
    ("c1", "Help", ["help"], "clarify"),
    ("b2", "I was charged twice", ["charged", "twice"], "billing"),
    ("a2", "I cannot get in", ["cannot", "get", "in"], "access"),
    ("t2", "The invoice file will not upload", ["invoice", "upload"], "technical"),
    ("c2", "Payment or password?", ["payment", "password"], "clarify"),
    ("b3", "Where is my refund?", ["refund"], "billing"),
    ("a3", "Cannot login to view an invoice", ["login", "invoice"], "access"),
    ("t3", "Export causes a crash", ["export", "crash"], "technical"),
    ("c3", "Invoice?", ["invoice"], "clarify")
]
gold = [row[3] for row in cases]
baseline = [route(row[2]) for row in cases]
candidate = ["billing", "access", "technical", "clarify", "billing", "access",
             "billing", "access", "billing", "access", "technical", "billing"]

This summary function requires aligned nonempty sequences of labels from the four-value taxonomy. It returns exact fractions so the denominator remains visible. Its undefined conditional error rate is None when nothing is routed; zero would incorrectly suggest evidence of error-free routing. Equal lengths do not prove case identity: assemble columns by stable identifier before calling it. The fixed ordered list above supplies that alignment.

from fractions import Fraction

LABELS = {"billing", "access", "technical", "clarify"}

def summarize(expected, predicted):
    for sequence in (expected, predicted):
        if not isinstance(sequence, (list, tuple)) or not sequence:
            raise ValueError("labels must be a nonempty list or tuple")
        if any(not isinstance(label, str) or label not in LABELS
               for label in sequence):
            raise ValueError("unknown label")
    if len(expected) != len(predicted):
        raise ValueError("unaligned labels")
    paired = list(zip(expected, predicted))
    routed = [(want, got) for want, got in paired if got != "clarify"]
    wrong = sum(want != got for want, got in routed)
    return {
        "agreement": Fraction(sum(want == got for want, got in paired), len(paired)),
        "coverage": Fraction(len(routed), len(paired)),
        "misroute_given_routed": Fraction(wrong, len(routed)) if routed else None
    }

print(summarize(gold, baseline))
print(summarize(gold, candidate))

The executed baseline agrees on 7/12 cases, routes 6/12 and misroutes 1/6 of those routed. The synthetic candidate agrees on 9/12, routes 11/12 and misroutes 3/11 of those routed. Its larger agreement score accompanies more wrong direct routes and less clarification. These fractions describe only the constructed list; they are not estimates of production frequency or statistically established model gains.

The baseline directly routes six cases with one error, while the invented candidate routes eleven with three errors; more agreement does not eliminate the coverage tradeoffView full-size image ↗

Both systems agree with gold on six cases. Only the baseline is correct on c2; only the candidate is correct on b2, a2 and a3. Both miss t2 and c3. Inspect these rows before assigning a monetary penalty or release threshold. The function gives every label mismatch equal weight, while a product may treat a wrong access route differently from an unnecessary clarification. It has no measured latency, token bill or user-frustration score to silently fold into an overall rating.

Anthropic’s Demystifying evals for AI agents, published January 9, 2026, distinguishes tasks, grading logic and recorded outcomes, and discusses combining graders for different aspects of success. Here the code grades supplied labels, while humans must justify the task policy and case construction. Executing the arithmetic does not establish that the candidate outputs are real observations.

Compare paired cases

Store both systems’ outputs for each test message. A paired view shows which cases the LLM solves that the rule misses and which cases regress. Inspect those groups before accepting an average gain. If the LLM handles paraphrases better but misroutes account-recovery requests, the aggregate result may hide an unacceptable tradeoff.

Keep actual model identifiers, generation settings, prompt text, and evaluation date. If the provider exposes a versioned identifier, record it. If a hosted alias may change, state that limitation. The evaluation best-practices guide, retrieved on September 20, 2026, recommends representative task data and evaluation throughout development.

Do not tune the prompt repeatedly against the final test set. Once a case influences a revision, it belongs to development history. Maintain an untouched set for the next honest estimate, or collect new independently labeled cases. Otherwise, the apparent improvement can reflect learning the evaluation examples.

Make a decision the evidence supports

For a low-volume internal queue, the simple rule may be adequate and easier to maintain. For varied international messages, the LLM may justify its cost if it handles paraphrases and languages the baseline misses. Those are hypotheses until the matched evaluation produces results.

A reasonable release condition can require fewer serious misroutes without an unacceptable increase in clarification or latency. Choose that condition before examining the final numbers. The result then answers an engineering question: whether this LLM system improves this routing task under the stated workload. It does not establish a universal ranking of models, and it should not be presented as one.

Sources & further reading

  1. OpenAI evaluation best practices
  2. Holistic Evaluation of Language Models
  3. Anthropic: Demystifying evals for AI agents, 9 January 2026
← Back to the journal
All notes

Illustration

100%