Split an LLM evaluation by source document and time

For an engineer evaluating an LLM application, split data according to the source of shared information, not merely by shuffling individual questions. Paraphrases from one document, neighboring turns from one conversation, and revisions of the same policy can reveal nearly the same answer. Placing them on both sides of a train-test boundary can make generalization look stronger than it is.

The worked example assumes basic Python and a manifest that records when each source became available. This matters even when you are not training model weights. Prompt examples, retrieval tuning, routing thresholds, and manual error analysis all adapt the system to development data. A final evaluation needs cases that did not guide those decisions.

See how a random split leaks a document

Consider a synthetic dataset with 300 documents and five questions generated from each document. That produces 1,500 questions. If each question independently enters the test set with probability 0.2, many documents will contribute questions to both development and test.

For one document, the probability that all five questions stay in development is 0.8^5. The probability that all five go to test is 0.2^5. Therefore, the probability of appearing in both partitions is 1 - 0.8^5 - 0.2^5 = 0.672. Across 300 such groups, the expected number spanning both partitions is 201.6.

The probability calculation can be reproduced with this small Python example:

from math import isclose

questions_per_document = 5
test_fraction = 0.2
mixed_probability = (1 - (1 - test_fraction) ** questions_per_document
                     - test_fraction ** questions_per_document)
expected_mixed_documents = 300 * mixed_probability
assert isclose(mixed_probability, 0.672)
assert isclose(expected_mixed_documents, 201.6)

These figures are an analytic calculation under the stated independent assignment rule, not an executed benchmark. They show why a familiar 80/20 row split can fail to separate the underlying information sources.

View full-size image ↗

The figure contrasts row-level shuffling with keeping a document’s questions together. Grouping reduces this form of leakage, though it does not automatically handle duplicate documents, shared templates, or related revisions.

Choose the grouping key carefully

A document identifier is a good start when every question derives from one document. If several files are revisions of the same policy, group by policy family or another stable lineage identifier. Otherwise, the development set may contain the old version of the answer while the test set contains a lightly edited new version.

For conversations, keep related turns and follow-up questions together. A later turn can reveal a clarification that makes an earlier question easy to answer. For coding tasks, issue descriptions, solution commits, and tests from the same change can share the answer even when they are stored as different records.

The BEIR paper evaluates retrieval across distinct datasets and domains. When designing an application evaluation, preserve similarly explicit boundaries around what is shared and what is held out. A score is interpretable only relative to that data construction.

Add time when freshness is part of the task

If the product must answer questions about new documents, a time-based split can be more realistic than a random group split. Use older material for development and later material for evaluation, with a clear cutoff and source timestamps. Keep revisions linked so a future update does not leak through an earlier duplicate.

A temporal split introduces its own confounders. Later documents may differ in topic, formatting, or difficulty. That may reflect the real deployment challenge, but report it. Do not attribute every score change to model quality when the evaluation distribution also changed.

Freeze the retrieval corpus used for each evaluation condition. If the test asks about a document that the production design would have indexed by that date, the evaluation should make it available in the appropriate condition. If the test is intended to measure unsupported-question handling, exclude it deliberately and label that purpose.

Combine a family boundary with a chronological cutoff

A family that spans the cutoff creates a choice: grouping alone can put future rows in development, while a strict date split can put the same family on both sides. The following small policy quarantines such families. It sacrifices their rows in this evaluation instead of pretending both requirements were met. That may be too wasteful for a small real dataset, but the tradeoff is visible.

The function accepts a nonempty list or tuple of manifest dictionaries with exactly id, family and available_on. Dates must be canonical ASCII ISO calendar dates. They describe source availability supplied by the dataset author, not dates inferred from filesystem modification times. Empty identifiers, duplicate row identifiers, malformed dates and one-shot iterators are rejected.

from datetime import date

def day(text):
    if not isinstance(text, str) or not text.isascii():
        raise ValueError("invalid date")
    try:
        parsed = date.fromisoformat(text)
    except ValueError:
        raise ValueError("invalid date") from None
    if parsed.isoformat() != text:
        raise ValueError("noncanonical date")
    return parsed

def split_families(rows, cutoff_text):
    if not isinstance(rows, (list, tuple)) or not rows:
        raise ValueError("manifest must be a nonempty list or tuple")
    cutoff = day(cutoff_text)
    groups, seen = {}, set()
    for row in rows:
        if not isinstance(row, dict) or set(row) != {"id", "family", "available_on"}:
            raise ValueError("invalid manifest row")
        if any(not isinstance(row[key], str) or not row[key]
               for key in ("id", "family")) or row["id"] in seen:
            raise ValueError("invalid or duplicate identity")
        seen.add(row["id"])
        groups.setdefault(row["family"], []).append((row["id"], day(row["available_on"])))
    result = {"development": [], "test": [], "quarantine": []}
    for family in sorted(groups):
        members = groups[family]
        dates = [when for _, when in members]
        partition = ("development" if max(dates) < cutoff else
                     "test" if min(dates) >= cutoff else "quarantine")
        result[partition].extend(case_id for case_id, _ in members)
    return {name: sorted(ids) for name, ids in result.items()}

manifest = [
    {"id": "refund-v1", "family": "refund", "available_on": "2026-05-12"},
    {"id": "refund-v2", "family": "refund", "available_on": "2026-07-04"},
    {"id": "access-q1", "family": "access", "available_on": "2026-05-10"},
    {"id": "access-q2", "family": "access", "available_on": "2026-05-10"},
    {"id": "export-q1", "family": "export", "available_on": "2026-07-01"},
    {"id": "export-q2", "family": "export", "available_on": "2026-07-01"},
    {"id": "export-q3", "family": "export", "available_on": "2026-07-01"}
]
print(split_families(manifest, "2026-07-01"))

Execution places both access cases in development, all three export cases in test and both refund revisions in quarantine. The cutoff is inclusive on the test side. Reordering the input does not change the sorted assignments. No row is lost: quarantine is a recorded outcome, not an unnoticed filter. The fixture also checks invalid identities and dates rather than silently dropping bad records.

View full-size image ↗

The GroupShuffleSplit documentation, retrieved September 21, 2026, clarifies that its size parameters refer to groups, not individual samples. It implements randomized grouped splitting, not this chronological quarantine rule. Group isolation and time realism are separate design choices, so a familiar class name is not evidence that both are enforced.

This manifest function cannot discover mislabeled families, copied paragraphs under different identifiers or incorrect availability timestamps. Inspect source lineage and duplicate evidence separately. Quarantining every family with a long history can also exclude important update questions and shift the test distribution. A separate, clearly labeled revision-following evaluation may be more suitable for that product behavior; it should not be described as unseen-family generalization.

For retrieval systems, held-out answer labels and accessible source documents play different roles. A July question may legitimately retrieve a July document if production would have indexed it by the request time. Keep the answer annotation out of examples and tuning, while providing the source under the documented retrieval condition. Removing every test source would instead measure missing-evidence handling. Record which condition you intended before interpreting the score.

Separate development roles

Use a development set for prompt examples and debugging. Use a validation set for choosing thresholds or comparing a small number of candidate designs. Keep a final test set untouched until the chosen system is ready for an estimate. These names matter less than recording how each set influenced decisions.

Once a test failure is inspected and used to revise the prompt, that case has become development information. Keep it as a regression case, but do not continue treating the same score as an untouched estimate. Collect or reserve new cases for later release decisions.

The HELM paper emphasizes transparent evaluation across scenarios and metrics. A transparent split manifest serves the same purpose at application scale: it tells readers which claims the evaluation can support and which forms of generalization remain untested.

Audit the boundaries with content evidence

Exact hashes can identify duplicate source files. Near-duplicate detection can flag templates or paraphrases that deserve manual review. Neither method proves semantic independence, but both can reveal obvious overlap before the evaluation becomes expensive.

Store group identifiers, source revision identifiers, timestamps, split assignments, and the reason for exclusions. Keep the assignment stable across repeated runs so a model comparison does not accidentally use different test cases. If a split changes, create a new dataset version and explain why.

For the synthetic document example, assign each policy family to one partition before generating or selecting its questions. Then inspect a sample of families near the boundary, especially those with copied sections or shared answer tables. The goal is to hold out the information relationship that the system could exploit, not merely a set of row numbers.

A clean split does not guarantee that a hosted model never encountered similar public material during training. That uncertainty should remain explicit when relevant. What the application team can establish is that its own examples, tuning, and evaluation construction did not knowingly pass the held-out answers back into the system.