Build a RAG pipeline with inspectable evidence
Trace a policy question through retrieval, context assembly, citations, and abstention to build a RAG pipeline whose evidence can be inspected.
For a developer comfortable with request handlers and document identifiers, retrieval-augmented generation, usually shortened to RAG, gives a language model external information while it prepares an answer. In an application, that usually means searching a document collection, selecting passages, and placing those passages in the model’s input. Each stage can fail independently, so a useful implementation preserves enough evidence to identify where an answer went wrong.
Consider a hypothetical employee question: “How many days do I have to submit a travel receipt?” The policy collection contains an old rule allowing 30 days and a current rule allowing 14 days. A fluent answer with the wrong number is a failure even if the retrieval engine found something related to receipts.
Keep document identity through ingestion
At ingestion, store a stable document identifier, revision identifier, publication or effective date when available, and access metadata. A chunk should retain the identity of the document it came from. If a passage contains “within 14 days” but loses the heading that says “travel receipts,” it may be hard to interpret correctly later.
For this example, imagine two records:
| Chunk | Revision | Text excerpt | Status |
|---|---|---|---|
| expense-r3-12 | Revision 3 | Submit travel receipts within 30 days | Superseded |
| expense-r4-09 | Revision 4 | Submit travel receipts within 14 days | Current |
These records are synthetic. The identifiers make an important property visible: the retrieval result is more than a string of text. The answer generator and reviewer need to know which revision supports the number.
Indexing can use lexical search, embeddings, or a combination. The original RAG paper combines a retriever with a generator for knowledge-intensive tasks. A product does not need to reproduce that paper’s training architecture to use external retrieval, but it should be precise about its own design when discussing results.
The figure shows a current-policy answer path. The authorization and revision filters belong before context assembly. Searching broadly and asking the model to disregard inaccessible documents would still expose those documents to the model and potentially to logs.
Retrieve candidates, then assemble evidence
A query might retrieve ten candidate passages. That list is not automatically the final prompt context. Remove duplicates, preserve useful neighboring material, and ensure that each selected passage includes the conditions needed to interpret it. A passage about “14 days” can be misleading if a nearby exception extends the deadline for a particular travel category.
For this policy question, the context could contain the current receipt rule and one exception paragraph. Give them stable labels such as P1 and P2. Ask the model to cite those labels beside the claims they support. The application should reject invented labels and map accepted labels back to actual source records.
This mapping matters because a plausible citation string is easy to generate. A link that looks official is not enough. The generator should choose from the evidence identifiers supplied by the application. Rendering a URL from that mapping also avoids trusting an arbitrary generated link.
The model input can state that the current rule is the controlling source and that superseded documents must not define the answer. Ideally, revision handling already removed the old rule from the candidate set. The instruction provides clarity; it should not be the only defense against stale records.
Make the answer auditable
A minimal answer might say, “The current travel policy requires submission within 14 days, unless the exception in the supplied policy applies [P1, P2].” Whether that wording is correct depends on what P2 actually says. If the exception applies only to international travel, the answer should preserve that condition.
Record the query, selected chunk identifiers, revision identifiers, prompt version, model identifier, and resulting answer under an appropriate privacy policy. Full document text may be unnecessary if the source snapshot is reproducible and access controlled. The goal is to reconstruct the evidence path without duplicating every sensitive document into an unrestricted log.
A reviewer can then ask separate questions. Did retrieval find the current rule? Did context assembly keep its conditions? Did the answer state only supported claims? Was the citation attached to the right claim? A single overall “good answer” label would conceal these distinctions.
Reproduce a lost exception
Make the exception concrete: revision 4 allows 21 days for international travel; other travel keeps the 14-day limit. Assign the exception chunk expense-r4-10. For a question specifically about international travel, annotate both current chunks as required. These are human-written expectations for this fixture, not labels inferred from a similarity score.
Suppose the retriever returns both chunks, but a context-length limit drops the exception. The answer’s citation to the 14-day rule can be a real, resolvable identifier while the answer is still incomplete. Checking only retrieved results would miss the loss. Preserve the context after the final trimming step, because an earlier assembly preview cannot prove what the generator actually received.
The following Python helper compares stage identifiers. Inputs are a nonempty set of required chunk IDs, a sequence of retrieved IDs, a mapping from final context labels to chunk IDs, and the answer’s citation labels. Assume the caller already validated these types and captured the actual request. Parent passages added during expansion must appear in the recorded retrieval/expansion inventory too.
def diagnose(required, retrieved, context, citations):
if not required:
raise ValueError("required evidence must be annotated")
found = set(retrieved)
supplied = set(context.values())
checks = {
"retrieval_miss": required - found,
"context_loss": (required & found) - supplied,
"unrecorded_context": supplied - found,
"unknown_citation": set(citations) - set(context),
}
return {name: sorted(ids) for name, ids in checks.items() if ids}
required = {"expense-r4-09", "expense-r4-10"}
retrieved = ["expense-r4-09", "expense-r4-10"]
context = {"P1": "expense-r4-09"}
print(diagnose(required, retrieved, context, ["P1"]))This run reports context_loss for expense-r4-10. Add P2 mapped to that chunk and the result becomes an empty dictionary. That means these identifier checks passed. It does not mean the answer mentioned international travel, stated 21 days, attached the correct citation, or refused an unauthorized approval request. Those judgments require the actual answer and source text.
The trace makes a small regression exercise reproducible. Remove the exception from both retrieval and context: expect retrieval_miss. Keep retrieval complete but omit its context entry: expect context_loss. Supply both passages but cite P9: expect unknown_citation. Finally, use valid labels with a deliberately wrong 30-day answer. The helper reports no identifier problem, so a separate claim audit must reject that answer. If a test treats the empty dictionary as factual approval, the test itself is incomplete.
Eugene Yan’s July 30, 2023 discussion of LLM system patterns treats retrieval as an external-knowledge component and evaluation as a separate system concern. That distinction informs this trace. The article also distinguishes ANN recall against exact neighbors from other quality and operating measures. Our policy fixture measures none of its cited systems’ performance; finding the nearest vectors is not equivalent to finding all evidence required for this question.
Design the no-answer path
Suppose the user asks, “Will finance approve my late receipt?” The documents may describe a deadline without granting the assistant authority to approve exceptions. The pipeline should explain the documented rule and identify the missing approval decision. It should not infer a manager’s decision from a nearby policy paragraph.
A retrieval score alone cannot settle this. High similarity can indicate that a passage concerns late receipts while offering no answer about approval. An answerability check needs to consider the requested claim and the available evidence. The assistant can ask for a relevant policy section or direct the user to the responsible reviewer without pretending to know an outcome.
The evaluation best-practices guide, retrieved on September 20, 2026, recommends task-specific evaluation sets. For this pipeline, include a current-rule question, a superseded-rule trap, an exception case, and an approval request that the documents cannot decide. Keep the expected evidence identifiers beside the answer requirements.
A small test set cannot establish quality across every policy or language. It can expose a specific broken stage. If the current passage never reaches the prompt, change retrieval or indexing. If it reaches the prompt and the answer still says 30 days, inspect the generation contract and conflicting context. Those are different repairs, and the stored evidence lets you choose between them.