Build a data inventory before a handbook assistant
Map source authority, permitted uses, audience and derived copies in a fictional handbook assistant, then design a concrete document-withdrawal exercise.
A useful first deliverable for a handbook assistant is a small inventory that explains what each source can support, who may use it and how changes will reach its copies. A folder of readable files does not answer those questions. Neither does a successful embedding job.
I would start with one reader question and trace its evidence before adding more documents. That makes data preparation a concrete engineering task: establish a source, preserve its identity, control its use and keep the answer current. The same habits help when learning an unfamiliar AI stack because they expose work that a model demonstration can hide.
Start with a question and five sources
Consider a fictional internal assistant answering, “Where should I request a laptop repair?” Its proposed audience is all staff. The source owner has approved the current handbook for this internal search use and the intended processing environment. This is a bounded learning example; it does not describe a real employer’s documents.
The candidate folder contains these records:
| Record | What it contains | Initial decision |
|---|---|---|
| H3 | Current staff handbook: use Helpdesk | Include after checking version and staff access |
| H2 | Retired handbook: contact Facilities | Exclude from current-policy answers |
| M1 | Restricted operations meeting notes | Exclude from the all-staff corpus |
| V1 | Public vendor troubleshooting page | Hold until the intended use and processing terms are resolved |
| S1 | Synthetic handbook written for the exercise | Use in the test fixture, visibly separate from real policy |
The table records decisions, not five equally trustworthy inputs. A newer file timestamp alone cannot establish that H3 replaced H2. Ask the source owner to identify the current authority and record that decision. A copied old handbook could have the newest modification time in the folder.
There is also a coverage boundary. H3 may identify the repair channel without specifying a repair deadline or reimbursement policy. Those questions need their own evidence. An inventory should make that gap visible before the assistant starts filling it with plausible prose.
Admission depends on purpose, authority and audience. Being readable by the importer is only one input to that decision.
Record decisions that another person can inspect
The Datasheets for Datasets paper proposes questions about motivation, composition, collection, preprocessing, intended uses, distribution and maintenance. Its questions include sensitive information, permitted uses, ownership, updates and obsolete versions. I find that structure useful for a small assistant even when a full dataset publication is unnecessary.
For H3, create an inventory entry with these fields:
- Identity: source ID, explicit revision, owner-approved authority and a checksum of the exact imported bytes.
- Purpose and permission: allowed use, intended processing destination, who approved it and where that decision is recorded.
- Audience: who may retrieve the document, where the access rule comes from and how changes reach the search system.
- Coverage: supported questions, unsupported questions and any sections omitted during import.
- Lifecycle: update contact, retention rule, withdrawal procedure and known derived copies.
A checksum answers a narrow question: whether two byte sequences match the recorded digest, subject to the chosen hash’s properties. It does not prove that the source is correct, licensed, current or approved. Keep those decisions as separate fields instead of using one “verified” checkbox.
Unknown permission is a useful finding. It means V1 stays outside this proposed corpus while the source owner or appropriate reviewer resolves the intended use. Public availability alone does not settle every reuse or processing question. Permission to support internal search also does not automatically establish permission to train a model or send content to a different provider. Record the specific use being considered instead of treating approval as unlimited.
The paper presents documentation as a process of reflection, not something a tool can settle automatically. A generated description of a folder may save typing. It cannot supply a missing owner’s decision.
Follow the document beyond the original file
Suppose H3 produces chunks C31 and C32, index records I31 and I32, and a cached answer A7 that cites C31. A separate exercise fixture S1 produces T1. Keep parent identifiers on these artifacts so that the relation can be followed in both directions.
The inventory can now answer a practical question: if H3 is withdrawn, which live artifacts depend on it? In this example, C31, C32, I31, I32 and A7 all need attention. T1 has no H3 parent and should remain available for the isolated exercise.
Dependencies need not form a simple tree. An answer may combine two documents. If A7 cites H3 and another still-approved source, withdrawing H3 still makes A7 stale for this purpose. Rebuild or remove the answer according to the product’s policy; do not preserve it merely because one parent remains valid.
The arrows describe provenance in a proposed system. They do not claim that deleting a parent automatically deletes its descendants.
Logs, exported evaluation fixtures and backups deserve explicit entries too. A log that stores retrieved text is another content copy. A vector index is a derived artifact; calling it an embedding does not justify assuming that it is anonymous or outside the withdrawal process.
This does not mean deleting every backup immediately. Retention and recovery policies may require a different treatment. Record the restriction, control access and make restoration reapply current withdrawals before restored data becomes available to readers. “Removed from live search” and “erased from every retained copy” are different claims. The inventory should show which one has actually been established.
Check access where retrieval happens
Document access belongs in the retrieval design. For this example, the service should use an authenticated caller identity and the applicable document permissions before supplying context to the model. Hiding a restricted citation after generation is too late if the restricted content has already reached the model or an unauthorized processing destination.
Azure AI Search’s document-level access overview illustrates a detail worth checking in any stack: permissions may be copied into the index. Its documented token-based approaches compare the caller’s claims with the permission metadata already stored there. The overview inspected in September 2026 marks those native approaches as preview and describes source-specific synchronization requirements. This is an example of a boundary to investigate, not a claim that every search product behaves identically.
A source permission change therefore needs an operational path. Ask when the indexed metadata changes, what happens during a delay and how a denied document is prevented from reappearing after restoration. An access check can be correctly implemented against stale metadata and still fail the product’s intended withdrawal deadline.
Cached answers need the same attention. An answer created for an authorized reader must not become an unrestricted shortcut for the next reader. Bind reuse to the required access conditions or avoid sharing that answer until the system can enforce them. This is a design requirement to implement and test, not a guarantee provided by the inventory itself.
Finish with a withdrawal exercise
Before expanding the corpus, rehearse three changes on the fictional fixture. Withdraw H3, change its allowed audience, and replace its repair-channel paragraph with an owner-approved H4 revision. For each change, write the expected behavior of current retrieval, existing cached answers and a restored older index.
For withdrawal, the five dependent live artifacts listed above must no longer supply the withdrawn content; the unrelated synthetic fixture should still work. For an audience change, test an affected caller and an unaffected caller. For H4, ask the same repair question and verify the new source identity as well as the answer text. Two versions might contain the same sentence, so a matching answer alone does not prove that the intended revision was imported.
These are proposed checks, not results from a deployed assistant. Their value is that each has an observable outcome and an owner. A small inventory costs time to maintain, and tracking every temporary object separately can become cumbersome. Start with durable identifiers and the copies that can reach users, processors or restored systems. Expand that record when a new pipeline creates another consequential copy.
The next useful learning step is then clear: implement one import and one withdrawal path for the authorized fixture, with evidence that its source, permissions and derived artifacts remain connected.