Build knowledge agents with traceable sources, version details, access controls, and clear responses when information is missing or conflicts.
Most teams can get a knowledge agent to produce a fluent answer in an afternoon. Point a model at a folder of documents, add a vector index, and the demo looks convincing. The trouble starts with the first real question where the answer matters, when someone asks, “Where does it say that?”
At work, an answer is only useful if a person can check it. A sales rep quoting a unit’s ceiling height, a support lead explaining a refund policy, an engineer reading a runbook at 2 a.m.: each needs to know which document the answer came from, which version, and whether anything contradicts it. An answer that cannot be checked has to be verified by hand, which reduces the value of the answer.
So the bar for a grounded answer is specific. It cites the passage and the document version. It says so when the sources don’t contain the answer. It flags when two sources disagree. Everything else in the system, from ingestion to the interface, exists to make those three behaviors reliable.
An original diagram of a grounded answer checked against its sources.
Ingestion decides what an answer can cite
Keep structure and metadata
A citation can only be as precise as what ingestion preserved. If a PDF is flattened into a stream of text, the system loses the section heading, the page number, the table a number came from, and the date the document took effect. The model can still answer, but it can no longer say where the answer lives.
Treat every chunk as a record, not a string. The useful fields are the document ID and title, the section path, the page, the version or revision, the effective date, the owner, and the permissions that apply. For a project’s architectural plans, that might mean the sheet number and revision letter. For a policy, the effective date and the team that owns it. These fields are what let the agent say “Sheet A-201, revision C, page 4” instead of “according to the documents.”
Tables, drawings, and scanned pages deserve their own handling. A unit schedule in a plan set is a table. Split it by line and it loses the header that says which column is square footage. Extract tables as tables, keep the header with each row, and store the page image so a person can see the original.
Chunk by meaning, not by size
Fixed-size chunks are simple to implement, but they can lose useful structure. They cut a clause in half, separate a heading from its content, and put two unrelated sections in the same chunk. Retrieval then returns a passage that is half right.
Chunk along the document’s own structure: sections, clauses, list items, table rows with their headers. Keep chunks small enough to be specific and large enough to stand alone. When a passage depends on its parent heading, include the heading path in the chunk text so both retrieval and the reader see the context. Store links to neighboring chunks so the system can expand a hit to the surrounding paragraph when it needs to.
Retrieval has to find the right passage
Combine keyword and semantic search
Embeddings are good at paraphrase. A question about “how tall the ceilings are” can match a passage that says “clear height.” They are weaker at exact identifiers: unit numbers, part codes, clause references, and names. Keyword search handles those well and misses the paraphrases.
Hybrid retrieval runs both and merges the results. A reranker, a model that scores each candidate passage against the question, then reorders a few dozen candidates and keeps the best handful. Test this two-stage approach on questions that mix exact vocabulary (“Unit 4B”) with intent (“does it have a balcony?”). A reranker adds cost and latency, so measure the benefit.
Metadata filters belong in this step too. If the user is asking about one project, filter to that project before ranking. For questions about current requirements, filter to the authoritative revision. Retrieve older versions separately when a comparison or conflict check needs them.
Know when not to retrieve
Some facts should never come from an index. Prices, availability, inventory, account status, and anything else that changes during the day belong in the system of record. A copy indexed last night is a stale answer waiting to happen.
This is where tools come in. Through MCP, the agent can call a live lookup, such as the current price list, a unit’s availability, or a customer’s plan, with the same permissions the user has. The design rule is simple: documents explain, systems of record state current facts. The agent retrieves the plan set’s description of a unit and calls the inventory tool to see whether it is still available. The answer cites both, including the time of the lookup.
Composition constrains claims to evidence
Answer only from what was retrieved
The generation step should be narrow. Give the model the retrieved passages with their IDs and metadata, and instruct it to make only claims it can attach to a passage. Every factual sentence carries a citation. A claim without one is removed or rewritten as uncertain.
Ask for structured output rather than free prose: the answer text, the citations for each claim, and a status field. That status is the hinge of the system. It says whether the question was answered, not found, or answered with a conflict. Structure also makes a post-check possible. Code can check that a citation exists in the retrieved set. Whether it supports the claim needs a separate evaluation, which may use a model or human review. A valid citation ID alone does not establish accuracy.
Treat abstention as a feature
“The plans don’t say” is a good answer when the plans don’t say. It tells the user exactly where the gap is and sends them to the right person. A confident guess about a ceiling height is worse than no answer, because someone will repeat it to a buyer.
Make abstention specific. Instead of a generic apology, the agent should say what it searched and what it found nearby: “The unit schedule lists floor area and orientation for 4B but not ceiling height. The section drawings on A-301 don’t cover this unit.” That turns a dead end into a lead.
Abstention needs tuning in both directions. An agent that refuses too often is useless, and one that never refuses is dangerous. Measure it, and treat both over-abstention and under-abstention as bugs.
Flag conflicts instead of resolving them silently
Real document sets disagree. The sales brochure says 990 square feet and the latest plan revision says 958. An older price sheet lists a finish that a later addendum removed. If the agent quietly picks one, it will eventually pick the wrong one.
Conflict detection starts at retrieval. When candidate passages describe the same attribute of the same thing, compare them. Differences in numbers, dates, and yes-or-no facts are the easy cases. Version metadata tells you which source is newer, but newer isn’t always authoritative. An addendum may govern over a brochure regardless of date. Encode that precedence where you can, and surface the disagreement either way: both values, both sources, and which one the system considers current. Conflicts are also useful signals for document owners, who usually want to know.
Access control belongs in retrieval
Filter before the model sees anything
A common shortcut is to retrieve broadly and tell the model, in the prompt, not to reveal documents the user shouldn’t see. That isn’t access control. A prompt is an instruction, not an enforcement point, and a well-phrased question or an instruction injected into a document can get around it.
Apply permissions at retrieval time. Each chunk carries the access rules of its source document, and every query runs with the identity of the person asking. If a passage isn’t visible to that person, it never enters the candidate set, never reaches the reranker, and never appears in the context window. Live tools follow the same rule: the server must enforce the user’s permissions. Using MCP does not provide that access policy automatically. Keep permissions in sync with the source system, and test the boundary with users who should and shouldn’t see a given document.
Evaluate retrieval and generation separately
Measure each stage on its own
When an answer is wrong, you need to know which stage failed. Did retrieval miss the passage, or did the model misread a passage it had? A single end-to-end score hides the difference, and the fixes are completely different.
Build a test set of real questions paired with the passages that answer them. Include questions that have no answer in the corpus and questions where sources conflict. Then measure:
- Retrieval recall: did the right passage appear in the top results?
- Citation accuracy: does each cited passage exist and support the claim attached to it?
- Faithfulness: does the answer contain claims that no retrieved passage supports?
- Abstention: on unanswerable questions, did the agent say so, and on answerable ones, did it avoid refusing?
- Conflict detection: on known disagreements, did the agent surface both sources?
Grade what you can automatically, and have a domain expert review a sample on a regular schedule. Version the test set alongside the corpus, because a new document revision can change the right answer. When a user reports a bad answer, add it to the set before fixing it.
Make checking a source a single click
Design the interface for verification
The interface should make source checking easy. If checking a citation means downloading a PDF and searching for a phrase, people will stop checking, and the design loses its point.
Each citation should open the source at the exact spot: the right page, with the passage highlighted, and the document title, version, and effective date visible. Abstentions should look different from answers. Conflicts should show both sources side by side. Live lookups should show when they were fetched. There should also be an easy way to report a wrong answer that captures the question, the retrieved passages, and the response, so the report arrives as evidence rather than a complaint.
A practical starting point
Start with one corpus and fifty questions
Pick one document set that people already ask about repeatedly: a project’s plans, a policy manual, a product’s runbooks. Collect fifty real questions from the people who ask them. For each one, write down the passage that answers it, or note that no passage does.
Then build the smallest version that respects the three behaviors:
- Ingestion that keeps page, section, version, and permissions.
- Hybrid retrieval with a reranker, filtered by the user’s access.
- Composition that cites every claim and can say “not found” or “these disagree.”
Run the fifty questions, measure retrieval and generation separately, and fix whichever stage fails most. Add live tools only for the facts that actually change. Expand the corpus once the numbers hold.
Make answers easy to check
A knowledge agent doesn’t earn trust by sounding sure. It earns trust by being easy to check, and by being honest when the evidence runs out or contradicts itself. Design for the question “where does it say that?” and most of the other decisions follow.
It’s the standard we build to at Xlinfe, including for sales teams who ask questions of a project’s plans and need the sheet and revision behind every answer.