When can you rely on an AI knowledge assistant?

01Why the demonstration misleads

Most knowledge assistants are built on the approach called retrieval-augmented generation (opens external site): look up the passages relevant to a question, then let a language model compose an answer based on them. Give it a tidy folder of fifty documents and you will have a persuasive demonstration within a week.

Real use is another matter. A Government department may hold years of circulars, procedures, minutes and case notes; a pharmaceutical plant thousands of SOPs in several versions; a bank compliance manuals that are revised every quarter. Much of it is duplicated, out of date or restricted. There the assistant can fail twice over. It answers wrongly and staff abandon it. Or, worse, it answers wrongly with assurance and staff act on what it said.

Whether such an assistant deserves trust depends less on the model than on five qualities of the system that surrounds it. Each one can be designed and, just as important, tested.

02Access rights: retrieve only what this user may read

The principle is straightforward: the assistant must never show anyone a passage that the original system would refuse to open for them. What counts is where the rule is enforced. Assume that any passage handed to the model may turn up in its answer. Access rights therefore have to be applied before the answer is written, when passages are retrieved, for the person asking.

Two risks on the OWASP list for language model applications apply here: sensitive information disclosure (opens external site) and vector and embedding weaknesses (opens external site), which includes leaks across permission boundaries. Four requirements follow.

• Keep each source document’s access rights on every chunk in the index, copied from the source system and not rebuilt by hand.
• Filter on those rights at the moment of the query, using the identity of the person signed in, before anything is ranked or generated.
• Pass changes in permissions through at least as fast as changes in content, and agree how quickly a withdrawn right must take effect.
• Test with the real roles. A test that checks whether a new recruit can see the pay scales belongs in your suite before go-live, not in a review after the leak.

03Sources: citations a reader can verify

Whatever the answer claims should be traceable to its passage: the document title, its version or date, its owner, and a link that opens at the right spot. Show the passage itself and not just the file name, so the reader sees at a glance whether it supports the sentence.

Shape the answer so that unsupported text is obvious. Where no retrieved passage backs a sentence, the system should drop it or flag it. Check this automatically too: hold each cited passage against the claim it is supposed to support, and send a sample of the results to a person for review. Citations that exist but do not back up the claim are a frequent failure that is easy to miss.

04Currency and the authoritative source

Knowledge inside an organisation comes in versions. Last year’s circular and the one that replaced it both match a question about the procedure for a refund, and they give opposite answers. Index the metadata that separates them: the date from which a document applies, its status (draft, approved, superseded) and its owner. Put current, approved documents first, leave out superseded ones unless the user asks for the history, and show the date in every citation.

Settle per subject which source has the final word. HR rules come from the HR portal, not from an email quoting an earlier version. Decide how often each source is indexed again, from close to real time for case data to once a day for an intranet, and monitor it: an index that has gone stale yields answers that look correct and are not.

05Honest gaps: saying “I could not find this”

Frequently the most useful reply is an admission that nothing reliable was found. Language models can write smooth statements that nothing supports, which NIST calls confabulation (opens external site) in its profile for generative AI. Basing the answer on retrieved passages makes this rarer but does not rule it out. OWASP treats the knock-on effect, people acting on false output, as a separate risk under misinformation (opens external site).

Fix a relevance threshold below which the assistant says it has no reliable source and points to where the user can ask instead. On questions about policy, entitlements or obligations, do not allow the model to fill gaps from general knowledge. Then follow the rate of unanswered questions per subject. In the first months that list is often the most valuable result of the project, because it shows document owners exactly where guidance is missing.

06How the system is put together

You do not need unusual technology for any of this. You need access rights, metadata and checks to be built into the pipeline from the start, not bolted on once the demonstration is over.

Personal data needs deliberate attention. Indexing documents still counts as processing the personal data inside them, so the GDPR principles of purpose limitation and data minimisation (opens external site) (Article 5) apply, under the supervision of the Information and Data Protection Commissioner (IDPC) in Malta. Exclude sources the use case does not need, make sure deletions in a source system reach the index, and cover the index when you reply to a subject access request.

07Quality measured before go-live and after

Trying a handful of questions in a meeting tells you nothing about trustworthiness. Agree quality criteria with the owners of the documents and measure each layer on its own, so that you know where a fault lies.

Assemble the test set from questions people really ask: search logs, helpdesk tickets, what new colleagues want to know in their first weeks. A few hundred questions with answers agreed by the document owners is a workable start. Run the whole set after every change, be it a new model, a new way of chunking or a new source. Quality slips without warning when content changes, and only a fixed test set brings the slip to light.

08What a user notices when it works

For the public officer, the quality manager or the compliance analyst using it, all of this comes down to a few things they can observe. The assistant answers only from documents they are entitled to read. Every claim carries a citation they can open. They can see how recent a source is. They hear plainly when no reliable answer exists. And they know quality is being watched, because they can flag a wrong answer and see it corrected.

An AI answer can still be wrong, and some questions call for a person rather than a search. Being open about that is no weakness in the design. It is the reason people are prepared to depend on the assistant for everything else.