Knowledge search a compliance team can rely on

01Why a demonstration convinces and production does not

Most knowledge assistants rest on a pattern called retrieval-augmented generation (opens external site). The system looks up the passages that fit a question, and a language model then composes a reply from them. Pointed at a tidy folder of fifty files, the result impresses within days.

Real collections look different: years of shared drives, intranet pages, tickets, contracts and internal policies, many copied, replaced or restricted. A compliance team at a bank, an insurer or a capital market intermediary adds its own layer. Think of Bank Negara Malaysia (BNM) policy documents such as Risk Management in Technology (RMiT), revised and in effect since 28 November 2025, the Securities Commission Malaysia (SC) Guidelines on Technology Risk Management, revised and in effect since 19 August 2024, and the internal procedures written to meet them.

In that setting an assistant can fail in two ways. It replies wrongly, and staff abandon it. Or it replies wrongly with assurance, and staff act on the reply. Trust therefore depends less on the model than on five qualities of the surrounding system. Each one can be designed, and each one can be tested.

02Access rights: retrieve only what the reader may open

The principle is plain. A reader must never see a passage, through the assistant, that the source system would refuse to open for them. What matters is where the check happens. Anything handed to the model can resurface in the reply, so access rights have to be applied before generation, during retrieval, for the person who asked.

For applications built on language models, OWASP ranks sensitive information disclosure (opens external site) as a principal risk, next to vector and embedding weaknesses (opens external site) such as leaks across permission boundaries. Four requirements follow.

• Each indexed chunk carries the access rights of its document, copied from the source system and never re-entered by hand.
• Queries are filtered on those rights with the signed-in reader’s identity, before ranking and before the model sees anything.
• Changes to permissions travel at least as fast as changes to content, and the team agrees how quickly a withdrawn right takes effect.
• Real roles are part of the tests. “Can a graduate trainee open the board papers?” is a test case, long before it becomes an incident review.

03Citations a reader can verify

Each statement should lead back to the passage it rests on, showing the document’s title, its version or date and its owner, with a link that lands on the right place. For a regulatory question, that is the paragraph of the policy document or guideline, not just its title. Display the passage itself, so the reader sees at a glance whether it supports the sentence.

Shape the answer so that unsupported wording stands out. Where no retrieved passage backs a sentence, the system drops it or flags it. Check this by machine too: test whether every cited passage really backs its claim, and pass a sample of the results to a person for review. A citation that exists yet does not support its claim is a frequent failure, and an easy one to miss.

04Versions in force and the authoritative source

Regulatory and internal texts both come in versions. When a regulator revises a policy document, the earlier issue and the revised one match the same question and may answer it differently. Index the details that set them apart: the owner, the effective date and the status, whether draft, approved or superseded. Rank texts in force and approved ahead of the rest, keep superseded ones out unless someone asks about history, and print the date beside each citation.

Settle per subject which source has the final word. The regulator’s current text comes from the regulator’s own site or from your controlled library, never from an email quoting a draft. Give each source its own re-indexing interval, from near real time for live case records to once a day for an intranet, and watch it. An outdated index produces replies that look correct and are not.

05Admitting gaps: “no reliable source found”

Often the most useful reply is an admission that nothing reliable turned up. Language models can write fluent statements that nothing supports; NIST’s generative AI profile describes this as confabulation (opens external site). Grounding replies in retrieved passages makes it rarer without ruling it out. OWASP treats the consequence, staff relying on untrue output, as a separate risk under the heading misinformation (opens external site).

Fix a relevance threshold under which the assistant reports that nothing reliable came back and suggests whom to ask. On policy, rights or regulatory obligations, never let the model fill a gap from general knowledge. Follow the no-answer rate per subject. In the early months, that list can be the most valuable thing the project produces, because it shows document owners exactly where guidance is missing.

06The pipeline behind the five properties

No unusual technology is required. What is required is that access rights, document details and checks are built into the pipeline from the first day, not bolted on once the demonstration is over.

07Documents that contain personal data

Indexing documents that hold personal data is processing in its own right. For an organisation in Malaysia, the Personal Data Protection Act 2010 (Act 709), the PDPA, applies to that processing as amended by the Personal Data Protection (Amendment) Act 2024 (Act A1727). Two of its principles matter most for an index.

• Purpose and retention. Personal data is processed for the purposes stated when it was collected and kept no longer than those purposes need. Leave out sources the use case can do without, and let records deleted at the source disappear from the index too.
• Security. The Security Principle asks for practical steps against loss, misuse and unauthorised access. Since 1 April 2025 it also binds data processors, such as a hosting or search service that runs the index on your behalf, so the contract with that service should say how it meets the principle.

A data access request covers the index as well, so search it when you respond. Where the documents hold personal data of people in the EU and the GDPR reaches your organisation, Article 5 sets the principles of purpose limitation and data minimisation (opens external site), which lead to the same design choices.

08Quality you measure, before and after go-live

A handful of questions tried during a meeting says little about reliability. Settle the quality criteria with the people who own the content, then measure each layer on its own, so a fault can be traced to where it sits.

Draw the test questions from real sources: search logs, helpdesk tickets and what new staff ask in their first weeks. For a compliance assistant, add the questions your team actually receives about the regulatory texts in force. Some hundreds of questions, each with an answer the content owners have accepted, make a workable start. Run the whole set after any change: a new model, a different chunking method or an extra source. Quality slips unnoticed as documents change, and only a stable test set makes the slip visible.

09What reliability looks like to the people using it

To its users, all of this appears as a few behaviours they can observe. Replies draw solely on files they are entitled to open. Each claim carries a citation they can follow. The age of every source is visible. When no reliable answer exists, the assistant says so plainly. And they can see that quality is watched, because a wrong reply they report gets corrected.

Even then, AI output is sometimes wrong, and some questions belong with a colleague rather than a search box. Designing for that in the open is not a flaw in the system. It is why staff are willing to depend on it for everything else.