AI agents and the approval step: when a person has to say yes

01Agents take actions, and that changes the question

With a chat assistant, the software writes and a person decides what happens next. An agent goes a step further and calls tools: it updates a case record, books an appointment, routes a request to the right section, sends a reply or prepares a payment. Once software acts, asking whether an answer is good is no longer enough. You also need to know who answers for each action and at which moment they agreed to it.

Security practitioners already have a term for the failure. The OWASP Top 10 for large language model applications counts excessive agency (opens external site) among the central risks: an agent given more functions, more permissions or more freedom than its job requires. Approval by a person is one of the controls against it. Put it on everything and work grinds to a halt while staff learn to click through. Leave it out and actions happen that nobody decided. The skill lies in placing it.

02Begin with the actions, not with the model

List every tool the agent is able to call and put four questions to each. They concern the action and its consequences on purpose, and say nothing about the model.

1. 01Can it be reversed? And what does reversing cost: one click, a letter of correction, a refund, a case in court?
2. 02Whom does it touch? An internal draft, a colleague, a resident or client, money, an outside party?
3. 03Does it bind you or have legal effect? A decision about a person, a payment, a contract, a message sent in the name of your ministry, council or firm.
4. 04How frequent and how fast is it? Something that runs thousands of times a day calls for a different control from something that happens twice a week.

The answers sort each action into one of four tiers. Most organisations discover that a small number of actions hold nearly all the risk, which helps: that is where the approval effort should go. At a local council, for example, tagging a resident’s request is a different matter from telling the resident that a permit was refused.

These tiers are a design tool, not a legal category, so test them against the rules you work under. The GDPR gives individuals particular rights where decisions rest solely on automated processing (opens external site) and have legal or similarly significant effects (Article 22); in Malta the Information and Data Protection Commissioner (IDPC) supervises those rights. For high-risk systems, the EU AI Act demands that people are able to oversee them effectively (opens external site) (Article 14). Those duties normally fall in tier 4.

03Signals that must always go to a person

Certain situations should lift an action into a higher tier, wherever it normally sits. Build them into the workflow as explicit checks, so nothing depends on the agent spotting them.

• The input falls outside what the agent was tested on: an unfamiliar document type, an unexpected language, fields that are empty or contradict each other.
• The sources the agent found disagree, or the proposal rests on something it cannot point to.
• The amount, the scope or the number of records involved passes a limit you have set.
• The case involves something sensitive: health data, a minor, a complaint, a dispute or an appeal still running.
• The agent has already tried again, or a system further down the line has returned an error.

The model’s own confidence is left off that list deliberately. Language models can say wrong things with great fluency, which the U.S. National Institute of Standards and Technology calls confabulation (opens external site) in its risk profile for generative AI. A score the model gives itself may count as one signal, but only beside outside checks such as validation rules and a comparison with the system of record, never in their place.

04An approval step people can genuinely use

A constant flow of approval requests trains people to approve. The AI Act addresses this risk head on: whoever oversees a high-risk system has to be able to stay alert to automation bias.

Four choices in the design turn that alertness into practice.

Put the evidence beside the proposal

Show the proposed action in plain language, the sources behind it with links that open at the relevant passage, the rule or policy applied, and precisely what will change in which system. If a reviewer must open three applications to check one proposal, the checking stops as soon as the queue gets long.

Make refusing as simple as accepting

Rejecting and amending are ordinary results, not exceptions. Place them beside the approve button, ask for a brief reason on every rejection, and use those reasons when you evaluate the agent. When amending a proposal takes longer than writing it afresh, people end up approving proposals that are not quite right.

Give the approval to a role that has authority

Approval belongs with a named role that has the competence, training and authority to overrule the system, which is what the AI Act asks of deployers of high-risk AI (Article 26(2) (opens external site)). Name a stand-in and an escalation route, and measure how long items wait. A queue without an owner turns into the bottleneck that tempts teams to work around the approval step.

Group only what carries little risk

Checking forty almost identical items one at a time does not make a reviewer more careful. Allow tier 2 actions to be reviewed in batches with random samples, and keep every tier 3 and tier 4 item separate.

05Enforce approval in the system, not in the prompt

A line in a prompt does not control access. Anything the agent reads, an email, an attachment or a web page, can carry instructions of its own, which OWASP calls prompt injection (opens external site). If a sentence in the agent’s instructions is all that stands between it and an action you cannot undo, expect that sentence to be overridden one day.

• Give the agent an identity of its own with the narrowest rights each tool needs, and let it act within the requesting user’s rights where the platform allows.
• Enforce approval in the integration layer: the service that issues the letter rejects the call unless a valid approval exists for that precise action.
• Set limits on value, volume and rate on the tools themselves, independently of the agent.
• Keep tools that read apart from tools that write, so the agent can be allowed to see more without being allowed to do more.

06Record what you would need to explain afterwards

For each run of the agent, keep the request, the sources it retrieved, every tool call and its parameters, the proposed action, the approver, the decision, any amendments and the times. Deployers of high-risk AI systems must retain the automatically generated logs for six months or more unless other law provides otherwise (Article 26(6) (opens external site)). Where no rule asks for them, keep them anyway, because the record is how you improve.

Go through the records at fixed intervals. Actions that are always approved without change can be considered for a lower tier. Actions that are amended often point to a weakness in the agent, or belong in a higher tier. Rejections that gather around one source, one form or one team usually expose a process problem that no model will solve. At an investment services firm preparing client due-diligence files, for example, repeated rejections on one document type are a question for compliance, not for the model.

07Start strict, then lower the tier where the record allows

Go live with the actions that matter most in tiers 3 and 4. Agree in advance what success means: which results count as correct, which mistakes you can accept and which you cannot. Measure against those criteria, using for instance the measurement functions of the NIST AI Risk Management Framework (opens external site) as a checklist.

Lower an action’s tier only on evidence: a period in which approvals needed no material change, tests on the cases that went wrong, and agreement from the process owner. Record the decision as you would any change to the system, and keep the option of reversing it.

An AI system will sometimes be wrong. That is no reason to keep agents away from real work; it is the reason to decide carefully where a person keeps the final say. Agents earn their place because they can act, and deliberate approval steps are what allow you to hand them more over time.