Where AI agents need human approval

01Agents act, so the question changes

A chat assistant drafts text and a person decides what to do with it. An agent goes further: it calls tools. It updates a record, books a slot, routes a case, sends a message or prepares a payment. Once software takes actions, the useful question is no longer only "is this answer good?" but "who is accountable for this action, and when did they agree to it?"

The security community has a name for getting this wrong. The OWASP Top 10 for large language model applications lists excessive agency (opens external site) as a core risk: an agent with more functionality, more permissions or more autonomy than its task needs. Human approval is one of the controls. Used everywhere, it slows work down and trains people to click "approve". Used nowhere, it leaves you with actions no one decided. The work is in placing it.

02Start from the actions, not from the model

Make an inventory of every tool the agent can call. For each one, answer four questions. They are deliberately about the action and its effect, not about the model.

1. 01Can it be undone? And at what cost: a click, a correction letter, a refund, a court case?
2. 02Who is affected? An internal draft, a colleague, a customer or citizen, money, a third party?
3. 03Does it create a commitment or a legal effect? A decision about a person, a payment, a contract, a message sent in your organisation's name.
4. 04How often and how fast? An action that happens thousands of times a day needs a different control than one that happens twice a week.

The answers place each action in one of four tiers. Most organisations find that a handful of actions carry almost all of the risk, which is good news: that is where approval effort belongs.

The tiers are a design device, not a legal classification. Check them against the rules that apply to you. The GDPR gives people specific rights around decisions based solely on automated processing (opens external site) that produce legal or similarly significant effects (Article 22). For high-risk systems, the EU AI Act requires that they can be effectively overseen by people (opens external site) (Article 14). Tier 4 is where those obligations usually land.

03Signals that should always reach a person

Some situations should move an action up a tier, whatever its usual place. Build these as explicit checks in the workflow, so they do not depend on the agent noticing them.

• The input is outside what the agent was tested on: a new document type, an unusual language, missing or contradictory fields.
• The sources the agent retrieved conflict, or the proposed action relies on something it cannot cite.
• The amount, scope or number of affected records is above a threshold you set.
• The case touches a sensitive category: health data, minors, a complaint, a dispute or an ongoing appeal.
• The agent has already retried, or a downstream system returned an error.

One signal is missing from that list on purpose: the model's own confidence. Language models can state wrong things fluently, which the U.S. National Institute of Standards and Technology describes as confabulation (opens external site) in its generative AI risk profile. A self-reported score can be one input, but it should sit next to external checks, such as validation rules and a comparison with the system of record, never replace them.

04Designing an approval people can actually give

A stream of "approve?" prompts teaches people to approve. The AI Act names the risk directly: people overseeing a high-risk system must be able to stay aware of automation bias.

Four design choices make that awareness practical.

Show the evidence, not only the answer

Present the proposed action in plain words, the sources it relies on (with links that open at the right passage), the rule or policy it applied and exactly what will change in which system. A reviewer who has to open three applications to check a proposal will, under time pressure, stop checking.

Make "no" as easy as "yes"

Reject and edit are normal outcomes, not exceptions. Put them next to approve, ask for a short reason when someone rejects, and feed those reasons into evaluation. If editing a proposal is harder than writing it from scratch, people will approve imperfect proposals instead.

Give approval to a role with authority

Approval belongs to a named role with the competence, training and authority to overrule the system; for high-risk AI systems the AI Act asks deployers for exactly this (Article 26(2) (opens external site)). Add a deputy and an escalation path, and measure how long items wait. A queue nobody owns becomes the bottleneck that pushes teams to bypass the gate.

Batch only what is truly low risk

Reviewing forty near-identical items one by one does not make anyone more careful. For tier 2 actions, allow batch review with random sampling. Keep tier 3 and tier 4 items individual.

05Enforce the gate in the system, not in the prompt

An instruction in a prompt is not access control. Text the agent reads, such as an email, a document or a web page, can contain instructions of its own, which OWASP describes as prompt injection (opens external site). If the only thing between an agent and an irreversible action is a sentence in its instructions, assume that sentence will one day be overridden.

• Give the agent its own identity with the least privileges each tool needs, and act within the requesting user's rights where the platform supports it.
• Enforce approvals in the integration layer: the service that sends the letter refuses the call unless a valid approval record exists for that exact action.
• Put value, volume and rate limits on tools, independent of the agent.
• Keep reading and writing tools separate, so you can widen what the agent may see without widening what it may do.

06Log what you would need to explain later

For every agent run, keep the request, the retrieved sources, each tool call with its parameters, the proposed action, the approver, the decision, any edits and the timestamps. For high-risk AI systems, deployers must keep the automatically generated logs for at least six months unless other law says otherwise (Article 26(6) (opens external site)). Where no rule requires it, keep them anyway: the log is how you learn.

Review the logs on a fixed rhythm. Actions that are always approved unchanged are candidates to move down a tier. Actions that are often edited point to a weakness in the agent, or belong a tier higher. Rejections that cluster around one source, one form or one team usually reveal a process problem that no model will fix.

07Start narrow, then move actions down a tier

Launch with most consequential actions in tiers 3 and 4. Agree beforehand what good looks like: which outcomes count as correct, which errors are acceptable and which are not. Then measure against those criteria, for example with the measurement functions of the NIST AI Risk Management Framework (opens external site) as a checklist.

Move an action to a lighter tier only with evidence: a period in which approvals needed no material edits, tests on the cases that went wrong, and sign-off from the owner of the process. Record that decision like any other change to the system, and keep the option to move it back.

AI output can be wrong. That is not a reason to keep agents away from real work; it is the reason to design where people stay in control. Agents are useful because they can act. Placing approval gates deliberately is what lets you give them more to do over time.