Where AI agents need human approval: maker-checker for agent actions

01From drafting to doing

A chat assistant produces text, and an employee decides what happens next. An agent is wired to tools, so it can take the next step itself: post an entry, update a vendor record, raise a service request, send a letter to a customer or stage a payment. Once software executes, the question an auditor asks shifts from “was the answer correct?” to “which officer authorised this action, and when?”

Security practitioners already track this failure. The OWASP list of risks for large language model applications includes excessive agency (opens external site): an agent holding more functions, wider permissions or more autonomy than its task requires. Requiring human approval addresses part of that risk, but it has a cost. Put it on every step and reviewers learn to click through without reading; leave it out and actions take place that nobody authorised. The real design work is deciding where it goes.

02Maker-checker, applied to agents

Indian financial institutions have long kept the maker, who enters or prepares a transaction, apart from the checker, who verifies and authorises it. Many enterprises apply the same rule to vendor payments, master data changes and credit decisions. The pattern carries over well: the agent becomes the maker, a named officer remains the checker, and the delegation of financial powers decides who that officer is and up to which limit.

Two illustrations show the shape. At an NBFC, an agent could assemble a loan file, check the documents against the credit policy and draft the sanction note, while sanction itself stays with the credit officer above a set amount. In accounts payable, an agent could match each supplier invoice with the purchase order, the goods receipt and, where the supplier falls under GST e-invoicing, the Invoice Reference Number (IRN), then pass clean matches for payment; an authorised officer releases anything above the limit or anything that does not reconcile. Both describe a design, not a completed project.

03Classify the actions before you look at the model

List each tool the agent is allowed to call. Then answer four questions per tool. They concern the effect in the real world, not the quality of the model.

1. 01Can it be reversed, and at what cost? A click, a credit note, a refund to a customer, or a dispute before a tribunal?
2. 02Whom does it touch? An internal working paper, a colleague, a customer or citizen, a bank account, a counterparty?
3. 03Does it bind the organisation? A decision about an individual, a payment, a contractual commitment, a communication issued on your letterhead.
4. 04What is the volume and pace? A step that runs ten thousand times a day calls for a different control from one that runs twice a week.

Those answers sort every action into one of four tiers. In most organisations a small number of actions account for nearly all of the exposure. That is useful: it tells you where your checkers’ time should go.

The tiers are an engineering tool, not a legal category, so test them against the law that applies to you. Where the agent serves people in the European Union, the GDPR gives individuals specific rights around decisions taken solely by automated means (opens external site) that have legal or similarly significant effects (Article 22), and the EU AI Act requires that high-risk systems allow effective oversight by natural persons (opens external site) (Article 14). Those duties usually sit in tier 4.

04When EU rules reach an agent built in India

The EU AI Act does not stop at companies established in the Union. It applies to a provider in a third country whose AI system is placed on the EU market or put into service there (Article 2(1)(a)), and to providers and deployers in a third country “where the output produced by the AI system is used in the Union” (Article 2(1)(c)). An Indian software company whose agent drafts claim decisions for a European insurer, or an Indian team whose agent output is used by a European business unit, should therefore check whether oversight duties travel with that output.

At home there is no AI statute. The India AI Governance Guidelines, released by the Ministry of Electronics and Information Technology (MeitY) in November 2025, are non-binding. The binding rules for most agent deployments come from other laws: the Digital Personal Data Protection Act, 2023 (DPDP Act) for personal data that flows into prompts, retrieved documents and tool calls, and the requirements of the Reserve Bank of India (RBI) and the Securities and Exchange Board of India (SEBI) for the entities they regulate.

05Triggers that always send an action to an officer

Certain conditions should lift an action into a higher tier regardless of where it normally sits. Code them as explicit rules in the workflow, so that they fire even when the agent fails to notice them.

• The input falls outside what the agent was evaluated on: an unfamiliar document type, a language it was not tested in, missing or conflicting fields.
• The retrieved sources disagree, or the staged action rests on a fact the agent cannot cite.
• The amount, the number of records affected or the scope crosses a limit set in the delegation of powers.
• The case involves sensitive matters: health information, a child’s data, a complaint, a dispute or a matter under appeal.
• The agent has retried, or a downstream system has returned an error.

The model’s self-assessed confidence is left off this list deliberately. Language models can produce fluent but unfounded statements, which the U.S. National Institute of Standards and Technology calls confabulation (opens external site) in its generative AI profile. A confidence score may inform routing, but it belongs beside independent checks such as validation rules and reconciliation with the system of record, never in their place.

06Designing a checker step that works

Reviewers who face a steady stream of approval requests start approving by reflex. For people overseeing a high-risk system, the AI Act addresses that tendency in so many words:

Four design choices turn that awareness into daily practice.

Put the evidence next to the proposal

Show the staged action in plain language, the documents it relies on with links that open at the relevant paragraph, the policy rule it applied, and precisely which fields will change in which system. If an officer must switch between three applications to verify one proposal, verification is the first thing to go when the queue grows.

Rejecting should take no more effort than approving

Amendments and rejections are routine outcomes. Place them beside the approve button, ask for a brief reason on rejection, and feed those reasons into evaluation. When correcting a staged action takes longer than doing the work by hand, officers will pass flawed proposals rather than fix them.

Give the checker role to someone who can overrule

The checker should be a designated role with the competence, training and authority to override the system. For high-risk systems the AI Act expects this of deployers (Article 26(2) (opens external site)), and it fits the delegation of financial powers most Indian enterprises already maintain. Name an alternate, define the escalation matrix and track how long items wait. An approval queue without an owner soon becomes the reason teams look for a way around it.

Batch review only where the risk is genuinely low

Reading forty almost identical items one after another does not sharpen anyone’s attention. Allow batch review with random sampling for tier 2. Keep tier 3 and tier 4 items as individual decisions.

07The checker step belongs in the system, not in the prompt

A sentence in the agent’s instructions does not control access. Content the agent processes, such as an email, a vendor’s PDF or a web page, can carry instructions of its own; OWASP calls this prompt injection (opens external site). If one line of instruction is all that stands between the agent and an irreversible step, plan for the day it is overridden.

• Run the agent under its own service identity with the narrowest privileges each tool requires, and act within the requesting user’s entitlements where the platform supports it.
• Enforce the checker step in the integration layer: the payment or messaging service refuses the call unless a valid approval record exists for that specific action.
• Set amount, volume and rate limits on tools that hold regardless of what the agent requests.
• Separate read tools from write tools, so that you can widen what the agent may consult without widening what it may change.

08Keep the record you will need when something goes wrong

For each agent run, retain the request, the sources retrieved, every tool call with its parameters, the staged action, the officer, the decision, any amendments and the timestamps. Under the EU AI Act, deployers of high-risk systems must hold the logs such a system generates automatically for no less than six months, unless another rule requires longer (Article 26(6) (opens external site)).

Indian rules point the same way. The CERT-In Directions of 28 April 2022, issued under section 70B(6) of the Information Technology Act, 2000, require listed cyber incidents to be reported to the Indian Computer Emergency Response Team (CERT-In) within six hours of noticing them, and require ICT system logs to be retained and clocks synchronised. An agent that can change records or move money is part of that picture: if it is misused, its records are how you establish what it did within the hours available. From May 2027, when the security safeguards of the DPDP Rules apply, a Data Fiduciary must also keep the personal data, traffic data and logs of its processing for at least one year (rule 8(3)), which includes personal data passing through prompts and tool calls.

Review the records on a fixed schedule. Actions that officers always approve unchanged are candidates for a lower tier. Actions that are often amended expose a weakness in the agent, or belong one tier higher. Rejections that concentrate on one source, one form or one branch usually reveal a process issue that no model change will solve.

09Begin with strict tiers and relax them on evidence

At go-live, keep most consequential actions in tiers 3 and 4. Agree acceptance criteria in advance: which outcomes count as correct, which errors are tolerable and which are not. Measure against them, for example with the measurement functions of the NIST AI Risk Management Framework (opens external site) as a checklist.

Lower an action’s tier only on evidence: a period in which officers made no material amendments, tests on the cases that previously failed, and written agreement from the process owner. Treat that decision as a change to the system, record it, and keep the ability to reverse it.

Agents will sometimes get things wrong. That argues for designing control into agent workflows, not for keeping agents away from real work. Agents earn their place because they can act, and a deliberate checker design is what lets you delegate more to them over time.