Connecting an AI agent to business software does not simply give it more context. It gives the agent the ability to modify a record, send a message, create an order or trigger a step that will be difficult to undo. At that point, the question is no longer “does the agent respond well?” It becomes: at what point should a person be able to review, correct or stop what it proposes?
The lazy answer is to require human validation everywhere. This is reassuring on paper, but it makes the tool so cumbersome that teams ultimately avoid it or click without reading. The opposite mistake is to confuse a successful demonstration with permission to act. A model may summarize a record correctly while misinterpreting a commercial exception, a date or a confidentiality rule. Validation must therefore focus on the consequences of the action, not on an abstract distrust of AI.
The CNIL recommends starting from a concrete need, establishing a framework for use and not entrusting decision-making to the system when that framework does not permit it. This approach has a very practical implication: before discussing autonomy, it is necessary to describe what the agent can read, what it can propose, what it can modify and what it must never do alone.
Prepare, recommend, execute: three statuses that are not equivalent
An agent can prepare work without making a decision. It can compile information from a record, retrieve passages from a procedure, extract information from a document or draft a response. Here, the employee retains the initiative: they use a proposal as a starting point. The main considerations are the quality of the sources, data confidentiality and clarity about what the tool has actually done.
The status changes when the agent recommends an action. If it suggests classifying a request as urgent, choosing a reason for a refund or prioritizing a customer, it is influencing a business decision. The person validating it must then be able to see the relevant information, challenge the recommendation and have an escalation path available. A screen that displays only “score: 0,91” does not fulfil this function; it hides arbitrariness behind a number.
Finally, the agent can execute: create a ticket, update a record, send a follow-up or trigger an order. Some actions are easily reversible and contained. Others commit the company in its dealings with a customer, modify reference data or have a financial impact. Placing them in the same category simply because they all go through an API is a design error.
This is why an AI agent and automation project must assign each action an owner, a scope, an audit trail and a stop rule. Who can authorize the agent to switch to active mode? What amount, customer type or status change requires a review? How can the action be reversed? If no one can answer these questions quickly, it is too early to connect the action.
Controls must reflect the potential harm, not the prestige of the task
An acknowledgement message can be automated within a very tightly defined scope. A response that quotes a price, accepts a contractual exception or closes a dispute must follow a different process. The same contrast applies to internal operations: preparing a CRM update is different from applying it to an entire database; flagging an invoice anomaly is different from approving the invoice for payment.
Effective human validation is therefore selective. It may take the form of a threshold, sampling, dual validation or mandatory review for defined cases. It must also address uncertainty. The agent should not invent a response when sources conflict, when required data is missing or when the case falls outside the intended scope. It must know how to hand off the case with the information it has found and an explanation of what it was unable to establish.
The NIST AI Risk Management Framework emphasizes risk management throughout the design, use and evaluation of a system. For an agent, this means testing scenarios that do not present it in the best light: two customers with the same name, an incomplete attachment, a conflicting instruction, an unavailable tool, or a person requesting an action they are not authorized to take. The right question is less “did the model succeed?” than “what happens when it cannot be certain?”
This approach also protects teams. Human validation is genuine only if the validator has the time, skills and authority to refuse. Asking a support representative to approve one hundred proposals without access to the source record does not create oversight; it creates oversight in name only. Effective AI training must therefore explain the system's limitations, escalation cases and stop procedures, not merely teach people to write better instructions.
A pilot must prove that the stop mechanism works
Autonomy can then be increased gradually. A repetitive, reversible and properly logged action can be authorized within a restricted scope. Ambiguous cases remain subject to validation. Every expansion must be accompanied by a test set, accessible logs, an accountable owner and a way to suspend the workflow without waiting for a complex change.
The NIST profile on generative AI focuses on the risks specific to generative systems. It reiterates an obvious point that is sometimes lost in demonstrations: a convincing output is not proof that the associated action is correct. This is why the pilot's exit criteria must include significant errors, exceptions, the validation workload and recovery capabilities, not just the adoption rate.
AI scoping brings these choices to the table before the agent becomes a habit that is difficult to reverse. The right objective is not to remove people from the process. It is to reserve their judgement for the points at which a relationship, a data item or a decision is at stake, and to give the agent a framework in which it also knows when not to proceed.