AI agent governance is the set of rules, approvals and records that decides what an AI agent may do and who answers for what it produces. IT governs identity and access. The business governs accountability: who approves the work before it runs, what the agent may commit, and whose name sits under the result.
The contract is ready for signature. IT has finished its security review, legal has read the terms, and the business case assumes agents will prepare the quarterly business review. One question is still open, and no security review answers it: when a figure in that review is wrong, whose name is on it?
The guidance that ranks for AI agent governance is written mostly for the people who manage identity, credentials and tool access. Those questions matter, and they belong to IT. The questions that decide whether an agent's work can be defended in a budget review or a supplier negotiation belong to whoever signs for the budget.
Leaving the business half unanswered has a price. Gartner's June 2025 forecast on agentic AI projects expects over 40% of them to be cancelled by the end of 2027, and it names inadequate risk controls as one of three causes, alongside escalating costs and unclear business value. The business half can be settled before the first project runs, with controls finance already owns.
In this article
- What is AI agent governance?
- How is agentic AI governance different from governing other AI?
- Which governance questions belong to IT, and which to finance and procurement?
- Who is accountable when an AI agent gets a figure wrong?
- What should an AI agent be allowed to commit, and what should it never commit?
- What records should exist after an AI agent finishes a piece of work?
- What do standards such as the NIST AI Risk Management Framework expect?
- What should finance and procurement require before an AI agent goes live?
- Frequently asked questions
- What should an agent product hand the person who signs for it?
What is AI agent governance?
AI agent governance is how an organisation decides what its AI agents may do, who approves their work and who answers for the result. Its technical half controls what an agent can reach, such as credentials and tools. Its business half controls what the agent's work may become: a board figure, a negotiating position or a commitment of spend.
Finance already governs delegated work, from junior analysts to consultants, with three instruments that carry over to AI agents:
- A delegation of authority, which says who may approve and commit what, and up to which limit.
- Segregation of duties, which keeps preparing, checking and approving a figure in different hands.
- An evidence file, which lets someone who was not there reconstruct how a decision was reached.
What changes with an agent is how much can happen between instructions before a person looks.
How is agentic AI governance different from governing other AI?
Agentic AI governance is AI governance extended from outputs to actions: because an agent plans steps, uses tools and passes work on without waiting to be asked, it must control what the agent does, not only check what it says. The risk moves from a wrong answer on a screen to a wrong step taken halfway through the work.
The design of where a person approves inside the project is a workflow question in its own right. For the person who signs off, four differences matter:
- What you review changes: nobody reads every intermediate step, so control moves to the plan before it runs and to the exceptions as they surface.
- Errors compound. A fare class mapped to the wrong cabin in step three flows into every figure built on it.
- The agent can act as well as answer, so authority to commit has to be withheld in writing.
- Work runs unattended. Agents can re-run on a schedule or when new data lands, so governance has to hold on the days nobody is watching.
The question for the business is organisational: who holds each approval, and who answers when one fails?
Which governance questions belong to IT, and which to finance and procurement?
IT owns what an agent can reach; finance and procurement own what its work may become. Identity, credentials, data access and tool permissions are IT's questions. Approval, commitment authority, accountability for a wrong figure and the evidence behind each decision belong to the budget holder, with legal reviewing the contract that allocates both.
The division matters because attention is uneven. McKinsey's March 2025 global survey on AI, fielded in July 2024 across 1,491 participants, found respondents at larger organisations much more likely than others to report managing cybersecurity and privacy risks from generative AI, but no more likely to report addressing the accuracy or explainability of its outputs.
Accuracy is the risk a finance lead signs for. The table divides the questions, names each owner and gives the evidence that settles it.
The first three rows sit with IT, and the rest of this article stays on the other six.
Who is accountable when an AI agent gets a figure wrong?
The named people who approved the work and signed off the deliverable are accountable, never the agent. An agent cannot be dismissed or held to a contract, so accountability that is not assigned to a named human has not been assigned at all. The vendor answers for its product working as contracted.
AI accountability for a wrong figure splits three ways:
- The approver answers for the question asked and the scope accepted. A plan that asks the wrong question gets a correct answer to it.
- The owner of the deliverable answers for how the figure is used: released to a board, taken into a negotiation or written into a budget.
- The vendor answers for the product behaving as documented, including any validation it says it runs.
Oversight at the top is often shared: in the same McKinsey survey, respondents reported two leaders in charge of AI governance on average. That works for policy, but a figure in a board pack needs one name beside it.
A worked example, invented for illustration: a procurement lead approves a plan to audit booked hotel stays against contracted rates, and the deliverable flags one property as overcharging because booked rates that include breakfast were compared with contracted rates that exclude it. Who answers for that was settled before the work began. Finding where the error entered takes longer, and only the record can show it.
The rule holds for an FP&A team too, however much of the monthly pack agents prepare, and what AI changes about the FP&A month is a separate question from who signs it.
What should an AI agent be allowed to commit, and what should it never commit?
An AI agent may read, analyse, draft and recommend, but it should commit nothing on its own authority: no spend, contract, supplier award or externally released figure without a named human's approval. Write that into the delegation of authority, where the agent's line reads none.
Treat an agent's authority as a ladder of five rungs, of which it may climb four.
A figure sent to a supplier is a commitment even when no money moves, because the supplier will negotiate against it from then on.
For a procurement lead, everything on the commit rung belongs among the procurement decisions that stay with people.
A spending limit can look like a sensible compromise. For analytical work in finance and procurement the right limit is none, because an agent with a small limit still commits, and commitments under a threshold are, by design, the ones that skip review.
What records should exist after an AI agent finishes a piece of work?
After every piece of agent work, five records should exist: the approved plan, each checkpoint decision, a trail tying every figure to its source query, the validation result, and the named sign-off. Together they let someone who was not in the room reconstruct how a number was reached, months later, without asking the agent.
IT's logs record which agent ran which query and under whose identity. The five records answer the business's questions instead: what was decided, by whom, and on what evidence.
- The request and the approved plan: what question was asked, what scope was accepted, and who accepted it.
- The checkpoint decisions: what the agents flagged as surprising or unverifiable, and what the person decided about each.
- The citation trail: every figure in the deliverable tied to the query and the source that produced it, so the number can be reproduced.
- The validation result: which independent check ran before the deliverable reached a person, and what it found.
- The sign-off: who released the deliverable, when, and to whom.
Keep the five together, and keep them for as long as you keep the other records behind the same decision, such as the contract file.
A quick test shows whether the record works. Pick one figure from a recent deliverable and ask a colleague who was not involved to trace it to its source using only the records. If they need the agents re-run, or the approver's memory, the record is incomplete.
What do standards such as the NIST AI Risk Management Framework expect?
Standards such as the NIST AI Risk Management Framework expect accountability for AI to be documented: clear roles, clear lines of responsibility, and executive leadership that owns decisions about AI risk. NIST's framework, released in January 2023, sets this out under its GOVERN function and is intended for voluntary use.
The NIST AI Risk Management Framework publishes its core document as NIST AI 100-1, which states the premise plainly: "Trustworthy AI depends upon accountability." Its GOVERN 2.1 subcategory asks for documented, clear roles and lines of communication for AI risk, which for an agent means a named approver and owner for each type of work; GOVERN 2.3 puts responsibility for decisions about AI risk with executive leadership, where the delegation of authority and the agent's limit of none belong. Whether any regulation applies to a particular agent depends on where you operate and what it does, which is a question for legal counsel.
What should finance and procurement require before an AI agent goes live?
Before an AI agent goes live, finance and procurement should write down five names: who approves its plans, who owns each deliverable, who commits spend on its work, who traces its figures, and who controls its access. None of the five should be the agent or a committee. If a name is missing, the deployment is not ready.
The Five Names Check
The Five Names Check is PredictX's pre-sign-off test: finance and procurement name one person for each of five roles around an AI agent and show the evidence that each role is working. It borrows one rule from the RACI (responsible, accountable, consulted, informed) matrix familiar from project management: exactly one person is accountable for each outcome.
Run it with IT and legal in the room, and treat a team in place of a name as a gap, as you would the agent in any role. The check has no partial pass, because a gap left open gets filled later by whoever is nearest when something goes wrong.
The contract should answer the same questions in its own language: who owns the outputs, what the provider warrants about their accuracy, and how your data may be used. PredictX's AI Addendum, a set of contract terms for AI services, sets out each of those clauses.
Frequently asked questions
What is an AI agent governance framework?
An AI agent governance framework is the written set of roles, rules and records an organisation uses to control its AI agents. A useful one names who approves agent work, what agents may commit, who owns each output and what evidence is kept, and it leaves identity, access and security design to IT.
Who should own AI agent governance inside a company?
Oversight of AI policy is often shared at the top. In McKinsey's March 2025 survey, respondents reported two leaders in charge of AI governance on average, and 28% of those whose organisations use AI named the CEO. Accountability for each agent's work belongs with the budget holder who relies on it, and IT owns access and security.
What are the risks of agentic AI?
For a finance or procurement lead, the main risks of agentic AI are a wrong figure travelling into a decision, an agent acting beyond its authority, work nobody can reconstruct afterwards, and spend without clear value. IT carries a separate set of risks around identity and access.
How do you prepare for AI agent governance?
Start with the controls you already run. Add the agent to your delegation of authority with a commitment limit of none, name an approver and an owner for each type of work, decide which records to keep, and have IT review identity and access before the first project runs rather than after it.
Does governing AI agents slow adoption down?
Governance settled before launch adds a short piece of work and takes a recurring question out of every review that follows. The larger risk runs the other way: Gartner's June 2025 forecast expects over 40% of agentic AI projects to be cancelled by the end of 2027, and names inadequate risk controls as one of three causes.
What should you ask a vendor about AI agent governance before signing?
Ask to see five things working rather than described: where a person approves the plan before work runs, what the agents can commit on their own, how each figure traces to its source, which independent check runs before a deliverable reaches you, and where the security and access controls are documented for IT to review.
Is AI agent governance a legal requirement?
Whether AI agent governance is a legal requirement depends on where you operate and what the agent does. Regulations such as the EU AI Act apply to some AI systems. NIST's AI Risk Management Framework is intended for voluntary use, and ISO/IEC 42001 is an international standard, not a law. Either way, accountability stays with you.
What should an agent product hand the person who signs for it?
An agent product should give its signer four things: an approval step before any work runs, a pause when the data surprises it, a trail from every figure to its source, and an independent check. It should hold no authority to commit. The delegation of authority, the five names and the final sign-off stay with the business.
Orchestra is PredictX's AI agent team for travel and expense: an AI project manager plans the work, directs nine specialist agents and returns a finished, cited deliverable. Nothing runs until a person approves the plan. PredictX states the principle as "autonomous in execution, human-gated in decision": nothing commits spend without a named human.
Set against the five records above, Orchestra supplies the steps behind four of them, and its citation trail and full audit log are records in their own right:
- The approved plan: you edit and approve it before any specialist runs.
- The checkpoint decisions: surprising findings pause for your review before the report is written.
- The citation trail: every figure tied to its source query, with every hand-off tracked in a full audit log.
- The validation result: a silent self-correcting layer, then a separate agent auditing the data health behind every deliverable.
- The sign-off: yours, against the name the Five Names Check put there.
Orchestra runs on the Cogent AI Framework, and IT's review of identity, access and security belongs with the Cogent AI Framework's security documentation. Orchestra launched in June 2026 and is deploying across enterprise travel programmes.
Related Posts
Agentic workflows, and how they differ from the automation you run
