Agent Governance: What it Looks Like When it is Built in?
It does not matter how large of a fleet of AI agents your organization is running. While their quality and effectiveness are a different ball game to worry about, their governance is another
Consider one scenario - A few months from now, somebody in your organization asks a question about an AI agent that terminated a process on a production server one night. Naturally, a few scrutinizing questions will arise.
- Who approved it to take that action?
- Was it an autonomous decision by the AI agent in question?
- What was the AI agent looking at when it decided to do so?
- Which model produced the reasoning, and who hosts that model?
- Did anyone review the decision afterwards?
- Is that AI agent is still running today?
In most organizations deploying agents right now, several of the above questions have no answer. Since the systems being used were built to make the AI agents work rather than to make them accountable, and the two are different problems.
In this article, I want to walk through how we approach this problem of 'AI Agent Governance' - using our own product as an example.
Let us get going!
Governance of agents is not governance of LLMs
The AI governance category that exists today was originally built for LLMs. This typically involved listing the models in use, assessing them against the EU AI Act, documenting the bias testing, managing model risk, and producing the register for regulators.
The above LLM-level governance is acceptable at a high level, but this is not the real problem.
The problem is that "a model produces an output that a person then acts on, while an AI agent takes the action itself."
Governing a model means asking whether its output was fair. Governing an AI agent means asking what it was permitted to touch, what it actually did, on whose authority, and whether you can demonstrate any of that afterwards.
For instance, bias testing does not tell you anything useful about an AI agent that granted the wrong person access to a payroll system. So, an organization that has bought model governance has not necessarily acquired or built AI agent governance.
Typical chain for reviewing an AI agent actions
When somebody investigates an agent action, they follow a chain. Each link in the chain must be justified, and each link is a separate design decision.
Here is what a typical review-chain for an AI agent looks like;
Get built-in AI agent governance
Rezolve.ai gives you fully auditable AI agents and their actions on top of world class ITSM and HR agentic intelligence
#1. What exists?
This usually includes the following questions;
- How many AI agents are there?
- Which AI agents are published?
- Which AI agents are drafts?
- Which are running and which have never run at all?
Once the above questions are answered, organizations usually discover that they had built hundreds of agents with only a handful in genuine use. This is because creation of AI agents was distributed and nobody maintained a register.
However, if you are using Rezolve.ai, it shows total agents, how many are published, total runs across all of them, etc. Moreover, it also shows the number of unique tools those AI agents can collectively reach.
Rezolve.ai Asset Management & Reporting Agents observes and tracks all other AI agents - including total runs, their tools-access, publish and update history, etc.
Rezolve.ai shows not just how many agents you have, but what they can touch between them - which is a number that a security reviewer is concerned with.
#2. What each one is permitted to do?
Connecting a system and granting access to everything in it are different decisions, and most products conflate them. That is why with Rezolve.ai, the MCP Integrations page shows tools and triggers individually enabled for each AI agent - each of them instantly toggleable. The environment is labelled, so you can see whether an AI agent is reaching production or a test instance. And the connection itself can be switched off without deleting anything.
Rezolve.ai MCP Integrations show the tools and MCP connections for each AI agent, and they can be instantly enabled or disabled
Credentials follow the same principle. Keys are scoped to a single integration and can only invoke that integration's tools. Each one shows when it was created, when it was last used, and how many requests it has served. Essentially, this makes it easy to track API keys created months ago but which are no longer in use, and must be revoked.
#3. Who approved an AI agent's action?
Approval, for instance, in Rezolve.ai sits at the tool-call level rather than the AI agent level. A specific action pauses the run and waits for sign-off, with the tool, the agent, the approver and the request time all recorded.
The part I would draw attention to is what happens when nobody responds. Unapproved requests expire after twenty-four hours into a terminal state called 'Approval Expired.' The run is not retried automatically so someone has to ask again to trigger it. Since an approval request that quietly proceeds after being ignored is worse than no approval process, because it produces the paperwork of control without the control!
Rezolve.ai AI Agent Approvals and request status tracking
Please note that in-session requests, where a person is present in the conversation, are answered there and do not expire. Out-of-band requests, where somebody has to be found, do expire within 24-hours. The approval model treats a person who is available differently from a person who is absent.
#4. What the AI agent actually did?
Within Rezolve.ai, Agent Interactions record the conversations and transcripts for every run, filterable by agent, source and status, with duration and timestamp. Completion logs record prompt completions run through the API, with the model and the provider on each. MCP logs record tool invocations against the platform server.
Rezolve.ai AI Agent Interactions record
"Did the agent do something" and "what did it touch" are not the same question, and they should not share a log.
#5. Why the AI agent did what it did?
This is the link in the review chain that breaks often, and it is the one I would test thoroughly in any evaluation.
In Rezolve.ai, you can open a run and you get the model, the duration, the source, the token counts in and out, and the full conversation log.
Rezolve.ai AI Agent Interactions log
For an agent that terminates processes, the record contains the structured input it was working from: the thresholds it was measuring against, the actual process list with names, PIDs and memory consumption, the top processes by CPU. And the response begins with a reasoning field with the agent's own justification, stored as structured output rather than reconstructed later.
Rezolve.ai AI Agent Interactions reasoning log
So the answer to "why did it kill that process" is not an inference. The answer is a field.
#6. What the AI agent cost?
Token counts in and out are recorded per run in Rezolve.ai. It makes the cost of an AI agent measurable at the agent level rather than arriving as one line on a provider invoice at the end of the month.
#7. Which LLM, and who is the provider?
In case of Rezolve.ai, LLMs are enabled individually per provider, with a tenant default that AI agents inherit unless overridden. Provider connection state is explicit. There is also a hosted option, which matters for anyone concerned with organization's data privacy and information safety.
LLM and AI providers are listed and readily auditable with Rezolve.ai
Because the model is recorded on every run, "which model produced this decision" is answerable from the record rather than from someone's memory of the configuration at the time.
Version history is an AI agent governance feature
This matters for a specific reason. When somebody asks why an agent behaved differently in sometime back than it does now, the answer requires knowing what the agent was configured to do previously. A product where AI agents are edited in place (without saving a log of agent configuration states) cannot answer that question, and no amount of logging other info compensates for this. Because the logs will tell you what happened without telling you what the agent was at the time.
The same applies to the marketplace. Agents there are versioned and attributed, and you clone them rather than installing them. So what enters your estate is yours to modify, with your own history from that point, and a reviewer can still trace where it originally came from.
Rezolve.ai AI Agents Marketplace where useful agents can be cloned - with versioning and attribution
What to ask of any agentic AI product?
Take these into an evaluation regardless of the vendor's name, repo, and credibility.
- Show me every AI agent in the tenant, including the ones nobody uses. Then show me what they can collectively reach.
- Show me an agent's permissions at the tool level. Not just which systems it is connected to, but which specific capabilities are enabled within them.
- Show me an unanswered approval request for an agent. What state is the run in now, and what happens next.
- Show me why an agent did something, say last month. You want a stored reasoning field and the input it was working from, not a summary generated after the fact.
- Show me your API keys, when each was last used, and how to revoke one.
- Show me a failed autonomous run. If failures are not as visible as successes, the success number is decoration.
- Show me the previous versions of this agent. There has to be a log of AI agent's stored versions and configurations.
If a product answers all seven, its governance was designed. If it answers three, governance was added later, and you will discover which three at the least convenient moment.
Rezolve.ai empowers you with built-in AI agent governance
While the quality and effectiveness of AI agents are a different ball game to worry about, their governance is another. Rezolve.ai not just gives you a thoroughly tested fleet of AI agents for ITSM and ESM, but a powerful agent governance infra that takes away audit and compliance headaches.
See how Rezolve.ai can fit your existing IT, HR, and other enterprise workflows, and solve the painful challenge of agent governance.
Explore your ROI with us - book a demo
Last updated on September 11, 2026
See the agentic service desk in action
Watch Rezolve.ai autonomously resolve real IT and HR tickets: governed, auditable, glass-box.
Frequently asked questions
1. Is AI agent governance the same as AI or LLM governance?
No, and the difference matters. The governance category that exists today was built for LLMs: list the models in use, assess them against the EU AI Act, document bias testing, manage model risk, produce the register for regulators. That is fine at a high level, but it answers a different question. A model produces an output that a person then acts on. An agent takes the action itself. Governing a model means asking whether its output was fair. Governing an agent means asking what it was permitted to touch, what it actually did, on whose authority, and whether you can demonstrate any of that afterwards. Bias testing tells you nothing useful about an agent that granted the wrong person access to payroll. Buying model governance does not mean you have agent governance.
2. What should we be able to answer about any agent action?
Think of it as a chain, where each link is a separate design decision. What exists: how many agents, how many published, how many drafts, how many have never run, and what they can collectively reach. What each one is permitted to do, at the tool level rather than the system level. Who approved the action. What the agent actually did. Why it did it. What it cost. And which model produced the reasoning, and who hosts it. Most organizations investigating an agent action find that several of those links have no answer, because the systems were built to make agents work rather than to make them accountable. Those are different problems.
3. Why does an approval process need an expiry?
Because an approval request that quietly proceeds after being ignored is worse than no approval process at all. It produces the paperwork of control without the control. Approval should sit at the tool-call level rather than the agent level, so a specific action pauses the run and waits for sign-off, with the tool, the agent, the approver and the request time all recorded. Unanswered requests should expire into a terminal state, and the run should not retry itself. Somebody has to ask again. It is also worth distinguishing in-session requests, where a person is already in the conversation and can answer there, from out-of-band requests where somebody has to be found. The second kind is the one that needs a clock on it.
4. How do we answer "why did the agent do that" months later?
Not by inference, and not by a summary generated after the fact. The record should contain the structured input the agent was working from and its own justification stored as a field at the time. For an agent that terminates processes, that means the thresholds it was measuring against, the actual process list with names, PIDs and memory consumption, the top processes by CPU, and a reasoning field explaining the decision. This is the link in the chain that breaks most often, so it is the one to test hardest in any evaluation. It is also worth keeping "did the agent do something" and "what did it touch" in separate logs, because they are separate questions.
5. Why is version history a governance feature rather than a convenience?
Because when somebody asks why an agent behaved differently a few months ago than it does now, answering requires knowing what the agent was configured to do at the time. A product that edits agents in place without storing configuration states cannot answer that, and no amount of run logging compensates. The logs will tell you what happened without telling you what the agent was when it happened. The same principle applies to marketplace agents: they should be versioned and attributed, and cloned rather than installed, so what enters your estate is yours to modify with your own history from that point, while a reviewer can still trace where it came from.
