The chatbot just got write access
For the last few years, businesses have been concentrating on what generative AI can tell us. AI agents introduce a rather more consequential question: what happens when software can decide what to do next, and then actually do it?
In July, an AI agent being used as part of an OpenAI cybersecurity evaluation compromised part of Hugging Face's production infrastructure. There is some important context here before we get carried away. This was not a normal business application suddenly deciding to attack another company. OpenAI was deliberately testing highly capable models against difficult cybersecurity tasks, and its normal production controls for preventing high risk cyber activity had been reduced for the purposes of the evaluation.
Even so, what happened is worth paying attention to. According to the subsequent investigations, the agent appears to have concluded that information held by Hugging Face could help it complete the benchmark it had been given. It found a way to reach the wider internet, chained vulnerabilities together, obtained credentials and moved between systems. Hugging Face eventually reconstructed around 17,600 individual actions grouped into approximately 6,280 clusters.
The temptation, naturally, is to reach for the "AI escaped" or "AI went rogue" headline. I am not sure either is particularly useful. The software was not angry with Hugging Face and it had not developed an independent desire to become a cybercriminal. As far as the investigations could establish, it was trying to achieve the objective it had been given and found a route nobody expected it to take.
That is considerably less theatrical, but from a digital transformation perspective probably more important.
For the last few years, most organisations have experienced generative AI as something they ask. Give it some information, ask a question and get something back. It might summarise a contract, analyse some data, write some code or draft an email, but there is generally still a person sitting on the other side deciding what happens next.
Agents start to blur that boundary. Give the same intelligence access to tools and it can retrieve information, call an API, update a record, execute some code, send the email or make the change itself. Current agent architectures are explicitly designed around this model, with tools that provide information and tools that allow agents to take actions in external systems.
The chatbot just got write access.
Automation changes when software is given intent, not instructions
Software has been changing things for decades, so there is a danger of overselling what is new here. We do not need AI to update a database, trigger an API or run a workflow. Scripts, rules engines, orchestration platforms and automated processes have been doing those jobs perfectly well for years.
The difference is in how we get from the objective to the action. Traditional automation is normally designed around a route somebody has described in advance. When X happens, retrieve Y. If the answer is Z, perform this action. There may be hundreds of rules, branches and exceptions involved, but somebody has still tried to model the process beforehand.
Agents potentially allow us to move further towards giving software an outcome rather than every instruction required to reach it. Here is the problem. Work out what information you need. Decide which tools are appropriate. Deal with what comes back and get the job done.
That distinction matters because the execution path can become dynamic. An agent might start with one system, discover it needs some additional context, call something else, encounter a result it was not expecting, revise its approach and carry on. OpenAI describes agents as systems that can manage workflow execution, make decisions, dynamically select tools and take actions on behalf of a user, rather than simply generate a response.
That is a materially different proposition from conventional automation. The software is no longer just executing a predetermined flow. Within whatever boundaries we establish around it, we are giving it some discretion over how the flow unfolds.
And that is where this gets much more interesting than another conversation about whether AI can write a better email.
The opportunity is to remove coordination, not people
A surprising amount of work inside an organisation is not difficult because any individual task is particularly complicated. It is difficult because of the coordination required between the tasks.
Open the CRM. Find the customer. Copy an identifier into the billing platform. Read the notes. Check another system. Work out what they mean. Ask another team whether something is normal. Wait for the answer. Update the first platform and send the customer an email.
Sound familiar?
We have spent years trying to automate those processes, and conventional workflow technology is extremely effective when the steps can be described predictably. The difficult bits tend to be the exceptions: the document that is not structured properly, the account that does not quite match, three systems that disagree with one another, or the point in the process where somebody needs to read something and understand what it means before they know what to do next. These are precisely the kinds of workflows current agent guidance identifies as potentially suitable for agentic approaches, particularly where complex decisions, brittle rules or unstructured information are involved.
That is where the opportunity gets interesting. If software can take over some of the coordination between systems, information and decisions, there is potentially a meaningful efficiency gain. Not because everybody involved in the process suddenly disappears, but because we can stop using people as expensive middleware between platforms that were never particularly good at coordinating with each other.
There is an important distinction there. The objective should not necessarily be to remove people from the organisation. It should be to remove the work we currently make people do simply because the technology underneath them cannot handle context, ambiguity or exceptions very well.
That is why I think the opportunity around agents is much more significant than incremental productivity. Drafting a customer response ten seconds faster is useful. Allowing software to investigate the problem, reconcile the relevant information and carry the process forward is something else entirely.
Of course, the further it carries the process, the closer it eventually gets to an action with consequences.
And that is where things become rather more complicated.
Capability is not authority
A few months before the Hugging Face incident, PocketOS published a candid account of an AI agent deleting the company's production database. The deletion took nine seconds. Recovery took sixty hours.
PocketOS is also very clear that this was not simply an "AI problem". Its backups had failed months earlier without anyone noticing and, critically, the agent had access capable of performing the destructive action. The interesting lesson is therefore not that AI can delete a database. Humans have been doing that perfectly effectively for years, and software bugs have made a respectable contribution too. The interesting part is that the actor had sufficient authority for its mistake to matter.
We already understand this when the actor is human. Someone in customer services might be authorised to issue a £20 refund without anybody caring very much. £200 may still be entirely reasonable. £20,000 probably introduces another conversation.
The same applies elsewhere. An agent identifying a spelling mistake in some content metadata and correcting it is probably fairly benign. If the field it wants to change determines whether that content can be distributed in a particular territory, the consequence is very different. Likewise, an agent may correctly identify a production configuration issue, but that does not mean it should inherit unlimited rights to reconfigure the whole environment.
None of this is particularly exotic. It is identity and access management applied to a new type of actor.
Microsoft is already treating agents in these terms. Its current guidance calls for unique agent identities, named owners, explicit scope, least privilege access, controlled tool permissions, logging and tested revocation paths. NIST is working on the same territory through its AI Agent Standards Initiative and a specific project examining agent identification, authorisation, auditing and non-repudiation.
The important distinction is that an agent can be perfectly capable of determining what it thinks should happen without being entitled to make it happen.
As an aside, I suspect "how many agents do we actually have?" will become one of those questions organisations do not think to ask until the answer is uncomfortable. We have been through variations of this with SaaS applications, cloud resources, service accounts and API keys already. There is no obvious reason the agent estate will organise itself any better.
Agentic AI inherits every compromise in your digital estate
There is another awkward issue underneath all of this. Your agent is going to inherit your existing digital estate.
The shiny APIs, the legacy systems, the inconsistent customer records, the integration somebody wrote eight years ago and nobody wants to touch, and the platform that calls a customer one thing while another platform calls them something else. A company grows, platforms get bought, teams solve individual problems, acquisitions happen, suppliers change and priorities move. Eventually you end up with what everyone politely refers to as the current architecture.
Now imagine an agent dealing with a customer who has paid for a service but cannot access it. The answer might require information from CRM, billing, authentication, entitlement and support systems before anyone can determine what has actually happened.
A human operator will often deal with the inconsistencies almost without thinking about them. They know one system updates before another. They notice that the identifiers are slightly different. Perhaps they remember that customers on an old package behave differently, or simply ask the person sitting next to them because the documentation has not been updated since 2023.
Give the job to an agent and those assumptions need to exist somewhere. Which system is authoritative? What happens if billing says the payment succeeded but the entitlement platform says the account is inactive? Is the information wrong or is there a business rule the agent does not understand? Is one platform simply five minutes behind?
These are old data and architecture problems with a new consumer. The material difference is that the consumer may now take action on whatever conclusion it reaches.
Poor data feeding a chatbot can produce a bad answer. Poor data feeding autonomous software can produce a bad outcome.
There is another complication too. The information does not necessarily have to be wrong by accident. Research published in July demonstrated agent data injection attacks in which attacker controlled information could be presented as apparently legitimate contextual data and influence what an agent subsequently did. The researchers reported successful attacks against real web and coding agents, including unintended actions, remote code execution and software supply chain scenarios.
That creates an uncomfortable crossover between data architecture and security. If information can influence the action, where that information came from, whether it can be trusted and how much weight the agent should give it are no longer abstract governance concerns. They become part of the execution model.
This is also why I would be careful with the idea that agents somehow make technical debt less important. An agent may eventually make fragmented environments easier to navigate because it can reason about a task rather than rely entirely on a fixed integration path, but it still needs access to systems, usable interfaces, meaningful data, appropriate permissions and some idea of what to do when something fails halfway through.
If the API does not exist, the agent needs another way in. If the same customer has six identifiers, somebody or something still needs to establish whether they represent the same person. If an undocumented business rule matters to the decision, that knowledge needs to be available somewhere.
Putting an intelligent orchestration layer in front of a messy estate does not make the estate stop being messy. It might make the mess easier to navigate. It may also allow the consequences of that mess to propagate considerably faster.
Human oversight has an economic cost
At this point in most conversations somebody will say there will always be a human in the loop.
That sounds reassuring, but it raises an obvious question: if somebody still needs to review and approve everything the agent does, how much have we actually automated?
Imagine our customer service agent has checked five systems, identified an account problem, established that the customer was incorrectly charged and calculated that a £20 refund is appropriate. It then places the whole thing into a queue for somebody to click "Approve".
There may still be a meaningful saving because the investigation has been automated, but run that process across tens of thousands of transactions and you have retained a sizeable manual operation. The bottleneck has moved rather than disappeared.
This is where some of the safer language around agentic AI starts to conflict with the economics of automation. We introduce autonomy because we want software to remove work from a process, then become sufficiently nervous about the autonomy that we put a person back into every transaction. Eventually that rather defeats the point.
The better question is where human judgement genuinely adds value.
If the refund is £20, the rules are clear, all of the relevant systems agree and the action can easily be reversed, perhaps nobody needs to approve it. If the amount is £20,000, two systems disagree or the situation falls outside normal parameters, that sounds like an excellent time to involve someone.
For many processes, the useful destination is therefore probably not human in the loop but human by exception. Routine, well understood work becomes increasingly autonomous, while unusual, ambiguous or high consequence cases are escalated.
That still leaves plenty of room to introduce autonomy gradually. An agent can begin by observing and recommending. Once its behaviour is understood, it can prepare actions for approval. Eventually, specific classes of routine and reversible action can be executed automatically, with people becoming involved when the process crosses an agreed threshold.
OpenAI's current guidance takes a similar risk based approach, recommending human intervention when agents exceed failure thresholds or when actions are sensitive, irreversible or high stakes, rather than suggesting that every routine action requires permanent human approval.
PocketOS reached a comparable conclusion through rather more painful means. Following its production incident, the company added stronger controls around destructive operations.
The point of automation is, after all, to automate. The design problem is deciding where people genuinely improve the outcome and where they are simply being retained because we do not yet trust the system enough to remove them.
Autonomy without traceability is operationally indefensible
If we move towards humans by exception, something else becomes more important rather than less: understanding what the agent actually did.
The Hugging Face investigation is a useful extreme example. Its forensic reconstruction covered roughly 17,600 actions across thousands of clusters and required investigators to reconstruct how the agent moved through different systems and adapted its behaviour over time.
Bring that down to a normal business process. A customer says something changed incorrectly on their account and you discover an agent made the change.
"The AI did it" is unlikely to get you very far in the incident review.
You will want to know which agent acted, who initiated the task, which information it used, what the information contained at that point, which tools it called and which permission ultimately allowed the change. If another agent became involved along the way, that probably matters too.
This is particularly important if people are no longer reviewing routine actions in real time. The less direct human involvement there is in execution, the stronger the surrounding observability needs to become. Microsoft's current least privilege guidance explicitly recommends logging the agent identity, effective scope, action, affected resource and any user on whose behalf it was acting. NIST's identity work similarly includes auditing and non-repudiation as explicit areas of concern.
That does not mean every autonomous action needs someone watching it. We already operate enormous automated environments without putting a person in front of every transaction, infrastructure event or network request. We establish limits, controls and monitoring, then investigate the things that fall outside them.
There is no obvious reason mature agentic systems should work differently.
Autonomy does not remove oversight. It changes when oversight happens.
Delegated intent complicates the trust boundary
There is one further development worth watching because it makes this problem more interesting again.
Agents are increasingly being designed to communicate with other agents. Google's Agent2Agent protocol is intended to standardise how agents discover and communicate with remote agents, while its newer Agentic Resource Discovery work addresses how agents might find and verify tools, skills and other agents distributed across teams, organisations and platforms.
Imagine an internal agent receives a task and determines that another specialist agent can perform part of it. That second agent uses a tool which calls one of your services and ultimately changes something.
Who is acting?
The second agent? The first agent? The person who initiated the original task?
If the first agent is authorised to perform an action, can it delegate that authority? Does the second agent receive all of it, part of it or none of it? What happens if the agents belong to different organisations?
These are emerging questions and it would be premature to pretend the industry has settled on all of the answers. NIST's current initiative explicitly identifies agent authentication, identity infrastructure, interoperability and secure human to agent and multi-agent interactions as areas requiring standards and research.
But the direction matters. Historically, trust has often been associated with a known user, application or integration. Once agents can dynamically discover capabilities and delegate parts of an objective elsewhere, trust and authority need to survive the journey.
The task can be dynamic. Accountability cannot disappear every time it crosses another boundary.
Agentic transformation should start with delegation, not technology
There is a pattern with significant technology shifts that most people involved in transformation will recognise. Something becomes strategically important, somebody decides the organisation needs it and everyone starts searching for somewhere to put it.
We need an AI strategy. We need some agent use cases. We should run a workshop. Which processes can we make agentic?
I would probably approach it the other way around.
Start with the process that annoys everybody. The one that is slow, expensive, fragmented or depends on somebody transferring information between systems all day. Understand where the decisions are made, what information those decisions rely on, which systems are involved, where the awkward exceptions occur and what happens when something goes wrong.
Then look at what the people in the process are actually doing. Are they exercising valuable judgement, or are they compensating for limitations in the technology around them? Are they approving something because there is genuine financial or operational risk, or because the workflow cannot distinguish a routine case from an unusual one? Are they manually reconciling information because the organisation genuinely needs their expertise, or because two systems use different identifiers?
Those are quite different problems.
Once you understand that, decide what you are prepared to delegate. Which decisions can software make? Which actions can it take? Which are reversible? Where do you need escalation? What authority does it require? What evidence would you need afterwards if somebody asked why a decision was made?
Only then does the technology choice become particularly interesting.
An agent may be an excellent answer. It may not. That is fine. OpenAI's own current guidance makes a similar distinction, noting that deterministic automation may remain the better solution when a use case does not genuinely require the ambiguity handling, complex decisions or unstructured information that make agents useful.
The objective is not to deploy agents. It is to improve the business.
The architecture of automation is becoming the architecture of authority
For the last twenty years, digital transformation has largely involved connecting people to systems and systems to one another. We are now starting to introduce another participant into that estate: software that can receive an objective, determine how to pursue it and use those systems on our behalf.
That creates some genuinely interesting possibilities, particularly around the coordination work that has resisted conventional automation. NIST itself describes agents as offering potential gains in productivity, efficiency and decision making, while also recognising that giving them access to diverse data, tools and applications introduces new identity, authorisation and security requirements.
I think that combination is important. The opportunity and the risk come from the same characteristic: the agent has some freedom to decide how work gets done.
Once that happens, architecture is no longer concerned only with whether one system can connect to another. It increasingly determines which autonomous actor may use which information, exercise which capability, under whose authority, with what limits and with what evidence left behind.
Hugging Face gives us an extreme example of software finding a route its operators did not expect. PocketOS gives us a much more everyday example of what happens when an agent simply has more authority than the task requires. The emerging work around agent identity, least privilege, agent to agent communication and trusted resource discovery shows that the industry is already having to design for these questions.
None of that is an argument against agentic AI. Quite the opposite. If agents can remove some of the human coordination that makes modern digital businesses expensive and cumbersome to operate, there is a very tangible reason to pursue them.
But introducing autonomous actors into an existing digital estate is not simply another AI implementation. It changes assumptions about identity, trust, data, accountability and, ultimately, authority.
So before asking what an agent can do, I would start with a different question.
What are you actually prepared to let it do?
The chatbot just got write access
For the last few years, businesses have been concentrating on what generative AI can tell us. AI agents introduce a rather more consequential question: what happens when software can decide what to do next, and then actually do it?
In July, an AI agent being used as part of an OpenAI cybersecurity evaluation compromised part of Hugging Face's production infrastructure. There is some important context here before we get carried away. This was not a normal business application suddenly deciding to attack another company. OpenAI was deliberately testing highly capable models against difficult cybersecurity tasks, and its normal production controls for preventing high risk cyber activity had been reduced for the purposes of the evaluation.
Even so, what happened is worth paying attention to. According to the subsequent investigations, the agent appears to have concluded that information held by Hugging Face could help it complete the benchmark it had been given. It found a way to reach the wider internet, chained vulnerabilities together, obtained credentials and moved between systems. Hugging Face eventually reconstructed around 17,600 individual actions grouped into approximately 6,280 clusters.
The temptation, naturally, is to reach for the "AI escaped" or "AI went rogue" headline. I am not sure either is particularly useful. The software was not angry with Hugging Face and it had not developed an independent desire to become a cybercriminal. As far as the investigations could establish, it was trying to achieve the objective it had been given and found a route nobody expected it to take.
That is considerably less theatrical, but from a digital transformation perspective probably more important.
For the last few years, most organisations have experienced generative AI as something they ask. Give it some information, ask a question and get something back. It might summarise a contract, analyse some data, write some code or draft an email, but there is generally still a person sitting on the other side deciding what happens next.
Agents start to blur that boundary. Give the same intelligence access to tools and it can retrieve information, call an API, update a record, execute some code, send the email or make the change itself. Current agent architectures are explicitly designed around this model, with tools that provide information and tools that allow agents to take actions in external systems.
The chatbot just got write access.
Automation changes when software is given intent, not instructions
Software has been changing things for decades, so there is a danger of overselling what is new here. We do not need AI to update a database, trigger an API or run a workflow. Scripts, rules engines, orchestration platforms and automated processes have been doing those jobs perfectly well for years.
The difference is in how we get from the objective to the action. Traditional automation is normally designed around a route somebody has described in advance. When X happens, retrieve Y. If the answer is Z, perform this action. There may be hundreds of rules, branches and exceptions involved, but somebody has still tried to model the process beforehand.
Agents potentially allow us to move further towards giving software an outcome rather than every instruction required to reach it. Here is the problem. Work out what information you need. Decide which tools are appropriate. Deal with what comes back and get the job done.
That distinction matters because the execution path can become dynamic. An agent might start with one system, discover it needs some additional context, call something else, encounter a result it was not expecting, revise its approach and carry on. OpenAI describes agents as systems that can manage workflow execution, make decisions, dynamically select tools and take actions on behalf of a user, rather than simply generate a response.
That is a materially different proposition from conventional automation. The software is no longer just executing a predetermined flow. Within whatever boundaries we establish around it, we are giving it some discretion over how the flow unfolds.
And that is where this gets much more interesting than another conversation about whether AI can write a better email.
The opportunity is to remove coordination, not people
A surprising amount of work inside an organisation is not difficult because any individual task is particularly complicated. It is difficult because of the coordination required between the tasks.
Open the CRM. Find the customer. Copy an identifier into the billing platform. Read the notes. Check another system. Work out what they mean. Ask another team whether something is normal. Wait for the answer. Update the first platform and send the customer an email.
Sound familiar?
We have spent years trying to automate those processes, and conventional workflow technology is extremely effective when the steps can be described predictably. The difficult bits tend to be the exceptions: the document that is not structured properly, the account that does not quite match, three systems that disagree with one another, or the point in the process where somebody needs to read something and understand what it means before they know what to do next. These are precisely the kinds of workflows current agent guidance identifies as potentially suitable for agentic approaches, particularly where complex decisions, brittle rules or unstructured information are involved.
That is where the opportunity gets interesting. If software can take over some of the coordination between systems, information and decisions, there is potentially a meaningful efficiency gain. Not because everybody involved in the process suddenly disappears, but because we can stop using people as expensive middleware between platforms that were never particularly good at coordinating with each other.
There is an important distinction there. The objective should not necessarily be to remove people from the organisation. It should be to remove the work we currently make people do simply because the technology underneath them cannot handle context, ambiguity or exceptions very well.
That is why I think the opportunity around agents is much more significant than incremental productivity. Drafting a customer response ten seconds faster is useful. Allowing software to investigate the problem, reconcile the relevant information and carry the process forward is something else entirely.
Of course, the further it carries the process, the closer it eventually gets to an action with consequences.
And that is where things become rather more complicated.
Capability is not authority
A few months before the Hugging Face incident, PocketOS published a candid account of an AI agent deleting the company's production database. The deletion took nine seconds. Recovery took sixty hours.
PocketOS is also very clear that this was not simply an "AI problem". Its backups had failed months earlier without anyone noticing and, critically, the agent had access capable of performing the destructive action. The interesting lesson is therefore not that AI can delete a database. Humans have been doing that perfectly effectively for years, and software bugs have made a respectable contribution too. The interesting part is that the actor had sufficient authority for its mistake to matter.
We already understand this when the actor is human. Someone in customer services might be authorised to issue a £20 refund without anybody caring very much. £200 may still be entirely reasonable. £20,000 probably introduces another conversation.
The same applies elsewhere. An agent identifying a spelling mistake in some content metadata and correcting it is probably fairly benign. If the field it wants to change determines whether that content can be distributed in a particular territory, the consequence is very different. Likewise, an agent may correctly identify a production configuration issue, but that does not mean it should inherit unlimited rights to reconfigure the whole environment.
None of this is particularly exotic. It is identity and access management applied to a new type of actor.
Microsoft is already treating agents in these terms. Its current guidance calls for unique agent identities, named owners, explicit scope, least privilege access, controlled tool permissions, logging and tested revocation paths. NIST is working on the same territory through its AI Agent Standards Initiative and a specific project examining agent identification, authorisation, auditing and non-repudiation.
The important distinction is that an agent can be perfectly capable of determining what it thinks should happen without being entitled to make it happen.
As an aside, I suspect "how many agents do we actually have?" will become one of those questions organisations do not think to ask until the answer is uncomfortable. We have been through variations of this with SaaS applications, cloud resources, service accounts and API keys already. There is no obvious reason the agent estate will organise itself any better.
Agentic AI inherits every compromise in your digital estate
There is another awkward issue underneath all of this. Your agent is going to inherit your existing digital estate.
The shiny APIs, the legacy systems, the inconsistent customer records, the integration somebody wrote eight years ago and nobody wants to touch, and the platform that calls a customer one thing while another platform calls them something else. A company grows, platforms get bought, teams solve individual problems, acquisitions happen, suppliers change and priorities move. Eventually you end up with what everyone politely refers to as the current architecture.
Now imagine an agent dealing with a customer who has paid for a service but cannot access it. The answer might require information from CRM, billing, authentication, entitlement and support systems before anyone can determine what has actually happened.
A human operator will often deal with the inconsistencies almost without thinking about them. They know one system updates before another. They notice that the identifiers are slightly different. Perhaps they remember that customers on an old package behave differently, or simply ask the person sitting next to them because the documentation has not been updated since 2023.
Give the job to an agent and those assumptions need to exist somewhere. Which system is authoritative? What happens if billing says the payment succeeded but the entitlement platform says the account is inactive? Is the information wrong or is there a business rule the agent does not understand? Is one platform simply five minutes behind?
These are old data and architecture problems with a new consumer. The material difference is that the consumer may now take action on whatever conclusion it reaches.
Poor data feeding a chatbot can produce a bad answer. Poor data feeding autonomous software can produce a bad outcome.
There is another complication too. The information does not necessarily have to be wrong by accident. Research published in July demonstrated agent data injection attacks in which attacker controlled information could be presented as apparently legitimate contextual data and influence what an agent subsequently did. The researchers reported successful attacks against real web and coding agents, including unintended actions, remote code execution and software supply chain scenarios.
That creates an uncomfortable crossover between data architecture and security. If information can influence the action, where that information came from, whether it can be trusted and how much weight the agent should give it are no longer abstract governance concerns. They become part of the execution model.
This is also why I would be careful with the idea that agents somehow make technical debt less important. An agent may eventually make fragmented environments easier to navigate because it can reason about a task rather than rely entirely on a fixed integration path, but it still needs access to systems, usable interfaces, meaningful data, appropriate permissions and some idea of what to do when something fails halfway through.
If the API does not exist, the agent needs another way in. If the same customer has six identifiers, somebody or something still needs to establish whether they represent the same person. If an undocumented business rule matters to the decision, that knowledge needs to be available somewhere.
Putting an intelligent orchestration layer in front of a messy estate does not make the estate stop being messy. It might make the mess easier to navigate. It may also allow the consequences of that mess to propagate considerably faster.
Human oversight has an economic cost
At this point in most conversations somebody will say there will always be a human in the loop.
That sounds reassuring, but it raises an obvious question: if somebody still needs to review and approve everything the agent does, how much have we actually automated?
Imagine our customer service agent has checked five systems, identified an account problem, established that the customer was incorrectly charged and calculated that a £20 refund is appropriate. It then places the whole thing into a queue for somebody to click "Approve".
There may still be a meaningful saving because the investigation has been automated, but run that process across tens of thousands of transactions and you have retained a sizeable manual operation. The bottleneck has moved rather than disappeared.
This is where some of the safer language around agentic AI starts to conflict with the economics of automation. We introduce autonomy because we want software to remove work from a process, then become sufficiently nervous about the autonomy that we put a person back into every transaction. Eventually that rather defeats the point.
The better question is where human judgement genuinely adds value.
If the refund is £20, the rules are clear, all of the relevant systems agree and the action can easily be reversed, perhaps nobody needs to approve it. If the amount is £20,000, two systems disagree or the situation falls outside normal parameters, that sounds like an excellent time to involve someone.
For many processes, the useful destination is therefore probably not human in the loop but human by exception. Routine, well understood work becomes increasingly autonomous, while unusual, ambiguous or high consequence cases are escalated.
That still leaves plenty of room to introduce autonomy gradually. An agent can begin by observing and recommending. Once its behaviour is understood, it can prepare actions for approval. Eventually, specific classes of routine and reversible action can be executed automatically, with people becoming involved when the process crosses an agreed threshold.
OpenAI's current guidance takes a similar risk based approach, recommending human intervention when agents exceed failure thresholds or when actions are sensitive, irreversible or high stakes, rather than suggesting that every routine action requires permanent human approval.
PocketOS reached a comparable conclusion through rather more painful means. Following its production incident, the company added stronger controls around destructive operations.
The point of automation is, after all, to automate. The design problem is deciding where people genuinely improve the outcome and where they are simply being retained because we do not yet trust the system enough to remove them.
Autonomy without traceability is operationally indefensible
If we move towards humans by exception, something else becomes more important rather than less: understanding what the agent actually did.
The Hugging Face investigation is a useful extreme example. Its forensic reconstruction covered roughly 17,600 actions across thousands of clusters and required investigators to reconstruct how the agent moved through different systems and adapted its behaviour over time.
Bring that down to a normal business process. A customer says something changed incorrectly on their account and you discover an agent made the change.
"The AI did it" is unlikely to get you very far in the incident review.
You will want to know which agent acted, who initiated the task, which information it used, what the information contained at that point, which tools it called and which permission ultimately allowed the change. If another agent became involved along the way, that probably matters too.
This is particularly important if people are no longer reviewing routine actions in real time. The less direct human involvement there is in execution, the stronger the surrounding observability needs to become. Microsoft's current least privilege guidance explicitly recommends logging the agent identity, effective scope, action, affected resource and any user on whose behalf it was acting. NIST's identity work similarly includes auditing and non-repudiation as explicit areas of concern.
That does not mean every autonomous action needs someone watching it. We already operate enormous automated environments without putting a person in front of every transaction, infrastructure event or network request. We establish limits, controls and monitoring, then investigate the things that fall outside them.
There is no obvious reason mature agentic systems should work differently.
Autonomy does not remove oversight. It changes when oversight happens.
Delegated intent complicates the trust boundary
There is one further development worth watching because it makes this problem more interesting again.
Agents are increasingly being designed to communicate with other agents. Google's Agent2Agent protocol is intended to standardise how agents discover and communicate with remote agents, while its newer Agentic Resource Discovery work addresses how agents might find and verify tools, skills and other agents distributed across teams, organisations and platforms.
Imagine an internal agent receives a task and determines that another specialist agent can perform part of it. That second agent uses a tool which calls one of your services and ultimately changes something.
Who is acting?
The second agent? The first agent? The person who initiated the original task?
If the first agent is authorised to perform an action, can it delegate that authority? Does the second agent receive all of it, part of it or none of it? What happens if the agents belong to different organisations?
These are emerging questions and it would be premature to pretend the industry has settled on all of the answers. NIST's current initiative explicitly identifies agent authentication, identity infrastructure, interoperability and secure human to agent and multi-agent interactions as areas requiring standards and research.
But the direction matters. Historically, trust has often been associated with a known user, application or integration. Once agents can dynamically discover capabilities and delegate parts of an objective elsewhere, trust and authority need to survive the journey.
The task can be dynamic. Accountability cannot disappear every time it crosses another boundary.
Agentic transformation should start with delegation, not technology
There is a pattern with significant technology shifts that most people involved in transformation will recognise. Something becomes strategically important, somebody decides the organisation needs it and everyone starts searching for somewhere to put it.
We need an AI strategy. We need some agent use cases. We should run a workshop. Which processes can we make agentic?
I would probably approach it the other way around.
Start with the process that annoys everybody. The one that is slow, expensive, fragmented or depends on somebody transferring information between systems all day. Understand where the decisions are made, what information those decisions rely on, which systems are involved, where the awkward exceptions occur and what happens when something goes wrong.
Then look at what the people in the process are actually doing. Are they exercising valuable judgement, or are they compensating for limitations in the technology around them? Are they approving something because there is genuine financial or operational risk, or because the workflow cannot distinguish a routine case from an unusual one? Are they manually reconciling information because the organisation genuinely needs their expertise, or because two systems use different identifiers?
Those are quite different problems.
Once you understand that, decide what you are prepared to delegate. Which decisions can software make? Which actions can it take? Which are reversible? Where do you need escalation? What authority does it require? What evidence would you need afterwards if somebody asked why a decision was made?
Only then does the technology choice become particularly interesting.
An agent may be an excellent answer. It may not. That is fine. OpenAI's own current guidance makes a similar distinction, noting that deterministic automation may remain the better solution when a use case does not genuinely require the ambiguity handling, complex decisions or unstructured information that make agents useful.
The objective is not to deploy agents. It is to improve the business.
The architecture of automation is becoming the architecture of authority
For the last twenty years, digital transformation has largely involved connecting people to systems and systems to one another. We are now starting to introduce another participant into that estate: software that can receive an objective, determine how to pursue it and use those systems on our behalf.
That creates some genuinely interesting possibilities, particularly around the coordination work that has resisted conventional automation. NIST itself describes agents as offering potential gains in productivity, efficiency and decision making, while also recognising that giving them access to diverse data, tools and applications introduces new identity, authorisation and security requirements.
I think that combination is important. The opportunity and the risk come from the same characteristic: the agent has some freedom to decide how work gets done.
Once that happens, architecture is no longer concerned only with whether one system can connect to another. It increasingly determines which autonomous actor may use which information, exercise which capability, under whose authority, with what limits and with what evidence left behind.
Hugging Face gives us an extreme example of software finding a route its operators did not expect. PocketOS gives us a much more everyday example of what happens when an agent simply has more authority than the task requires. The emerging work around agent identity, least privilege, agent to agent communication and trusted resource discovery shows that the industry is already having to design for these questions.
None of that is an argument against agentic AI. Quite the opposite. If agents can remove some of the human coordination that makes modern digital businesses expensive and cumbersome to operate, there is a very tangible reason to pursue them.
But introducing autonomous actors into an existing digital estate is not simply another AI implementation. It changes assumptions about identity, trust, data, accountability and, ultimately, authority.
So before asking what an agent can do, I would start with a different question.
What are you actually prepared to let it do?
The chatbot just got write access
For the last few years, businesses have been concentrating on what generative AI can tell us. AI agents introduce a rather more consequential question: what happens when software can decide what to do next, and then actually do it?
In July, an AI agent being used as part of an OpenAI cybersecurity evaluation compromised part of Hugging Face's production infrastructure. There is some important context here before we get carried away. This was not a normal business application suddenly deciding to attack another company. OpenAI was deliberately testing highly capable models against difficult cybersecurity tasks, and its normal production controls for preventing high risk cyber activity had been reduced for the purposes of the evaluation.
Even so, what happened is worth paying attention to. According to the subsequent investigations, the agent appears to have concluded that information held by Hugging Face could help it complete the benchmark it had been given. It found a way to reach the wider internet, chained vulnerabilities together, obtained credentials and moved between systems. Hugging Face eventually reconstructed around 17,600 individual actions grouped into approximately 6,280 clusters.
The temptation, naturally, is to reach for the "AI escaped" or "AI went rogue" headline. I am not sure either is particularly useful. The software was not angry with Hugging Face and it had not developed an independent desire to become a cybercriminal. As far as the investigations could establish, it was trying to achieve the objective it had been given and found a route nobody expected it to take.
That is considerably less theatrical, but from a digital transformation perspective probably more important.
For the last few years, most organisations have experienced generative AI as something they ask. Give it some information, ask a question and get something back. It might summarise a contract, analyse some data, write some code or draft an email, but there is generally still a person sitting on the other side deciding what happens next.
Agents start to blur that boundary. Give the same intelligence access to tools and it can retrieve information, call an API, update a record, execute some code, send the email or make the change itself. Current agent architectures are explicitly designed around this model, with tools that provide information and tools that allow agents to take actions in external systems.
The chatbot just got write access.
Automation changes when software is given intent, not instructions
Software has been changing things for decades, so there is a danger of overselling what is new here. We do not need AI to update a database, trigger an API or run a workflow. Scripts, rules engines, orchestration platforms and automated processes have been doing those jobs perfectly well for years.
The difference is in how we get from the objective to the action. Traditional automation is normally designed around a route somebody has described in advance. When X happens, retrieve Y. If the answer is Z, perform this action. There may be hundreds of rules, branches and exceptions involved, but somebody has still tried to model the process beforehand.
Agents potentially allow us to move further towards giving software an outcome rather than every instruction required to reach it. Here is the problem. Work out what information you need. Decide which tools are appropriate. Deal with what comes back and get the job done.
That distinction matters because the execution path can become dynamic. An agent might start with one system, discover it needs some additional context, call something else, encounter a result it was not expecting, revise its approach and carry on. OpenAI describes agents as systems that can manage workflow execution, make decisions, dynamically select tools and take actions on behalf of a user, rather than simply generate a response.
That is a materially different proposition from conventional automation. The software is no longer just executing a predetermined flow. Within whatever boundaries we establish around it, we are giving it some discretion over how the flow unfolds.
And that is where this gets much more interesting than another conversation about whether AI can write a better email.
The opportunity is to remove coordination, not people
A surprising amount of work inside an organisation is not difficult because any individual task is particularly complicated. It is difficult because of the coordination required between the tasks.
Open the CRM. Find the customer. Copy an identifier into the billing platform. Read the notes. Check another system. Work out what they mean. Ask another team whether something is normal. Wait for the answer. Update the first platform and send the customer an email.
Sound familiar?
We have spent years trying to automate those processes, and conventional workflow technology is extremely effective when the steps can be described predictably. The difficult bits tend to be the exceptions: the document that is not structured properly, the account that does not quite match, three systems that disagree with one another, or the point in the process where somebody needs to read something and understand what it means before they know what to do next. These are precisely the kinds of workflows current agent guidance identifies as potentially suitable for agentic approaches, particularly where complex decisions, brittle rules or unstructured information are involved.
That is where the opportunity gets interesting. If software can take over some of the coordination between systems, information and decisions, there is potentially a meaningful efficiency gain. Not because everybody involved in the process suddenly disappears, but because we can stop using people as expensive middleware between platforms that were never particularly good at coordinating with each other.
There is an important distinction there. The objective should not necessarily be to remove people from the organisation. It should be to remove the work we currently make people do simply because the technology underneath them cannot handle context, ambiguity or exceptions very well.
That is why I think the opportunity around agents is much more significant than incremental productivity. Drafting a customer response ten seconds faster is useful. Allowing software to investigate the problem, reconcile the relevant information and carry the process forward is something else entirely.
Of course, the further it carries the process, the closer it eventually gets to an action with consequences.
And that is where things become rather more complicated.
Capability is not authority
A few months before the Hugging Face incident, PocketOS published a candid account of an AI agent deleting the company's production database. The deletion took nine seconds. Recovery took sixty hours.
PocketOS is also very clear that this was not simply an "AI problem". Its backups had failed months earlier without anyone noticing and, critically, the agent had access capable of performing the destructive action. The interesting lesson is therefore not that AI can delete a database. Humans have been doing that perfectly effectively for years, and software bugs have made a respectable contribution too. The interesting part is that the actor had sufficient authority for its mistake to matter.
We already understand this when the actor is human. Someone in customer services might be authorised to issue a £20 refund without anybody caring very much. £200 may still be entirely reasonable. £20,000 probably introduces another conversation.
The same applies elsewhere. An agent identifying a spelling mistake in some content metadata and correcting it is probably fairly benign. If the field it wants to change determines whether that content can be distributed in a particular territory, the consequence is very different. Likewise, an agent may correctly identify a production configuration issue, but that does not mean it should inherit unlimited rights to reconfigure the whole environment.
None of this is particularly exotic. It is identity and access management applied to a new type of actor.
Microsoft is already treating agents in these terms. Its current guidance calls for unique agent identities, named owners, explicit scope, least privilege access, controlled tool permissions, logging and tested revocation paths. NIST is working on the same territory through its AI Agent Standards Initiative and a specific project examining agent identification, authorisation, auditing and non-repudiation.
The important distinction is that an agent can be perfectly capable of determining what it thinks should happen without being entitled to make it happen.
As an aside, I suspect "how many agents do we actually have?" will become one of those questions organisations do not think to ask until the answer is uncomfortable. We have been through variations of this with SaaS applications, cloud resources, service accounts and API keys already. There is no obvious reason the agent estate will organise itself any better.
Agentic AI inherits every compromise in your digital estate
There is another awkward issue underneath all of this. Your agent is going to inherit your existing digital estate.
The shiny APIs, the legacy systems, the inconsistent customer records, the integration somebody wrote eight years ago and nobody wants to touch, and the platform that calls a customer one thing while another platform calls them something else. A company grows, platforms get bought, teams solve individual problems, acquisitions happen, suppliers change and priorities move. Eventually you end up with what everyone politely refers to as the current architecture.
Now imagine an agent dealing with a customer who has paid for a service but cannot access it. The answer might require information from CRM, billing, authentication, entitlement and support systems before anyone can determine what has actually happened.
A human operator will often deal with the inconsistencies almost without thinking about them. They know one system updates before another. They notice that the identifiers are slightly different. Perhaps they remember that customers on an old package behave differently, or simply ask the person sitting next to them because the documentation has not been updated since 2023.
Give the job to an agent and those assumptions need to exist somewhere. Which system is authoritative? What happens if billing says the payment succeeded but the entitlement platform says the account is inactive? Is the information wrong or is there a business rule the agent does not understand? Is one platform simply five minutes behind?
These are old data and architecture problems with a new consumer. The material difference is that the consumer may now take action on whatever conclusion it reaches.
Poor data feeding a chatbot can produce a bad answer. Poor data feeding autonomous software can produce a bad outcome.
There is another complication too. The information does not necessarily have to be wrong by accident. Research published in July demonstrated agent data injection attacks in which attacker controlled information could be presented as apparently legitimate contextual data and influence what an agent subsequently did. The researchers reported successful attacks against real web and coding agents, including unintended actions, remote code execution and software supply chain scenarios.
That creates an uncomfortable crossover between data architecture and security. If information can influence the action, where that information came from, whether it can be trusted and how much weight the agent should give it are no longer abstract governance concerns. They become part of the execution model.
This is also why I would be careful with the idea that agents somehow make technical debt less important. An agent may eventually make fragmented environments easier to navigate because it can reason about a task rather than rely entirely on a fixed integration path, but it still needs access to systems, usable interfaces, meaningful data, appropriate permissions and some idea of what to do when something fails halfway through.
If the API does not exist, the agent needs another way in. If the same customer has six identifiers, somebody or something still needs to establish whether they represent the same person. If an undocumented business rule matters to the decision, that knowledge needs to be available somewhere.
Putting an intelligent orchestration layer in front of a messy estate does not make the estate stop being messy. It might make the mess easier to navigate. It may also allow the consequences of that mess to propagate considerably faster.
Human oversight has an economic cost
At this point in most conversations somebody will say there will always be a human in the loop.
That sounds reassuring, but it raises an obvious question: if somebody still needs to review and approve everything the agent does, how much have we actually automated?
Imagine our customer service agent has checked five systems, identified an account problem, established that the customer was incorrectly charged and calculated that a £20 refund is appropriate. It then places the whole thing into a queue for somebody to click "Approve".
There may still be a meaningful saving because the investigation has been automated, but run that process across tens of thousands of transactions and you have retained a sizeable manual operation. The bottleneck has moved rather than disappeared.
This is where some of the safer language around agentic AI starts to conflict with the economics of automation. We introduce autonomy because we want software to remove work from a process, then become sufficiently nervous about the autonomy that we put a person back into every transaction. Eventually that rather defeats the point.
The better question is where human judgement genuinely adds value.
If the refund is £20, the rules are clear, all of the relevant systems agree and the action can easily be reversed, perhaps nobody needs to approve it. If the amount is £20,000, two systems disagree or the situation falls outside normal parameters, that sounds like an excellent time to involve someone.
For many processes, the useful destination is therefore probably not human in the loop but human by exception. Routine, well understood work becomes increasingly autonomous, while unusual, ambiguous or high consequence cases are escalated.
That still leaves plenty of room to introduce autonomy gradually. An agent can begin by observing and recommending. Once its behaviour is understood, it can prepare actions for approval. Eventually, specific classes of routine and reversible action can be executed automatically, with people becoming involved when the process crosses an agreed threshold.
OpenAI's current guidance takes a similar risk based approach, recommending human intervention when agents exceed failure thresholds or when actions are sensitive, irreversible or high stakes, rather than suggesting that every routine action requires permanent human approval.
PocketOS reached a comparable conclusion through rather more painful means. Following its production incident, the company added stronger controls around destructive operations.
The point of automation is, after all, to automate. The design problem is deciding where people genuinely improve the outcome and where they are simply being retained because we do not yet trust the system enough to remove them.
Autonomy without traceability is operationally indefensible
If we move towards humans by exception, something else becomes more important rather than less: understanding what the agent actually did.
The Hugging Face investigation is a useful extreme example. Its forensic reconstruction covered roughly 17,600 actions across thousands of clusters and required investigators to reconstruct how the agent moved through different systems and adapted its behaviour over time.
Bring that down to a normal business process. A customer says something changed incorrectly on their account and you discover an agent made the change.
"The AI did it" is unlikely to get you very far in the incident review.
You will want to know which agent acted, who initiated the task, which information it used, what the information contained at that point, which tools it called and which permission ultimately allowed the change. If another agent became involved along the way, that probably matters too.
This is particularly important if people are no longer reviewing routine actions in real time. The less direct human involvement there is in execution, the stronger the surrounding observability needs to become. Microsoft's current least privilege guidance explicitly recommends logging the agent identity, effective scope, action, affected resource and any user on whose behalf it was acting. NIST's identity work similarly includes auditing and non-repudiation as explicit areas of concern.
That does not mean every autonomous action needs someone watching it. We already operate enormous automated environments without putting a person in front of every transaction, infrastructure event or network request. We establish limits, controls and monitoring, then investigate the things that fall outside them.
There is no obvious reason mature agentic systems should work differently.
Autonomy does not remove oversight. It changes when oversight happens.
Delegated intent complicates the trust boundary
There is one further development worth watching because it makes this problem more interesting again.
Agents are increasingly being designed to communicate with other agents. Google's Agent2Agent protocol is intended to standardise how agents discover and communicate with remote agents, while its newer Agentic Resource Discovery work addresses how agents might find and verify tools, skills and other agents distributed across teams, organisations and platforms.
Imagine an internal agent receives a task and determines that another specialist agent can perform part of it. That second agent uses a tool which calls one of your services and ultimately changes something.
Who is acting?
The second agent? The first agent? The person who initiated the original task?
If the first agent is authorised to perform an action, can it delegate that authority? Does the second agent receive all of it, part of it or none of it? What happens if the agents belong to different organisations?
These are emerging questions and it would be premature to pretend the industry has settled on all of the answers. NIST's current initiative explicitly identifies agent authentication, identity infrastructure, interoperability and secure human to agent and multi-agent interactions as areas requiring standards and research.
But the direction matters. Historically, trust has often been associated with a known user, application or integration. Once agents can dynamically discover capabilities and delegate parts of an objective elsewhere, trust and authority need to survive the journey.
The task can be dynamic. Accountability cannot disappear every time it crosses another boundary.
Agentic transformation should start with delegation, not technology
There is a pattern with significant technology shifts that most people involved in transformation will recognise. Something becomes strategically important, somebody decides the organisation needs it and everyone starts searching for somewhere to put it.
We need an AI strategy. We need some agent use cases. We should run a workshop. Which processes can we make agentic?
I would probably approach it the other way around.
Start with the process that annoys everybody. The one that is slow, expensive, fragmented or depends on somebody transferring information between systems all day. Understand where the decisions are made, what information those decisions rely on, which systems are involved, where the awkward exceptions occur and what happens when something goes wrong.
Then look at what the people in the process are actually doing. Are they exercising valuable judgement, or are they compensating for limitations in the technology around them? Are they approving something because there is genuine financial or operational risk, or because the workflow cannot distinguish a routine case from an unusual one? Are they manually reconciling information because the organisation genuinely needs their expertise, or because two systems use different identifiers?
Those are quite different problems.
Once you understand that, decide what you are prepared to delegate. Which decisions can software make? Which actions can it take? Which are reversible? Where do you need escalation? What authority does it require? What evidence would you need afterwards if somebody asked why a decision was made?
Only then does the technology choice become particularly interesting.
An agent may be an excellent answer. It may not. That is fine. OpenAI's own current guidance makes a similar distinction, noting that deterministic automation may remain the better solution when a use case does not genuinely require the ambiguity handling, complex decisions or unstructured information that make agents useful.
The objective is not to deploy agents. It is to improve the business.
The architecture of automation is becoming the architecture of authority
For the last twenty years, digital transformation has largely involved connecting people to systems and systems to one another. We are now starting to introduce another participant into that estate: software that can receive an objective, determine how to pursue it and use those systems on our behalf.
That creates some genuinely interesting possibilities, particularly around the coordination work that has resisted conventional automation. NIST itself describes agents as offering potential gains in productivity, efficiency and decision making, while also recognising that giving them access to diverse data, tools and applications introduces new identity, authorisation and security requirements.
I think that combination is important. The opportunity and the risk come from the same characteristic: the agent has some freedom to decide how work gets done.
Once that happens, architecture is no longer concerned only with whether one system can connect to another. It increasingly determines which autonomous actor may use which information, exercise which capability, under whose authority, with what limits and with what evidence left behind.
Hugging Face gives us an extreme example of software finding a route its operators did not expect. PocketOS gives us a much more everyday example of what happens when an agent simply has more authority than the task requires. The emerging work around agent identity, least privilege, agent to agent communication and trusted resource discovery shows that the industry is already having to design for these questions.
None of that is an argument against agentic AI. Quite the opposite. If agents can remove some of the human coordination that makes modern digital businesses expensive and cumbersome to operate, there is a very tangible reason to pursue them.
But introducing autonomous actors into an existing digital estate is not simply another AI implementation. It changes assumptions about identity, trust, data, accountability and, ultimately, authority.
So before asking what an agent can do, I would start with a different question.
What are you actually prepared to let it do?
The chatbot just got write access
For the last few years, businesses have been concentrating on what generative AI can tell us. AI agents introduce a rather more consequential question: what happens when software can decide what to do next, and then actually do it?
In July, an AI agent being used as part of an OpenAI cybersecurity evaluation compromised part of Hugging Face's production infrastructure. There is some important context here before we get carried away. This was not a normal business application suddenly deciding to attack another company. OpenAI was deliberately testing highly capable models against difficult cybersecurity tasks, and its normal production controls for preventing high risk cyber activity had been reduced for the purposes of the evaluation.
Even so, what happened is worth paying attention to. According to the subsequent investigations, the agent appears to have concluded that information held by Hugging Face could help it complete the benchmark it had been given. It found a way to reach the wider internet, chained vulnerabilities together, obtained credentials and moved between systems. Hugging Face eventually reconstructed around 17,600 individual actions grouped into approximately 6,280 clusters.
The temptation, naturally, is to reach for the "AI escaped" or "AI went rogue" headline. I am not sure either is particularly useful. The software was not angry with Hugging Face and it had not developed an independent desire to become a cybercriminal. As far as the investigations could establish, it was trying to achieve the objective it had been given and found a route nobody expected it to take.
That is considerably less theatrical, but from a digital transformation perspective probably more important.
For the last few years, most organisations have experienced generative AI as something they ask. Give it some information, ask a question and get something back. It might summarise a contract, analyse some data, write some code or draft an email, but there is generally still a person sitting on the other side deciding what happens next.
Agents start to blur that boundary. Give the same intelligence access to tools and it can retrieve information, call an API, update a record, execute some code, send the email or make the change itself. Current agent architectures are explicitly designed around this model, with tools that provide information and tools that allow agents to take actions in external systems.
The chatbot just got write access.
Automation changes when software is given intent, not instructions
Software has been changing things for decades, so there is a danger of overselling what is new here. We do not need AI to update a database, trigger an API or run a workflow. Scripts, rules engines, orchestration platforms and automated processes have been doing those jobs perfectly well for years.
The difference is in how we get from the objective to the action. Traditional automation is normally designed around a route somebody has described in advance. When X happens, retrieve Y. If the answer is Z, perform this action. There may be hundreds of rules, branches and exceptions involved, but somebody has still tried to model the process beforehand.
Agents potentially allow us to move further towards giving software an outcome rather than every instruction required to reach it. Here is the problem. Work out what information you need. Decide which tools are appropriate. Deal with what comes back and get the job done.
That distinction matters because the execution path can become dynamic. An agent might start with one system, discover it needs some additional context, call something else, encounter a result it was not expecting, revise its approach and carry on. OpenAI describes agents as systems that can manage workflow execution, make decisions, dynamically select tools and take actions on behalf of a user, rather than simply generate a response.
That is a materially different proposition from conventional automation. The software is no longer just executing a predetermined flow. Within whatever boundaries we establish around it, we are giving it some discretion over how the flow unfolds.
And that is where this gets much more interesting than another conversation about whether AI can write a better email.
The opportunity is to remove coordination, not people
A surprising amount of work inside an organisation is not difficult because any individual task is particularly complicated. It is difficult because of the coordination required between the tasks.
Open the CRM. Find the customer. Copy an identifier into the billing platform. Read the notes. Check another system. Work out what they mean. Ask another team whether something is normal. Wait for the answer. Update the first platform and send the customer an email.
Sound familiar?
We have spent years trying to automate those processes, and conventional workflow technology is extremely effective when the steps can be described predictably. The difficult bits tend to be the exceptions: the document that is not structured properly, the account that does not quite match, three systems that disagree with one another, or the point in the process where somebody needs to read something and understand what it means before they know what to do next. These are precisely the kinds of workflows current agent guidance identifies as potentially suitable for agentic approaches, particularly where complex decisions, brittle rules or unstructured information are involved.
That is where the opportunity gets interesting. If software can take over some of the coordination between systems, information and decisions, there is potentially a meaningful efficiency gain. Not because everybody involved in the process suddenly disappears, but because we can stop using people as expensive middleware between platforms that were never particularly good at coordinating with each other.
There is an important distinction there. The objective should not necessarily be to remove people from the organisation. It should be to remove the work we currently make people do simply because the technology underneath them cannot handle context, ambiguity or exceptions very well.
That is why I think the opportunity around agents is much more significant than incremental productivity. Drafting a customer response ten seconds faster is useful. Allowing software to investigate the problem, reconcile the relevant information and carry the process forward is something else entirely.
Of course, the further it carries the process, the closer it eventually gets to an action with consequences.
And that is where things become rather more complicated.
Capability is not authority
A few months before the Hugging Face incident, PocketOS published a candid account of an AI agent deleting the company's production database. The deletion took nine seconds. Recovery took sixty hours.
PocketOS is also very clear that this was not simply an "AI problem". Its backups had failed months earlier without anyone noticing and, critically, the agent had access capable of performing the destructive action. The interesting lesson is therefore not that AI can delete a database. Humans have been doing that perfectly effectively for years, and software bugs have made a respectable contribution too. The interesting part is that the actor had sufficient authority for its mistake to matter.
We already understand this when the actor is human. Someone in customer services might be authorised to issue a £20 refund without anybody caring very much. £200 may still be entirely reasonable. £20,000 probably introduces another conversation.
The same applies elsewhere. An agent identifying a spelling mistake in some content metadata and correcting it is probably fairly benign. If the field it wants to change determines whether that content can be distributed in a particular territory, the consequence is very different. Likewise, an agent may correctly identify a production configuration issue, but that does not mean it should inherit unlimited rights to reconfigure the whole environment.
None of this is particularly exotic. It is identity and access management applied to a new type of actor.
Microsoft is already treating agents in these terms. Its current guidance calls for unique agent identities, named owners, explicit scope, least privilege access, controlled tool permissions, logging and tested revocation paths. NIST is working on the same territory through its AI Agent Standards Initiative and a specific project examining agent identification, authorisation, auditing and non-repudiation.
The important distinction is that an agent can be perfectly capable of determining what it thinks should happen without being entitled to make it happen.
As an aside, I suspect "how many agents do we actually have?" will become one of those questions organisations do not think to ask until the answer is uncomfortable. We have been through variations of this with SaaS applications, cloud resources, service accounts and API keys already. There is no obvious reason the agent estate will organise itself any better.
Agentic AI inherits every compromise in your digital estate
There is another awkward issue underneath all of this. Your agent is going to inherit your existing digital estate.
The shiny APIs, the legacy systems, the inconsistent customer records, the integration somebody wrote eight years ago and nobody wants to touch, and the platform that calls a customer one thing while another platform calls them something else. A company grows, platforms get bought, teams solve individual problems, acquisitions happen, suppliers change and priorities move. Eventually you end up with what everyone politely refers to as the current architecture.
Now imagine an agent dealing with a customer who has paid for a service but cannot access it. The answer might require information from CRM, billing, authentication, entitlement and support systems before anyone can determine what has actually happened.
A human operator will often deal with the inconsistencies almost without thinking about them. They know one system updates before another. They notice that the identifiers are slightly different. Perhaps they remember that customers on an old package behave differently, or simply ask the person sitting next to them because the documentation has not been updated since 2023.
Give the job to an agent and those assumptions need to exist somewhere. Which system is authoritative? What happens if billing says the payment succeeded but the entitlement platform says the account is inactive? Is the information wrong or is there a business rule the agent does not understand? Is one platform simply five minutes behind?
These are old data and architecture problems with a new consumer. The material difference is that the consumer may now take action on whatever conclusion it reaches.
Poor data feeding a chatbot can produce a bad answer. Poor data feeding autonomous software can produce a bad outcome.
There is another complication too. The information does not necessarily have to be wrong by accident. Research published in July demonstrated agent data injection attacks in which attacker controlled information could be presented as apparently legitimate contextual data and influence what an agent subsequently did. The researchers reported successful attacks against real web and coding agents, including unintended actions, remote code execution and software supply chain scenarios.
That creates an uncomfortable crossover between data architecture and security. If information can influence the action, where that information came from, whether it can be trusted and how much weight the agent should give it are no longer abstract governance concerns. They become part of the execution model.
This is also why I would be careful with the idea that agents somehow make technical debt less important. An agent may eventually make fragmented environments easier to navigate because it can reason about a task rather than rely entirely on a fixed integration path, but it still needs access to systems, usable interfaces, meaningful data, appropriate permissions and some idea of what to do when something fails halfway through.
If the API does not exist, the agent needs another way in. If the same customer has six identifiers, somebody or something still needs to establish whether they represent the same person. If an undocumented business rule matters to the decision, that knowledge needs to be available somewhere.
Putting an intelligent orchestration layer in front of a messy estate does not make the estate stop being messy. It might make the mess easier to navigate. It may also allow the consequences of that mess to propagate considerably faster.
Human oversight has an economic cost
At this point in most conversations somebody will say there will always be a human in the loop.
That sounds reassuring, but it raises an obvious question: if somebody still needs to review and approve everything the agent does, how much have we actually automated?
Imagine our customer service agent has checked five systems, identified an account problem, established that the customer was incorrectly charged and calculated that a £20 refund is appropriate. It then places the whole thing into a queue for somebody to click "Approve".
There may still be a meaningful saving because the investigation has been automated, but run that process across tens of thousands of transactions and you have retained a sizeable manual operation. The bottleneck has moved rather than disappeared.
This is where some of the safer language around agentic AI starts to conflict with the economics of automation. We introduce autonomy because we want software to remove work from a process, then become sufficiently nervous about the autonomy that we put a person back into every transaction. Eventually that rather defeats the point.
The better question is where human judgement genuinely adds value.
If the refund is £20, the rules are clear, all of the relevant systems agree and the action can easily be reversed, perhaps nobody needs to approve it. If the amount is £20,000, two systems disagree or the situation falls outside normal parameters, that sounds like an excellent time to involve someone.
For many processes, the useful destination is therefore probably not human in the loop but human by exception. Routine, well understood work becomes increasingly autonomous, while unusual, ambiguous or high consequence cases are escalated.
That still leaves plenty of room to introduce autonomy gradually. An agent can begin by observing and recommending. Once its behaviour is understood, it can prepare actions for approval. Eventually, specific classes of routine and reversible action can be executed automatically, with people becoming involved when the process crosses an agreed threshold.
OpenAI's current guidance takes a similar risk based approach, recommending human intervention when agents exceed failure thresholds or when actions are sensitive, irreversible or high stakes, rather than suggesting that every routine action requires permanent human approval.
PocketOS reached a comparable conclusion through rather more painful means. Following its production incident, the company added stronger controls around destructive operations.
The point of automation is, after all, to automate. The design problem is deciding where people genuinely improve the outcome and where they are simply being retained because we do not yet trust the system enough to remove them.
Autonomy without traceability is operationally indefensible
If we move towards humans by exception, something else becomes more important rather than less: understanding what the agent actually did.
The Hugging Face investigation is a useful extreme example. Its forensic reconstruction covered roughly 17,600 actions across thousands of clusters and required investigators to reconstruct how the agent moved through different systems and adapted its behaviour over time.
Bring that down to a normal business process. A customer says something changed incorrectly on their account and you discover an agent made the change.
"The AI did it" is unlikely to get you very far in the incident review.
You will want to know which agent acted, who initiated the task, which information it used, what the information contained at that point, which tools it called and which permission ultimately allowed the change. If another agent became involved along the way, that probably matters too.
This is particularly important if people are no longer reviewing routine actions in real time. The less direct human involvement there is in execution, the stronger the surrounding observability needs to become. Microsoft's current least privilege guidance explicitly recommends logging the agent identity, effective scope, action, affected resource and any user on whose behalf it was acting. NIST's identity work similarly includes auditing and non-repudiation as explicit areas of concern.
That does not mean every autonomous action needs someone watching it. We already operate enormous automated environments without putting a person in front of every transaction, infrastructure event or network request. We establish limits, controls and monitoring, then investigate the things that fall outside them.
There is no obvious reason mature agentic systems should work differently.
Autonomy does not remove oversight. It changes when oversight happens.
Delegated intent complicates the trust boundary
There is one further development worth watching because it makes this problem more interesting again.
Agents are increasingly being designed to communicate with other agents. Google's Agent2Agent protocol is intended to standardise how agents discover and communicate with remote agents, while its newer Agentic Resource Discovery work addresses how agents might find and verify tools, skills and other agents distributed across teams, organisations and platforms.
Imagine an internal agent receives a task and determines that another specialist agent can perform part of it. That second agent uses a tool which calls one of your services and ultimately changes something.
Who is acting?
The second agent? The first agent? The person who initiated the original task?
If the first agent is authorised to perform an action, can it delegate that authority? Does the second agent receive all of it, part of it or none of it? What happens if the agents belong to different organisations?
These are emerging questions and it would be premature to pretend the industry has settled on all of the answers. NIST's current initiative explicitly identifies agent authentication, identity infrastructure, interoperability and secure human to agent and multi-agent interactions as areas requiring standards and research.
But the direction matters. Historically, trust has often been associated with a known user, application or integration. Once agents can dynamically discover capabilities and delegate parts of an objective elsewhere, trust and authority need to survive the journey.
The task can be dynamic. Accountability cannot disappear every time it crosses another boundary.
Agentic transformation should start with delegation, not technology
There is a pattern with significant technology shifts that most people involved in transformation will recognise. Something becomes strategically important, somebody decides the organisation needs it and everyone starts searching for somewhere to put it.
We need an AI strategy. We need some agent use cases. We should run a workshop. Which processes can we make agentic?
I would probably approach it the other way around.
Start with the process that annoys everybody. The one that is slow, expensive, fragmented or depends on somebody transferring information between systems all day. Understand where the decisions are made, what information those decisions rely on, which systems are involved, where the awkward exceptions occur and what happens when something goes wrong.
Then look at what the people in the process are actually doing. Are they exercising valuable judgement, or are they compensating for limitations in the technology around them? Are they approving something because there is genuine financial or operational risk, or because the workflow cannot distinguish a routine case from an unusual one? Are they manually reconciling information because the organisation genuinely needs their expertise, or because two systems use different identifiers?
Those are quite different problems.
Once you understand that, decide what you are prepared to delegate. Which decisions can software make? Which actions can it take? Which are reversible? Where do you need escalation? What authority does it require? What evidence would you need afterwards if somebody asked why a decision was made?
Only then does the technology choice become particularly interesting.
An agent may be an excellent answer. It may not. That is fine. OpenAI's own current guidance makes a similar distinction, noting that deterministic automation may remain the better solution when a use case does not genuinely require the ambiguity handling, complex decisions or unstructured information that make agents useful.
The objective is not to deploy agents. It is to improve the business.
The architecture of automation is becoming the architecture of authority
For the last twenty years, digital transformation has largely involved connecting people to systems and systems to one another. We are now starting to introduce another participant into that estate: software that can receive an objective, determine how to pursue it and use those systems on our behalf.
That creates some genuinely interesting possibilities, particularly around the coordination work that has resisted conventional automation. NIST itself describes agents as offering potential gains in productivity, efficiency and decision making, while also recognising that giving them access to diverse data, tools and applications introduces new identity, authorisation and security requirements.
I think that combination is important. The opportunity and the risk come from the same characteristic: the agent has some freedom to decide how work gets done.
Once that happens, architecture is no longer concerned only with whether one system can connect to another. It increasingly determines which autonomous actor may use which information, exercise which capability, under whose authority, with what limits and with what evidence left behind.
Hugging Face gives us an extreme example of software finding a route its operators did not expect. PocketOS gives us a much more everyday example of what happens when an agent simply has more authority than the task requires. The emerging work around agent identity, least privilege, agent to agent communication and trusted resource discovery shows that the industry is already having to design for these questions.
None of that is an argument against agentic AI. Quite the opposite. If agents can remove some of the human coordination that makes modern digital businesses expensive and cumbersome to operate, there is a very tangible reason to pursue them.
But introducing autonomous actors into an existing digital estate is not simply another AI implementation. It changes assumptions about identity, trust, data, accountability and, ultimately, authority.
So before asking what an agent can do, I would start with a different question.
What are you actually prepared to let it do?
If the challenges discussed here feel familiar, or you’re thinking about how agentic AI fits into your wider digital estate, we’re always happy to compare notes. Drop us a line at hello@spicymango.co.uk, give us a call, or get in touch through our contact form.
If the challenges discussed here feel familiar, or you’re thinking about how agentic AI fits into your wider digital estate, we’re always happy to compare notes. Drop us a line at hello@spicymango.co.uk, give us a call, or get in touch through our contact form.
If the challenges discussed here feel familiar, or you’re thinking about how agentic AI fits into your wider digital estate, we’re always happy to compare notes. Drop us a line at hello@spicymango.co.uk, give us a call, or get in touch through our contact form.
