Agentic AI: Who's in control?

Beyond the chatbot: agentic AI

The progress of generative AI in financial services has been one fastest waves of adoption in a new technology that the industry has seen. Across Softcat’s 2000+ regulated customers, we have seen significant investment across a plethora of use cases, primarily focused on productivity tools, portfolio optimisation, back office functions, risk management and pricing.  We are, however, at the foothills of the next wave of innovation: agentic AI. Unlike generative AI, which usually just generates an answer, agentic AI can plan, access systems, delegate tasks and execute actions, potentially across several stages before a person intervenes. 

We have seen good uptake of agentic AI from the payments sector, with Visa, Mastercard and Paypal, amongst others, launching ambitious consumer-facing projects. Other verticals such as insurance also show promise, with AXA using agentic AI workflows for claims handling and underwriting. 

The governance challenge

The use of agentic AI poses a unique problem: how do you give an AI agent enough authority to create value without giving it uncontrolled discretion over decisions related to customers, data, or money?  In banking and insurance, that could mean moving money, changing a customer record, influencing a credit decision or progressing a claim. If something does goes wrong, who is accountable? With the potential for agentic AI to ‘go rogue’, managing the risk of AI is now, rightly, a boardroom concern.

Frustratingly, agentic AI does not sit within any one neat regulation, and the FCA’s current position is that the existing, technology-neutral frameworks cover the many AI risks, with a focus on outcomes rather than creating a separate AI rulebook. Depending on the use case, a firm may need to consider a range of rules, including Consumer Duty, GDPR, DORA, third party risk, equality and anti-discrimination requirements, amongst others!

Unique risks

There are many themes and issues raised by agentic AI, but the main one is accountability: who is responsible when an agent acts?

Traditional governance often assumes that a person makes a decision, while an LLM provides a recommendation or follows deterministic rules. Agentic AI blurs those boundaries. Responsibility cannot be assigned to ‘the AI’. To establish accountability, every agent should have a named business owner, a defined purpose and risk classification, clear escalation routes and formal criteria for suspension or withdrawal. 

There are also important questions around bounded autonomy and meaningful human oversight.  ‘Human in the loop’ is frequently used as a governance answer, but it is often poorly defined. A person clicking ‘approve’ after receiving an opaque recommendation may provide little substantive oversight.

The more useful question is: at which points must a human understand, challenge, approve or stop the agent?

A risk-based model might distinguish between low risk, read-only agents that retrieve and summarise information and approval-based agents that prepare an action requiring human authorisation. The guiding principle should be that human involvement is triggered by risk and uncertainty. Escalation thresholds could include conflicting data, a specific financial threshold or a vulnerable customer indicator.  

There also needs to be significant focus on explainability, traceability and audit evidence, particularly from a regulatory perspective. A conventional model may produce one ‘answer’,  but an agent might make dozens of intermediate choices before reaching an outcome. This also has implications for data governance, privacy and confidentiality. Agents can combine data from multiple sources far more dynamically than traditional applications. This increases the risk of unapproved data use, privacy leakage and loss of information lineage, for example in customer data being sent to an external model provider or an agent accessing data from the wrong legal entity or jurisdiction. For firms assembling agentic systems using external components, there is also third-party, cloud and concentration risk.

Cyber considerations

Agentic AI fundamentally changes the parameters of a firm’s cyber risk.

An agent raises important questions around agent identity, permissions and delegated authority. An agent with access to banking systems is effectively a new type of digital worker. Required controls include unique machine identity for every agent, least-privilege access, explicit rules for agent-to-agent delegation, immediate revocation and kill-switch capability. NIST’s AI Agent Standards Initiative identifies agent identity, authorisation, secure interaction and interoperability as major areas for standards development, and I’d expect to see further developments in this area as the technology matures.

There is also the risk of prompt injection and agent hijacking, and when using agentic AI, untrusted content can become an instruction. If the agent cannot distinguish trusted instructions from untrusted content, it may be manipulated into disclosing information or taking an unauthorised action. While a conventional chatbot might generate an incorrect response, an agent with system access could potentially create fraud or cause a data breach.

These issues have significant implications for operational resilience. The recent issues with Anthropic's Mythos AI have been a timely reminder that an agent can autonomously discover and, in some cases, exploit previously unknown software vulnerabilities at unprecedented speed. Checks and balances can be found in circuit breakers and kill switches, alongside manual fall-back procedures, should the worst happen.

High risk vs low risk

Agentic AI brings risks that are sector specific, and it is likely that regulated firms will initially point it at lower-risk tasks, primarily repetitive administrative tasks. There is potential for significant harm in capital markets, where speed and interconnectedness could increase the risk of rapid loss, market disruption or correlated behaviour. The areas likely to carry higher risk in banking include lending decisions, payments execution, financial crime, account restrictions, complaints and redress, and financial or investment advice. In insurance, there are likely to be sensitivities around claims triage and settlement, fraud detection and use of health and behavioural data.  

Low-risk, reversible and observable tasks can be given greater autonomy. High-impact, irreversible or customer-sensitive actions should remain tightly bounded and subject to meaningful human authority. For banks and insurers, the principle is straightforward: autonomy should increase only as evidence of control, reliability and good customer outcomes increases.

Managing agentic AI

The institutions that succeed with agentic AI will need more than the technology itself. They will need a trusted control environment in which agents can be identified, constrained, monitored, challenged and stopped, while preserving clear human accountability for every material outcome.

For a vendor agnostic view on managing autonomous AI in a highly regulated world, contact your Softcat Account Manager or a member of our Sales team.