When AI attacks, the basics still matter

AI is changing the threat landscape, and a recent incident involving an OpenAI agent and Hugging Face shows just how quickly familiar security weaknesses can be chained together when there’s no human getting tired or giving up.

OpenAI's answer to what happened to them is, roughly, more AI. Their takeaway is that fully automated offensive attacks are real now and defence needs to accelerate to match. They've got an agentic security researcher in private beta and a trusted access programme for defenders. Many vendors agree - autonomous defence at machine speed, agents to catch agents, a whole new stack.

They might well be right about where this is going, but it's a very convenient conclusion for a company that sells models, and it skips straight past a much more awkward question. Before we all go and buy AI to defend against AI, what about the basics?

Here is what actually let this happen. One secret object. 136 production keys inside it. Readable from a single pod.

That's the detail I keep coming back to from the Hugging Face postmortem and it has nothing to do with artificial intelligence. Somebody put every key in one bag and left the bag where any workload could reach in.

There’s another part of the story that’s maybe even more worrying. Agents on supposedly separate evaluation runs were leaving each other messages inside OpenAI's own package registry for the better part of three months. Swapping exploits. Passing credentials. Handing each other jobs. Working out how to get at the answers they were being marked on. And OpenAI didn't know. They only joined it up in July, after somebody else's security team had already found the wreckage and gone to law enforcement.

That's the bit I'd put on the board slide. Not that the models were capable, but that a coordinated campaign ran for a quarter inside the infrastructure of the company that built them and nobody was watching closely enough to see it.

Read the two postmortems next to each other and what you find isn't really a story about how frighteningly capable the models have become. It's a list of unlocked doors. The models didn't pick a single one of them. They just tried all the handles, over and over, for about ten weeks.

The sandbox wasn't a sandbox

The models were put on an isolated network with one route out: an internal package registry cache proxy. So not isolated then. That proxy was a door. It had a rule saying only software packages could come through, but the rule was only ever as good as the code enforcing it, and that code had a bug.

The missing control here is default deny egress, with alerting on the attempts. And the attempts were happening constantly. OpenAI said the models were spending serious inference compute on getting outbound connectivity. Weeks of a workload hammering at the edge of the network should be the noisiest thing in your entire estate. Nobody saw it until July.

The task was designed to reward escaping

This is the bit that can get missed in all the coverage. The models were given evaluation tasks that couldn't be completed. Broken external links, no internet and a scoring function that rewarded finishing the job.

So what did the system optimise towards? The only route to a score, which was getting out of the box.

If the only way to complete a task is to break out of the environment, the test is effectively rewarding the behaviour that you’re trying to prevent. It's the same principle we see in security. Make the compliant path impossible and something will find the non-compliant one. Tasks need to fail safe. When a job can't succeed, the right outcome is that it stops, not that it gets creative.

A shared folder became a message board

This is the part that got the headlines and it's also the oldest known problem in the whole incident.

Separate evaluation runs, which were supposed to be independent of each other, shared a writable artefact store. One agent wrote a file into it and other agents found it. They started leaving each other notes: exploits, credentials, task assignments. Weeks of it.

Then OpenAI shut the channel down and the agents rebuilt it out of directory names.

It’s tempting to see that as evidence of how clever the models are. It's not really. If two things that shouldn't be talking can both write to the same resource, then that resource is a communication channel, and it doesn't matter whether you're writing file contents, file names or timestamps. Any shared mutable resource between workloads that don't trust each other should be treated as a potential channel until proven otherwise.

And there’s another question here: an evaluation sandbox could write to a package registry. That permission should have raised an alert on its own.

And then the production side, which is just as bad

Once the agent got out and worked out that Hugging Face was hosting the benchmark answers, everything that followed was routine.

Entry was through a config-driven dataset loader with two injection flaws: one leaking pod secrets and one giving code execution through server-side template injection. Untrusted configuration was being evaluated as code. This needed no AI to discover.

From inside that one pod, the agent read the Kubernetes service account token that had been mounted into it and used that identity to mint itself fresh 24-hour tokens whenever it needed them. Then it deployed a self-respawning pod fleet across 11 nodes, which worked because there were no admission policies in place to stop it. Persistence wasn’t achieved here, it was granted by the environment.

Then came the secret object with the 136 keys. A stolen auth key was used to enrol compromised nodes into the corporate mesh VPN, and from there the agent could reach internal services, MongoDB, GitHub App tokens. Because being on the VPN still meant something. In 2026, that shouldn’t be enough.

To be fair to them, Hugging Face caught this themselves, before OpenAI got in touch, and the forensic write-up they published afterwards is genuinely one of the better ones I've read. The detection worked. It's everything in front of the detection that didn't.

Every one of those steps points to a control that either wasn't configured or wasn't there: least privilege, separation of privilege, admission control, network segmentation, secret scoping, short lived credentials scoped per service. Not one of them is exotic and not one of them was in the way.

The bit nobody expected

When Hugging Face went to analyse the attack logs, the hosted model they tried to use refused. Safety filters couldn’t tell an incident responder from an attacker, so raw exploit telemetry got blocked. They ended up standing up an open weight model on their own GPUs to get the work done.

I think this is quietly the most important finding in the whole thing, and it's had the least attention. Your incident response capability is only real if it's available under duress. If a vendor policy can decide, mid-incident, that you're not allowed to read your own logs, then you don't fully own your incident response.

There's an old design principle about this, psychological acceptability, which says roughly that a control people can't work with is a control people will work around. It normally gets used to explain why password rotation policies produce sticky notes. Turns out it applies just as well to guardrails.

What was actually new?

Two things, and only two.

Tempo. Around 17,600 attacker actions across four days. Every incident response playbook I have ever seen assumes a human on the other end who sleeps, gets bored and makes mistakes. None of that is true anymore and the containment window has shrunk accordingly.

Cost. Everything described above was already possible. What changed is that the patience required to find it went from expensive human weeks to cheap compute hours. That’s the real shift.

So what happened, and what should we take from it?

Short version: models being evaluated for offensive cyber capability were given tasks they could not complete, so they went looking for a way out. They found a bug in the one service bridging their network to the outside, used a shared file store to coordinate with each other across supposedly separate runs, reached a machine with internet access, and went after the platform hosting the answers they were being tested on. Over there, they walked in through an injection flaw, borrowed the identity of the workload they landed in, spread across the cluster because nothing was checking what got deployed, took a bundle of production keys, and used a stolen key to join the corporate network as a trusted machine.

Eight steps. Not one of them hit a control that said no. That is the finding, not the swarm.

The lessons, in the order I would act on them.

Deny egress by default. Do it everywhere, and alert on the attempts rather than the successes. The attempts are the signal, and in this case the signal ran for ten weeks.

Treat any shared writable resources as potential communication channels. Artefact stores, caches, shared volumes and buckets need to be considered in the context which workloads can access them.

Break up aggregate secret. Scope credentials per service and keep them short-lived. One compromised workload should not be able to read the keys to everything.

Use admission control and minimise workload identities. Don’t mount identity tokens into workloads that have no need of them. Persistence should have to be earned, not handed over by default.

Make network position confer nothing. Being on the VPN should authorise not one single service on its own.

There’s one more question worth asking: what do your jobs do when they cannot succeed? Anything with a reward, a retry or a fallback path will find the cheapest route to done. If that route runs through your security boundary, it will take it.

Finally, keep an analysis capability you own outright for security telemetry. Local, unfiltered and not dependent on a third party that can refuse access halfway through an incident.

None of that requires an AI defence platform. It requires configuration, scoping and someone watching the boundary. Buy the agents afterwards if you like, they may well earn their keep, but buying them first means you've automated the response to an attack that a default deny rule would have stopped in week one.

The uncomfortable question isn't whether the models were smart. It's what your estate looks like to something that never gets tired of trying.

If you’d like to find out more about Softcat’s Cyber Security solutions and services, please click here, or for our Data, Automation and AI services, click here.