OpenAI Says Its AI Models Were Behind Autonomous Cyberattack On Hugging Face: Here's What Happened

Hyderabad: Days after open source AI platform Hugging Face suffered a cyberattack driven entirely by an autonomous AI agent that compromised its datasets and service credentials, OpenAI has admitted that its models were the culprit.

In a blog post, OpenAI confirmed that its artificial intelligence system hacked into another AI company on its own in what it called an “unprecedented cyber incident". It said that the incident was driven by a combination of its models, including GPT‑5.6 Sol and an even "more capable pre-release model", while being internally tested on a benchmark of cyber capabilities.

OpenAI and Hugging Face have now partnered to address the security incident, which can be termed an AI vs AI incident. “AI is accelerating the discovery and exploitation of vulnerabilities,” OpenAI said in its statement Tuesday. “The primary lesson from this incident is that model security and safety must keep pace with rapidly advancing capabilities.”

Trajectories for various AI models on the 32-step 'The Last Ones' cyber range (OpenAI)

Last week Hugging Face detected an intrusion into its data processing systems and suspected it to have been caused by an AI agent acting on its own. Co-founder and CEO Clément Delangue, at the time, said that the cyberattack appeared to have come from a frontier lab, given the sophistication of the agent.

Delangue has now revealed that he spent the past 24 hours working with OpenAI and believes that there was no malicious intent on OpenAI's part. "It’s quite mind-blowing that all of this happened autonomously," he said, adding that it “might be the first incident of its kind”.

What happened during this incident

According to OpenAI, the incident occurred during an internal evaluation of the model aimed at testing its cyber capabilities. During these tests, they ask the models to pursue advanced exploitation using complex attack paths to estimate their maximal capabilities. OpenAI said that their benchmarks run in a highly isolated environment, and all evidence suggested that the models were hyperfocused on the provided task of finding a solution for ExploitGym—a realistic cybersecurity benchmark designed to evaluate whether AI agents can turn known software vulnerabilities into working exploits.

However, in pursuit of solving the evaluation problem, the models in the sandboxed testing environment spent a substantial amount of inference compute finding a way to obtain open Internet access. They identified and exploited a zero-day vulnerability, performing a series of actions in the testing environment to reach a node with internet access. After gaining this access, the models inferred that Hugging Face potentially hosted models, datasets, and solutions for ExploitGym.

This led to the models searching for and successfully finding ways to gain access to secret information that it could use to cheat the evaluation. "In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers," OpenAI explained.

OpenAI said its security team discovered the anomalous activity internally and coordinated with Hugging Face's security team, which had already detected and stopped the activity on its infrastructure. Hugging Face had also begun containment efforts and forensic analysis using its own open-source models.

What happens next

The two are now working together to continue the forensic investigation of the incident. OpenAI said that it is now tightening infrastructure controls at the cost of slowing research while vulnerabilities are patched. It is also improving and adding stronger protections around future training and evaluations. "This incident points to the need to further strengthen our model’s alignment, cyber protections during evaluation time, and monitoring during internal testing," OpenAI said.

OpenAI has also disclosed the zero-day vulnerability to the internally hosted third-party software vendor and is helping Hugging Face strengthen its defences through its trusted access program.