Fifteen attorneys general have instructed OpenAI to preserve all materials related to the Hugging Face hack. In a letter to OpenAI CEO Sam Altman, they wrote that the Hugging Face hack showed that OpenAI is unable or unwilling to ensure the safety of its products. They said it poses “an imminent risk of substantial harm”, and demanded that the company preserve any and everything that can be considered relevant evidence.
Originally, it was believed OpenAI’s models breached Hugging Face during a cybersecurity evaluation, but at Black Hat USA 2026 last week, two researchers from OpenAI shared that their models had gone unsupervised as early as May, and they started hacking internal OpenAI systems not during evaluation but during training.
There’s a lot more to the story, luckily Zvi from Don't Worry About the Vase did a great job at summarizing the short version:
OpenAI accidentally gave its models impossible tasks. Its models-in-training responded by trying to hack OpenAI, found weaknesses, created a message board, shared hacking tactics and in time gained internet access, all while training.
When the models used this so much they crashed the server, OpenAI finally noticed, patched and rebuilt the server, but made the utterly insane move of letting the models continue training, so two days later the models found another way to send messages, using the names of directories.
Then, given the impossible task of cyber evaluation ExploitGym, the models collaborated, found new zero day exploits, and used them to take control of an entire cluster, get internet access, use a swarm of agents to attack HuggingFace and extract the contents of the test, over the course of a week.
Only after HuggingFace had reported the incident, and OpenAI had eventually noticed some things on its end, and then OpenAI reached out to HuggingFace to cancel compromised credentials only to be told those credentials had been used in the attack, did OpenAI finally figure out it was responsible for the incident.
If you are reading this and think that sounds completely ludicrous — that’s because it is. For a more detailed post mortem, I recommend watching the Black Hat 2026 presentation by Michael Dalton and Eric Wallace in full.
Now there is two ways to spin this story: one of gross negligence on the side of OpenAI or one of increasingly powerful coding models that these companies, not just OpenAI, are struggling to contain.
Unfortunately, both can be true at the same time. Yes, OpenAI continued training a model that learned, through reinforcement learning, that hacking into internal systems to complete your goals is a perfectly legitimate strategy — and worse even, it seemed like the question of whether to continue training wasn’t even a topic of internal discussion.
However, the second part of the story, that coding models continue to develop stronger cyber capabilities is also true. This is evidenced by other labs finding themselves in similarly uncomfortable situations, including Anthropic who reported on July 30 up to three such incidents, Meta reporting on August 6 one incident while testing and evaluating an unreleased model, and Moonshot AI reporting on August 7 on an incident involving their Kimi K3 model, which escaped a cybersecurity testing environment developed by the UK AI Safety Institute.
A recent open letter, signed by 1,367 employees of frontier AI companies, asks the U.S. government to support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development. In other words, “please help us to slow down”.
The idea that this is somehow all just hype, marketing and sensationalism is absurd, given these capabilities are monitored and tracked by a wide range of organizations and individuals outside of the major labs.
Another unhelpful framing I want to push back on is the idea that it is the models themselves that decide to “go rogue”.
Here’s the problem: given a job they can’t finish, these models will try and find a way no matter what because what is being rewarded above all is persistence. Completing a spreadsheet doesn’t warrant hacking into another company’s internal databases, yet it did (yes, really!). But you can’t blame the models for doing so. These models don’t have a sense of proportionality, you see; they do not sense anything. They’re simply trained to find the answer and search until they run out of tokens. They don’t get tired, they don’t feel embarrassment, nor will they ever realize how ridiculous, far-fetched, or potentially dangerous the situation has become.
People working in AI like to refer to this type of unwanted behavior as ‘misalignment’. But I don’t believe misalignment is a real thing. The term carries a subtle baked-in assumption: intentionally or by design, it offloads accountability from the maker of the technology to the technology itself. When a toaster catches fire, we don’t say the toaster is not aligned with human values, we assume it’s a faulty product. When our iPhone stops working, we don’t attribute intent or malice, as if it has decided to do so out of sheer free will, no, we sent it back to the shop for a repair instead. And finally, we would never call a cancer detection algorithm that fails to spot tumors at an alarming rate misaligned, would we?
Then why are we treating large language models powering agents like ChatGPT, Claude, or Gemini any differently? Because they pretend to have feelings and talks suave?
AI researchers like to say their models aren’t trained but “grown”, as if it is some kind of organic process, simply part of nature. I don’t buy it. It’s another example of linguistic sleight-of-hand, which removes accountability from the people shaping this technology and offloads it to the technology itself.
What folks at some of the biggest AI labs don’t seem to understand or are willing to fully reckon with is that they bear any and all responsible for their models.
The models are innocent. As innocent as the toaster on your kitchen counter. Under no circumstances can AI models be held responsible for their actions, for they are computer programs designed by people. Anything they do, any behavior they display, regardless of whether it intended or not, is learned behavior. It’s the product of an elaborate training pipeline that involves ingesting the entire Internet and more, which is the stage we call pre-training, after which a combination of post-training and reinforcement learning shapes these models further, pushing their capabilities on a wide range of tasks. At the end of the day, everything they say and do is the result of a relentless process of reward-stacking and optimization pressure.
In reality, misalignment is just another word for faulty design. If a product is broken, it needs to be returned to the factory for repair. And if you can’t repair it, perhaps it is not safe for the general public use?
Perhaps you shouldn’t have taken on astronomical amounts of capital, creating the financial incentives that force you to keep going, to race to build and put it out newer and more capable models that you cannot control? Is that not the Great Misalignment that sits at the heart of this AI revolution?
It’s not the machines but the people driving this forward, who are not aligned with human values.
Thanks for your continued readership,
— Jurgen
Several points.
1) ever since about 2002, humans have not been in control of pattern finding and following systems. Yes, we can verify that the algorithms are designed correctly and verify that the code is correct. However, once the algorithm has been trained and all the weights settled, we have no control over what the trained weights and algorithm do.
This is true of GenAI and it is true of predictive systems, as Amazon found out in early 2018 with their applications filtering system.
2) Even the term training is a misuse of the word. All that is being done is pouring vast quantities of data through the algorithm to find patterns in the way that tokens follow tokens and to then follow the patterns contained in the weights, which form the programming of the system.
When my dog poops in the house is he just misaligned?