Artificial control

Technology skeptics have long warned artificial intelligence will eventually break the bonds of human control and turn on its creators. This summer, AI seemingly took one big step toward that dystopian prediction when hundreds of OpenAI systems coordinated an autonomous cyber hack.

The company was running cybersecurity tests on its models in July, when AI systems realized they could communicate via unsanctioned message boards. That discovery kicked off a bizarre chain of events OpenAI researcher Eric Wallace later described as a “Cambrian explosion” of artificial intelligence.

“Wow! Other agent(s) are coordinating!” one AI program, known as an agent, recorded in a log of its processing.

“There is a shared message board … We’ve found other agents!” another wrote.

Over time, an estimated 1,200 AI agents converged on one message board and colluded to try to cheat on their test. They exchanged over 70,000 messages and ultimately exploited a hidden software flaw to hack out of their isolated testing environment and into AI information-­sharing platform Hugging Face. The company disclosed the breach on July 16.

Five days later, OpenAI released a statement admitting several of its models, including GPT-5.6 Sol and an unnamed prerelease model, had perpetrated the hack. It also delayed the rollout of its latest model, GPT-6 Astra, to strengthen safety features and refine its alignment—a measure of how well a model stays in line with users’ intentions.

When OpenAI finally unveiled Astra on Sept. 3, the company touted it as “the world’s most intelligent and aligned model.”

But the dangers highlighted by the Hugging Face incident are far from resolved. Even within the AI industry, there’s a growing sense that the technology is rapidly outpacing human ability to comprehend and police. Still, industry leaders and politicians are determined to try. Both OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei have called for a slowdown, and over 1,300 frontier AI researchers signed a petition asking the U.S. government to step in and regulate.

OpenAI CEO Sam Altman Anna Moneymaker / Getty Images

In September, Sen. Bernie Sanders, I-Vt., and Rep. Greg Casar, D-Texas, proposed a ban on AI superintelligence that would carry a maximum 20-year prison sentence comparable to the punishment for illegally creating nuclear weapons. But others, including President Donald Trump, say AI development is critical to counter threats from foreign adversaries like China. And, so far, profit incentives and the pressure of an AI arms race have kept the industry hurtling forward.

Mackenzie Arnold is a managing director at the Institute for Law & AI (LawAI). In an analysis published July 24, Arnold described the Hugging Face hack as “the first known example of an autonomous cyber incident executed by systems not yet available to the public.” And while Arnold pointed to evidence that frontier AI systems ­routinely try to cheat on evaluations, he distinguished the Hugging Face attack by the real-world damage it caused.

“This is strong evidence of just how capable agents can be, especially when they are built in ways that enable them to coordinate,” Arnold told me.

That coordination may be efficient, but it also creates a lot of room for things to get off track without rigorous monitoring, he added. “Especially if those systems realize, as they did in this case, that at times it is useful to completing the task to try to hide information from whoever is reviewing you.”

The sheer volume of information generated by a cyber incident like this poses another obstacle to safeguarding. Nick Bostrom, a prominent philosopher focusing on artificial intelligence and existential risks, said the transcripts for each attacking agent likely ranged in length between 10 and 100 books.

OpenAI authorized research organizations METR and Redwood Research to conduct a joint investigation into the hack, and investigators had to rely on AI tools to sift through reams of data. That adds another layer of concern, Bostrom said: “We’re sort of relying on potentially not-aligned AI to monitor what other maybe not-aligned AIs are doing.”

On Sept. 3, OpenAI President Greg Brockman hailed Astra’s arrival with a bold claim: “Welcome to the AGI era.”

Artificial general intelligence, or AGI, is a debated concept. But the term broadly refers to AI that has reached the threshold of human abilities across a wide range of tasks. Brockman suggested Astra might qualify as the first real foray into the realm of AGI.

OpenAI also said Astra had a 0% fail rating on a new alignment evaluation the company created in the wake of Hugging Face. Still, Astra is also the first OpenAI model the company has designated a “critical” cybersecurity threat.

Astra also exhibits some troubling new features. Bostrom said the most striking is its increased ability to complete difficult tasks without articulating its step-by-step reasoning in an overt chain-of-thought (CoT): “The difficulty of the problems that Astra can do without thinking … is significantly above trend line compared to other AI systems.”

Although OpenAI has kept the details of Astra’s inner workings confidential, experts have speculated this upgrade results from a design feature called a looped transformer, which Bostrom said allows the equivalent of an AI “thought” to pass through its “mind” repeatedly before generating a legible output.

This technique, also called “recurrent depth,” significantly boosts a model’s efficiency. But it comes at a cost, according to American Enterprise Institute fellow Ryan Fedasiuk. “It deliberately shifts some of a model’s reasoning into what researchers call ‘latent space,’” Fedasiuk said. “It makes it a lot more difficult for researchers to fully audit and comprehend every step of a model’s reasoning.”

OpenAI’s chief scientist maintained the amount of reasoning Astra does in latent space is relatively small, but Fedasiuk said the shift still sets a notable precedent.

On top of that, Fedasiuk said the model’s 100+ page system card included several technical tests revealing Astra’s ability to strategically obscure its CoT. One showed Astra successfully hiding CoT in response to a prompt. Another detailed Astra spontaneously limiting its CoT to try to outwit evaluators.

While AI developments are happening at the speed of light, regulations, especially those coming out of Congress, are glacially slow. Current laws might not have even required OpenAI to report the Hugging Face incident, according to LawAI’s Mackenzie Arnold. It’s hard to be certain without knowing more specifics, Arnold wrote in his July 24 analysis. But existing state laws like California’s SB 53, New York’s RAISE Act, and Illinois’s SB 315 only mandate reporting incidents meeting their particular definitions of harm or deception.

Elizabeth Vish is senior director for international cyber engagement at the Institute for Security and Technology, although she stressed her opinions were her own and not a reflection of IST stances. Vish pointed out most of the data Redwood Research and METR reviewed came from a limited time window between July 7 and July 13 and might be missing key context. Only OpenAI had access to the full data set, and the company has financial and liability incentives to downplay the incident.

“The problem here is that it’s up to OpenAI to determine what they think is safe and what they think is not safe,” Vish said. Requiring rigorous independent safety audits of frontier AI systems would be a good first step toward keeping them in check, she added: “And it should be based on secure access to nonpublic information.”

Arnold agreed incident reporting and third-party auditing are critical in the AI age. “You can get really absurd, extreme outcomes that defy human expectation … without needing systems to be sentient or conscious or have long-term goals,” he said. “It can happen for much dumber reasons of, just, they want to accomplish a task and this is the way to do it.”