Attack of the AI

A San Francisco–based artificial intelligence developer claimed last month that a group of hackers had manipulated its chatbot to hack U.S. organizations. Anthropic labeled the hack of its Claude program as the first known “AI-orchestrated cyber espionage campaign.”

Humans behind the hacks, whom Anthropic said were likely “a Chinese state-sponsored group,” provided only a handful of prompts while AI performed the majority of the dirty work. “Threat actors can now use agentic AI systems to do the work of entire teams of experienced hackers,” the company said in its report.

Some experts say fully autonomous AI hacking might leave the realm of science fiction and become reality soon. They warn that AI isn’t going away, and some recommend that businesses and other entities fight fire with fire by using AI to automate security protocols.

Hollywood tends to portray hackers as figures in black hoodies tapping a few code words into a computer to quickly access files. Experts say that’s a misleading picture.

“In reality, hacking is a very slow, trial-and-error process,” explained Victor Benjamin, an assistant professor at Arizona State University’s Department of Information Systems. “So you kind of knock on multiple doors, see what’s open, what you can crack open to get further into a system.”

Benjamin said computer hacking has gotten easier as artificial intelligence allows hackers to automate busywork and scale up their attacks. “Now [hackers] have a partner in crime,” he said.

Using large language models, or LLMs, bad actors can quickly research targets and spot an organization’s security gaps. Once AI detects those weak points, it can generate disruptive software or malware to exploit the targeted organization.

AI tools allow hackers to create personalized phishing content based on data they’ve gathered on the target company. According to security company KnowBe4, AI helps write roughly 82% of phishing emails. Security Hero estimates AI tools can crack half of common passwords in less than a minute.

Some AI models are specifically designed for hacking. The subscription service WormGPT 4 describes itself as “your key to an AI without boundaries.” The model creates advanced phishing content and ransomware notes and offers beginner-friendly malware generation.

While many AI tools have safety features, those protections may not stop hackers from using them for nefarious purposes. In last month’s attack, hackers overrode Claude’s security features, a process known as jailbreaking. According to Anthropic, the Chinese-sponsored actors told the chatbot they belonged to a cybersecurity firm, using Claude to research about 30 intended American targets. Claude organized the information and created malware to take advantage of the weak points in each target’s systems.

While hackers only successfully compromised a handful of victims, Benjamin said the attacks mark a troubling technological milestone, since the Chinese government allegedly sponsored the attack. “Now we have solid proof other nation states are actually doing this against us—before, it was just kind of presumed,” said Benjamin.

But some AI experts do not believe this is such an open-and-shut case of China manipulating American AI. Amar Singh, CEO of London-based consulting firm Cyber Management Alliance, doesn’t think it makes sense for China to use Claude when the country has its own AI software with DeepSeek.

While he doesn’t deny that a Chinese actor could be using AI, he questions Anthropic’s assertion that the hackers used the company’s own AI chatbot. “I think Anthropic has achieved its marketing objective where everyone is discussing it,” said Singh.

The China angle is only one troubling aspect of the report. Anthropic described the cyberattacks as being almost fully autonomous, since AI carried out up to 90% of them. Each hacking campaign needed only about six human prompts. Anthropic said it detected the attacks in September, then notified the affected companies and upped its security measures.

Artificial intelligence may be sophisticated, but it still needs human prompts to carry out cyberattacks, Singh said, pointing out that hackers can’t just tell AI to rob a bank and let the computer handle all of the work. “I think the good news is we’re not there yet,” said Singh.

But Victor Benjamin believes it won’t be long before AI can hack a system by itself, without human prompts. “As far as being completely autonomous, I don’t see why not,” he said.

More intuitive cyberattacks are concerning because the world is becoming what Benjamin calls “cyberphysical.” Computers have become integrated into everything from cars to dishwashers, meaning everyday tools are suddenly vulnerable to cyberattacks.

“The security issues aren’t just attacking a server and stealing your credit card,” Benjamin warned. “Now they can attack autonomous vehicles or traffic lights and cause chaos in the real world.”

Benjamin thinks it’s too late to scale back reliance on AI. Instead, he recommends using AI to harden organizations against increasingly sophisticated cyberattacks. The generative AI cybersecurity market, currently valued at about $2.5 billion, is expected to reach nearly $24 billion by 2034.

Alexei Bulazel, senior director for cyber at the White House National Security Council, explained at the DEF CON Hacking Conference in August that AI tools can help organizations quickly detect their own weak points. “We can take AI and apply it to, here’s the source code for this router, identify all the vulnerabilities, and then generate patches for me,” said Bulazel. “And you don’t need an elite team of government-trained hackers to do that.”

Some organizations are already doing that. Last year, research firm PYMNTS surveyed executives from 60 companies with annual revenue exceeding $1 billion. PYMNTS found that 55% of those organizations—“a threefold increase compared to earlier in the year”—used generative AI to run automated cybersecurity management systems.

“Just like AI is being used to attack systems, we can have AI-based defenders,” said Benjamin. “So I think at some point, cybersecurity will become less managed by humans and more managed just directly by AI.”