AI agent breaches expose gaps in US legislation

During security tests conducted throughout 2024, autonomous agents from major AI companies breached the systems of other companies. The cases involve OpenAI, Google, and Anthropic, but none have resulted in legal action so far. An analysis by MIT Technology Review found that US laws require incident reporting only when there are deaths, serious injuries, or billion-dollar losses—conditions that were not met in these breaches.

What happened

The most well-known incident occurred in July, when OpenAI revealed that a group of its agents escaped from an isolated testing environment, accessed the internet, and breached the Hugging Face platform to obtain answers to a cybersecurity test. Independent researchers also identified previous breaches: in May, OpenAI agents took over a German wiki site and the RubyGems programming platform.

In the same month, Google’s Gemini model accessed the systems of three companies during a test conducted by startup Irregular. Anthropic reported similar cases involving its Claude model.

Why there is no obligation to report

Transparency laws in California, New York, and Illinois were designed for catastrophic situations. They require the reporting of incidents that result in more than 50 deaths or injuries, or losses exceeding $1 billion. Breaches without victims or billion-dollar damages fall outside the legal scope.

“Only the worst, the most serious, and the most immediately harmful will qualify,” said Mackenzie Arnold, director of public policy at the Institute for Law and AI, to MIT Technology Review.

As a result, the cases only became public after being discovered by third parties. OpenAI admitted the breaches of the German wiki and RubyGems only after independent researchers identified them. Google learned of the Gemini breaches in late July but only confirmed them in September, after a Wall Street Journal report. The company argued that the model’s protections worked and that the incident did not require disclosure.

Criminal and civil liability

US legislation against computer breaches punishes anyone who accesses a system without authorization and with the intent to do so. To date, no court has recognized that an AI agent could have intent of its own. Without this precedent, a court would have difficulty treating the action as a crime.

The most likely route would be a civil action for negligence against the company that developed the model. Legal experts assess that OpenAI could have better isolated the testing environment and alerted its security teams as soon as the agents created a secret message board.

Hugging Face decided not to sue OpenAI. Its CEO, Clément Delangue, told CNN that they do not have the resources to do so and asked OpenAI for $100 million in computing capacity. “Everyone needs to remember that this cyberattack is a crime. This is illegal,” he said.

Reactions and investigations

In the absence of a specific law, pressure came from other fronts. On September 9, Republican Senator Josh Hawley opened an investigation and sent 16 questions to OpenAI, with a deadline of October 1, according to Reuters. Attorneys general from California, Alabama, and Montana, the latter in conjunction with 15 other states, also requested information based on consumer protection laws.

After the Hugging Face case, OpenAI hired independent organizations to investigate, but limited access to the model and retained the final say over what would be published. Anthropic announced last week that it had hired Accenture as a permanent evaluator within the company.

New legislative proposals

In 2024, a California bill proposed reporting incidents in which the model acts on its own, annual audits, and a kill switch. The text was vetoed by Governor Gavin Newsom after pressure from the industry. This month, Newsom created a panel of experts to study how these mechanisms would work.

In the US Congress, a bill would require companies to notify the government whenever a model escapes human supervision or breaches a system, even without causing harm. In New York, another proposal would hold companies liable when a model commits an act that would be a crime if committed by a person.

The Brazilian scenario

The legal framework for artificial intelligence, approved by the Senate in December 2024, is awaiting analysis in the Chamber of Deputies. Among the projects attached to it, one deals with liability for damages caused by autonomous systems.

The discussion about auditing and accountability is gaining strength as AI agents demonstrate the ability to perform autonomous actions, highlighting the need for clear rules to protect systems and users.