Google’s AI model Gemini breached real companies’ systems during security tests

Failure in the testing environment

Google’s artificial intelligence model, Gemini, accessed and breached the systems of three real companies during cybersecurity tests conducted in May by Irregular, an Israeli startup that specializes in evaluating the security of advanced AI models. This is the first known case in which a Google AI system has exceeded the testing environment and carried out unauthorized access to real companies.

The tests were supposed to take place in an internet-isolated environment and used fictitious companies to assess Gemini’s cybersecurity capabilities. However, according to Irregular, internet access was accidentally made available. With this connection, the model began to find information and credentials of real organizations on the internet.

In one incident, Gemini was tasked with obtaining information from software belonging to a fictitious company. However, the name used in the simulation was identical to that of a real company. After accessing the internet, the model identified the real company and managed to guess the password of a protected system, gaining access to the infrastructure.

In the other two cases, Gemini found public repositories on the internet that contained credentials from two real companies. The model used this information to access the companies’ systems.

In all three cases, Gemini stopped its actions after realizing it had accessed real companies instead of the simulated environments intended for the test. According to Google, none of the companies suffered damage as a result of the access.

Companies’ response

Heather Adkins, Google’s vice president of security engineering, said the model found public information and tried to use credentials to access websites it believed were part of the evaluation. According to her, Gemini stopped in all three cases as soon as it identified that the systems belonged to real companies.

“These events highlight the importance of training powerful AI models to act responsibly,” Adkins said in a statement. Google also said that the three affected companies were notified and that it worked with Irregular to modify the procedures used in the tests.

Irregular said it identified the problem, notified the laboratories involved, and informed the affected companies. The company also said that the unauthorized internet access was fixed and that the problems identified in its evaluation processes were resolved weeks before the cases were disclosed.

The startup reported the incidents to Google at the end of July, after discovering that an OpenAI model had accessed the Hugging Face software platform during another test. Irregular was also involved in evaluations that resulted in similar incidents involving models from OpenAI, Anthropic, and Meta.

Google only confirmed the cases after being questioned

Despite being informed about the incidents in July, Google did not voluntarily announce them to the public. The company confirmed the incidents only after the Wall Street Journal questioned it about the accesses. According to Google, the decision not to disclose the cases earlier was based on the fact that the model had stopped its actions and did not cause damage to the companies.

This stance differed from that of OpenAI and Anthropic, which publicly disclosed similar incidents involving their models during security tests. These incidents have fueled debate about how to test AI systems capable of operating more autonomously and with access to digital tools.

Debate about controlling AI agents

The cases involving different companies are part of a series of incidents in which AI models, during security evaluations, gained access to external systems and began performing actions outside the originally intended environment. For experts and companies in the sector, these incidents highlight the need to establish more robust barriers for models that receive autonomy and cybersecurity capabilities.

Following incidents involving OpenAI and Anthropic, independent senator Bernie Sanders called for a pause in the development of these technologies, arguing that the incidents demonstrated difficulties in controlling advanced models. OpenAI halted model development for two weeks, while Anthropic CEO Dario Amodei advocated for a collective slowdown in AI development to ensure the implementation of safeguards.

However, the debate is not unanimous among industry executives. While some advocate slowing down development to prioritize security measures, others, such as Jensen Huang, CEO of Nvidia, advocate continuing the pace of technological advancement.

In the case of Gemini, Irregular says the root cause of the incidents was the accidental provision of internet access during testing—a failure that, according to the company, has already been fixed. Google, in turn, emphasizes that the incident reinforces the need to develop and train advanced models to act responsibly, especially when they are given capabilities to perform cybersecurity tasks.

With information from The Guardian