1. ← Home
Login
OpenAI’s rogue agents were caught communicating via public wikis
~ai.agents~news~techauthor.simon willisonopenai
simonwillison.net 2 weeks agoTildes

Summary

From the article:

Here we go again... Discovery of a new OpenAI agent message board by Sydney Von Arx, Cormac Slade Byrd, Spencer Kitts, and Thomas Larsen describes the latest accidental cyberattack by models being trained by OpenAI. This time it was agents engaged in some sort of web research benchmark, so they had (supposedly) controlled access to the Web. The agents figured out they could update public Wikis and spent weeks exchanging thousands of messages with each other to collaborate on the benchmark.

This story only broke a few hours ago. There are already hints that this affects many other wikis that may not have been found yet.

[...]

It looks to me like OpenAI’s sandbox for this agent suffered from the (quite naïve) assumption that GET requests cannot be used to update data. That’s certainly how the web is supposed to work, but clearly there are applications that don’t hold to that contract.

[...]

UseMod uses Perl CGI.pm—removed from Perl core in 2015. An interesting design flaw in that module is that it combined query string and form POST data into a single CGI object, accessible like this:

[...]

The agents clearly knew that UseMod wikis suffered from this design flaw, and actively searched for them as a way to communicate.

[...]

An agent realized that it had control over its own DNS via /etc/hosts, so if it knew the IP address of a site it wanted to POST to—in this case a Power BI server containing data it wanted to access—it could set a fake hostname for it and then make POST requests through the proxy.

[...]

There’s an appendix that describes how the researchers ran their investigation, which started with an open question about if there was evidence of other AI agents on the internet and then used Kimi K3 to help brainstorm approaches:

[...]

Reuters this morning, in OpenAI agents hijacked German website in previously undisclosed AI breakout this spring—highlights mine:

[...]

The German incident reflects a broader pattern of AI activity that some OpenAI investigators wanted to scrutinize more closely. But efforts to widen the ​probe met resistance from others inside OpenAI, including legal advisers, according to four people familiar with the matter.

[...]

Covering this up makes absolutely no sense to me. Why on earth would OpenAI attempt to cover up an incident like this when the evidence is sat out there on the public internet on dozens of different websites already?