Red-Teaming Evaluators Found 'Novel' Vulnerabilities in OpenAI's Internal Agent Monitoring System

CNBC reports: Nonprofit evaluators METR and Redwood Research on Thursday said they conducted a pilot review of an internal monitoring system at OpenAI... METR and Redwood said the exercise was designed for OpenAI to test what it would look like to give external researchers "a deep level of access" to its internal infrastructure. From July 21 to Aug. 11, an employee from each organization had access to OpenAI's internal codebase, documentation and some technical staffers. The employees were chaperoned while on-site and used OpenAI-issued devices that "never left the premises," according to a release.

The evaluators discovered several "novel vulnerabilities," including an exploit that would allow agents to prevent "hundreds" of recent actions from being monitored. OpenAI told METR and Redwood that it has patched the identified vulnerabilities that impact its production safeguards, according to the release... OpenAI CEO Sam Altman pledged to grant independent evaluators "employee-like access" to the company in September, after Anthropic CEO Dario Amodei proposed the idea as part of his plan to slow the pace of AI development. Anthropic has since announced agreements with organizations like Accenture, METR and SecureBio. OpenAI said earlier this month that it's "actively finalizing contracts with third-party safety assessors."

Am I missing something?

[fuzzyfuzzyfungus]

I fully admit that this might be a 'me' problem, not a them problem; but I just don't understand why this agent monitoring setup was so vulnerable and is being treated as so novel.

Sure, the monitoring of whatever goes on inside the LLM is comparatively new and exotic; but when you set it loose on the internet it's going out through one or more of your WAN links, presumably with your router and firewall and so on in the way, and is mostly doing HTTP(s) stuff. Keeping an eye on what an internal host is doi

Re:

[martin-boundary]

Yes. What you're missing is that AI researchers are, by and large, dumb. There's a handful of scientist types who are subject matter experts and think this makes them smart enough to make decisions in another subject they know nothing about. Then there's a much larger handful of groupies who worship the scientist types and will just go along whatever is proposed, no matter how sketchy. Finally, there's the handful of lowly IT guys who are treated as morons by the researchers, and are not allowed to interfer

OpenAI's tools fail OpenAI

[GanjaManja]

Does anyone else find it hilarious that with OpenAI's hype, their tools are finding vulnerabilities in Other systems (eg. PHP), but Humans had to find the vulnerabilities within OpenAI itself? And specifically, those leaving OpenAI open to hacks by it's own AI systems?