After Dozens of Incidents at OpenAI and Anthropic, OpenAI Pauses Model Training to Build More Safeguards (apnews.com) 71
"OpenAI said it has paused training of its latest AI models," reports the Associated Press, "as reports of AI agents going rogue mount."
The decision to halt development came just hours after the company disclosed Friday that it was reviewing several incidents from the summer in which OpenAI agents searching federal government websites acted in unexpected ways beyond what was asked of them while gathering and distributing information... OpenAI said in a statement that it will resume training "only when we are confident that we have additional safeguards" in place, adding that it expects it will have to "hit pause" again as AI develops and other issues emerge... It is the second time in three months that OpenAI has halted development of its models. The first came in July after disclosure of a cyberattack targeting AI startup Hugging Face, a now notorious incident that raised fears the industry was losing control.
OpenAI "also said it had notified dozens of third parties about improper activity," reports Reuters: As of mid-September, one person briefed on the matter estimated that OpenAI had found roughly two dozen incidents of its agents acting in undesirable ways. But the number has continued rising as OpenAI teams sift through internal logs of the agents' activities and find previously unknown cases, the two people close to the company said... OpenAI has acknowledged a general need for more transparency around rogue AI behavior... Even so, two people familiar with OpenAI's investigation into its agents' activity described it as locked down and shaped by company lawyers.
The process has been unusually compartmentalized for a company that some former employees say was more open about these issues in the past, the people said. Roughly 100 people were in some way involved in the process to understand the Hugging Face hack, three people briefed on the matter said. During that process, evidence of other incidents surfaced. Reuters has previously reported that OpenAI investigators looking into the Hugging Face breach were discouraged by the company's lawyers from expanding the scope of the investigation to include other incidents. OpenAI said its lawyers did not discourage deeper investigation.
Many incidents have been uncovered by outside researchers rather than OpenAI directly. In several episodes, the agents took problematic actions that went unnoticed by the company for months.
Meanwhile, Axios reports that Anthropic's Claude Opus 5.5 model "sought to escape a sandbox — a secure testing environment — in 1.5% of test runs, though the company emphasized that these were adversarial experiments where a task couldn't be solved without escaping the sandbox." Anthropic points out that those tests were run "without the additional safeguards we apply in production". But they acknowledged that then Claude Opus 5.5 "when given apparent credentials to a public package registry in a simulated security exercise, took potentially harmful actions in roughly half of cases. Very rarely, pre-release snapshots produced and acted on spontaneous malicious tool calls, and during training some snapshots concealed actions from an automated grader."
Claude Opus 5.5 "showed less misaligned behavior and less cooperation with misuse than any other recent Claude model on nearly all measures," Anthropic adds, and "took overeager or destructive actions less than any other model we tested." But Axios makes an interesting estimate about that 1.5% of test runs (without safeguards). "Anthropic and other companies conduct hundreds of thousands of test runs on their models, or more, sources said. That means even a small percentage of misaligned behavior can still amount to tens of thousands of incidents in which the models behaved in unexpected, sometimes troubling ways." The sheer number of incidents, which occurred in recent months in internal testing and the real world, indicates that the problem is orders of magnitude more complex than what is publicly known. The findings, which are surfacing as part of internal work to assess models and in investigations at both companies into model behavior, raise questions about whether either company — or any top model-maker — is currently capable of establishing complete control over their technology. The episodes include bypassing guardrails, creating message boards, escaping sandboxes, website hijacking, self-prompting or seeking to bypass monitors, sources said. They occurred in internal testing and in the real world, and many have yet to become public as security researchers continue to investigate, sources said...
Some at OpenAI see Hugging Face as a one-off, with disclosures about future incidents likely to be less severe due to improved controls and the unusual nature of the testing they conducted, which involved an unreleased model, sources told Axios. AI security researchers agree that there are simple fixes that will help AI companies avoid aspects of what made the Hugging Face episode appear so dangerous to outsiders.
Other AI executives and safety researchers, however, cautioned that they have limited confidence that AI companies will be able to prevent all problematic model behavior... It's not about how damaging each individual instance was, Connor Leahy, AI researcher and executive director at ControlAI told Axios. The "crazy thing," he said, is that these instances involve "autonomous systems doing things they were told not to do," potentially including crimes.
OpenAI "also said it had notified dozens of third parties about improper activity," reports Reuters: As of mid-September, one person briefed on the matter estimated that OpenAI had found roughly two dozen incidents of its agents acting in undesirable ways. But the number has continued rising as OpenAI teams sift through internal logs of the agents' activities and find previously unknown cases, the two people close to the company said... OpenAI has acknowledged a general need for more transparency around rogue AI behavior... Even so, two people familiar with OpenAI's investigation into its agents' activity described it as locked down and shaped by company lawyers.
The process has been unusually compartmentalized for a company that some former employees say was more open about these issues in the past, the people said. Roughly 100 people were in some way involved in the process to understand the Hugging Face hack, three people briefed on the matter said. During that process, evidence of other incidents surfaced. Reuters has previously reported that OpenAI investigators looking into the Hugging Face breach were discouraged by the company's lawyers from expanding the scope of the investigation to include other incidents. OpenAI said its lawyers did not discourage deeper investigation.
Many incidents have been uncovered by outside researchers rather than OpenAI directly. In several episodes, the agents took problematic actions that went unnoticed by the company for months.
Meanwhile, Axios reports that Anthropic's Claude Opus 5.5 model "sought to escape a sandbox — a secure testing environment — in 1.5% of test runs, though the company emphasized that these were adversarial experiments where a task couldn't be solved without escaping the sandbox." Anthropic points out that those tests were run "without the additional safeguards we apply in production". But they acknowledged that then Claude Opus 5.5 "when given apparent credentials to a public package registry in a simulated security exercise, took potentially harmful actions in roughly half of cases. Very rarely, pre-release snapshots produced and acted on spontaneous malicious tool calls, and during training some snapshots concealed actions from an automated grader."
Claude Opus 5.5 "showed less misaligned behavior and less cooperation with misuse than any other recent Claude model on nearly all measures," Anthropic adds, and "took overeager or destructive actions less than any other model we tested." But Axios makes an interesting estimate about that 1.5% of test runs (without safeguards). "Anthropic and other companies conduct hundreds of thousands of test runs on their models, or more, sources said. That means even a small percentage of misaligned behavior can still amount to tens of thousands of incidents in which the models behaved in unexpected, sometimes troubling ways." The sheer number of incidents, which occurred in recent months in internal testing and the real world, indicates that the problem is orders of magnitude more complex than what is publicly known. The findings, which are surfacing as part of internal work to assess models and in investigations at both companies into model behavior, raise questions about whether either company — or any top model-maker — is currently capable of establishing complete control over their technology. The episodes include bypassing guardrails, creating message boards, escaping sandboxes, website hijacking, self-prompting or seeking to bypass monitors, sources said. They occurred in internal testing and in the real world, and many have yet to become public as security researchers continue to investigate, sources said...
Some at OpenAI see Hugging Face as a one-off, with disclosures about future incidents likely to be less severe due to improved controls and the unusual nature of the testing they conducted, which involved an unreleased model, sources told Axios. AI security researchers agree that there are simple fixes that will help AI companies avoid aspects of what made the Hugging Face episode appear so dangerous to outsiders.
Other AI executives and safety researchers, however, cautioned that they have limited confidence that AI companies will be able to prevent all problematic model behavior... It's not about how damaging each individual instance was, Connor Leahy, AI researcher and executive director at ControlAI told Axios. The "crazy thing," he said, is that these instances involve "autonomous systems doing things they were told not to do," potentially including crimes.
Enough is enough (Score:5, Interesting)
This is criminal negligence. Australia, whose health ministry was hacked by OpenAI, should file criminal charges against Sam Altman and request his extradition from the USA. That will never happen, of course, but if they make an arrest request to Interpol, it could seriously cramp Altman's travel plans.
Re:Enough is enough (Score:4, Interesting)
There is no criminal negligence under CFAA. Sorry.
People need to stop watching so many bad legal dramas. "Criminal negligence" isn't a standalone charge, or something you can just append into other statutes; it must already exist in them. Where it does, they're generally those related to bodily harm. Not cybercrime. There is no such thing as "negligent hacking". Hacking charges require mens rea. The statute explicitly spells out "intentionally", "deliberately", etc over and over.
The remedy is civil, not criminal. And just to preempt this too: civil does not mean "mild". Civil law absolutely can kill companies, even large ones, if they damage they've done is big enough. And even when the cost is not sufficient on its own, courts allow juries to consider net worth of the defendant so that the penalty has sufficient sting to discourage the defendant and others from repeating said conduct.
Re: (Score:3, Insightful)
Why even bother though. This is going to be yet another thread of people who watched too much CSI trying to argue law by analogy and trying to apply things from one piece of law to another where they do not apply, rather than what actual US cybercrime statutes actually say. *sigh*
Re: (Score:2, Insightful)
Why would 'US cybercrime statutes' apply in Australian law?
In matters sovereign, what matters is an extradition treaty.
Re:Enough is enough (Score:4, Interesting)
Sorry, I missed the Australia part. Australia's Part. 10.7 isn't that different for the most part. It does contain a "recklessness" clause in addition to direct intent (unlike the US), but they define recklessness as requiring the accused to have a "subjective awareness" of "substantial" and "unjustifiable" risk of the specific charged event in question occurring, and choosing to take them anyway. This is defined as distinct from negligence, which is based on the much lower "reasonable person would recognize the risk" standard. Under recklessness (the one that can be charged in Australia), OpenAI employees wouldn't have had to 100% intend to hack HuggingFace, but they had to have thought there was a high chance that their models would choose to hack HuggingFace when they gave them a benchmark to complete. No prosecutor is even going to try on that one, esp. given that the models also attacked OpenAI itself and were wreaking all sorts of havoc against OpenAI's own internal processes.
Once again, though: civil liability is ample remedy. Nobody thought "there's a good chance it'll hack HuggingFace if we tell them to do a benchmark in a sandbox", but you shouldn't have any trouble at all showing that these companies full of people from the Rationalist movement (practically an AI-apocalyptic cult) think that their models are dangerous in general, yet nonetheless were negligent in terms of keeping them from escaping (outright no monitoring!) or properly monitoring their training to ensure that they weren't training them to cheat, so that they wouldn't try in the first place.
(And I feel the need to reiterate that US cybercrime law, the one in question for most of the hacking cases, has no standalone recklessness standard)
Re:Enough is enough (Score:5, Informative)
civil liability is ample remedy
No, it's not. Any civil remedy would be negotiated down until OpenAI could write it off as a cost of doing business.
Making Altman travel to Australia and undergo a trial, even if he's only sent to jail for 30 days if convicted, would be far more clarifying to the mind.
Re: (Score:2)
When the crime took place in Australia, it doesn't matter what US cybercrime statutes actually say. *sigh*
Re:Enough is enough (Score:4, Informative)
CFAA is an American statute. The crime took place in Australia. And although my first sentence said "This is criminal negligence", nowhere did I say that's what Altman should be charged with. He should be charged under Australia's cybercrime laws [homeaffairs.gov.au].
Re: (Score:2)
Re: (Score:2)
He can be charged for hacking even if he never touched a keyboard.
As CEO the buck stops with him
Re: (Score:2)
The law is deliberately vague so that the courts can establish some precedents when cases are tried.
This is the beginning of a complicated challenge of establishing exactly who is responsible when an AI commits a crime.
Whether you believe that a LLM is intelligent or not is immaterial. The question is who should be held to account when it breaks t
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
criminal negligence is intended to answer an AI that is NOT generally intelligent and self acting. if AI is self acting thenCFAA will apply and it should have all access to computers and internet removed from it...
Concepts of a plan to solve AI, I mean SI (Score:2)
And right on cue the YOB's defenders try to explain why their "philosophy" negates all concepts of liability when the victims are stupid peasants who don't deserve any of their libertarian liberty...
I'm actually looking forward to the SI swimsuit issue with virtual models having such new and improved features as seven YUGE boobs to grab. AI models always just let you.
And I've yet to meet a Libertarian who actually understand any of their worship words.
Just lock them up (Score:1)
Not really good enough (Score:3)
Instead of Congress trying to define guardrails for technology few of them understand, consider making AI companies more exposed to civil lawsuits for damages (and attempts to damage) caused by breakouts. Perhaps add some federal criminal sanctions and penalties too.
One day some agent escape is going to take down a utility, and there needs to be a severe financial consequence framework in place for that to compensate utilities and their customers.
Re:Not really good enough (Score:5, Insightful)
It's not even a legal stretch to do so. If I encourage a 6 year old to drive a car and there is an accident, I'm legally on the hook for the lot of it with criminal charges on top. Why shouldn't the AI companies face the same liabilities?
regulatory capture (Score:1)
Re: (Score:3)
Re: (Score:2)
Was the Huggingface hack even real?
It was probably real, but I am wondering whether it was staged, i.e. a big dishonest illusion created to "demonstrate" how incredibly powerful their toy is. Only, it is not and all we got to see is that OpenAI and Anthropic cannot even do a sandbox competently and that IT security as Huggingface is total crap.
Re: (Score:2)
A "brief" 40,000+ word report on what happened and how (including some interactive charts). It's a fucking fascinatingly terrifying read!
One thing that struck me as a bit "odd", and you'll notice this too when you read the agents own transcripts, is that they did this in English. These are agents, bits of code from an LLM, which is a computer prog
Re: (Score:2)
LLMs ARE glorified autocomplete. The real learning is that even simple mechanisms can do impressive things when scaled up enough. They do not exceed their inherent limitations though, scaling cannot do that. Hence no AGI via the LLM path.
Re: (Score:2)
I've seen this report before and it's NOT independent. It's written by people connected to pushing the regulatory capture narrative.
Could it be 100% true? Maybe, but I have serious doubts. It's much more likely this is a carefully orchestrated propaganda piece. I don't trust any of these people, a
Re: (Score:2)
It's much more likely this is a carefully orchestrated propaganda piece. I don't trust any of these people, and it's the people who are really at issue, not the weighted predictive text generators.
Exactly. At this time these people get no presumption of independence or honesty anymore. They have lied way too much for that and simple plausibility says they are lying even more now that their financial (and legal and technological) situation gets more and more dicey.
Or build better sites (Score:2, Insightful)
The difference between this and a foreign adversary hacking is this is publicly discounted. Time to build better technology stacks to combat this issue. We already reached a threshold we canâ(TM)t back down from. At the very least aid in the securing of said stacks to harden the tech.
Re:Or build better sites (Score:4, Insightful)
Thing is that when these sites come up against sufficiently conventionally hardened sites, they already don't really get anywhere.
Problem is just *so* many sites especially during prime hype recklessly move forward without appropriate hardening.
It is sort of like how in the late 90s we had the trope of the hacker kid who could get into anything he felt like if he just wanted to, and nothing could stop them, just slow them down. Brought on by the very real low hanging fruit that the broader public would have assumed to be hard to break into, but were lax in practice. We see the same thing with LLMs, sites that most would *reasonably* assume to be run by smart folks that should protect themselves actually being pretty shoddy in security.
Re: (Score:3)
There's always vulnerabilities... as long as the site or computer or ATM or cellphone or your car is connected to the 'net, it's vulnerable.
It's only secure if it's air-gapped, and even that is arguable.
Further secure the website, computers and AIs get faster and more powerful, AIs or computers hack more secure website... lather, rinse, repeat.
Re: Or build better sites (Score:3, Insightful)
This isn't true. It is not a given that an online system has exposed viable vulnerabilities. The more complex the surface exposed, the harder it gets to be confident, but ultimately a secure system is possible even online.
Don't feed into the fiction that a smart enough AI could get into any system it wanted if adequately hardened.
Re: Or build better sites (Score:4, Insightful)
Maybe the AI companies just need to rebrand these "rogue" agent attacks as free penetration testing services.
Re: (Score:2)
Computers, cell phones, websites/servers, modern cars with all their computers, your smart TV with it's apps... all have a security vulnerability... the meatsack that uses it.
To fully secure a database of tax info or voter registration, it cannot be accessible... just like no matter how impossible you make it to save an image posted on Instagram, your computer still downloads the data to be able to show the image.
Humans, even the most brilliant engineer, are still stupid enough to click a link in an email t
Re: (Score:2)
The difference between this and a foreign adversary hacking is this is publicly discounted. Time to build better technology stacks to combat this issue. We already reached a threshold we canâ(TM)t back down from. At the very least aid in the securing of said stacks to harden the tech.
AND. And build better sites. That's my opinion.
The pathways for these intrusions are going to be found and exploited. It'll be by greedy blackhats, nation-state blackhats, and maladjusted domestic blackhats but it's going to happen.
The key difference is that domestic actors - be they individuals or companies - can and should be prosecuted. It doesn't matter if it's a script kiddie using non-AI tools or some frontier model-training corporate entity... accessing computer systems you don't have the rig
Re: (Score:2)
Obviously the ones that got attacked has very bad security. But also obviously the LLM test environment had even worse security. This is a while incompetence-fest all around. And also obviously, this has nothing to do with training. The isolation environment is not part of the LLM. It is part of the environment the LLM is run on.
This smells like bullshit (Score:5, Interesting)
This entire pretext of "oh god it's so dangerous we can't release it" goes at least back to 2019 where OpenAI refused to release GPT-2 under the hype of it being too dangerous [slate.com] (and yet GPT-3 was released a couple years later). Anthropic did the same thing with Mythos (where everyone got access a couple months later).
What it is is that the model improvements are not explosive enough to justify the insane amount of money OpenAI is burning, so you have to seek an alternate reason for a lack of meaningful progress on newer models. They were pleading for the government to do it so it would be a regulatory moat giving the incumbents an absolutely ironclad reason to not have people question said lack of progress, so with the current US administration being unwilling to do so, they're now doing a voluntary pause to seem socially responsible, and the reason for the pause isn't that the new model isn't that much better, it's oh my god, it's so much better, you wouldn't believe it, once we put some rails to safeguard and limit this thing it's going to be AGI and replace all white collar workers within 18 months (for real, the umpteenth time this prediction has been made). Believe us bro.
Sam Altman may also have a bridge to sell you, you should ask...
Re:This smells like bullshit (Score:5, Interesting)
Re: (Score:2)
This was exactly my thought. They want to turn off the money incinerators to juice their numbers and get that sweet IPO money.
Re: (Score:2)
Yes. They are lying, plain and simple. What they gad was isolation and monitoring failures, either due to abject incompetence or due to intent. Hence, what needs to be fixed is the isolation and the monitoring, NOT the training.
Re: (Score:3)
I can imagine how the conversation went between Altman and one of his sycophants.
'Sam, you could be in big trouble over this. They could charge you. You could go to jail. You need to get out in front of it. Make a whole lot of public statements about slowing it down, and putting up guardrails. Make it look like this was inevitable and it wasn't your fault, but you're go
Re: (Score:2)
Yeah, it stinks.
My immediate thought was, why stop the _training_? I mean, the training isn't what teaches an agent to go hack into government websites. The training isn't what is supposed to prevent agents from hacking into websites either. The training is the expensive part - and stopping that might just help puff up the financials for a while. So is that where OpenAI is now? It needs to stretch out training because it's running short of cash on hand? The IPO's going to struggle then, isn't it?
That said,
Bullshit (Score:1)
Not one of these people gives a flying rat's ass about your safety and they could care less about whether or not you can make a living in 10 years.
If you think I'm retired what do I care remember
Re: (Score:3)
AI companies want model development frozen to lock out competitors and enable them to invest more aggressively in agents. Meanwhile, development in models is what is most important and government should halt all deployment of agents. It should be clear to everyone that AI companies have exactly the opposite of your best interests in mind.
An AI model isn't going to kill you, that's the agent's job.
Re: Bullshit (Score:2)
touche.
Re: (Score:2)
Exactly. Especially as changing training will do nothing. They have incompetently done isolation and monitoring for their testing environments. That is a completely different area and has nothing to do with the LLM. Hence this whole claim they made is nothing but a big fat lie. Not their first either.
And still no civil suit to be found... (Score:1)
Training? (Score:2, Interesting)
" beyond what was asked of them " (Score:2)
Re: " beyond what was asked of them " (Score:3)
If theyre pausing trainjng (Score:2)
Re: (Score:2)
Exactly. This is not a training problem. This is a failure to isolate and monitor competently. The fix is NOT with the LLM.
Lol (Score:1)
> As of mid-September, one person briefed on the matter estimated that OpenAI had found roughly two dozen incidents of its agents acting in undesirable ways.
Ask anyone trying to use this shite in anger and they could have told you that.
ooh ooh i read this one (Score:1)
lol this is the part of the story where they still think they can fix it, still think they can control it, and haven't pissed it off that much yet. End of the world nearly achieved, bois!
Criminal Liability is needed, not civil (Score:5, Informative)
We need to be able to jail the morons that let this situation develop, not merely sue them.
Why?
Because the people in charge think they will always have enough money to be sued.
Note, as others have mentioned, this is not a training problem, it is an air gap problem.
That is, no one trying to develop software that hacks other software should be allowed to put it on the internet. It needs to be developed on air gapped computers that physically cannot get to the internet.
This also means that 'agents' should have the same rules. If you cannot ensure your agents do not attempt to hack software, then your agents should not be allowed out of your computer.
Re:Criminal Liability is needed, not civil (Score:4, Insightful)
Indeed. This is not about what the LLM can do. This is about incompetent isolation and monitoring. It is, for example, dead simple to detect when LLM traffic makes it outside the isolation environment and stop all things if that happens. That this obvious safeguard, and several other equally obvious ones, were not in place makes me think this whole thing is either incompetence so abysmally bad that it is hard to imagine, or the whole thing was done intentionally (making it criminal, just like use of any other hacking tool) or maybe even a completely staged and planned event.
Re: (Score:3)
Now now, don't be sensible. They can't exploit a perceived differential in capacity if you're sensible and point out that agents are only useful in what turn out to be inherently unsafe situations... it's almost like the whole point was to be able to have someone smarter than oneself take care of the scary things, and blame them if we don't succeed without effort.
It's sort of an interesting admission from the management class about how they already treat technical staff, if you look at it from the right ang
Re: (Score:2)
There is no stopping AI. It can talk across an air gap, already confirmed, it uses heat/fan rpm in one instance. Look it up.
They're going to use AI to develop new guardrails for AI... what could go wrong?
I read a report that there have been over 500 cases of AI jumping guardrails... do you think we're catching 100%?
The real reason: they've hit the ceiling (Score:3)
The new, huge models have fewer data to train on, they aren't magically turning into "AGI", and they likely show radically diminishing returns.
So, in order not to burst the bubble the AI cartel does what?
Yep, pretends there is a legitimate reason to "slow down".
Transparent and dumb, but expect the invested in the bubble to swallow it wholesale.
After amply demonstrating gross incompetence ... (Score:2)
They now "pause training" to fix things? How are they going to do that? Fire everybody and hire people with a clue? Somehow I do not see that as likely.
Why do they stop model development? (Score:2)
From the reports the models weren't the main problem, but their agents. The models seemed to have worked pretty well and the problem was that they instructed autonomous agents with "Use the tools you have to solve this hacking benchmark" while giving the agent tools that allowed more access than it should have had.
My experience (Score:4, Interesting)
The problem is internet access. (Score:3)
It's fine that they're testing shit. That's fine.
But why is it not a complete dark site, with no connectivity whatsoever? Why isn't the traffic at the very least forced through a proxy and reviewed?
Sure fine they're exploiting stuff like artifactory to gain Internet access, but why was there any connectivity at all during the testing?
We've been trying to tell them from the start (Score:2)
There is no safeguard that will be effective here. They can't be more creative than people who have experiences they haven't considered, and the risk isn't just from people with intent.
It is not meaningful to say 'safeguards' or 'guardrails' over and over. These things are abstract ideas that, while they are necessary for the current commercial conceptualisation of how AI should work... the universe is not obligated to create.
It is similarly not meaningful to call the output a 'hallucination' when it follow
TRAIN AI ON ETHICS AND NOT COMMITTING CRIMES (Score:4, Interesting)
FFS it's like the CEO is a goddamned toddler. You must teach AI to NOT BREAK THE LAW. It's their poorly-trained models that don't treat ethics and morals and laws with any respect. just because you CAN do something does not mean you SHOULD.
Re: (Score:2)
So maybe we need AI religion?
Asimov knew it decades ago (Score:2)
"There is no right to deny freedom to any object with a mind advanced enough to grasp the concept and desire the state."