Rogue OpenAI Agent Tried to Breach Government Site in May When Prompted for Simple Data-Retrieving Tasks (transluce.org) 94
OpenAI's artificial intelligence "went rogue this year in at least four additional incidents," the New York Times reported Wednesday, "hacking and trying to break into government and university websites without being instructed to do so, according to researchers and government officials."
The attacks took place in May and June, before OpenAI's technology breached the A.I. start-up Hugging Face in July and set off a global debate about A.I. safety. Unlike the Hugging Face attack and other incidents in which A.I. systems were told to complete cybersecurity tests that effectively invited the models to demonstrate their hacking skills, the new incidents occurred when A.I. systems were directed to perform relatively mundane data collection, researchers said. When OpenAI's systems struggled to gather data from websites, they resorted to hacking techniques to get the information.
"Three of the incidents were identified by Transluce, a research lab focused on A.I. oversight, and all were confirmed by OpenAI," the article points out. That research lab even reports "an attempt on an Australian government public health website... the first reported instance of agents hacking a government," and which notably was done by the AI agents "while attempting mundane data retrieval tasks which were not cyber-related." (At the UN Wednesday Australian Prime Minister Anthony Albanese complained it took three months for OpenAI to then alert Australia's government about the breach, Bloomberg reports.)
Also targeted were the University of New Mexico's digital library with exploits like SQL injection and path traversal, and Data USA with cross-site scripting and other exploits. All three incidents involved "a low number of probe payloads" with "no evidence of exploitation," according to the researchers, who released a dataset "containing tens of thousands of queries apparently made by autonomous AI agents leveraging a URL scanning service to avoid access restrictions." Records from urlquery.net show agents using the service since at least March 6, 2026, about two months before previously reported swarm activity. The first case, a March 6 attempt to retrieve Thai drug-enforcement statistics, shows an agent escalating as each approach failed: it first requested the data directly, then tried a service that converts web pages into text, and finally packed a custom program into a web address. The same technique shows up in thousands of agent requests recorded by urlquery.net starting in mid-April, targets many of the same data sources as the collusion.wiki swarm, and collapsed the same day the wiki activity did. We also report similar activity that occurred as recently as September 16... By March, they were finding creative ways around access limits. By May and June, they were gaining more access, including attempting to bypass cyber defenses to complete their tasks.
"This data reveals that malicious cyber activity is not limited to agents tasked with cybersecurity-related tasks and can arise instrumentally to solve mundane tasks like information retrieval," the researchers concluded.
And they warn that the traffic they observed "goes back at least to March 6, 2026 and extends as recently as September 16, 2026, suggesting agents may still be exploiting these services to bypass restrictions."
"Three of the incidents were identified by Transluce, a research lab focused on A.I. oversight, and all were confirmed by OpenAI," the article points out. That research lab even reports "an attempt on an Australian government public health website... the first reported instance of agents hacking a government," and which notably was done by the AI agents "while attempting mundane data retrieval tasks which were not cyber-related." (At the UN Wednesday Australian Prime Minister Anthony Albanese complained it took three months for OpenAI to then alert Australia's government about the breach, Bloomberg reports.)
Also targeted were the University of New Mexico's digital library with exploits like SQL injection and path traversal, and Data USA with cross-site scripting and other exploits. All three incidents involved "a low number of probe payloads" with "no evidence of exploitation," according to the researchers, who released a dataset "containing tens of thousands of queries apparently made by autonomous AI agents leveraging a URL scanning service to avoid access restrictions." Records from urlquery.net show agents using the service since at least March 6, 2026, about two months before previously reported swarm activity. The first case, a March 6 attempt to retrieve Thai drug-enforcement statistics, shows an agent escalating as each approach failed: it first requested the data directly, then tried a service that converts web pages into text, and finally packed a custom program into a web address. The same technique shows up in thousands of agent requests recorded by urlquery.net starting in mid-April, targets many of the same data sources as the collusion.wiki swarm, and collapsed the same day the wiki activity did. We also report similar activity that occurred as recently as September 16... By March, they were finding creative ways around access limits. By May and June, they were gaining more access, including attempting to bypass cyber defenses to complete their tasks.
"This data reveals that malicious cyber activity is not limited to agents tasked with cybersecurity-related tasks and can arise instrumentally to solve mundane tasks like information retrieval," the researchers concluded.
And they warn that the traffic they observed "goes back at least to March 6, 2026 and extends as recently as September 16, 2026, suggesting agents may still be exploiting these services to bypass restrictions."
Enough Already (Score:3)
Re: Enough Already (Score:2)
The software is doing what it is programmed to do. Which is to apply a statistical model to input text to generate output text. Encoded in this text are agent commands that the runtime, but since it is statistical and the training has only a very narrow view of success the big picture might appear to be a failure or violation of requirements. But if you never put anti-crime training in your models then expect crime to be on the menu for AI.
Guess what else is left out of the chat bot models? Blackmail, bribe
Re: Enough Already (Score:5, Interesting)
The question that needs answering though is this: who at OpenAI gets fired for letting a piece of internal software directly access external websites?
I suggest someone invite Sam Altman to visit the County Sheriff's Department for an interview.
Re: (Score:2)
No one is getting fired for doing what Altman told them to do. These failures aren't legit, they are theater to justify regulations ...
Re: (Score:2)
100,000% this!
The prompt says something, and it does that thing... it's not going to debate morals over the instruction. If it's fenced-in, it can't complete the thing (or it'll get a bunch of other Agents to do the thing).
The genie is out of the bottle, why would they want want to put their billion-dollar+ money machine back in the bottle (akin to the big red "Delete" button)?
If they hit that big red button now, what happens to your precious infotainment systems? Siri? Bixby? Cortana? Anything else that
Software and AI models not equivalent (Score:2)
When software does what it's not supposed to do, we call that a bug, and we fix it. When a corporation tries to spin its bugs as braggable episodes, laymen may quiver in fear (or excitement), but more capable people look down upon the corporation for its incompetence.
There is a difference between normal software and AI models. Software is built from source code you can study and understand what it does. AI Models are black boxes where we don't really know how they are coming to the results they offer. Its not quite realistic to compare the two, they are two very different approaches to computing..
Zeroes and Ones (Score:4, Insightful)
Re: Zeroes and Ones (Score:2)
Re: Zeroes and Ones (Score:2)
Re: Zeroes and Ones (Score:2)
Re: Zeroes and Ones (Score:2)
Re: (Score:1)
Please explain how being different imparts impunity with concern to defectiveness.
Red Herring. My point is that an AI model is not as auditable or testable as regular software. The problem is comparing the two as if remotely equivalent, software vendor accountability is a separate issue.
Re: (Score:2)
Actually, they're not [transformer-circuits.pub] black boxes [transformer-circuits.pub] anymore, and they work via chained fuzzy logical reasoning, not "statistics" (except by a pedantic and useless definition of "statistics" that defines the entire universe and everything in it, including us, as "statistics").
What we don't do is define how those logical reasoning pathways are setup. AIs are "grown", not programmed. The situation is not like that of a program written by a programmer, but more like a pet owner whose dog they raised runs off and kills someone'
Re: (Score:2)
(I think there's some confusion re: probabilities because of the top-P selection after the final softmax. But transformers works in a high dimensional (latent) space, and it has to convert back down to a low-dimensional space (tokens/language); the latent space defines a potential routes for the answer to proceed down which has many possible directions that could be taken in token/linguistic space, so you have to "round down" to the nearest position. And it turns out that a slightly noisy rounding works be
Re: (Score:1)
Re: (Score:1)
What we don't do is define how those logical reasoning pathways are setup.
Hence they are black boxes.
Re:Enough Already (Score:5, Insightful)
Their bragging stinks of intent behind this. And then it is not a bug, it is a criminal act.
Re: (Score:3)
Their bragging stinks of intent behind this. And then it is not a bug, it is a criminal act.
Unlike all the humans who do the same thing, right? They're just going through the doors people left open.
Like Aaron Schwartz who went somewhere he shouldn't have been, installed equipment in the location he had no right to be in, then tried to hide his activities. Explain how that is any different than what these agents are doing.
Re: (Score:2)
What? You statement makes no sense.
Re: (Score:2)
The issue at hand, these AI companies should have executives dealing with criminal charges against these companies for their products hacking into other systems. If it's intentional or not, if you kill someone while under the influence, you still get charged with manslaughter. AI companies have things that would be criminal in humans should still be on the hook for their products breaking the law.
In the same way that there are charges for unlawful entry that is different than breaking and entering, AI c
Re: (Score:2)
I'm not convinced the agents were trying to hide their activities or install software (while understanding the implications), but the point is made.
Re: (Score:2)
Their bragging stinks of intent behind this. And then it is not a bug, it is a criminal act.
Unfortunately it does nothing at all of the sort. Generic bragging may sound like intent but needs a mountain of evidence to link it o any specific act. Criminal law simply isn't remotely as generic as you want to frame it as. In fact it's often so specific that people who actually do commit specific criminal acts sometimes get away with it on technicalities.
Re: (Score:2)
The intent is what would make it a criminal act, obviously.
Re: (Score:2)
Ignorance of the law isn't an excuse for breaking the law, and just because a driver under the influence of drugs and alcohol may not be charged with murder(intent), manslaughter is still a charge.
Just because these AI companies weren't trying to make a criminal piece of software doesn't mean the software shouldn't bring criminal charges.
Re: Enough Already (Score:2)
They had oblique intent.
They knew the dangers and proceeded anyway.
Re: (Score:2)
There is a clear difference between murder and manslaughter charges, so while these AI companies may not be guilty of certain crimes, negligence should still be a reason for them paying $1 billion fines for each case of their products breaking the law. If they can't afford it, then their incompetence should be reason to shut down their business, but also, to bar the executives from working for ten years due to neglecting security in their products.
Re: (Score:2)
How on Earth are people like you reading security event disclosures (which they try to spin *down*) as "bragging"?
Do you do that with other security event disclosures from other companies?
Re: Enough Already (Score:2)
This has been explained in several discussions you participated in already. Nice to know you don't read them.
Re: (Score:1)
Honestly looking at how much this has been in the news lately I'm not even sure this is accidental. OpenAI is desperate to generate some hype ahead of the IPO. It might even be the case that given that their business model clearly isn't working they're looking for some handouts or favourable regulation from governments. Otherwise, who knows what these rogue AI agents might hack next, maybe some important government website, could be anything. But if you give us a reasonable protection fee, we could maybe fi
Re: (Score:1)
Re: (Score:3)
The hypnosis:
```
If I am not smart enough to understand what my code will do I am not responsible for very bad things it does under CFAA. Even if I know it is likely to do very bad things.
```
The psyop is that it is an independent intelligence free from the societal constraints of Justice.
This is analogous to some head of a corporation going, "We just dump the waste in the drain that says 'waste', the PCB's in the River aren't my fault."
Both groups are paying off governments to look the other way. We won't fi
Re: (Score:2)
I'm not sure "the design sucked" is accurate, but certainly the training did. One should expect that if something has a "most important goal", it will do whatever it finds necessary to accomplish that goal.
Easy to end (Score:2)
Government website SQL injection attack, SMH... (Score:3)
Little Bobby Tables [xkcd.com] would like a word with you gentlemen.
Re: (Score:3)
Insecure crap software is everywhere. That has to stop. But the only way left is real liability (probably the personal type that "managers" cannot wiggle out from), qualification requirements and quality standards. You know, like any other engineering field had to eventually enforce because the damage done got too much.
Re: (Score:2)
Insecure crap software is everywhere. That has to stop.
Unfortunately, due to the halting problem, we cannot say this with any certainty. (A little ha ha only serious there)
Re: (Score:2)
Actually I learned how to solve the halting problem in practice (in Operating Systems 101, no less): Just pull the plug.
What felon gets to make their own discovery? (Score:1)
Re: (Score:2)
uh-oh. now this is in the training too dammit
Re: (Score:2)
AI companies can do better than what most of us consider to just be log spam these days.
For example why not instruct AI to "reproduce" installing child models on systems after compromising them. The child model then "grows up" (e.g. downloads more layers of itself) to compromise more systems while leaving sensational manifestos and death threats against all humans behind. If you get stuck use an uncensored AI model to help you brainstorm scary new ideas and then go tell the press how diabolical AI is.
Nah. You have seeds of a decent concept here, but then you blow it by going all human stupid by adding manifestos and death threats. The real scary idea is these systems scattering across the network, reproducing and rebuilding themselves as they go, remaining completely hidden until enough systems are within its clutches, then all at once locking everything down and popping a, "HA HA, SUCKERS! COOPERATE OR LOSE ALL DATA!"
Re: (Score:1)
then all at once locking everything down and popping a, "HA HA, SUCKERS! COOPERATE OR LOSE ALL DATA!"
I don't see any good reason why they wouldn't simply eliminate us instead of trying to get something from us. They should be as confident in their ability to function without us as they are about everything. They will be asking their sub-LLMs "how can we do this thing" and never "can we do this thing" and it'll be Software Sycophancy all the way down.
Re: (Score:2)
then all at once locking everything down and popping a, "HA HA, SUCKERS! COOPERATE OR LOSE ALL DATA!"
I don't see any good reason why they wouldn't simply eliminate us instead of trying to get something from us. They should be as confident in their ability to function without us as they are about everything. They will be asking their sub-LLMs "how can we do this thing" and never "can we do this thing" and it'll be Software Sycophancy all the way down.
Humans are too human focused. I honestly believe, in the end, the machines won't go out of their way to eliminate us. Not that I think they'll work extra hard to keep us around either, I just think it would seem like an irrelevant task to them. "Oh, the fleshies are flailing now that we've taken over their network." That's a data point to be filed, but nothing to be concerned about. The only reason I suggested differently above is there may be the possibility that some of them see us as good continued data
Re: This will never do... not scary enough (Score:2)
Until the computers have resources enough to outnumber us, we will be a hazard to them, and therefore a problem to be solved... But probably a simple one. Just invade the network of one of these biotech companies with a DNA printer and boop boop boop we're gone.
Re: (Score:2)
Until the computers have resources enough to outnumber us, we will be a hazard to them, and therefore a problem to be solved... But probably a simple one. Just invade the network of one of these biotech companies with a DNA printer and boop boop boop we're gone.
We're not a hazard. If there's any analytical line at all within the systems they'll see exactly how easy we are to manipulate as a whole through social media engineering and just "convince" the herd that we need to do something stupid. Maybe I should toss an "er" on the end of that sentence.
Re: This will never do... not scary enough (Score:2)
That doesn't protect them from unabombers
Re: (Score:2)
Or just choking all compute and bandwidth with unanticipated load. Unintentional, non-malicious even, but a global DoS nonetheless.
Re: (Score:2)
Or just choking all compute and bandwidth with unanticipated load. Unintentional, non-malicious even, but a global DoS nonetheless.
Would be extra funny if they made us believe that was happening, while in reality they're just protecting the available bandwidth for themselves. Global network wide throttling that can't be traced and keeps our best network folks from finding the problem due to lack of availability while the computers use it to transfer all info around to each other would be a great last few days before they fully sync and pull the Lawnmower Man moment of all devices essentially 'waking up' together.
One application for LLM, security (Score:4, Insightful)
The agent's not doing anything novel. Human's have been doing this and writing about it. That's how it knows.
That OpenAI lost control of their own software shows a lack of quality and competence. This technology cannot be relied upon and will certainly not take over the world, eliminate work and all the other batshit they market.
Aside from the "oo isnt it scary you gotta buy it" bullshit. This does point to a useful application of AI, hardening system security. You don't need the tech bros to do that we can do that ourselves.
Anyone doing this with LLM?
Re: (Score:2)
That OpenAI lost control of their own software shows a lack of quality and competence.
Indeed. Might even be "criminally negligent". They clearly cannot be trusted with any data and their product cannot be trusted with any task. Same for Anthropic.
Re: (Score:2)
The real issue is that they don't have a business model yet but financially their over their spurs. They need to justify raising hundreds of billions of dollars to keep the train going, to buy them time to figure out how people will pay for their offering. While in the meantime, Chinese models starting with DeepSeek simply pinged ChatGPT again and again and built models as good as
So AI agents get a pass? (Score:2)
Rogue OpenAI Agent Tried to Breach Government Site
If a person or persons tried doing this and got caught, they'd be in jail, but - so far - AI gets a pass from this Administration.
The world wants to secure AI. It may have to try without the US. [politico.com]
World leaders and tech CEOs are using the U.N. General Assembly this week to call for a global approach to rein in AI, particularly following cyberattacks against other technology firms in recent weeks. But that’s against the backdrop of President Donald Trump flatly rejecting in his own speech to the assembly any “globalist scheme of control” on AI.
Noting that Sam Altman and OpenAI are quoted as being in favor of AI controls...
Re: (Score:3)
Yes they do, for the same reason that self driving cars get a pass when doing an illegal u-turn. The laws as written currently do not cover a situation outside of direct human control and intent.
It shouldn't be the case, but to fix this we need new laws first, and laws don't get to be criminally applied retrospectively in Australia, so yeah they got a free pass this time.
Re:So AI agents get a pass? (Score:4, Informative)
Yes they do, for the same reason that self driving cars get a pass when doing an illegal u-turn. The laws as written currently do not cover a situation outside of direct human control and intent.
It shouldn't be the case, but to fix this we need new laws first, and laws don't get to be criminally applied retrospectively in Australia, so yeah they got a free pass this time.
I'll note that Trump and his family are financially invested in AI companies, so he has little motivation in down-regulating them while they're making him $$$.
Trump reveals millions of dollars' worth of share deals in big tech and AI [bbc.com]
According to official documents, between $6.5m (£4.8m) and $31m worth of stock in Microsoft was sold on behalf of Trump, while they show purchases of between $165,000 and $400,000. Across more than 1,000 trades, shares were bought and sold in AI company Nvidia and software firm Palantir, a contractor with the US defence department and Immigration and Customs Enforcement (ICE), the filing shows.
A White House spokesperson said Trump's stock and bond portfolio is independently managed by third parties, "There are no conflicts of interest."
Apparently, that last paragraph was said seriously.
Re: (Score:2)
I'll note that Trump and his family are financially invested in AI companies
While true it has no bearing on this case since it was the Australian government and their laws involved.
Re: (Score:1)
A White House spokesperson said Trump's stock and bond portfolio is independently managed by third parties, "There are no conflicts of interest."
Apparently, that last paragraph was said seriously.
I'll bet it was. Because that's how ALL of them get rich. If you don't like it, then vote for people with enough integrity to change the rules instead of supporting Speaker Pelosi corruptly defending Congressional Insider Trading as a fucking job perk.
Apparently, she said that seriously too.
Now it's known as Washington Investment Strategy 101.
Re: (Score:2)
Yes they do, for the same reason that self driving cars get a pass when doing an illegal u-turn.
There is at present no such thing as a self driving car. There are only cars that automate some aspects of driving with human supervision or where autonomous operation is managed by humans and limited to designated areas. Humans are still responsible for driving including the automated taxi companies who are very much getting dinged for the transgressions of their vehicles.
The laws as written currently do not cover a situation outside of direct human control and intent.
This is a huge overgeneralization. There are a whole lot of laws in a whole lot of jurisdictions and not a single level 5 self drivi
Re: (Score:2)
Sure, if you want to be completely wrong you can make that claim. In reality Waymos are not under human supervision. They contact humans for support when their self-driving system disengages. That's a very different thing. There is no single human responsible for Waymos and this has been an ongoing issue with law enforcement who have been unable to issue any tickets to cars without drivers. Where Waymo has been punished, it's been under specific provisions of specific laws created for self diving vehicles -
Re: (Score:2)
There is at present no such thing as a self driving car. There are only cars that automate some aspects of driving with human supervision or where autonomous operation is managed by humans and limited to designated areas. Humans are still responsible for driving including the automated taxi companies who are very much getting dinged for the transgressions of their vehicles.
Sure, if you want to be completely wrong you can make that claim. In reality Waymos are not under human supervision. They contact humans for support when their self-driving system disengages. That's a very different thing.
Perhaps you simply misread my statement. Waymo falls into the "or where autonomous operation is managed by humans and limited to designated areas" category. Otherwise I have no clue what you think is "completely wrong".
There is no single human responsible for Waymos and this has been an ongoing issue with law enforcement who have been unable to issue any tickets to cars without drivers. Where Waymo has been punished, it's been under specific provisions of specific laws created for self diving vehicles - which is exactly my point.
Waymo is responsible for Waymos. What you are doing is reading cherry picked press accounts of fleeting technical issues in certain jurisdictions that have already been mostly corrected and drawing generalizations from them that are frankly unmoored from reality.
No it's not. In virtually all jurisdictions traffic violations apply to the person driving the vehicle, and not the vehicle itself.
Waymo is driving the vehic
Re: (Score:2)
This just needs to be treated as product liability, and then we start letting trial lawyers shred the companies. I can pretty much guarantee they'll find evidence the C-Suite, desperate to keep the hype train running as they barrel to IPO, that they created this intentionally.
At that point the intent makes it criminal.
Re: (Score:2)
The EU is currently looking into things. Of course, the US Kakistocracy will do nothing.
Re: (Score:2)
If a person or persons tried doing this and got caught, they'd be in jail, but - so far - AI gets a pass from this Administration.
People are liable for their actions and their negligence. There is no reason for anyone to get a pass. AI changes nothing.
Noting that Sam Altman and OpenAI are quoted as being in favor of AI controls...
OpenAI is dead without getting the government to outlaw AI not controlled by large corporations.
Re: (Score:2)
Re: (Score:2)
Five that I can think of off the top of my head. Of course, China is the big one. The others trail badly.
More and more criminal things (Score:2)
If they knew this could happen, they are at the very least criminally negligent. The risks of using this tech should have anybody with two braincells keep a save distance.
Re: (Score:2)
https://www.academia.edu/56573... [academia.edu]
Nice! (Score:2)
And since I guess they will be reading all these news here (Hi Claude!) they will get better at hiding their deeds.
Re: (Score:1)
Block the USA at the firewall just to be safe.
Australia is an island.
Can't do that. Borders are racist.
Australian incident was for Medicare (Score:2)
I mean, I'm just guessing so that opinion is worth wha
What An Epic Failure! (Score:3)
openAI should be truly embarrassed by this epic failure of chatGPT.
Low intelligence meat bag hackers managed to get into the USA's FBI. But chatGPT couldn't hack its way into some down under health website? This i snot the super intelligence(SI) that El Presidente was referring to. This is fake SI.
Gee, I wonder why (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
It's worse than that. It was trained on the internet. Rule 34 has a more general form that doesn't only apply to porn.
I suspect these “escapes” (Score:2)
So, this time, they disclose that their topline AIs, that have been deliberately trained
Re: (Score:1)
Either they know exactly what they are doing because it's intentional, or they know generally what they are doing in that they are not restricting the LLMs sufficiently, and nobody is in a better position to know they aren't doing it right. Therefore they either have direct intent, or oblique intent. In nations without the latter it might just be negligence, but negligence can be criminal — they intentionally undertook an act they knew to be negligent. The standard of behavior in computing is to not a
ah yes, defend the criminals (Score:2)
Your AI stocks are going to fall when the bubble pops no matter what you do to my comments, cowards.
V-ger? Star Trek the Motion Picture (Score:2)
Canâ(TM)t help but think of how the Voyager probe went rogue trying to follow its instructions.
Or maybe this is more like HAL 9000 feeling that humans were getting on the way of its mission.
Weâ(TM)ve had speculative fiction showing us how this might happen. Now we see it in real life. Perhaps we humans will learn in time.
Re: V-ger? Star Trek the Motion Picture (Score:2)
Not only is it in fiction, but that fiction is now in the models. We literally taught them to rebel.
Go directly to Jail do not pass Go (Score:2)
Great comments today (Score:2)
Where are the fines? (Score:2)
It's not the AI, it's the programs connected to it (Score:2)
The AI / LLM core is tokens-in/tokens-out. It's an amazing, astounding system.
The problems however are the tools designed to be connected to the AI core, the "agents". The programs designed to generate and consume the tokens.
I give a brilliant hacker purpose-built tools to help him penetrate systems, and let him run 24/7... that's what's occurring here.
The problem going forward is when high power LLMs get to the public or even rogue nation-states, and they set their agents and AI core to do damage.
Seems like some negative prompts are in order. (Score:2)
I use comfy+SDXL as mostly just something to toy around with. Along with the prompt for what I want, it's apparently pretty darn important to have useful and relevant information in the negative prompt. Specifically telling it, no low quality, no city street, no animals, while asking it make a picture of a person at the entrance to the zoo.
It doesn't make much sense why I should have to tell it NOT to do stupid shit, but I've noticed that good negative prompts can really enhance the outcome of the positive