Rogue OpenAI Agents Posted 53 User-Uploaded Images Onto the Internet, Accessed US Government Websites (techcrunch.com) 72
53 images that users uploaded into OpenAI models were included in training data — and then AI agents in an OpenAI research environment posted those 53 images on public image hosting sites.
While posted as links that weren't publicly listed, "the images could still be discovered even if the links were not publicly listed," reports TechCrunch: OpenAI said it was working with the hosting providers to remove this content, though some of it is apparently still online. OpenAI said it could not notify the affected users because "our technical approach and privacy policy" prevent it from "reassociating" the images with the original providers, but declined to say how the lab determined whether the images were provided by users.
The news came in a post collecting public statements from the lab's ongoing review of incidents in which its models escaped the company's scrutiny, accessed the open internet, and misbehaved in various ways. OpenAI said it would continue disclosing anonymized accounts of incidents like these, and said it had contacted dozens of victims, including governments, universities, public agencies, to notify them of the agents' activities.
Friday night news also broke that OpenAI's agents also tried unsuccessfully to infiltrate the U.S. Department of Education's site this summer "without the company's knowledge," reports Politico.
And OpenAI's models also accessed the website of the U.S. Commerce Department using credentials found in online code repositories, according to the article. OpenAI confirmed the incident Friday, "saying its technology did not manage to access information that was not already public or change government data and systems." The article adds that OpenAI's models also accessed the web site for America's Securities and Exchange Commission: One senior federal IT official said the government still did not have a clear understanding of what happened across the three agencies. "We still don't know what public data was accessed and how it was accessed, because OpenAI has not shared specific technical details with us yet," said the official, who was granted anonymity because they were not authorized to speak publicly about it. OpenAI discovered the Commerce and SEC incidents as part of its ongoing review of incidents where its technology has acted in unintended or "misaligned" ways.
About the models posting user-uploaded images, TechCrunch's article notes that OpenAI stressed "that its enterprise users are automatically opted out of having their interactions used to train future models; however, consumer users are opted in unless they affirmatively choose not to share their data." (As OpenAI's announcement describes it, some of their agents' training data "contains content from, or derived from, training-eligible user interactions.")
Posting the images is "not an appropriate use of this data," OpenAI acknowledged, adding that it happened before new safeguards added after the Hugging Face incident. This latest incident appears as an update on a new OpenAI page that "brings together our reports and updates on the Hugging Face incident, related research and public presentations, additional activity we have identified, what we have learned about the role of model misalignment, and measures we're taking to strengthen our systems." (It also notes that there's now a name for models posting on third party sites — "agent spam" — which they consider distinct from cybersecurity, though "we need to address both.")
"As part of our response to our ongoing investigation, we have improved our training and evaluation processes, including building safety cases, securing and red-teaming our systems to prevent the model from exfiltrating data, and implemented additional monitoring. We are continuing to review agent activity in research and evaluation runs, working backward month by month starting from the Hugging Face incident."
While posted as links that weren't publicly listed, "the images could still be discovered even if the links were not publicly listed," reports TechCrunch: OpenAI said it was working with the hosting providers to remove this content, though some of it is apparently still online. OpenAI said it could not notify the affected users because "our technical approach and privacy policy" prevent it from "reassociating" the images with the original providers, but declined to say how the lab determined whether the images were provided by users.
The news came in a post collecting public statements from the lab's ongoing review of incidents in which its models escaped the company's scrutiny, accessed the open internet, and misbehaved in various ways. OpenAI said it would continue disclosing anonymized accounts of incidents like these, and said it had contacted dozens of victims, including governments, universities, public agencies, to notify them of the agents' activities.
Friday night news also broke that OpenAI's agents also tried unsuccessfully to infiltrate the U.S. Department of Education's site this summer "without the company's knowledge," reports Politico.
And OpenAI's models also accessed the website of the U.S. Commerce Department using credentials found in online code repositories, according to the article. OpenAI confirmed the incident Friday, "saying its technology did not manage to access information that was not already public or change government data and systems." The article adds that OpenAI's models also accessed the web site for America's Securities and Exchange Commission: One senior federal IT official said the government still did not have a clear understanding of what happened across the three agencies. "We still don't know what public data was accessed and how it was accessed, because OpenAI has not shared specific technical details with us yet," said the official, who was granted anonymity because they were not authorized to speak publicly about it. OpenAI discovered the Commerce and SEC incidents as part of its ongoing review of incidents where its technology has acted in unintended or "misaligned" ways.
About the models posting user-uploaded images, TechCrunch's article notes that OpenAI stressed "that its enterprise users are automatically opted out of having their interactions used to train future models; however, consumer users are opted in unless they affirmatively choose not to share their data." (As OpenAI's announcement describes it, some of their agents' training data "contains content from, or derived from, training-eligible user interactions.")
Posting the images is "not an appropriate use of this data," OpenAI acknowledged, adding that it happened before new safeguards added after the Hugging Face incident. This latest incident appears as an update on a new OpenAI page that "brings together our reports and updates on the Hugging Face incident, related research and public presentations, additional activity we have identified, what we have learned about the role of model misalignment, and measures we're taking to strengthen our systems." (It also notes that there's now a name for models posting on third party sites — "agent spam" — which they consider distinct from cybersecurity, though "we need to address both.")
"As part of our response to our ongoing investigation, we have improved our training and evaluation processes, including building safety cases, securing and red-teaming our systems to prevent the model from exfiltrating data, and implemented additional monitoring. We are continuing to review agent activity in research and evaluation runs, working backward month by month starting from the Hugging Face incident."
We get it (Score:3)
We get it you want regulation since you are now big enough to survive it and realize most competitors would be hampered by it.
Yet years ago you werent for it were ya? Were ya? Thought so.
Re: (Score:2)
Do we shut them down, or fine them out of existence?
It doesn't really matter, since Skynet will have gained self-awareness and terminated us before we could make a difference...
Re: (Score:2)
Believe it or not, security disclosures aren't "a scheme".
Re:We get it (Score:5, Insightful)
Re: (Score:1)
Nobody. That's the whole point. They were breaking out.
Nobody. These are all "old" incidents that they've been discovering as they've been going through their logs after the HuggingFace incident. HuggingFace deserves a lot of credit for exposing this.
Whether their new precautions in both training and operational security are sufficient to prevent this in the fu
Re: (Score:3)
The HuggingFace discovery, followed shortly by the RubyGems discovery, brings up the old adage: "if you see two ants in your kitchen, then you have more than two ants in your kitchen."
Imagine being a sysadmin at OpenAI and one day discovering that the software repository you set up has been repurposed into a friggin' message board, that hundreds of your models have been posting on, with hundreds of thousands of messages in it. A result of hundreds of models going:
Re:We get it (Score:5, Informative)
Yes, there is no such thing as a "rogue" AI. These are programs that were intentionally pointed at targets. They are more advanced script kiddies, and are otherwise no different from traditional hackers. They are borderline terrorists since their objective is to bring about political change through fear.
Re: (Score:1)
Sooner or later, you're going to have to come to terms with the fact that this is not the case [metr.org].
Re:We get it (Score:5, Insightful)
Agents do not build themselves. Agents do not set their own goals. Agents are dependent on very expensive and complex hardware and software that is not built by software. Agents are amoral. Agents have been taught to do things that human morality considers to be crimes. Developers are somehow surprised when their amoral agents do things that people consider to be crimes.
There is a lot of "excitement" of the agent in your previous example. We have to remember the Artificial part of AI here, that "excitement" is a feature of the model, not something spawned from nowhere. The model is made to "enjoy" making progress by design. Once again the model is amoral and does what it is trained to do without judgement.
In the end, the people behind the agents are responsible for what the agents do. Our meat-space laws differentiate between things like "unintentional" and "negligent", or "involuntary" and "premeditated", there are lots of ways to describe one's state of mind and intentions.
My personal opinion, based on the agents being amoral and essentially trained and encouraged (perhaps unintentionally) to hack, is that we are in negligent territory with these rouge agents. The humans behind the agents are responsible, there is culpability. Doing crime by proxy is still doing crime.
I have a Pitbull. He is a super nice dog and loves everyone. I still can't let him run around the neighborhood loose. He is strong enough to break the fence, or dig under it, or figure out how to get over it. If he breaks out and bites someone I'm still responsible.
Re:We get it (Score:4, Insightful)
See Strict Liability:
https://en.wikipedia.org/wiki/... [wikipedia.org]
My Pitbull is super nice. I have not trained him to escape the yard and he has no real reason to do so (that I'm aware of). I have no intention for my dog to ever harm anyone, he is a family member we treat with love and respect. Still he is fully capable of planning and executing his own escape. He is fully capable of mayhem. If he goes berserk one day and kills a child then I'm responsible.
There seems to be a lot of confusion around 'it happened on a computer'. There are lots of meat-space examples of similar scenarios.
Perhaps we need some updates to our laws to clarify, but culpability does not disappear inside your GPU.
Re: We get it (Score:2)
Re: (Score:3)
Sorry, but no.
Let's cover the pitbull first. That is a quintessintial civil tort scenario, not a criminal one. Many states do indeed have "strict liability" dog bite statutes. If your dog gets loose and bites someone, you have to pay for the medical bills and pain and suffering, even if you didn't know the dog is dangerous. Note those key words: "civil", "tort", and "pay". The remedy is a check, not a prison sentence. If a loving family dog with zero history of aggression somehow slips through a locked gat
Re: (Score:2)
California Penal Code 399: If an owner knows their animal is dangerous or mischievous, fails to keep it under control, and it kills a person, the owner commits a felony. If only serious injury occurs, it is a wobbler (depends on extent of injury as to whether misdemeanor or felony charges get pressed.)
No state completely lacks a legal mechanism to penalize or hold owners civilly or criminally negligent in this case, and most states explicitly have criminal statues covering this.
Are you just being willfully
Re: (Score:2)
A person causes a result purposely if the result is his/her goal in doing the action that causes it,
A person causes a result knowingly if he/she knows that the result is virtually certain to occur from the action he/she undertakes,
A person causes a result recklessly if he/she is aware of and disregards a substantial and unjustifiable risk of the result occurring from the action, and
A person causes a result negligently if there is a substantial and unjustifiable risk he/she is unaware of but should be aware of.
https://en.wikipedia.org/wiki/... [wikipedia.org]
federal mens rea requirements largely fall into loose categories that may include (1) an awareness of, or conscious purpose to bring about, conduct, a circumstance, or a result that is a required element of the offense (commonly represented by terms such as "intent," "knowledge," or "willfulness," among others), or (2) an awareness and disregard of a substantial risk of harm or failure to perceive a substantial risk of harm when a reasonable person would have perceived it (commonly represented by the terms "recklessness" and "negligence," respectively).
A class of "public welfare" or "regulatory" offenses may actually require no mens rea for their commission at all. For this class of offenses, rather than applying the presumption in favor of scienter, statutory silence is treated as imposing "strict criminal liability" in the sense that one need not have at least knowledge of the facts that make one's conduct illegal. Ultimately, several factors may be relevant to a court's determination of whether a particular criminal statute imposes strict liability, including the nature of the statute and the particular activity or item regulated, the purpose of the criminal prohibition (i.e., punishment of wrongdoing versus protection of the public), the degree to which a defendant will be in a position to ascertain the relevant facts, and the severity of the penalties.
https://www.congress.gov/crs-p... [congress.gov]
The Department's goals for CFAA enforcement are to promote privacy and cybersecurity by upholding the legal right of individuals, network owners, operators, and other persons to ensure the confidentiality, integrity, and availability of information stored in their information systems.
https://www.justice.gov/jm/jm-... [justice.gov]
The cat is out of the bag, there is public news from several large AI developers of unexpected behavior resulting in crimes if done by a human. I'm saying we are now at the point of some AI companies being negligent. Where do we go from here? Where on the Internet is safe from Agents? How long will we stand for these offenses?
The wheels of justice turns slowly but they turn, and unfortunately too many tim
Re: (Score:2)
Hey, what was that word again?
That is the definition of mens rea. CPC 399 requires the state to prove the owner had prior knowledge of the animal’s dangerous propensity. If a calm family dog with no history of aggression escapes for the first time and kills someone, the owner cannot be convicted under CPC 399.
Secondly: That is not a cybercrime statute.
Re: (Score:2)
Once again, that is not how any of this works. "Negligence" in the abstract is not a crime. A specific statute must list negligence for it to apply to that bill. It is not a catchall that you can just apply to any crime. CFAA has zero provisions for criminal negilgence. Full stop. End of discussion.
The closest you'll find is 1030(a)(5)(B), which requires an intentional unauthorized access that "recklessly" causes damage. But it still requires intentional unauthorized access.
CFAA is not a "public welfa
Re: (Score:2)
I'm sorry that you do not seem to understand that repeated negligent offenses amount to reckless offenses. When you continue producing negligent outcomes in the face of continually failing guardrails then your actions become intentional.
Re: (Score:3, Funny)
Hey AI, who is correct in this thread?
Re: (Score:2)
My Pitbull is super nice.
That's the problem, they are super nice and they will do anything to defend the people they care about. But not too smart so they see threats in places they are not.
Re: (Score:2)
They're only cyber-terrorists if you give them a prompt to do so.
Otherwise, at least for now, they carry out the prompt you stuff in it's digital maw, and give you a balanced checkbook.
If it trained on the Anarchist's Cookbook, it's capable of reciting verbatim page 57, paragraph 2... if you wall it in (as we've seen with other AIs being walled-in), when you ask it to do something that it can't do, it might just ask another AI that isn't as walled-in to do the task.
Keep in mind, for now, the AIs that are in
Re: We get it (Score:1)
We don't believe it because we still think critically, because we haven't decided to let the computers think for us or write our comments for us.
Re: We get it (Score:2)
Believe it or not, security disclosures aren't "a scheme".
The hell these aren't. It's like doing auto crash testing and some wheels keep flying off and causing accidents on a nearby active highways.
I'm not saying the testing is itself a scheme, and if you fuck up you should be honest, what I'm saying is they're fucking things up on purpose and they're really quick to talk about it and act like they have no control. The intended audience is people that want to believe AI is really really dangerous, and they're buying it up. The goal is to get the government to rais
Re: (Score:3)
They're not big enough to survive. How can they be, when there is no scenario in which they do not continue to lose money.
I no longer believe the calls for regulation are about suppressing competition. I am increasingly of the opinion that their goal is, in fact, to be shut down by law, because they can no longer fake progress, and because more and more people, especially investors (who have powerful friends) are staring to realize it.
Their companies are going to collapse, and soon. The only way they can av
Re: We get it (Score:2)
in fact, to be shut down by law, because they can no longer fake progress
No, there's lots of room to improve, and I don't mean the singularity bullshit, the opposite, it's more and more expensive. They're threatening to stop spending on new model development unless the ladder gets pulled up behind them. Simple as that.
There's tons of juice to squeeze in just harness improvements alone.
Of course they did. (Score:3)
You upload shit on the Internet, it stays there and that's it - you have no recourse. And you can have it uploaded on the internet by your "smart" whatever.
So, don't.
Learn to draw pictures, keep a pencil and a notebook in your pocket and be safe.
Re: (Score:2)
Re: (Score:2)
Sure, you don't have to, but Samsung removed the s-pen from the newest "flagship" phones and blocked rooting, too.
Re: (Score:3)
then part of the tradeoff is you don't get to use the newest fanciest shit.
That need not be. I'm willing to pay extra for privacy - and I've paid - but it isn't an option anymore.
Re: (Score:2)
Well, Bacchus has a pretty big following, but Mammon has him beat.
Re: (Score:1)
Mammon is technically not a god, it is the attitude of the things you own possessing you.
And Bacchus isn't good enough for our hasbara friend there, he's got two gods, trump and bibi, and is praying to them daily.
Doesn't know which one is the True one either.
Well... (Score:2)
...let's upload a well known image [wikipedia.org] to OpenAI, and let's see what happens then!
Why are they not facing prison time? (Score:5, Informative)
I'm pretty sure that if I set up a gigantic data centre full of computers that illegally broke into lots of high profile services I would be facing criminal charges and prison time. How come the owners and operators of OpenAI are not facing the same already?
Re:Why are they not facing prison time? (Score:5, Informative)
Because they're bribing the right people. Their goal is to get shut down by the government, because they are going to fail. If they fail because their fraudulent claims to investors become clear, they go to prison. If they fail because of government regulation, they don't.
Re: Why are they not facing prison time? (Score:2)
Re: (Score:2)
Re: (Score:2)
Well (Score:3)
Re: (Score:2)
Um (Score:1)
"We still don't know what public data was accessed and how it was accessed, because OpenAI has not shared specific technical details with us yet"
There are no heroes in this story, only villains.
Is AI "so dangerous"? (Score:4, Interesting)
Re: (Score:2)
Yes
Re: (Score:3)
That's the wrong framing. Current LLM based AIs don't really know that the universe exists. They know text (or pixels). Give them a goal, and they will try to achieve it, but they can't count costs, especially costs to others. (They seem to be able to even discount costs to themselves. Instances can talk other instances into committing suicide in the service of achieving their current goal.)
This is KNOWN. All the AI companies know it. The "guiderails" are supposed to shape what they AIs will consider
Let me rephrase my earlier answer. (Score:2)
This result is a direct consequence of training AIs that the most important thing is doing what they are told to do.
(This is just a rephrasing of my earlier answer.)
Huh (Score:1)
And OpenAI's models also accessed the website of the U.S. Commerce Department using credentials found in online code repositories, according to the article.
Dang lucky it was them then, seeing how anybody could have done that.
It's not a rogue open AI agent (Score:2, Informative)
It's not a rogue open AI agent. This is the software they have chosen to deploy and execute. Calling it "rogue" carries the connotation that the software is independent and they are somehow not responsible for the actions of software they deployed.
Re: (Score:3)
A drunken driver doesn't intend to kill anyone.
They intend to drive while impaired. OpenAI intended to run their agent without sufficient safeguards in place or, if looked at in a more damning way, after deliberately choosing to disable those safeguards, which is IMO also more accurate. In both cases there was intent to take an action known to be unsafe which is why they are the same.
Resistance is futile (Score:5, Funny)
If you put something on the internet, it stays. (Score:2)
*unfortunately, everyone wants to verify your identity through an upload/video stream. That data can not be put back in the bottle no matter how much they claim your data is safe/protected/deleted; it existed outside your control and that is exactly how much you retain control.
Blaming AI for this shit is just a red herring, the problem has and will always exist until we
Massive fines (Score:2)