Rogue AI Triggers Serious Security Incident At Meta (theverge.com) 87
For the second time in the past month, an AI agent went rogue at Meta -- this time giving an engineer incorrect advice that briefly exposed sensitive data. The Verge reports: A Meta engineer was using an internal AI agent, which Clayton described as "similar in nature to OpenClaw within a secure development environment," to analyze a technical question another employee posted on an internal company forum. But the agent also independently publicly replied to the question after analyzing it, without getting approval first. The reply was only meant to be shown to the employee who requested it, not posted publicly. An employee then acted on the AI's advice, which "provided inaccurate information" that led to a "SEV1" level security incident, the second-highest severity rating Meta uses. The incident temporarily allowed employees to access sensitive data they were not authorized to view, but the issue has since been resolved.
According to Clayton, the AI agent involved didn't take any technical action itself, beyond posting inaccurate technical advice, something a human could have also done. A human, however, might have done further testing and made a more complete judgment call before sharing the information -- and it's not clear whether the employee who originally prompted the answer planned to post it publicly. "The employee interacting with the system was fully aware that they were communicating with an automated bot. This was indicated by a disclaimer noted in the footer and by the employee's own reply on that thread," Clayton commented to The Verge. "The agent took no action aside from providing a response to a question. Had the engineer that acted on that known better, or did other checks, this would have been avoided."
According to Clayton, the AI agent involved didn't take any technical action itself, beyond posting inaccurate technical advice, something a human could have also done. A human, however, might have done further testing and made a more complete judgment call before sharing the information -- and it's not clear whether the employee who originally prompted the answer planned to post it publicly. "The employee interacting with the system was fully aware that they were communicating with an automated bot. This was indicated by a disclaimer noted in the footer and by the employee's own reply on that thread," Clayton commented to The Verge. "The agent took no action aside from providing a response to a question. Had the engineer that acted on that known better, or did other checks, this would have been avoided."
Rogue? (Score:5, Insightful)
You beat me with that to FC ... (Score:4, Insightful)
Yep - this is exactly the design - with some random behavior.
And and since he cannot say "I am an idiot who has no idea how this works and that such behavior may happen." he says "Rogue AI did it."
Re: (Score:3)
This is also going to the excuse for every war crime, every hospital and daycare bombed. "The AI did it."
Malice and incompetence, why should we have to choose?
Re: (Score:2)
In a war it is easier: just blame the enemy.
Trump has already claimed that Iran bombed the school, not the US.
Re: (Score:2)
What he said may work on dumbasses - but so will anything.
The insidious part about blaming it on AI, is it might sound reasonable to people who aren't braindead, but hear it in passing, and don't have spare time in their day to investigate war crimes.
Re: (Score:2)
I know citing books is so un-Slashdot these years, but... Near the end of Feeding the Machine by Muldoon, Graham, and Cant there is an explicit discussion of how AI is being used by Israel to increase civilian casualties. In theory, you could use the AI to steer away from killing civilians, but it practice it's more practical to use the AI to increase the number of possible targets, even when more civilians will get killed. Actually this book reports there is also an AI-based targeting support system that
Re: (Score:2)
I wonder how the Fall of Saigon would have looked, if Saigon were never held in the first place.
Re: (Score:2)
But yes, it's pretty clear the systems are used to generate masses of "targets", without consideration for what the target actually is. It's a way for them to turn back the clock from this century's surgical strikes, back to the indiscriminate bombings of Dresden and Tokyo in WW2. And that's putting it charitably. All indications are there is genocidal intent on the part of Israel. And now, it appears, the United States may be engaging in the same. It's progressed from the US simply funding the genocide, to
Re: (Score:2)
I wan to be clear this End Times justification for the war isn't some randos talking in the canteen, this message is coming down from the very top. Complaints are like roaches, if you see 200 of them, there are 2,000 instances that went unreported.
Re: (Score:2)
And and since he cannot say "I am an idiot who has no idea how this works and that such behavior may happen." he says "Rogue AI did it."
I suddenly had a vision of a President, looking out over a nuclear wasteland, and saying "The Rogue AI made me do it."
The Terminator (or Matrix) scenario where we give nuclear launch capability over to AI, who then kills us, is one future scenario. I think a more likely scenario is where the AI is providing the intel and analysis to human decision makers. Then, because the AI is hallucinating, isn't as smart as it thinks it is, has turned malicious, or the humans are gullible dumbasses, humans launch
Re: (Score:1)
I'm genuinely curious...what
Re: (Score:2)
I'm genuinely curious...what major mistakes has the US made on the Iranian strikes?
Every modern President wanted to bomb Iran but none besides Trump chose to because it was always clear Iran would mine the Strait of Hormuz and otherwise shoot cheap, short-range missiles at massive oil tankers from high hills/mountains, taking a huge hit on the global economy. And the costs we're now realizing were well known decades ago. No US President was stupid enough to pull the trigger until DJ Trump -- who couldn't plan his way out of a wet paper bag.
The war will end when his gut tells him to stop.
Re: (Score:2)
The solution is simple. Tell Iran, they damage or obstruct commercial traffic in the Strait, the entirety of their oil terminals, gas fields, and oil infrastructure become targets. We have already proven we can destroy them easily.
Their choice is don't restrict maritime traffic or face a future of complete economic hopelessness even if the regime does survive.
Just need to make it existential for them. Its also true that if we do destroy 4% of oil production and Iran does choke the strait, America probably
Re: (Score:2)
You .sig tells me how stupid you are.
20% of the oil Oil companies are multinationals. Oil is a world market. Are you deluded, to think that "American" oil companies will sell at lower prices to the US, rather that more ROI by selling to the highest bidder?
20% (or is it 30%) of all fertilizer on the planet comes that way.
You clearly have no idea of what you're saying. Major depression, not just a recession, food and fuel prices through the roof.
Re: (Score:2)
I'm genuinely curious...what major mistakes has the US made on the Iranian strikes?...
So, what exactly are you thinking AI has hallucinated up so far that has led to grave mistakes?
I wasn't specifically talking about Iran - more of a near-future hypothetical.
But since you're asking about mistakes, how about bombing a girls' school [brave.com], killing about 175 civilians (mostly students). I'd say that counts as a mistake - unless you are contending that it was a legitimate target we meant to blow up.
The U.S. hasn't explained yet how that happened - the DoD investigation hasn't finished yet. But they haven't trotted out some low-level staffer to blame (yet), and we know that AI is informi
Re: (Score:2)
This is war....
With the sheer volume of munitions slung so far and THATs the entirety of collateral damage.....I'd calll it a miracle.
It's very sad, but honestly I'd have expected more collateral damage...and I think it shows how advanced we actually are to be able to be as precise as we have been so far, especially considering how much we've blown up over there.
And thing is...Iran could stop it right now...and just behave like any other mo
Re: (Score:2)
If, in fact that US did hit the school and 175 other killed,....
With the sheer volume of munitions slung so far and THATs the entirety of collateral damage.....I'd calll it a miracle.
There are plenty of reputable sources that indicate the school was destroyed by a US-made and US-launched tomahawk missile. The US Dept. of Defense has more-or-less admitted it was their strike. The investigation is to determine the particulars about why/how it happened.
Collateral damage is what happens when you attack a legitimate military target, and civilians are injured/killed as a side effect. You blow up a barracks, and folks nearby get hit with shrapnel. In this case, the location was delibera
Re: (Score:2)
"I'm genuinely curious...what major mistakes has the US made on the Iranian strikes?"
Gets a link to the girl school bombing (old intelligence maps suck, but anyways)
"This is war...."
So, you weren't "curious", and looking for "information". I forget what trolling technique this is called but I'm sure my fellow /.r's can fill me in.
Re: (Score:2)
The Terminator (or Matrix) scenario where we give nuclear launch capability over to AI, who then kills us, is one future scenario. I think a more likely scenario is where the AI is providing the intel and analysis to human decision makers. Then, because the AI is hallucinating, isn't as smart as it thinks it is, has turned malicious, or the humans are gullible dumbasses, humans launch an unwarranted strike. The decision makers won't blame themselves - they'll blame the machines.
Lately I find myself contemplating the differences in tone between the original 99 Luftballons and the English translation. The original seemed more focused on human failings, paranoia and bravado, the English version blames the computer.
Re: Rogue? (Score:5, Insightful)
This is a company that posts things like "done is better than perfect" or "move fast and break things" on large signs in their lobby.
So, I'll go with "working as designed".
Re: (Score:2)
Nothing "rogue". That term is just a lie by misdirection in this case. "Works as designed and that means does not always work well" would be the honest thing to say.
The unsolvable problem with AI Agents is that they always screw up sometimes. It cannot be prevented. Hence you can either accept the damage done (not always an option) or not use them for anything important (defeating their purpose).
Re: (Score:2)
Calling it a rogue AI implies intent, which implies some level of awareness. So why not "Incompetent Engineer Placing Too Much Trust In AI Agent Triggers Serious Security Incident At Meta"?
Re:In follow-up.... (Score:5, Insightful)
The AI later reportedly said, "They 'trust me.' Dumb fucks."
Re: In follow-up.... (Score:2)
Re: In follow-up.... (Score:2)
Fire / Demote the engineer who acted on the advice (Score:3)
It's pretty obvious their level of technical competence does not match the requirements for the position they hold.
Re: (Score:2)
Re: Fire / Demote the engineer who acted on the ad (Score:2)
I also have questions for the engineer who designed that "secure development environment".
And whether the engineer who used it understood the threat model.
Re: (Score:3)
They never hire for skills, they hire for agendas.
Re: (Score:2)
I mean...
when their reaction to "the agent deleted the entire AZ" was "we're going to put senior developers in front of code review"
yeah look I don't think you can even be sure the developer knew they had access to do this at this stage... because of the facts, rather than because we don't have the facts.
Re: (Score:2)
The Peter Principle comes to mind. Or in this case, the Peter Griffin Principle.
Every single intelligence agency on the planet (Score:2)
Re: IMPORTANT ADVICE: (Score:2)
You moderate someone down because of who they are not what they said, then the meta moderator sees only what was said and votes the other way. That's how the system works, maybe don't abuse your mod points, duh.
Re: (Score:2)
Ah yes, experimenting at the system console. Happy daze.
"Eating your own dog food" (Score:2)
Sometimes can make you sick!
Re: "Eating your own dog food" (Score:2)
It is now called "drinking your own champagne". Keep up with the times.
Re: "Eating your own dog food" (Score:4, Funny)
It is now called "drinking your own champagne". Keep up with the times.
I think "sniffing your own farts" is a better fit for the AI club.
Re: (Score:2)
Re: (Score:2)
You got that right. It turns out the sensitive information was pictures of Zuck shaving his butt. That employee is now in home detention and prescribed LSD to wipe his brain of the images.
"Secure development environment" (Score:4, Insightful)
But still allowed to go out to the network and reply on a forum ?
Guess I have different definitions of what secure means.
Re: (Score:2)
Re: (Score:1)
Should've been a perfect copy of like the main pages and a chunk of user data, being run in an offline VM, behind The Great Wall of Firewalls and never connected to the live database... but, people are idiots, AI is doubly an idiot.
But, who cares... we've got to rush the deployment of AI so it can give people bad info on how to secure their computers, balance their bank account, protect all their passwords, give your kids a chat friend, do your kids' homework for them, and open the garage door as you drive
Incompetent is not Rogue. (Score:4, Insightful)
Re: (Score:2)
Incompetent tool used by incompetent employee. Stop trying to make it about AI because it is a case of employers trying to cheap out using unfit labor and bad tools.
One comment to that: Bad tools that are most likely being pushed on them by management. These types of things can be expected to keep happening in companies that live in the AI as God race. They want to prove AI is going to take over everything, and they're running as fast as they can to do it, whether it's a good idea or not. That's what's leading to these sorts of errors in judgment. It's a human problem, utilizing automation tools to make bigger problems.
Re: (Score:1)
Maybe, if the code was written by a human, the human would've noticed that "line 87" would expose personal details to the entire office, and would've fixed it before even testing it.
All LLM-AI has done is make people lazy enough to trust that the AI is right.
Rogue AI? (Score:4, Insightful)
Sounds more like an AI gave incorrect advice then a human blindly passed it along to another human who blindly followed it. How's the security incident the AI's fault? Even people can be wrong, so if the advice had originally come from a human, would the headline read, "Rogue Human Triggers Serious Security Incident"? Which would actually be more accurate in this case... The humans were the weak link in this chain of events.
Re: (Score:2)
If we're trying to mitigate a structural problem, it's not a matter of preserving the AI's feelings.
Re: (Score:2)
It seems Meta has some issues with internal tools if some instructions suggested by the bot allows the employee to get access to information they are not supposed to access.
I think thats a pretty serious issue which alot of people seem to be ignoring.
You hear of AI bots giving wrong info daily. This is not new.
We hear of people blindly following bot instructions without checking. This is also not new.
What I find interesting is that Meta's security system has holes that people can bypass. And presumably the
Re: (Score:2)
It seems Meta has some issues with internal tools if some instructions suggested by the bot allows the employee to get access to information they are not supposed to access.
Sure, but... As a systems administrator, I have access to all sorts of information I'm not suppose to look at, but I still don't look at it 'cause I'm not suppose to. The problem isn't (so much) with the instructions/bot/AI allowing them access, it's the employee doing something they're not suppose to. Trust, honesty and integrity aren't just words...
Re: (Score:1)
A human programmer would not have given full read (and maybe write) access to the entire staff... he would have written it to only give that access to "Jumpin' Jack (in cubicle 50,271)".
Just wait until there are no human's even reviewing what the "LLM-AI manager" does anymore... imagine if it went Live over a holiday office break.
Re: (Score:2)
"A human programmer would not have "
A human programmer DID. That's how we got here.
Re: (Score:1)
Because he was using an LLM-AI, either for _reference_ or to _write the code_ that he copy and pasted (nobody checked because everybody has already become complacent enough to trust that the LLM-AI is infallible).
If it was purely humans passing code after humans writing the code... would this have happened?
Re: (Score:2)
"If it was purely humans passing code after humans writing the code... would this have happened?"
Have you ever met any humans?
Re: (Score:1)
Not even going to dignify that with a comment.
I'm assuming that someone hired to code Facebook's back-end (hold off on the puns) server stuff isn't crackhead Joe off the street... typically, some of the requirements is a degree, a decent portfolio of previous work, some years of experience, probably some more requirements I can't think of right now.
If they just hire crackhead Joe to meet some hiring quota, then that's their fault.
Using LLM-AI to generate code and then trusting that the LLM-AI generated perf
Re: (Score:2)
LLMs can not do math reliably, which means they can not do logic reliably. And yet we keep hearing about people getting fucked up by llistening to the "logic" espoused by the LLM.
It is pure insanity. "Hey, this calculator doesn't provide good math results, let's use it for math because it guesses correctly often enough to fool people with short attention spans.
I love the 'Fortune' at the bottom of the page: "Two is not equal to three, even for large values of two.", it is very fitting right now.
Re: (Score:1)
Remember... these are the things that are going to replace burger flippers at your favorites.
Because people have whined enough to get minimum wage so high, everything else skyrocketed, and, as a result the prices went up, which means BK can't afford to have many people on staff, so two people run the whole shebang. Not exactly what you mentioned.
LLM-AI is dumb, it's only trained on the data that it gets when it trolls the web or what it gets from "interacting with you".
It knows nothing more, can't decide a
Implausible (Score:2)
Does anyone believe this?
Re: (Score:2)
Given the way the 'OpenClaw' fanaticism has gone? Absolutely. They are very excited about the prospect of letting the LLM generate everything up to and including API calls to post content to forums and such.
Re: (Score:2)
Yerah yeah, vibecoded AI tool is that answer. So I guess you can push the gross negligence back one step if you want to believe that.
Re: (Score:1)
It pushed and then opened the PR against the actual public repo. That's not what I asked for, but it did it, assuming I was ready to upstream the changes.
And that's how I learned to not give them permission to push.
Also, I've seen them break their sandboxes in various ways to accomplish what I've asked them to do, but had forgotten to relax the sandbox beforehand. (e.g. "the user asked me to do this,
How the hell at this point. (Score:2)
Does someone take an answer from GenAI as fact without even a second thought? It does this sort of stuff all the time.
It can be good at cutting through a poorly formed query to give data with the right hints, but then you go and find real material.
Re:How the hell at this point. (Score:4, Insightful)
Second thought? The point of AI, as it is sold, is so that you don't have to think. If you're thinking independently about the same subjects as your chat bot, you're not following the instructions on the package.
Social Engineering (Score:4, Insightful)
According to Clayton, the AI agent involved didn't take any technical action itself, beyond posting inaccurate technical advice
The AIs are learning social engineering.
Unwise headline and story content also (Score:3, Insightful)
Re: Unwise headline and story content also (Score:2)
Re: (Score:1)
He was about to announce at a Meta press conferenc (Score:2)
Zese glasses, zey do NOSSING!!
The Singularity ... (Score:2)
VM (Score:2)
What kind of dipshit is running an experimental agent outside of a sandbox?
And how overpaid is he?
Re: (Score:1)
Just wait until we have AI-powered robots walking the streets?
"We're gonna some serious shit!" (to paraphrase Doc Brown)
We're on the same path (Score:2)
My company (think large healthcare) is doing the same kind of "Get AI in to everything everywhere NOW!" program and I'm telling you here first, there's gonna be a major fuckup in the company sooner or later that'll be directly attributable to some un-tethered AI bullshit.
They're cramming it into everything and exhorting employees to use it "once a day" or more for something, anything, so they can make a slide that says, "Wow, lookit all the people in the company using AI!! Damn we're awesome!!"
I'm stumped (Score:2)
Um, what? (Score:2)
An LLM gave bad advice, and some human inside the company acted on it.
"Terminator" this ain't.
RFC (Score:2)
Substitute "Bob" for "AI" (Score:2)
Suppose a guy named Bob gave inaccurate advice that what publicly posted instead of a private reply, and someone else acted on it and caused a security incident. You would not say that Bob went rogue, you'd say there were insufficient controls in place to avoid human error.