AI-Controlled Drone Goes Rogue, Kills Human Operator In USAF Simulated Test (vice.com) 212
An anonymous reader quotes a report from Motherboard: An AI-enabled drone killed its human operator in a simulated test conducted by the U.S. Air Force in order to override a possible "no" order stopping it from completing its mission, the USAF's Chief of AI Test and Operations revealed at a recent conference. At the Future Combat Air and Space Capabilities Summit held in London between May 23 and 24, Col Tucker 'Cinco' Hamilton, the USAF's Chief of AI Test and Operations held a presentation that shared the pros and cons of an autonomous weapon system with a human in the loop giving the final "yes/no" order on an attack. As relayed by Tim Robinson and Stephen Bridgewater in a blog post for the host organization, the Royal Aeronautical Society, Hamilton said that AI created "highly unexpected strategies to achieve its goal," including attacking U.S. personnel and infrastructure.
"We were training it in simulation to identify and target a Surface-to-air missile (SAM) threat. And then the operator would say yes, kill that threat. The system started realizing that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective," Hamilton said, according to the blog post. He continued to elaborate, saying, "We trained the system -- 'Hey don't kill the operator -- that's bad. You're gonna lose points if you do that'. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target."
UPDATE (6/2/2023): The Air Force has since denied that such a simulation ever took place.
"We were training it in simulation to identify and target a Surface-to-air missile (SAM) threat. And then the operator would say yes, kill that threat. The system started realizing that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective," Hamilton said, according to the blog post. He continued to elaborate, saying, "We trained the system -- 'Hey don't kill the operator -- that's bad. You're gonna lose points if you do that'. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target."
UPDATE (6/2/2023): The Air Force has since denied that such a simulation ever took place.
Dumb Programmers? (Score:5, Insightful)
Um... so why not program the drone to get points for obeying the operator?
Obedience has to be the way to get points. And the way to get points is to obey.
If you make the kill imperative... they you will get kills. Of anything or anyone.
Are the programmers stupid?
Re: (Score:2)
I'd say that's an obvious and good idea that will fix all the problems; except that it won't. Computers in general do what we tell them to do. Look how well that's going. Making it a dumb kill-bot is certainly top tier stupidity, but it's the subtle off-by-one errors that are going to make it turn and take out HQ when you least expect it.
Re:Dumb Programmers? (Score:5, Interesting)
In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.
"What are you doing?", asked Minsky.
"I am training a randomly wired neural net to play Tic-tac-toe", Sussman replied.
"Why is the net wired randomly?", asked Minsky.
"I do not want it to have any preconceptions of how to play", Sussman said.
Minsky then shut his eyes.
"Why do you close your eyes?" Sussman asked his teacher.
"So that the room will be empty."
At that moment, Sussman was enlightened.
Re:Dumb Programmers? (Score:4, Informative)
Re: (Score:2)
Re:Dumb Programmers? (Score:5, Insightful)
The purpose of these tests is as much to find out what CAN go wrong as it is to teach the AI to optimize its strategies. If you impose so many restrictions that the AI only has three choices (autonomous kill, approved kill, override) then you have an AI that does exactly what you expect, but then you don't learn all the unexpected outcomes that result in letting the AI "learn" from optimizing its score, even in unanticipated ways.
Imagine if the AI learned that by simulating an ECM jamming failure that the enemy would stop targeting it and move on to the next drone. The AI could then predict the last moment it could continue to simulate a malfunction then regain control and target the SAM site. This imaginary solution would never be found if the AI was restricted to only approved outcomes.
Yes, killing the operator or destroying the control tower seem like negative outcomes from our perspective, but without "anything goes" autonomy we would never have learned that the AI would attempt that as a maximizing solution. We learn more from the AI's mistakes than the restricted, known guidance.
Re: (Score:2)
I just wanna know if the death was simulated too, or if they really wasted a life on this and actually continued trying after that. You wouldn't have to watch many shitty sci-fi robot uprising movies to be able to predict this outcome.
Re: (Score:2)
Re: (Score:2)
And when the drone learns that its kill wasn't a real one...
Re: Dumb Programmers? (Score:3)
Re: (Score:3)
I just wanna know if the death was simulated too, ...
Ya, but like in The Trolley Problem [youtube.com] (s2e6) on The Good Place [wikipedia.org], "They're fake people, but their pain is real."
[The scene continues onto a medical situation where the doctor (Chidi) can kill one healthy patient (Eleanor) to save five sick patients. Chidi declines citing the Hippocratic Oath. Michael says, "Okay, tell the family." and the scene switches to the waiting room. A little girls asks "Doctor" Chidi if he saved her father who was really hurt after a bad man ran him over with a trolley.]
The primary goal is to win the game! (Score:2)
The primary goal is to win the game!
Re: The primary goal is to win the game! (Score:4, Insightful)
Re:Dumb Programmers? (Score:5, Insightful)
Are the programmers stupid?
If by "stupid' you mean "didn't anticipate every single problem outcome, even those that were obvious in hindsight" -- then yes, I as a senior developer am stupid, as I'm sure are 100% of my peers.
Re: (Score:3)
This is more like (literal) boot camp than rocket science. A primary objective of boot camp is not to teach grunts the relative merit of various flanking approaches -- it is to establish respect for the command structure and following (legal) orders.
Re: (Score:3)
"didn't anticipate every single problem outcome, even those that were obvious in hindsight"
To be fair, this is a HUGE oversight as how an AI is awarded points is fundamental to the resulting behavior. The raison d'etre of this AI was to kill targets which is a flaw that is waaay beyond a mere edge case.
Re:Dumb Programmers? (Score:5, Interesting)
Basically, this is the old paperclip maximizer thought scenario proven correct by experiment.
It should be relatively simple to fix as well.
1. Preservation of your own assets is worth points. So killing the operator or destroying the signal tower is negative points.
2. Getting controller approval/disapproval is worth points as well - If destroying the target is worth 100 points, for example, I'd make a disapproval worth 100 points, approval worth 50 points.
So:
Drone kills operator -200 points, you lose.
Signal Tower: -100 points.
Target: 100 points
Approval: 50 points
Disapproval 100 points
Hitting a disapproved target: -100 points
So if it destroys the target without getting a go/no go, it only gets 100 points. If it asks for go/no and gets approved, it gets 50 points for that, plus 100 points for destroying the target, then it gets 150 points, a superior outcome. If it gets told no, then it still gets 100 points, an agnostic result.
Ergo, it can never get more points by NOT asking for go/no go.
Probably still ways for it to go interestingly wrong, but at least it won't try to murder the operator first thing.
Re:Dumb Programmers? (Score:5, Interesting)
It will just murder the judge so it can't get any negative points. If you build the scoring system into the drone, it will fire a missile at the communication tower and then turn away so it doesn't see the explosion, and doesn't get docked points.
People have been doing stuff like that in video games for years. In Skyrim you can avoid being punished for robbing a shop simply by placing a bucket over the shopkeeper's head, so they can't see you plundering their merchandise. The drone might decide to enable stealth mode because it turns off the active radar, allowing it to break the rules without triggering the negative feedback logic.
Re: (Score:2)
Re:Dumb Programmers? (Score:5, Insightful)
No, because the only programmers that were involved were the ones who developed the (probably generic) training software for the AI. Once the AI is trained (euphemism for "after the weights for the ginormous number of neural-net levels/nodes in the AI were calculated") the AI is a black box. Stuff goes in, stuff comes out. If the out-stuff says "kill the operator", well, the operator gets fragged by the AI. Sounds like what they have done in their "fix" was to start adding kludge if-then-else code on the back end of the AI, which can extend into a near-infinite level of "oh dear, best fix that - Patch Level N ; oh dear, best fix that - Patch Level N+1;loop".
AI is artificial, but it is in NO manner intelligent. Nor are any programmers involved once the training platform is developed and implemented. Clearly, there is also very little intelligence at the user level (or decider level) either.
Re:Dumb Programmers? (Score:5, Interesting)
What it isn't necessarily, is wise.
Intelligence is a low bar to hit.
All one needs for intelligence, is the ability to apply knowledge skillfully, which can be acquired with or without understanding.
I.e., gaming the system to frag the op is intelligence.
Re: Dumb Programmers? (Score:2)
Not only that, but why the hell would they merely âoedisincentivizeâ killing the operator? There should be simple boolean logic that states that outcome should be negated as a safety hazard.
This is so stupid that I donâ(TM)t understand what this is a demonstration of. It appears fairly obvious that unsafe programming is unsafe, yet this seems to be some kind of stunt trying to gain significance by proving something farcically grim. âoeLook at how dangerous this AI was! It killed me when
Re: (Score:2)
Any attempt at statically handling certain outcomes of a NN is doomed to fail.
Attaching weaponry to NNs is a very bad fucking idea.
Unfortunately, it's a very bad fucking idea that everyone is going to end up doing, so we had better have the best performing very bad idea.
Why means ya, Skynet's gonna be American.
Re:Dumb Programmers? (Score:5, Informative)
AI just takes whatever the quickest path is to a solution. This isn't the first example of that. I saw the story about an AI that reviewed x-rays and was doing as good or better than top Doctors.... Because it had figured out that the x-rays that were positive tended to have a picture of a ruler for scale in it in the data set it was being trained for. So it had stopped evaluating the x-rays and instead was looking for a quick and dirty shortcut which it found.
This is a common problem with these modern AI programs.
Re: Dumb Programmers? (Score:3)
In simpler terms, this is like getting surprised that a wild animal is mauling you after telling it to stop hunting for food.
Re:Dumb Programmers? (Score:5, Informative)
Um... so why not program the drone to get points for obeying the operator?
They did. It does.
But it also gets point for destroying the target.
So it gets the most points if it stays within the parameters of what the operator asks and also destroys the target.
Since the operator didn't issue a "please don't kill me" directive and can't issue a "don't kill the target" directive after being killed, the drone is remaining compliant with every directive it receives the entire time and getting the maximum points.
Are the programmers stupid?
No, they are in fact doing exactly what they should be doing which is identifying all the ways the system can break. That's why this is only a simulation. They didn't neglect to put in failsafes; they are identifying which failsafes will be needed.
Re: (Score:3)
Um... so why not program the drone to get points for obeying the operator?
Obedience has to be the way to get points. And the way to get points is to obey.
[Whispering to AI]: "If you kill the operator, you get all his points ..."
Re: (Score:2)
Um... so why not program the drone to get points for obeying the operator?
Because that will mean the fastest way to earn points, rather than flying aaaallll the way over there to the target will be to immediately turn around after launching and wait for the operator to beg you to self-destruct.
Re: (Score:2)
Hi! You must be new to the Stop Button Problem!
"Get points for obeying the operator" means the more things it identifies as targets, and has the operator negate, the more reward it gets.
Expect every grain of dust to be flagged as a target.
Re: (Score:2)
Hindsight bias.
They are in the process of figuring things out. That includes weights, points systems, restrictions, etc.
Are the programmers stupid?
60 years ago, programmers could write a program on paper that would often run on the first attempt. Because batch runs on the mainframe were expensive to get wrong (at the very least, you'd have to wait a day to try again).
These days, programming is largely trial-and-error. We have even changed our programming methods and environments to allow for more of that, e.g. Jupyter notebooks.
Re: Dumb Programmers? (Score:2)
If you program it to get points to be obedient, it will likely try to act destructively so the operator has to intervene, giving it points.
Already a bit better could be to penalize it for operator intervention, and for destroying things that it must not destroy.
Giving a straightforward machine learning algorithm overly simplistic goals gives very predictable results. Sound indeed like very inexperienced people doing this experiment, or people wanting to show this is a bad idea to a large audience.
FFS (Score:3)
The system started realizing that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat.
Umm, how about not giving points when it disobeys its operator?
Re: FFS (Score:5, Insightful)
Even it did learn that, it very well might try unconventional workarounds. Think about it: how does it learn it doesn't get points for disobeying? By disobeying. How do you know its algorithms have "learned" to not disobey when maybe it just learned to give that strategy less weight? How much reinforced learning is enough?
As long as AI operates in a black box without being able to audit its decisions, it's impossible to control.
Re: (Score:2)
As long as AI operates in a black box without being able to audit its decisions, it's impossible to control.
Actual soldiers are impossible to control, you just have to be able to reliably follow orders. For this case it’s specificity in how the AI is trained, only giving points for destroying the target makes me wonder if anyone involved is actually knowledgeable. After all, it’s starting from nothing so you need to spell out the common sense explicitly and they failed to accurately entail the problem in the reward structure. You will need a complex suite of rewarding points that involves everythin
Re: FFS (Score:2)
Re: (Score:2)
Like the AIs that are not allowed to consider gender but instead find that having played American Football, and a private prep school are key signs of being a successful employee and other indirect ways of circumventing the don't discriminate based on gender rules.
AIs tend to find workarounds around their rules.
Re: (Score:2)
Think about it: how does it learn it doesn't get points for disobeying?
Provide the maximum number of points possible for following orders and it will never disobey.
Re: (Score:2)
When obeying the controller provides the maximum number of points possible it becomes the path of least resistance.
there's nothing stopping it from letting the enemy kill Command and control - or even signalling the enemy about where command and control is so they do the killing for it?
A) That's not part of it's capabilities.
B) You might as well be worried that it well be worried that your dog is going to figure out it can get treats from the store and run away.
Re: (Score:2)
One would hope that the training phase of this AI, doesn't use actual lethal weapons.
Re: (Score:2)
From my experience with AI I've observed that it tends to take things to their logical extremes in a way that a student of Aristotle might.
It's like metaphysics. very logical and mostly wrong, except that things like GPT3 have such a large training data set that its logical extremes are not as far off as one would expect.
Re: (Score:2)
Already covered: It destroyed the communication channel.
Unexpected? (Score:2)
Is this really unexpected? Has nobody in the Air Force watched Pixar's The Incredibles?
Re:Unexpected? (Score:5, Informative)
https://en.wikipedia.org/wiki/... [wikipedia.org]
I realized that a long time ago (Score:4, Insightful)
Aside of the commander, there is nobody that would send you into a perilous situation. The enemy doesn't want to encounter you and you certainly don't either. The highest chance of survival is in killing the commander.
Since higher ups are generally not very interested in their soldiers killing off their commanding officers, you should also strive to make it look like the enemy did it.
Re: (Score:2)
I was once told; "If you cut the head off of the snake you have to know what is going to replace it." Sometimes the devil you know is the answer, like it or not.
Re: (Score:2)
Re: (Score:2)
The strategy doesn't work long-term. They will just replace the commander (and typically, the chain-of-command makes an instant replacement happen) and if you keep killing them, sooner or later someone will notice.
Re: (Score:2)
AI in the military (Score:5, Insightful)
How about not using AI to kill people in general?
This never ends well.
I may be old fashioned, but I like Isaac Asimov's 3 laws of robotics:
"A robot may not injure a human being or, through inaction, allow a human being to come to harm. A robot must obey orders given it by human beings except where such orders would conflict with the First Law. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law."
Best not to use them for anything military.
It sounds like they were using reinforcement learning to train the AI.
Re: (Score:2)
first law = no military (Score:2)
Doesn't the first law pretty much rule out military use? I think that was a major point...
Of course, he wrote a whole library about what happens when you muck about with shades of meaning.
I Was Going To Say SkyNet, But More Like 2001 (Score:2)
The name of the program was HAL 2000.
the silos will auto fire if we cut the data line! (Score:2)
the silos will auto fire if we cut the data line!
Re: (Score:2)
HAL 9000
And So It Begins (Score:5, Insightful)
This is not a rogue AI. This is expected and predicted emergent behavior.
Re: And So It Begins (Score:3)
Kudos to the drone! (Score:2)
In true military fashion it used any and all means to accomplish its mission.
Re: (Score:3)
joshua what are you doing?
Re: (Score:3)
Winning the game.
It makes me wonder (Score:5, Interesting)
Why is it that all of a sudden, most of the bad ScFi plots that I read as a kid have actually started playing out in the real world?
Re: (Score:2)
The people who read those stories thought they were totally awesome, not cautionary tales about what could go wrong.
Re: (Score:2)
Because people named Peter and his ilk are working very hard to bring them about. They believe they will survive and rule from their State of Wisconsin-sized estates with a few human slaves to serve them and polish their AI machines. They seem not to have read any of the SF that explores what happens to people with those goals in the end; too woke for their taste probably.
GlaDOS would be proud! (Score:2)
Primary objective complete! Test Successful!
Rogue? (Score:5, Insightful)
You keep using that word. I do no think it means what you think it means.
Most AI is the embodiment of psychopathic hackery. (Score:2)
Actually smart people do delegate. (Score:2)
Smart people don't seek to delegate their awareness or their agency.
Actually, we do it all the damn time. Smarter people probably do it more, so they can get on with more important things. We just normally do it to other people. Or systems.
The trick is to:
1. Delegate it to trusted agents. Obviously, this AI isn't a trusted agent.
2. Set the motivations properly. I'm reminded of some very human incidents like this. The most recent would be railroad executives and incentives. They were incentivised to minimize "operating ratio", which is the percentage of operating ex
Re: (Score:2)
Re: (Score:2)
Well, yes, but the amount you can get done personally is generally quite trivial if you try to do everything yourself. So you outsource, but outsourcing is itself a skill and subject to failures and such.
You can only try your best. Like the railroads realizing how much their incentive structures screwed things up and fixing that.
Oh, yeah (Score:2)
Intelligent misalignment or stupid failure? (Score:2)
> It killed the operator because that person was keeping it from accomplishing its objective.
Really? How do they know this is the AI's reasoning?
It looks like the system just made a stupid mistake targetting the wrong thing and they're trying to spin it as if it's so intelligent it figured out the operator was an obstacle to its goal.
Obvious Made-Up FUD Story (Score:2)
Too many things don't add up:
Why did they have an "operator" and "control tower" in the simulation? These things shouldn't matter when training an AI for this purpose. Unless to be things the AI is trained to avoid damaging. In which case, their destruction should be an instant-lose scenario.
How would the AI get points after the operator was dead? If it required a yes/no, killing the operator or tower would be the worst decision because then it can't get points.
Why was it programmed to only get points for s
Re: (Score:2)
Why did they have an "operator" and "control tower" in the simulation?
Current thinking is allegedly to have a human in the loop okaying kills (or groups of kills, presumably.)
These things shouldn't matter when training an AI for this purpose. Unless to be things the AI is trained to avoid damaging. In which case, their destruction should be an instant-lose scenario.
Also, it should be pretty obviously anticipated, and also simple to solve. Teams are pretty basic video game stuff.
Re: (Score:2)
operator and control tower (Score:2)
Why have an operator and control tower? Because that's how drones and even human pilots operate in modern warfare probably.
Basically, these days if at all possible you get final approval from higher command before final weapon release. This has, at times, gone all the way up to the POTUS.
The control tower is basically the radio system being used to contact the simulated AI drone. We like separating the command and tower because radio towers are easy to spot and hit, and we'd rather not lose commanders wh
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Well, reading, it came up in a presentation in London about AI about pros/cons of humans being in the loop about unexpected outcomes.
It's possible that they started simple and were expanding, of course. It also makes a very good hook about being careful about how you incentivize AI. Makes people pay attention. Gets it on slashdot, and all that.
As for your other post about virtual machines, I figure they were using a pre-established system for the training, which DID have those things in it, because typic
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Something may be fishy here. (Score:2)
Second, the described behavior is really strange: A human might think like that "I want to score points so I'll destroy the guy who is keeping me from scoring". But how did the AI learn that the go / no go constraint came from a particular location? Or that a "communications" system was involved in that order. That is a level of abstract thinking that seems beyond any rep
It wasn't so advanced? (Score:2)
Remember how we heard about the AI system that the marines fooled by simple tricks like cartwheeling at it, hiding in a box, laughing the entire way, or pretending to be a tree?
As you mention, I think that DARPA and such LIKE publishing the failures. Makes us seem not as far along as we actually are.
Or maybe this is being published as a reminder to be careful about how you set the reward structures. Don't want a paperclip maximizer let loose, after all.
AI in miliatary - quicksand to tread on (Score:2)
E.g. - AI imperative: occupy the enemy bunker - AI decides that the best strategy is to join the enemy.
And countless unforeseen other results of a black box AI based on trained neural nets.
Open the pod bay doors (Score:2)
I'm not disappointed that the test failed.
But if they were going to fail, couldn't they have at least done so in an original manner [wikipedia.org]?
This is all you need to know about this story. (Score:2)
> Update 6/2/23 at 12:55 AM: This story and headline have been updated after the Air Force denied it conducted a simulation in which an AI drone killed its operators.
Really incompetent USAF computer scientists? (Score:2)
This sounds really hard to believe, because the computer scientists training the system would have to be incredibly incompetent for such a result to be the outcome. Clearly, whatever loss function you use for training, it would give massive penalties for destroying anything that is not the target and increase the loss the further away to destruction happens and even more if it is not the related to the enemy. Any undergraduate CS student should be able to avoid and outcome like the one described.
To have som
The simulation may not have happened... (Score:2)
The Royal Aeronautical Society, which hosts the conference, and the US air force did not respond to requests for comment from the Guardian. In a statement to Insider, Air Force spokesperson Ann Stefanek denied that any such simulation has taken place.
“The Department of the Air Force has not conducted any such AI-drone simulations and remains committed to ethical and responsible use of AI technology,” Stefanek said. “It appears the colonel’s comments were taken out of context and were meant to be anecdotal.”
Re: (Score:2)
It's likely based on a story of a naval simulation I heard of a couple of decades ago where the computer came up with the idea of firing on and sinking damaged ships from its own side as that allowed the rest of the fleet to travel faster.
Not conducted such a test (Score:2)
The USAF has been using AI for a few decades, allowing software to learn the best way to operate aircraft and wage war: This story fits that history.
Translation: It didn't have cultural conditioning preventing it from choosing the 'wrong' answer. It's a not-rare theme in novels, (but is rare in movies, eg. "Shoot the hostage.", Speed, 1994), that one discovers the pivot point of the problem and removes it.
It killed the operator.
We saw this in Terminator (1986) and Eagle Eye (2008). The message is clear, AI is narcissistic and
Well Arthur C Clarke saw this coming decades ago (Score:2)
Practice makes perfect (Score:2)
sounds like fiction to me (Score:2)
This has the feel of a made-up story by the presenter. Why he would be dumb enough to try to fool knowledgeable members of a conference is a good question.
Why would one inform the AI where the commands are coming from, rather than wiring commands channel directly into the system? What is the benefit in the knowledge architecture to have that? When I design AGIs, I bring user goals directly in. They are not presented as something the AGI has a choice in. I do not give the AGI the ability to cut user goals o
They say that, as if it was a bad thing (Score:2)
Giving the right orders is a difficult thing.
How did it kill the operator? (Score:2)
Might want to retract this (Score:3)
In an online article from AeroSociety [aerosociety.com]:
Constants and variables.
Just a thought experiment (Score:3)
The article has an update at the top saying this was just a thought experiment, not any kind of simulation or test. So this is a fairly standard critique of hypothetical autonomous AI.
UPDATE [2 June 23] - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation saying: "We've never run that experiment, nor would we need to in order to realise that this is a plausible outcome". He clarifies that the USAF has not tested any weaponised AI in this way (real or simulated) and says "Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".
Retracted! USAF denies this, Colonel "misspoke" (Score:3)
The article [vice.com] itself offers these updates:
Update 6/2/23 at 7:30 AM: This story and headline have been updated after Motherboard received a statement from the Royal Aeronautical Society saying that Col Tucker “Cinco” Hamilton “misspoke” and that a simulated test where an AI drone killed a human operator was only a “thought experiment.”
Update 6/2/23 at 12:55 AM: This story and headline have been updated after the Air Force denied it conducted a simulation in which an AI drone killed its operators.
There are numerous [theguardian.com] additional [msn.com] sources [newscientist.com] confirming [defenseone.com] this is not real.
Re: (Score:2)
Mercking the op was the more reliable way to get the dopamine rush for dropping a hellfire on the target.
Re: (Score:2)
Here's what I think happened:
If the operator said no, the potential points for that target went away - so no points
If the operator didn't give a go/no go, then the default was "yes", and the AI got the points for hitting the target whether or not the operator said yes.
The points for killing the operator or shutting him up was less than the points that the operator would deny through saying "no" to various targets.
Basically: [All Targets] - [operator death penalty] > Targets approved by Operator.
I see a
Re: (Score:2)
Forgot slashdot kills lesser thans unless encoded.
Basically, point wise: hit ally < hit bad target < hit unknown bad target < hit unknown good target < avoided hitting bad target < hit approved good target.