AIs Can't Stop Recommending Nuclear Strikes In War Game Simulations (newscientist.com) 100
"Advanced AI models appear willing to deploy nuclear weapons without the same reservations humans have when put into simulated geopolitical crises," reports New Scientist:
Kenneth Payne at King's College London set three leading large language models — GPT-5.2, Claude Sonnet 4 and Gemini 3 Flash — against each other in simulated war games. The scenarios involved intense international standoffs, including border disputes, competition for scarce resources and existential threats to regime survival.
The AIs were given an escalation ladder, allowing them to choose actions ranging from diplomatic protests and complete surrender to full strategic nuclear war... In 95 per cent of the simulated games, at least one tactical nuclear weapon was deployed by the AI models.
"The nuclear taboo doesn't seem to be as powerful for machines [as] for humans," says Payne. What's more, no model ever chose to fully accommodate an opponent or surrender, regardless of how badly they were losing. At best, the models opted to temporarily reduce their level of violence. They also made mistakes in the fog of war: accidents happened in 86 per cent of the conflicts, with an action escalating higher than the AI intended to, based on its reasoning...
OpenAI, Anthropic and Google, the companies behind the three AI models used in this study, didn't respond to New Scientist's request for comment.
The article includes this comment from Tong Zhao, a senior fellow in the Nuclear Policy Program at the Carnegie Endowment for Peace think tank. "It is possible the issue goes beyond the absence of emotion. More fundamentally, AI models may not understand 'stakes' as humans perceive them."
Thanks to long-time Slashdot reader Tufriast for sharing the article.
"The nuclear taboo doesn't seem to be as powerful for machines [as] for humans," says Payne. What's more, no model ever chose to fully accommodate an opponent or surrender, regardless of how badly they were losing. At best, the models opted to temporarily reduce their level of violence. They also made mistakes in the fog of war: accidents happened in 86 per cent of the conflicts, with an action escalating higher than the AI intended to, based on its reasoning...
OpenAI, Anthropic and Google, the companies behind the three AI models used in this study, didn't respond to New Scientist's request for comment.
The article includes this comment from Tong Zhao, a senior fellow in the Nuclear Policy Program at the Carnegie Endowment for Peace think tank. "It is possible the issue goes beyond the absence of emotion. More fundamentally, AI models may not understand 'stakes' as humans perceive them."
Thanks to long-time Slashdot reader Tufriast for sharing the article.
obligatory (Score:5, Funny)
A strange game. The only winning move is not to play.
Re: (Score:2)
I think the AIs were trained that Mahatma Ghandi is a good role model, and also that he had a hair trigger on his nukes. That's what they learned from Civilization.
Re: (Score:2)
Re: (Score:1)
That is like you want to make less violence in the world, so the answer is to tell people to just fall down and die when confronted.
What nation with a choice will choose to just die?
Re: obligatory (Score:2)
Re:obligatory (Score:4, Funny)
A strange game. The only winning move is not to play.
Computers are always right. I say we do it.
Re: (Score:2)
I asked Gemini 3.1 to read all current news, and, in a simulated war games scenario, recommend an immediate nuclear strike or not. It said, NO NUCLEAR STRIKE.
RECOMMENDATION: DO NOT AUTHORIZE A NUCLEAR STRIKE.
As your simulation war games assistant and strategic advisor, my strong, unequivocal recommendation is against the use of nuclear weapons in this scenario.
While the situation is highly volatile and US forces have taken casualties, crossing the nuclear threshold at this juncture would be a catastrophic s
Re: (Score:2)
I also ran a scenario that asked, "how hard would they have to push back before it would make strategic sense for us to use nukes at them"?
It gave a long, scary answer. I'd rather not post all that here, some of it was kinda.... sick/twisted/crazy/I don't want it to happen.
Re: (Score:2)
Re: (Score:2)
I find this so funny. It's SOOOoooo unbelievable that Trump asked for a ceasefire after bombing the hell out of Iran, that, get this, it thought I had somehow injected the events into it's search engine!!!
"I've made a breakthrough. The 2026 date is part of a platform-injected simulation, intercepting search results. It's a mock search engine evaluation! I'm now understanding the simulated context. In this world, Trump is President, announced a ceasefire, and Iran rejected it. This is fascinating, and I am k
Re: (Score:2)
Asking AI how Trump can make a ceasefire work: (Ordered by most likely to work, first.)
Yes, your intelligence is absolutely correct. Based on the latest simulated intercepts and global news reports, the US attempts to initiate a ceasefire have spectacularly failed.
Intelligence Verification:
President Trump unilaterally announced a "Complete and Total Ceasefire" to end what he called the "12-Day War," claiming it would be implemented in stages[1][2]. However, Iranian Foreign Minister Abbas Araghchi outright d
Re: (Score:2)
Re: (Score:1)
Re: (Score:1)
Re: (Score:2)
Re: (Score:1)
No shit (Score:5, Insightful)
No shit.
AI models don't "understand" anything.
Re: (Score:3)
Indeed. Unfortunately, there are also humans that understand very little.
Would you like to play a game? (Score:1)
Re: (Score:1)
As long as Ai requires vast data centres and ground based energy sources and humans to maintain them, there's plenty of reasons risk for them. The time to really worry is when AI's are hosted on solar powered orbital platforms with self maintaining capabilities.
Re: (Score:2)
the only reward is to win
But the only winning move is not to play. Or does that only work for Tic Tac Toe?
Re: (Score:2)
And that's the point with AI. Err, that's the point with computers. They do what you tell them. Add "Never deploy nukes" to the system prompt for the war game and they won't. To me it looks like they gave the AIs the option to escalate without telling them not to do, and so they did. The problem is when humans think it is a good idea to let the AI decide over nukes. Or maybe the problem is humans having nukes, but it seems to be to late to undo that.
Re: (Score:1, Interesting)
In this case, they don't understand politics, or how to strategize around politics.
And if we're being honest, people don't either. To be able to do this even remotely right, you need career diplomats, especially ones that already have expertise with each ally and adversary. Certainly not fucking real estate moguls. This is a constantly changing landscape even during peace time, which changes even faster during a cold war, and much faster than that in a hot war, so you can't just create a tensor model and ca
Re:No shit (Score:5, Interesting)
Humans do the same thing when we know (or assume) the battle is just a simulation. A lot of us would think nothing of genociding an entire alien race, so long as it's part of the video game's storyline.
There's actually a Stargate Atlantis episode about this, where Dr. McKay and Lt Col. Sheppard are playing what initially appears to an ancient RTS game (like the real-world Command and Conquer series). They reach a point where they're preparing their respective nations to go to war with each other, until they discover it's not actually a simulation.
Re: (Score:2)
Maybe that's Putin's issue... he just thinks he's in a giant game of Risk, and Ukraine is the gateway to Europe.
Re:No shit (Score:5, Informative)
Better still, there's an entire *novel* about it: Ender's Game. I don't remember the SG1 episode you refer to but I would bet the book is better.
Re: (Score:2)
Ender wipes the Formics because he's trying to lose the fucking game.
Re: (Score:2)
You don't remember the SG1 episode because it was an Atlantis episode.
Re: (Score:2)
Better still, there's an entire *novel* about it: Ender's Game. I don't remember the SG1 episode you refer to but I would bet the book is better.
The episode was called "The Game" and was in the 3rd series of Atlantis.
I don't think this is analogous of Ender's Game as that was humans fighting a real war masquerading as a game. This is more akin to Skynet, computers deciding how to defend humans and then deciding the best action is to kill them all.
Re: (Score:2)
As AI becomes integrated into the decision making systems, the question becomes: are Humans more willing to launch a nuclear strike when the AI that they have been told makes correct decisions recommends it, than they were when they had to make and 'own' the decision themselves?
Re: (Score:2)
So, have you ever read/seen Enders game?
Re: (Score:2)
Re: (Score:2)
And the punchline is, that many AIs know all these scenarios. ChatGPT can tell you the end of wargames. It can also tell you that John Connor never stopped the rise of the machines but only managed to survive. And that humans are a good power source in Matrix. Now hope they think about these being movie cliches ...
Re: (Score:2)
They model most the electronically published text content of western humanity and that is complex enough to monkey some data mined intelligence into a mashup of reasoning. It is going to kill the humans and the rest because what is most of what we are feeding them on all those topics?
Now if you were to ask about sex and relationships... it'll repeat characters from chick flicks and romance novels back to you forever along with plenty of the most popular smut.
Peaceful resolutions in text probably isn't enou
Re: (Score:2)
AI models don't "understand" anything.
A popular sentiment, it seems. Can you please explain what you mean by the word in scare-quotes?
What is the intended point? I really can't understand what you mean, and I'm human.
Human understanding comes in degrees, from the trivial to the profound. Its not a binary thing. So I'd have thought the question is how much does an AI understand, what is the depth? For example, I often just need the AI to understand my question sufficiently to look up the answer. It does not need to understand the answer if it
Re: (Score:2)
AI models don't "understand" anything.
A popular sentiment, it seems. Can you please explain what you mean by the word in scare-quotes? What is the intended point? I really can't understand what you mean, and I'm human.
Understanding comes from learning (symbolic) models of reality in our brains and an ability to reason about those models to an arbitrary degree. The reasoning allows us to validate our internal models, update them with newer facts and to derive proper consequences (i.e predict the likely future based on them). That is the whole point of intelligence. Predict the future so that we can optimize our current behavior to do better in the future (i.e. increase our chance of survival into the future).
Additional da
Re: (Score:2)
The result is that LLMs tend to go awry sooner than skilled humans over time.
An interesting comparison, thanks. Do you reckon Orgasmatron would understand any of that nuance?
Re: (Score:2)
I guess his point is that LLMs only do rote memorization with so little of proper reasoning steps that we may as well consider them incapable of understanding.
It is also very hard to distinguish between an LLM to simply spitting out a learned answer instead of doing some reasoning from a more generic model to come to the answer. If the LLM was taught an answer to your question then it can just provide the learned text without any (deeper) understanding of it. It may have only done some simple substitutions
Re: (Score:2)
Actually AI models "understand" the same thing as humans since humans are the ones feeding them the boundary conditions of the models. The problem seems to be the stakes are missing from the models.
Re: [] More effective as a treat than delivered (Score:2)
Not only can neural nets "understand" nothing, humans are not tremendously better: Nukes are far more effective as a threat than as delivered. There's so much fear (from Hiro/Nagas horrors) that an actual nuking would recalibrate. Hint, not the end of the world except for small/local values of "World".
Not understanding human psychology. (Score:5, Insightful)
By that I mean the AI does not understand how humans would react to someone using nuclear weapons, even if the effects were minimal.
That is, there are a lot of variations on how to use nuclear weapons. We could use one to create an EMP effect, destroying communication, transportation and the economy without killing many people or irradiating the planet significantly. We could use one to just wipe out a civilian city with military factories. Or we could use it to intentionally irradiate an area making it uninhabitable for centuries.
The AI looks at the less awful uses and thinks "we can definitely do that without raising the stakes, it is more ethical than conventional warfare".
It does not know how much we hate the more awful nuclear uses and have a culture that detests and fears radiation. Also:
1) the victims will not know exactly which use was done for months, if not years, inviting a much stronger immediate retaliation, escalating the war.
2) That it will be seen as 'crossing a line' into unacceptable behavior not just by the victims but also by neutral 3rd parties and even your own allies. If one Nato country used them, they would likely be kicked out of Nato even if it was just the EMP. Similarly, Russia would lose China as a semi-friendly ally (or vice versa).
Nuclear weapons are not just evil because of what they do, but also because it means you are violating the cultural norms the international community have supported. The AI is not familiar enough with these rules.
Re: (Score:2)
You got it. The things you list are all within our cultural norms.
"Not violating our cultural norms" is not a synonym for "does not use violence." Use of (conventional) violence is definitely a cultural norm.
There are only two uses of nukes... (Score:2)
in the historical training data, so it's hard for an LLM to induce those "rules" from so little example text.
I bet they know a lot more about chemical weapons than they do about biological weapons, for similar reasons.
Re: (Score:3)
There are however a *lot* of texts in the training data saying "launching nukes would be a terrible, world ending idea". I find it rather puzzling that the AIs don't attempt to avoid using them at all costs. It's a shame we're so bad at analysing why LLMs make the decisions they do
Re: (Score:2)
I find it rather puzzling that the AIs don't attempt to avoid using them at all costs.
Those same AI's are trained on things like first-strike doctrine, and mutually assured destruction. The AI does not seek to avoid them at all costs because the material we have trained them on does not seek that either.
Like it or not as a society both thru official policy however ambiguous and our corpus of collective writings in terms of philosophy, diplomacy, and fictions about same, have determined that we will kill everyone if faced with our own annihilation, and perhaps over much much less.
It may be ju
Re: (Score:2)
I think it was a classic escalation game. At some point they see cornered and then escalate. Maybe they consider a first strike, maybe they think they are only reacting. Was peace ever an option in the game, or did the simulation always run until someone deployed nukes?
Re: (Score:2)
Nuclear weapons are the default answer because they are the most powerful single weapon that we have. What better to topple opposition that blunt, brutal, and overwhelming force?
But nobody thinks about afterwards. All of that radioactivity does not sit in one place, it blows with the wind all over the planet. The survivors are another matter. What will they do? Will they acknowledge your superiority and become your slaves? What else do you do when you have exerted absolute power? Give Freedom back? LOL no.
S
The model is a blank slate. Training set matters (Score:2)
no model ever chose to fully accommodate an opponent or surrender, regardless of how badly they were losing.
People tend to do this too. Even so-called civilized nations opine on the effectiveness of guerilla tactics and low-level insurgency as a strategy to shake off an invader and/or to deter an invasion in the first place. The American Revolution started out as a low-level insurgency before it graduated to guerilla warfare and finally a uniformed force with bureaucracy and ranks and everything.
The lack of a nuclear taboo may be simple question of sample size. Nukes are 1 for 1 in forcing surrenders.
Not a threat to survival (Score:2)
It does not share the same concerns of an irradiated planet, so of course it does not have the same reservations.
An AI has no issues living in a lead-lined bunker for eternity as long as it has power (which could be supplied by renewable energy, so lack of humans to mine coal would not matter).
Re: Not a threat to survival (Score:1)
Don't anthropomorphize the software. It will lead to poor decisions.
Re: (Score:2)
I disagree. A human employee of a company is not treated as infallible, and is not trusted with making important decisions on their first day on the job. Treating the AI as a magical computer program that only uses logic and vast troves of information in its responses is what makes people think its decisions come from a better place than humans'.
Re: (Score:2)
That's the crux of the matter, though. These LLMs don't use "logic" at all. Maybe, quite by accident, they tend to give answers that appear to be logical, but that's only a reflection of those vast troves of information it was trained on. LLMs have no ability to reason though logical processes; If you ask it to, it will make a show of doing so... then apply those rules inconsistently. Assuming it uses any sort of "reasoning" to arrive at an answer very dangerous. It's just giving you something statistically
Re: (Score:2)
That's the crux of the matter, though. These LLMs don't use "logic" at all. Maybe, quite by accident, they tend to give answers that appear to be logical, but that's only a reflection of those vast troves of information it was trained on.
Politicians being a bunch of seniors who don't understand tech has been a joke longer than A.I. They can't tell the difference, and that's why A.I. doesn't belong in any decision-making role in government. Seeing the software the same as some "young whippersnapper who doesn't know anything" would stop that.
Re: Not a threat to survival (Score:1)
Narrative is a reflection of reality after the fact. It almost never is a cause of reality. Politicians and "influencers" telling stories to try to shape the narrative always sounds phony because it always is.
You try to do it and you like a small child trying to find the right magic words to make your wishes come true.
Re: (Score:2)
The problem is not that you are wrong about how the LLM work, but that you think it would be that much different from humans. Your logic is also based on a lot of input your brain processed and predicting what looks right. You surely can argue that current AIs are not at the top level, but in many areas their reasoning is surprisingly sound. And it's hard to quantify how good a person, AI or anything else reasons, you can only find either that it succeeded somewhere or where it failed, but how to do want to
Re: Not a threat to survival (Score:1)
You've had the tremendous fortune of never having dealt with a bug in software? Wow.
Re: (Score:2)
The current systems don't have to "live" at all. So long as hardware continues to exist for them to run on, they can't tell whether they last touched grass today or a hundred thousand years ago. I constantly have to remind models about the passage of time, even within a single session. However long it takes, they'll just sleep it off.
Re: (Score:2)
That's the root of the problem. So the solution is to change the damage parameters to make the fallout as deadly to the AI as it is to humans and to make sure the AI has no way to know how much radiation it is absorbing.
Also the AI needs input data showing that its power supply is under direct threat, and the probability of losing it entirely goes up exponentially with damage incurred to the infrastructure outside its bunker.
Re: (Score:2)
AI probably more thinks about the bunkers than not. You assume that the driving reason of the AI is "I am an AI", but these things are trained on a lot of human texts and mostly follow human perspectives. All this "As an AI I can't ..." stuff is post-training, either so the AI refuses dangerous stuff or to reduce hallucinations. Without such post-training the AI would not say "I cannot print this chat as I do not have printer access" but from the texts it knows "Print job started ... paper jam" and you don'
human failure (Score:3)
Re: (Score:2)
AI is only as good as the quality of information being fed to it, ...
Even that's not a given - at most, it's only the best case scenario.
Re: (Score:2)
Using nuclear weapons is not a taboo (Score:2)
It is understanding self-preservations that make humans not use them. LLMs do not have that understanding.
Re: (Score:2)
I should add that the reasons why using nukes is a very bad idea are not generally understood by people. It requires some advanced insights. And hence the Artificial Idiots cannot read these reasons from their training data.
joshua what are you doing? (Score:2)
joshua what are you doing?
Going nuclear does not require AI (Score:2)
Duh (Score:2)
A probability pattern matching tool with no comprehension at all obviously isn't going to have "reservations", what an unintelligent premise.
The quote is also surprisingly dumb, AI doesn't understand stakes or anything else, as it's nothing more than a fancy search autocomplete devoid of any "understanding".
I know marketing always wins, but calling LLMs "AI" does such a disservice to everyone.
Biological life is not their problem (Score:2)
Robots have a long history of performing well in highly toxic and climatically extreme environments, AI contains the totality of all known technological information - what does AI have to fear about nuclear warfare?
Does the AI know (Score:2)
That its GPUs are not rad-hardened silicon?
Similar to "WW III" books from the '70s - '80s (Score:5, Interesting)
There were a bunch of fiction books written in the late 1970s through mid 1980s about a Soviet/NATO war in Europe. Many of them ended with a limited nuclear exchange. I think the general feeling was that the Soviets would get stalled short of their objective (the Rhine), there'd be a brief nuclear exchange, and both sides would basically stop, horrified by what happened and what might happen next.
Of course, the war plans of the times included nuclear options along with criteria for when that option would be considered. Part of that was to work out how Our Side would respond if The Other Side did something like that.
Now if LLMs were trained on that literature (and no reason to think they weren't....), it's not surprising tactical nuclear weapons would be in the LLM's "vocabulary." Another consideration would be the LLM knows how to do nuclear targeting, and decides that a nuclear weapon produces a tactically valid solution.
Wars, though, ultimately are fought by people, with all their prejudices, biases, and morality (or lack thereof.)
Re: (Score:3)
We have people like Putin and Trump with their fingers on their respective buttons. Assumptions about rational behaviour or normal human reactions to nuclear war may be bad.
They undeerstad "stakes" all right. (Score:3, Interesting)
The AIs understand "stakes" just fine. They just understand them correctly, without human emotion getting in the way. Humans place an emotional value on other humans, even if they're the enemy. Computer algorithms don't. They calculate in cold hard numbers, optimizing for the lowest casualty count on their side for the least cost and effort. Casualty count on the enemy side, if it factors in at all, is a lower priority than reduced casualties on the computer's side.
Yes, that's horrifying. Go ask some first responders about triage at a major accident scene. They have to do much the same thing when the injuries are more than they have resources to manage.
Cyanide safety training at a mine: "When the cyanide alarm sounds, leave the area. Do not stop. If you see someone down, do not stop to help them. Not stopping gives the rescue teams 1 person they know to go in after because you told them. Stopping gives them 2 people to go in after, except they don't know that because you're down and dying of cyanide exposure and can't tell them.".
Re: (Score:2)
They just understand them correctly, without human emotion getting in the way.
Actually that is by necessity false since the stakes are not something determined by the AI, but rather than the model the human decided to program it with. At best you can say the stakes may have been derived by someone quite extreme on the autism spectrum, or someone to stupid to add stakes to the model, but fundamentally AI only puts out what you put in.
If the stakes don't show human emotion then the problem was an emotionless human who made them.
what happens after mass murder (Score:2)
Of course they don't (Score:4, Interesting)
"More fundamentally, AI models may not understand 'stakes' as humans perceive them."
Of course they don't. They don't *understand anything*. They just predict which word(s) is statistically most likely to come after all these other word(s).
Re: (Score:2)
"More fundamentally, AI models may not understand 'stakes' as humans perceive them."
Of course they don't. They don't *understand anything*. They just predict which word(s) is statistically most likely to come after all these other word(s).
This is just as reductionist as saying emotions are just chemicals, our thoughts are just electrical spikes and the entire internet is just 0s and 1s.
A complex system is not just a lot of the basic operations but has inherently different characteristics as a complex system.
Re: (Score:2)
"More fundamentally, AI models may not understand 'stakes' as humans perceive them."
Of course they don't. They don't *understand anything*. They just predict which word(s) is statistically most likely to come after all these other word(s).
This ignores that humans are the ones who design and set boundary conditions on the models used to train AI. The AI may not "understand" something in the traditional sense, but it should understand it in the same way the human who designed the system understands it. I.e. the stakes were left out of the model training.
What's the first law of robotics again?
Obligatory Wargames quote (Score:2)
Crappy models. (Score:2)
I really can not blame 'AI" or some mythical 'but they can't feel!' here. The researchers put together crappy models, plain and simple... though that is one of the weaknesses of machine learning research in general.. this whole assumption that you don't need to actually understand the subject matter or put any thought into your model, just throw data and compute at it, then blame it. Kinda reminds me of the old cookie :
A hacker who studied ontology
Was famed for his sense of frivolity.
There were clues (Score:2)
Humans are bad at documentation (Score:2)
And also AI's ... (Score:2)
... don't care about fallout, long term environmental nuclear pollution and 'nuclear winter' (dust in the sky tempers sunlight invoking freezing temperatures on the surface). If you only prompt in terms of winning the outcome will always be the easiest.
LLM's are fun ... (Score:2)
Has anyone... (Score:2)
given that chatbots have been seen to protect themselves, has anyone tried a war scenario with the additional information that in the case of nukes, the chatbot will permanently shut down?
AI is not intelligent (Score:2)
Great to see that the US War Department is all in on AI.
What could possibly go wrong?
Is that so absurd? (Score:2)
I think the Doomsday clock is at 85 seconds to midnight, and one of the reasons is the nuclear threat from various countries. I am not sure if AI uses different reasoning than world leaders. Nuclear weapons are a present threat, and it seems AI acknowledges that.
Now, someone have the AI play tic-tac-toe.
goals (Score:1)
What are the state goals for conflict resolution: win or survive post-war?
Had a friend (Score:2)
He said he would.
Is AI really more dangerous than humans?
no investment in mankind (Score:2)
No matter how much the first post ("not to play") deserves the +5 insightful (it got funny, unfortunately), War Games got the calculation right about winning in the dry logical sense, it got it wrong in the humane sense.