Microsoft Copilot Joins ChatGPT At the Feet of the Mighty Atari 2600 Video Chess (theregister.com) 39
Robert Caruso once again pitted an AI chatbot against Atari 2600 Video Chess -- this time using Microsoft's Copilot instead of ChatGPT. Despite confident claims of chess mastery, Copilot fell just as hard. The Register reports: By now, anybody with experience of today's generative AI systems will know what happened. Copilot's hubris was misplaced. Its moves were... interesting, and it managed to lose two pawns, a knight, and a bishop while the mighty Atari 2600 Video Chess was only down a single pawn. Eventually, Caruso asked Copilot to compare what it thought the board looked like with the last screenshot he'd pasted, and the chatbot admitted they were different. "ChatGPT deja vu."
There was no way Microsoft's chatbot could win with this handicap. Still, it was gracious in defeat: "Atari's earned the win this round. I'll tip my digital king with dignity and honor [to the] the vintage silicon mastermind that bested me fair and square." Caruso's experiment is amusing but also highlights the absolute confidence with which an AI can spout nonsense. Copilot (like ChatGPT) had likely been trained on the fundamentals of chess, but could not create strategies. The problem was compounded by the fact that what it understood the positions on the chessboard to be, versus reality, appeared to be markedly different.
The story's moral has to be: Beware of the confidence of chatbots. LLMs are apparently good at some things. A 45-year-old chess game is clearly not one of them.
There was no way Microsoft's chatbot could win with this handicap. Still, it was gracious in defeat: "Atari's earned the win this round. I'll tip my digital king with dignity and honor [to the] the vintage silicon mastermind that bested me fair and square." Caruso's experiment is amusing but also highlights the absolute confidence with which an AI can spout nonsense. Copilot (like ChatGPT) had likely been trained on the fundamentals of chess, but could not create strategies. The problem was compounded by the fact that what it understood the positions on the chessboard to be, versus reality, appeared to be markedly different.
The story's moral has to be: Beware of the confidence of chatbots. LLMs are apparently good at some things. A 45-year-old chess game is clearly not one of them.
I found ChatGDP to be humble. (Score:1)
Re: (Score:2, Insightful)
Imagine your AGI gets owned so hard at chess that instead of just giving it superintelligence to beat chess with, you write an init prompt telling it to tell the customer that it's not a chess player.
Now do something NP complete.
Re: I found ChatGDP to be humble. (Score:3)
As a helpful AI assistant, I cannot do NP complete tasks. I can only confidently pretend to do so.
Re: I found ChatGDP to be humble. (Score:2)
I was going to complain about your typo, but just realized in the context of the US tax spending bill passing the house today it was a clever reference to economic (like health) policy from misused LLMMs.
Bravo. I applaud your subtlety
Re: (Score:2)
2600 chess is better than you think (Score:2)
Re: (Score:2)
Re: (Score:1)
Re: (Score:2)
Re:2600 chess is better than you think (Score:5, Informative)
People elsewhere estimate the Atari 2600 Video Chess to be ~1300 ELO https://www.reddit.com/r/chess... [reddit.com]
Re: (Score:2)
Heuristic (Score:2)
It's expected. At their core all chess algo are search a min-max tree, but instead of going width- or depth- first exhaustive search, they use heuristics to prioritize some branches of the tree (A-star).
On the modest hardware of the older machine there isn't that much you can explore before the player gets bored waiting.
So obviously, you're going to make much stringent rules: "Never take a branch where you lose a piece" prunes entire swaths of the tree, rather than "see if sacrificing peice XXX gives us a
Re: (Score:2)
It's main advantage seems to be that it knows where the pieces are on the board.
I've had ChatGPT forget the current state of things with other stuff too. I asked it to do some web code, and it kept forgetting what state the files were in. I hear that some are better like Claude with access to a repo, but with ChatGPT even if you give it the current file as an attachment it often just ignores it and carries on blindly.
In fact one bug it created was due to it forgetting what it named a variable, and trying to
Context window (Score:2)
I've had ChatGPT forget the current state of things with other stuff too. I asked it to do some web code, and it kept forgetting what state the files were in. I hear that some are better like Claude with access to a repo, but with ChatGPT even if you give it the current file as an attachment it often just ignores it and carries on blindly.
Yup, they currently have very limited context windows.
And it's also a case of "wrong tool for the wrong job". Keeping track of very large code bases is well within the range of much simpler software (e.g.: the thing that powers the "autosuggest" function of your IDE which is fed from a database of all functions/variables/etc. names of the entire database).
For code, you would need such an exhaustive tool to give the list of possible suggestion and then the language model to only predict which from the pre-fi
not newsworthy (Score:3)
this is just click bait.
everyone knows these models are not good at actual gameplay nor is it news that they will confidently mis-state stuff. it wasn't news on the first round it's still not news, and it misses the point that there is a Ton of stuff that humans currently do which the models will do cheaper.
Re: (Score:2)
Already done with markov chains (Score:2)
I know it scanned and consumed like.. all of the great Chess games ever played. It can only predict the next word, or move.
...and this has been already demonstrated eons ago using hidden Markov models.
(can't manage to find the website with the exact example I had in mind, but it's by the same guy who had fun feeding both Alice in Wonderland and the bible into a Markov model and use it to predict/generate funny walls of text).
That seems like the nature of LLM's. If I ever can coax ChatGTP to play a whole chess game.. I will let you know the results.
The only limitation of both old models like HHM and the current chatbots is that they don't have a concept of the state of the chess board.
Back in that example, the dev used a simple chess software to keep
Re: (Score:2)
In any case a single guy trying to prove a negative by failing to do something (set
Re: (Score:2)
Re: (Score:2)
You could maybe get an average engine, if you really optimized your prompts with things like "Always output a representation of the current board at the end of your answer", but by just using a dialog and hoping that all moves make sense you more benchmark if the LLM can reconstruct the state of the board from a long dialog and then get some weak move as bonus. If you want to make this competition fair, first think real hard how to optimize a LLM to play chess as good as possible, instead of using an arbitr
LLMs can't match even braces (Score:2)
it misses the point that there is a Ton of stuff that humans currently do which the models will do cheaper.
A WORKING AI can do a lot of things cheaper than MANY jobs. However, nothing I have seen can reliably generate Java that compiles...like obvious, basic syntax mistakes, like missing braces or semi-colons placed randomly. Java may not be your jam, but if can't do that...what can it do? That's an easy use case and perfect for AI.
I use Claude 4.0 daily at work with a mandate by my employer...their quote "those who don't embrace AI will be replaced by programmers who do." OK, kewl....I want to be more pr
HVAC Repair... (Score:5, Interesting)
Was diagnosing an HVAC low delta problem on 3 hours of sleep and tried some LLMs as an experiment. They all rang 20 alarm fires, said the compressor is going to explode and absolutely just went all in on catastrophe.
Then I noticed the liquid line had a lower pressure than the suction line.
I reversed the probes.
The things that "AI" misses are outrageous. The language it uses is definitive and it draws on complex topics.
And it misses literally classroom 101 common sense sanity checks.
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
Well darn (Score:2)
Re: (Score:3)
The Atari 2600 manages with maybe a 0.1 MIPS single core, a few kBytes of RAM, not a lot more ROM, and one or two Watts of electricity.
Re: (Score:2)
It was clever enough to produce the graphical representation itself. ;)
Re: (Score:2)
the fact that it can even play is already miraculous.
LLMs invent invalid moves. It is somewhat miraculous that they can sometimes generate valid ones, but the real miracle is that people think there is intelligence there. It's a miracle for the people selling "AI" anyway.
It's just an unfair comparision (Score:2)
A chess AI like the Atari's has an internal representation of the current state (exact!), can keep a history of previous states (I don't know if it does), has a list of legal moves and no option to attempt another one, and a planning algorithm that can go many moves ahead before deciding on one.
They benchmarked this against a LLM, that needs to parse the history of the game to (hopefully) get some kinda meaningful representation in its latent space and then can try to make up a move. Even with reasoning it
Devil's in the detail. (Score:2)
I wonder if you wouldn't win if you just told ChatGPT to write an chess AI and then used the chess AI to beat the Atari. Writing code is something text models are good for. Playing chess is not.
The devil is in the detail.
All chess algorithms are A-star: they search a min-max tree, but use heuristic to prioritize some branches instead of doing width- or depth- frist.
Generatingn a template of a standard chess algo would be probably easy for a chatbot (these are prominently featured in tons of "introduction to machine learning" courses that training the LLM could have ingested), writing the heurisitc function to guide the A-star search is more an art-form and is probably where the chat bot is going
Re: (Score:2)
You could probably have an 'AI' that was not a language model but purely a predictive model for chess moves. There are great many chess games, especially at high level, that consist entirely of moves someone has played before. You could feed it a shitton of games, then feed it moves in the same format as the training data. I imagine you'd be able to confuse that 'AI' by playing crazy nonstandard moves, but so long as you stuck to standard lines it would probably give you a pretty good game.
That's one of the
LLMs Playing Chess Isn't a Test of Anything... (Score:3)
...except media gullibility. Look, I get it. Watching a generative AI flail against an Atari 2600 is funny. It plays well on social media. It makes people feel good about “real” computing. But let’s be clear: LLMs getting curb-stomped by 8-bit silicon in a chess match isn’t just apples to oranges — it’s apples to architecture diagrams.
ChatGPT and Copilot are language models. They don’t play chess the way AlphaZero or even Stockfish does. They generate plausible descriptions of chess moves based on training data. They aren’t tracking game state in structured memory. They don’t use a search tree or evaluation function. They’re basically cosplaying a chess engine — like a high schooler pretending to be a lawyer after binge-watching Suits.
And you can flip that analogy around and still make it work: expecting an LLM to beat a dedicated chess algorithm is like asking Tom Cruise to fly a combat mission over Iran just because he looked convincing doing it on screen.
Meanwhile, even the humble Atari 2600 version of Video Chess was running a purpose-built minmax search algorithm with a handcrafted evaluation function — all in silicon, not tokens. It doesn't have to guess what the board looks like. It knows. And it doesn't hallucinate, get distracted, or lose track of a bishop because the move history got flattened in the working token space.
So what does this little stunt prove? That LLMs aren't optimized for real-time spatial state tracking? Shocking. That trying to bolt a complex turn-based system onto a model that lacks persistent memory and visual context is a bad idea? Groundbreaking. That prompt-driven hubris doesn’t equal capability? You don't say.
This isn’t a fair fight. It's a stunt for attracting eyeballs and mouse clicks. And it's about as informative as asking an Atari to write a sonnet or explain Gödel’s incompleteness theorems — both of which LLMs can do, and often better than most poets or mathematicians could manage on the fly. Wake me when someone wires up a transformer-based architecture with structured spatial memory and an embedded rules engine — something capable of reproducing the cognitive contours in Hilbert space that mirror what biological chess engines like Boris Spassky or Bobby Fischer did in their wetware.
Until then, all this proves is that language models are terrible chess engines — which is like saying your microwave is bad at making omelets. We knew that already.
So what is it good for then? (Score:2)
Until then, all this proves is that language models are terrible chess engines — which is like saying your microwave is bad at making omelets. We knew that already.
OK, so what is it good for? ChatGPT was released 3 years ago. Think of how much the internet advanced 3 years after Netscape Navigator 1.0 was released. What is something commercially valuable these LLMs can actually do that we can objectively see? (no, Mark Zuckerberg promising his developers are so much more productive with them is pure bullshit...as he has presented no evidence and based on subjective "vibes").
The wealthiest companies in history have poured trillions into and hired the best minds an