How Badly Did ChatGPT and Copilot Fail to Predict the Winners of the Kentucky Derby? (courier-journal.com) 29
In 2016, an online "swarm intelligence" platform stunned horse-racing fans by making a correct prediction for the Kentucky Derby — naming all four top finishers in order. (But the next year its predictions weren't even close, with TechRepublic suggesting 2016's race just had an unusual cluster of obvious picks.)
Since then it's become almost a tradition — asking AI to predict the winning horses each year, then see how close it came. So before today's race, a horse named "Journalism" was given the best odds of winning by professional bookmakers — but could AI make a better prediction? USA Today reports: The USA TODAY Network asked Microsoft Copilot AI to simulate the order of finish for the 2025 Kentucky Derby field based on the latest, odds, predictions and race factors on Thursday, May 1. Journalism came out on top in its projection. The AI-generated response cited Journalism's favorable post position (No. 8), which has produced the second-most Kentucky Derby winners and a four-race winning streak that includes last month's Santa Anita Derby.
ChatGPT also picked the exact same horse, according to FanDuel. But in fact, the winning horse turned out to be "Sovereignty" (a horse Copilot predicted would finish second). Meanwhile Copilot's pick for first place ("Journalism") finished in second.
But after that Copilot's picks were way off...
Since then it's become almost a tradition — asking AI to predict the winning horses each year, then see how close it came. So before today's race, a horse named "Journalism" was given the best odds of winning by professional bookmakers — but could AI make a better prediction? USA Today reports: The USA TODAY Network asked Microsoft Copilot AI to simulate the order of finish for the 2025 Kentucky Derby field based on the latest, odds, predictions and race factors on Thursday, May 1. Journalism came out on top in its projection. The AI-generated response cited Journalism's favorable post position (No. 8), which has produced the second-most Kentucky Derby winners and a four-race winning streak that includes last month's Santa Anita Derby.
ChatGPT also picked the exact same horse, according to FanDuel. But in fact, the winning horse turned out to be "Sovereignty" (a horse Copilot predicted would finish second). Meanwhile Copilot's pick for first place ("Journalism") finished in second.
But after that Copilot's picks were way off...
- Copilot's pick for third place was a horse named Rodriguez — which hours later was scratched from the race altogether. (And the next day Copilot's pick for 10th place was also scratched.)
- Copilot's pick for fourth place was "Sandman" — who finished in 18th place.
- Copilot's pick for fifth place was "Burnham Square" — who finished in 11th place.
- Copilot's pick for sixth place was "Luxor Cafe" — who finished in 10th place
- Copilot's pick for seventh place was "Render Judgment" — who finished in 16th place...
An online racing publication also asked "a trained AI LLM tool" for their predictions, and received a wildly uneven prediction:
- Burnham Square (finished 11th)
- Journalism (finished 2nd)
- Sandman (finished 18th)
- Tiztastic (finished 15th)
- Baeza (finished 3rd)
Well obviously reality is at fault (Score:2)
AI is wonderful, AI is powerful, AI should be everywhere, in everything, all the time.
Re: (Score:2)
AI is wonderful, AI is powerful, AI should be everywhere, in everything, all the time.
I'm curious if different people ask the same AI the same predictive question at the same time do they all get the same answer every time?
Re: (Score:2)
Re: (Score:3)
Re: (Score:2)
No. It is reliably hallucinating all the time and not only about the future. Ask the "AI" anything that is a series of numbers twice in a slightly different manner and you're practically guaranteed to get quite different answers.
Re: (Score:2)
Starts at 1:51, or you can find their original paper [transformer-circuits.pub] (sorry it's not in PDF).
Re: (Score:2)
"I'm curious if different people ask the same AI the same predictive question at the same time do they all get the same answer every time?"
It depends.
You get the same probabilities every time. If you do greedy sampling, you get the same result every time. If you have an AI with control over the samplers, set temperature to 0 or use a TopK sampler with k=1.
If you do random sampling, it depends on how likely the answer is. If you ask if the sky is blue, then the word "potato" is a pretty unlikely answer, whil
Re:Well obviously reality is at fault (Score:4, Insightful)
It also fails every week to predict the lotto numbers.
Re: (Score:2)
AI is wonderful, AI is powerful, AI should be everywhere, in everything, all the time.
Lets see how well it will excel at leading a Gamblers Anonymous meeting then.
Obvious shitty problem, is obvious.
Re: Well obviously reality is at fault (Score:2)
I hate to be the one defending AIs, but these things aren't made to predict basically random events. Anyone who tries to use AIs as gambling assistants is monumentally dumb, even dumber than the AIs.
To these who will say that a horse race isn't random, the AI as a basically statistical machine has no way of knowing the real factors which determine the race's outcome.
Re: (Score:2)
I hate to be the one defending AIs, but these things aren't made to predict basically random events. Anyone who tries to use AIs as gambling assistants is monumentally dumb, even dumber than the AIs.
If a gambling system has been legalized because of the design elements that give the player a 1% better chance of winning over the house, don’t assume it’s going to be anything other than the machine that excels at winning consistently.
To these who will say that a horse race isn't random, the AI as a basically statistical machine has no way of knowing the real factors which determine the race's outcome.
You mean the statistical machine won’t use statistics to predict a winner more accurately? Odd, because that’s how everyone else gains an advantage. Does AI even know HOW to gamble? Serious question.
How badly did the editor fail to read the winners? (Score:2)
ChatGPT at least has the excuse of making a prediction before the race. The "person" writing TFA got the finishing order wrong. (I assume the submission of actually from AI.)
Sandman finished 7th, not 18th. Burnham Square finished 6th, not 11th.
Re: How badly did the editor fail to read the winn (Score:4, Funny)
"I assume the submission of actually from AI."
What kind of dressing goes with AI word salad?
Re: How badly did the editor fail to read the win (Score:4, Funny)
Re: (Score:2)
Re: How many horses did they put down? (Score:2)
Well, at least they're not eaten. Usually. Humans are cruel towards animals. This is the way of humanity.
Re: How many horses did they put down? (Score:2)
Forgot to mention that humans are cruel to their fellow humans too... Have you watched a MMA event lately?
Nonsense (Score:1)
I don't think AI predictions on horse racing is something an average slashdot reader is interested in. This is as yellow and useless as "tech news" can get.
Re: (Score:2)
PLEASE DO POST (Score:1)
PLEASE DO POST everyone's predictions for some stupid 1800s horse race that were either correct or incorrect.
Start with real people who have real money who gambled.
Then go with real peple who didn't gamble, but IF THEY DID GAMBLE, this is how they'd do it.
Then add ChatpGPT, Copilot, and other glorified spellcheckers.
Seriously, Slashdot on a weekend.
Wake me up when there's somethig new.
Why should it be able to predict it? (Score:2)
Why should it be able to predict it? What crucial information does it have to make a good prediction? I mean bring back the kraken who predicted soccer results, he wasn't that bad at it.
AI can only excel if there are patterns (Score:2)
You can ask AI about things, which can be known by deduction. If the problem is random, like "I'll roll my dice, how many eyes will I see?", then AI has no edge on humans or even random guesses. Expecting AI to excel at this is just a sign of someone not understanding what AI can and cannot do.
Therefore throwing a problem at AI is a good marker, how much randomness an experimental outcome contains. The Kentucky horse race seems to be a mixed bag of randomness and visible patterns, AI was somewhat correct bu
This is a success (Score:2)
Based on this article, these two did just as poorly at picking the winner or other places as humans do. This means AI in on par with humans.
Congratulations.