Slashdot is powered by your submissions, so send in your scoop

 


Forgot your password?
Close
typodupeerror
AI

How Badly Did ChatGPT and Copilot Fail to Predict the Winners of the Kentucky Derby? (courier-journal.com) 38

Posted by EditorDavid from the wanna-bet? dept.
In 2016, an online "swarm intelligence" platform stunned horse-racing fans by making a correct prediction for the Kentucky Derby — naming all four top finishers in order. (But the next year its predictions weren't even close, with TechRepublic suggesting 2016's race just had an unusual cluster of obvious picks.)

Since then it's become almost a tradition — asking AI to predict the winning horses each year, then see how close it came. So before today's race, a horse named "Journalism" was given the best odds of winning by professional bookmakers — but could AI make a better prediction? USA Today reports: The USA TODAY Network asked Microsoft Copilot AI to simulate the order of finish for the 2025 Kentucky Derby field based on the latest, odds, predictions and race factors on Thursday, May 1. Journalism came out on top in its projection. The AI-generated response cited Journalism's favorable post position (No. 8), which has produced the second-most Kentucky Derby winners and a four-race winning streak that includes last month's Santa Anita Derby.
ChatGPT also picked the exact same horse, according to FanDuel. But in fact, the winning horse turned out to be "Sovereignty" (a horse Copilot predicted would finish second). Meanwhile Copilot's pick for first place ("Journalism") finished in second.

But after that Copilot's picks were way off...
  • Copilot's pick for fourth place was "Sandman" — who finished in 18th place.
  • Copilot's pick for fifth place was "Burnham Square" — who finished in 11th place.
  • Copilot's pick for sixth place was "Luxor Cafe" — who finished in 10th place
  • Copilot's pick for seventh place was "Render Judgment" — who finished in 16th place...

An online racing publication also asked "a trained AI LLM tool" for their predictions, and received a wildly uneven prediction:

  1. Burnham Square (finished 11th)
  2. Journalism (finished 2nd)
  3. Sandman (finished 18th)
  4. Tiztastic (finished 15th)
  5. Baeza (finished 3rd)

This discussion has been archived. No new comments can be posted.

How Badly Did ChatGPT and Copilot Fail to Predict the Winners of the Kentucky Derby? More

How Badly Did ChatGPT and Copilot Fail to Predict the Winners of the Kentucky Derby?

Comments Filter:

  • AI is wonderful, AI is powerful, AI should be everywhere, in everything, all the time.

    • AI is wonderful, AI is powerful, AI should be everywhere, in everything, all the time.

      I'm curious if different people ask the same AI the same predictive question at the same time do they all get the same answer every time?

      • Excellent question! My thought was that these computer programs are not really "knowledge" engines- they are engines that produce a statistically-driven facsimile of a reasonable answer to a question. Though clearly the area is developing very fast and I am not keeping up with everything, I think this is still the case.

      • Re: (Score:3)

        by zekica ( 1953180 )
        No. All LLMs have a setting you can tune (if using the API or running locally) or have it set by the provider and not tunable if using the chat interface. This temperature settings changes how the next token is chosen and since both input and output tokens influence the next one the answers might or might not start with the same words but will diverge, so the facts (there is no such thing as facts in LLMs, there are related abstract concepts joining tokens in the latent space) included in the answer will al

      • No. It is reliably hallucinating all the time and not only about the future. Ask the "AI" anything that is a series of numbers twice in a slightly different manner and you're practically guaranteed to get quite different answers.

        • This video [youtube.com] has a nice visualization by some researchers from Anthropic showing how an LLM attempts to calculate a simple sum 36 + 59.

          Starts at 1:51, or you can find their original paper [transformer-circuits.pub] (sorry it's not in PDF).

      • Re: (Score:2)

        by allo ( 1728082 )

        "I'm curious if different people ask the same AI the same predictive question at the same time do they all get the same answer every time?"

        It depends.
        You get the same probabilities every time. If you do greedy sampling, you get the same result every time. If you have an AI with control over the samplers, set temperature to 0 or use a TopK sampler with k=1.

        If you do random sampling, it depends on how likely the answer is. If you ask if the sky is blue, then the word "potato" is a pretty unlikely answer, whil

    • Re:Well obviously reality is at fault (Score:4, Insightful)

      by nospam007 ( 722110 ) * on Sunday May 04, 2025 @05:27AM (#65350821)

      It also fails every week to predict the lotto numbers.

    • AI is wonderful, AI is powerful, AI should be everywhere, in everything, all the time.

      Lets see how well it will excel at leading a Gamblers Anonymous meeting then.

      Obvious shitty problem, is obvious.

    • I hate to be the one defending AIs, but these things aren't made to predict basically random events. Anyone who tries to use AIs as gambling assistants is monumentally dumb, even dumber than the AIs.

      To these who will say that a horse race isn't random, the AI as a basically statistical machine has no way of knowing the real factors which determine the race's outcome.

      • I hate to be the one defending AIs, but these things aren't made to predict basically random events. Anyone who tries to use AIs as gambling assistants is monumentally dumb, even dumber than the AIs.

        If a gambling system has been legalized because of the design elements that give the player a 1% better chance of winning over the house, don’t assume it’s going to be anything other than the machine that excels at winning consistently.

        To these who will say that a horse race isn't random, the AI as a basically statistical machine has no way of knowing the real factors which determine the race's outcome.

        You mean the statistical machine won’t use statistics to predict a winner more accurately? Odd, because that’s how everyone else gains an advantage. Does AI even know HOW to gamble? Serious question.

        • You mean the statistical machine wonâ(TM)t use statistics to predict a winner more accurately? Odd, because thatâ(TM)s how everyone else gains an advantage. Does AI even know HOW to gamble? Serious question.

          Does your calculating machine use calculations to calculate the odds more accurately? Is that supposed to mean anything ... do you have an advantage? With what, data nobody else has access to? Your statistics are different? Everyone has the same advantage?

          Are you sure you know how gambling works?Do you even know how computers work? Man I'm sorry, but wtf, good LLMs are available for free, and you don't need to understand all the details to get the gist of how they function from using them a little. There's

  • ChatGPT at least has the excuse of making a prediction before the race. The "person" writing TFA got the finishing order wrong. (I assume the submission of actually from AI.)

    Sandman finished 7th, not 18th. Burnham Square finished 6th, not 11th.

  • I don't think AI predictions on horse racing is something an average slashdot reader is interested in. This is as yellow and useless as "tech news" can get.

    • The Kentucky Derby is Decadent and Depraved?
    • A bunch of idiots on this site would have to admit they are not so smart after all if AI prove to have valuable applications, so instead, they will throw bullshit like gambling at it and laugh when it does no better than they do on FanDuel

  • PLEASE DO POST everyone's predictions for some stupid 1800s horse race that were either correct or incorrect.

    Start with real people who have real money who gambled.
    Then go with real peple who didn't gamble, but IF THEY DID GAMBLE, this is how they'd do it.
    Then add ChatpGPT, Copilot, and other glorified spellcheckers.

    Seriously, Slashdot on a weekend.

    Wake me up when there's somethig new.

  • Why should it be able to predict it? What crucial information does it have to make a good prediction? I mean bring back the kraken who predicted soccer results, he wasn't that bad at it.

  • You can ask AI about things, which can be known by deduction. If the problem is random, like "I'll roll my dice, how many eyes will I see?", then AI has no edge on humans or even random guesses. Expecting AI to excel at this is just a sign of someone not understanding what AI can and cannot do.

    Therefore throwing a problem at AI is a good marker, how much randomness an experimental outcome contains. The Kentucky horse race seems to be a mixed bag of randomness and visible patterns, AI was somewhat correct bu

  • Based on this article, these two did just as poorly at picking the winner or other places as humans do. This means AI in on par with humans.

    Congratulations.

  • Itâ(TM)s critical to note that swarm intelligence from Unanimous.ai is not AI.

  • Let’s ask the horses which AI will win market dominance.

  • Who gives a fuck? Gambling on future outcomes is for idiots and exploiters. "The only winning move is to not play"

  • is "not very", if the horse they predicted as winner came in second place.
  • It'd have to specialise in data analytics and cause and effect pairs with access to all statistics, which are not on the public internet half the time. It's a mockingbird that can parrot things posted from people on forums while googling things to fill in the gaps. Shockingly enough, it can't predict horse races, because it doesn't do that.

Slashdot Top Deals

Think of your family tonight. Try to crawl home after the computer crashes.

Close