Forgot your password?
typodupeerror
IBM Software

IBM's Question-Answering System "Watson" Revisited 170

Posted by timothy
from the can-only-give-you-answers dept.
religious freak writes "IBM has created and made the question answering algorithm, Watson, available online. Watson has competed in and won a majority of (mock) matches against humans in Jeopardy. Watson does not connect to the Internet to answer his questions, but rather seeks answers using many different algorithms then employs a ranking algorithm to choose the best answer." We mentioned Watson last year as well.
This discussion has been archived. No new comments can be posted.

IBM's Question-Answering System "Watson" Revisited

Comments Filter:
  • by LostCluster (625375) * on Friday June 18, 2010 @01:05AM (#32609984)

    Part of the deal with Jeopardy! is that they will have as part of the 2010-2011 season be a televised episode in which record-breaking champ Ken Jennings will play against Watson, with a to-be-named-later champion in the third slot. This has been in the works since 2009, but the staff of the show finally thinks the system is ready for it's televised match.

    One key factor is how the human behavior will change when prize money is at stake. Jennings has proven in numerous appearances on GSN that he's willing to play in any test of knowledge and the fact that he knew he was Jeopardy's first millionaire in regular season play didn't stop his long Jeopardy! run. He also studied for the show, particularly alcoholic beverages (which he doesn't drink) because he had seen the Potent Potables category on TV.

    But, what about that player-to-be-named later? Will they know more than the grad students... and play the game not as if it's for points but real dollars?

  • by LostCluster (625375) * on Friday June 18, 2010 @01:07AM (#32610004)
    and see students from the MIT Robotics Lab test their machine that they say can avoid the Bankrupts and find that Million Dollar wedge on the Wheel of Fortune!
    • Re: (Score:3, Funny)

      by LostCluster (625375) *

      Elsewhere on the TV dial...

      H&R Block's mainframe system has computed that all of the the offers on Deal or No Deal are bunk, you're always statistically better off sticking with your case through most of the game... but they're still unsure whether you should take Howie's offer to switch your case with the last one left in the hands of the models.

      A tragedy as the Stanford-built computer made to play Russian Roulette was caught not in the lead when time was called in the second round, and was dropped by

      • H&R Block's mainframe system has computed that all of the the offers on Deal or No Deal are bunk, you're always statistically better off sticking with your case through most of the game... but they're still unsure whether you should take Howie's offer to switch your case with the last one left in the hands of the models.

        That's interesting to me for a couple reasons.

        I loathe that show. I'm not part of the anti-pop-TV brigade, I just find it incredibly boring. I tell my wife it's like watching someone throw dice against the wall for 42 minutes (DVR!), interspersed with crap dialogue.

        Having admittedly never thought about the last two cases scenario (the one you started with, and a final model's case). I would have thought that the probability maths behind this would be pretty simple.. do you have a link for that H&R block

        • by Ambiguous Puzuma (1134017) on Friday June 18, 2010 @06:19AM (#32611072)

          I don't care for the show myself, but don't forget about the nonlinear utility function of money. As an illustration, given that I'm comfortable but not wealthy:

          If given a choice between a guaranteed $400 and a 50-50 shot at $1,000, I'd choose the latter. The money wouldn't have a major impact on my life, so I'd go for the option with the best expected return.
          If given a choice between a guaranteed $400,000 and a 50-50 shot at $1,000,000, I'd take the guaranteed $400,000, even though the expected return of the latter situation is $500,000. $400,000 would give me a substantial amount of freedom and security. An additional $600,000 beyond that would be nice but would provide relatively few benefits compared to the initial $400,000.

          Now, if I were already a millionaire, I'd most likely choose the 50-50 shot at $1,000,000.

          • A good formula for "Expected Happiness", from the Wizard of Odds [wizardofodds.com] (4th question down):

            When the prizes become life-changing amounts, the wise player should play conservatively at the expense of maximizing expected value. A good strategy should be to maximize expected happiness. A good function to measure happiness I think is the log of your total wealth. Let's take a person with existing wealth of $100,000 who is presented with two cases of $0.01 and $1,000,000. By taking "no deal" the expected happiness is

          • by HiThere (15173)

            You're forgetting taxes.

            That $400,000 is probably closer to $200,000, and it means you're going to need to fight off a simply incredible number of get-rich-quick artists.

            Being comfortably well off is extremely desirable. Being quite rich has lots of good points. Winning a huge monetary prize is usually quite destructive. (I'm not sure, though, that $400,000 counts as huge. That's less than the cost of a good house.)

            • Re: (Score:3, Interesting)

              by Hatta (162192)

              Winning a huge monetary prize is usually quite destructive.

              There's a solution for that. Don't be a douchebag. I realize this is difficult for a large proportion of society.

        • - You pick one of the [26] cases at the beginning. Instead of focusing on amounts let's call one a "winning" case (top prize), all the others losers.

          This is a bad approach, since the amounts have a lot to do with it. The amounts mean that there isn't a "winning case" and 25 "losers", but instead 26 cases of various degrees of "win".

          Also, the offers are some kind of function of the average of the remaining cases (haven't actually checked mathematically, but I'm sure it amounts to something like that), so which "losing cases" you eliminate changes your strategy drastically.

          In short, the best strategy for the game is not to "pick the winning case", but

        • by Hatta (162192)

          This sounds a lot like the Monty Hall [wikipedia.org] problem. Turns out that the probability is not 50/50. It's always better to switch.

        • Re: (Score:3, Interesting)

          by inio (26835)

          Don't the same forces at work in the Monty Hall Problem [wikipedia.org] make it not 50/50, or does the fact that the eliminated "losers" were randomly selected without foreknowledge that they're losers make it 50/50?

    • Re: (Score:3, Interesting)

      by tlhIngan (30335)

      and see students from the MIT Robotics Lab test their machine that they say can avoid the Bankrupts and find that Million Dollar wedge on the Wheel of Fortune

      With a little empirical testing, it should be possible. Bankrupts are at known positions on the wheel, and you know the starting location. If you can model the physics of the wheel well enough, you can easily avoid them. (Unless the wheel has external influences - e.g., a brake and a motor that randomly apply and remove energy from the wheel making it

  • I'll be impressed (Score:5, Interesting)

    by Kitkoan (1719118) on Friday June 18, 2010 @01:19AM (#32610054)
    If it can properly rate if these people are hot or not [hotornot.com].
  • Watson has competed in and won a majority of (mock) matches against humans in Jeopardy.

    Against mock humans? I killed it, easily.

            Brett

    • by brianerst (549609)

      It was ridiculously easy to beat - so much so that I wondered if there was something wrong with the website. My final score was 59-11, and that was only because I typoed two of the answers (Watson is, I'm sure, a much better typist than me).

      Even when I mentally tried to score it for "late" correct/incorrect answers (the website shows you Watson's answer even if you answer correctly), I'm pretty sure Watson would have ended up in the high-teens.

      I once owned a Viszla named Watson - I'm not sure, but I think t

      • by bws111 (1216812)

        Well yeah it's easy to beat, it's not playing. This is just a canned flash demo of answers (and possible answers) that it came up with. It gives you the opportunity to answer first every time, so it is never going to 'beat' you if you know the answer. Looking at the possible answers it considered is way more interesting than trying to beat this demo. For instance, one of the 'what me worry' answers was 'scratching' (which it did not get right), but one of the answers it considered (along with eczema) w

    • Re: (Score:3, Insightful)

      by KingSkippus (799657)

      I killed it, too, but mainly because of the rules of the game in which I get first crack at the answer.

      Still, I think the point is that it's impressive the number of questions it gets right. I really didn't miss very many. My mental tally had it getting around 70% or so, which is pretty damn good. I got around 80%, but again, I had first crack at the answer, so Watson could have only possibly scored around 20% of the answers at best. If it tallied your score and Watson's score without actually competing

    • Re: (Score:3, Interesting)

      by moeinvt (851793)

      Back to the drawing board my arse. You may have beaten the web page game easily, but consider the fact that you wouldn't get first shot at every single question in a real match. If you assume 50-50 on the ring-in time when both players think they have the right answer, I think Watson looks pretty good on this game (although they wouldn't have put an example of a particularly bad match on the web).

      It also has a probability threshold below which it won't attempt a response. The real machine must be set up

  • by bobdotorg (598873) on Friday June 18, 2010 @01:31AM (#32610102)

    What is the air speed velocity of an unladen swallow?

  • I wonder if a website where people subscribed to artificial friends, shrinks, lovers would be a viable business model if it was as good at mimicking these things in conversations. An Eliza frontend on this Jeopardy beast might work. Plus Eliza was always giving questions as answers too!! I'd rather talk to a computer program about certain things anyway......and this one *would* be connected to the internet and would hone into your tastes quickly.
  • Eat that, you big blue lump of metal!

    (And it took me two questions what the hell how "before and after" worked)

  • without seeing the architecture of the application.

    Anybody can write an quiz game that beats you every time. What is (might be) impressive is how you are beaten. If I understand this, the impressive part is supposed to be that the system processes the questions as natural language, generates a number of matches, then chooses the one that it "thinks" matches the sense of the question most closely.

    That's a pretty tall order, but still as a demo the app isn't impressive, because the app designer chooses the

  • by peter303 (12292) on Friday June 18, 2010 @11:00AM (#32613180)
    Both Watson and Google relied on statistically analyzed word occurrences with a veneer of procedural/rule intelligence. Waston seeks the correct answer, while Google the most popular document containing the answer. Neither incorporate "deep" understanding like CYC's rule ontology. But it may not be that necessary if the statistics are large enough.
    • Re: (Score:3, Informative)

      by nine-times (778537)

      But it may not be that necessary if the statistics are large enough.

      It's possible, though, that the statistics can never be "large enough". I remember seeing an article here about natural language speech recognition (oh, here [posterous.com] it is) and about how many companies had hoped to continually feed more and more examples of language use into a computer and, through statistical analysis, be able to develop human-level speech recognition. The article indicated that these companies found a point after which additional examples didn't help. The statistical analysis (at least the met

  • ...it's a "pre-recorded" session. I played the NYTimes game twice in a row. The first time, the computer beat me. The second time, given the exact same questions, I handily beat the computer. I learned from my mistakes and was able to apply that knowledge to a similar scenario improving the outcome on my end. It's definitely an interesting study.

If a thing's worth having, it's worth cheating for. -- W.C. Fields

Working...