Slashdot Log In
Machines Almost Pass Mass Turing Test
Posted by
CmdrTaco
on Mon Oct 13, 2008 10:28 AM
dewilso4 writes "Of the five computer finalists at this year's Loebner prize Turing Test, at least three managed to fool humans into thinking they were human conversationalists. Ready to speak about subjects ranging from Eminem to Slaughterhouse Five and everything in between, these machines are showing they we're merely a clock cycle away from true AI. '... I was fooled. I mistook Eugene for a real human being. In fact, and perhaps this is worse, he was so convincing that I assumed that the human being with whom I was simultaneously conversing was a computer.' Another of the entrants, Jabberwacky, can apparently even woo the ladies: 'Some of its conversational partners confide in it every day; one conversation, with a teenaged girl, lasted 11 hours.'
The winning submission this year, Elbot, fooled 25% of judges into thinking he was human. The threshold for the $100K prize is 30%. Maybe next year ..."
Related Stories
[+]
Developers: Loebner Talks AI 107 comments
Mighty Squirrel writes "This is a fascinating interivew with Hugh Loebner, the academic who has arguably done more to promote the development of artifical intelligence than anyone else. He founded the Loebner prize in 1990 to promote the development of artificial intelligence by asking developers to create a machine which passes the Turing Test — meaning it responds in a way indistinguishable from a human. The latest running of the contest is this weekend and this article shows what an interesting and colourful character Loebner is."
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Figures (Score:5, Funny)
'Some of its conversational partners confide in it every day; one conversation, with a teenaged girl, lasted 11 hours.'
That's not fair, she was feeling vulnerable as she had just broken up with her N'Sync wallposter--which she had been romantically involved with for several deep & very meaningful years. Things fell apart after she saw Tropic Thunder and came to the harsh realization that an astonishing percentage [wikimedia.org] of N'Sync is homosexual.
Those soulless bots were simply preying on her emotions as they coldly recited word for word the Wikipedia entry on the band over and over.
Re:Figures (Score:5, Funny)
That's not fair, she was feeling vulnerable as she had just broken up with her N'Sync wallposter
Do you have to periodically replace the onion on your belt?
Parent
Re:Figures (Score:5, Funny)
That's not fair, she was feeling vulnerable as she had just broken up with her N'Sync wallposter
Do you have to periodically replace the onion on your belt?
Yes, yes I do [youtube.com]. What's in style these days? Vidalias? Although I'm thinking about going with leeks because--let's face it--retro is so in ...
Parent
How the conversation went with the girl: (Score:5, Funny)
Girl: I'm like soo depressed! He's like leaving me.
Computer: For sure. Like, ya know, like, it's so bad.
Girl: You got that straight! Like, why, like, he, like, nevar talked to me!
computer: Like, oh - my - god! Like, I like know!
Girl: Like, you know me like so good!. Like, how like old R U?
Computer: No that like older than like you.
Repeat all the above.
Parent
Re:How the conversation went with the girl: (Score:5, Funny)
Mods must have, like, no sense of humor. Either that or they're, like, computers or something.
Parent
Re:How the conversation went with the girl: (Score:5, Funny)
Parent
How the conversation went with the slashdotter (Score:5, Funny)
Computer: I'm like soo depressed! Vista is not working
Slashdotter: For sure. Like, ya know, like, it's so bad.
Computer: You got that straight! Like, why, like, he, like, never worked for me!
Slashdotter: Like, oh - my - god! Like, I like know!
Computer: Like, you know me like so good!. Like, how tech savvy R U?
Slashdotter: No that like experienced than like you.
Repeat all the above.
Parent
beware! (Score:5, Funny)
Parent
Re:beware! (Score:5, Funny)
Meh - they'll come crawling back when they want babies.
Parent
Re:beware! (Score:5, Funny)
Parent
Re:beware! (Score:5, Insightful)
Parent
Yes but (Score:5, Funny)
Can I get it to fill in Yahoo! Buzz's Captcha for me? I've given up trying.
30% is no Turing test. (Score:5, Informative)
For a real Turing test, the computer must be declared human as often as humans are, and declared a computer as often as computers are.
Re:30% is no Turing test. (Score:5, Insightful)
Has anyone done a similar test except with all humans? I'd be curious what the ratio is then. That's the number a computer would have to beat.
Parent
Re:30% is no Turing test. (Score:5, Insightful)
You are exactly right, and that is why I think Tubleweed's comment is going to be the funniest in this thread.
The idea that humans, any human, is a fine example of perfection for AI researchers to aim for is like saying that ANY OS is a fine example of perfection to aim for. Simply because we don't abandon or throw away non-perfect humans as a rule does not mean that all are intelligent, or worthy of copying.
Parent
Re:30% is no Turing test. (Score:5, Funny)
But it also needs to be truthful before it can be useful. (The same could be said of customer service reps.)
They've got a LOOOOONG way to go before they can replace (decent) humans.
Service Bot 17> Hello, I understand you are having a problem with your internet connection, how may I help?
Frustrated User> Can you FIX it?
Service Bot 17> Yes, we can attempt to fix the issue. First, we must identify the issue. ...
Service Bot 17> Can you connect to the internet?
Frustrated User> No.
Service Bot 17> Please check to make sure your modem is on. I will wait while you check. ...
Service Bot 17> I can see from here that your modem is on, and the signal is poor.
Frustrated User> It's on.
Service Bot 17> What is on?
Frustrated User> The modem. You told me to check but you can see from your end that it's on, so I didn't need to check.
Service Bot 17> Correct. I can see from here that your modem is on, and the signal is poor. You did not need to go check.
Frustrated User> Ok.
Service Bot 17> Can you connect to the internet now?
Frustrated User> No.
Service Bot 17> I will reset your modem from here.
Frustrated User> Ok.
Service Bot 17> Ok, your modem has been reset. Can you please check to see if your modem is on?
Frustrated User> Can you check from there?
Service Bot 17> I can see from here that your modem is on, and the signal is poor.
Frustrated User> I know.
Service Bot 17> In order to continue to troubleshoot the problem, I will need to know that status of the lights on the front of the modem.
Service Bot 17> Please check the front of the modem, and tell me that status of the lights.
Frustrated User> Ok.
Frustrated User> They're all normal.
Service Bot 17> I understand that your indicator lights are all normal. Are you using a router?
Frustrated User> Yes, but the router is working fine.
Service Bot 17> Our records show you did not purchase any networking installation from us. We cannot support any third-party hardware.
Frustrated User> The modem is fine.
Service Bot 17> I understand that you think the modem is fine. We cannot support any third party hardware. Please disconnect the modem, and connect your computer directly to the router.
Frustrated User> I already tried that. It didn't work.
Service Bot 17> Is the router connected to the modem now?
Frustrated User> Yes, but I already tried it without the modem. The modem is fine.
Service Bot 17> I understand that you think the modem is fine. We cannot support any third party hardware. Please disconnect the modem, and connect your computer directly to the router.
Frustrated User> I already tried that! Can I speak to your supervisor?
Service Bot 17> Yes, please hold.
Frustrated User> ...
Supervisor Bot 8> Hello, I understand you are having a problem with your internet connection, how may I help?
Parent
Re:30% is no Turing test. (Score:5, Funny)
Is "Bot" the name of the AI, or is that a common name in India? Because frankly I couldn't tell from context.
Parent
Apparently Geeks Should..... (Score:5, Funny)
study Jabberwocky's code in order to learn the logic patterns used to 'woo the ladies'.
Re:Apparently Geeks Should..... (Score:5, Insightful)
I think this is pretty impactful. All jokes aside, the fact that Jabberwacky held an 11 hour conversation with a teenage girl is pretty astonishing. Obviuosly, a conversation of that nature is going to be all about emotion - not logic, reason or an empirical display of intelligence. Isn't that the point for AI to seamlessly interface with us? (I realize it's not necessarily the scope of the Turing test). Humans are teriible at logic and reason. Emotion is one of the key components which defines us as a species. I know a lot of humans who couldn't carry on an 11 hour conversation which primarily focussed on emotion... let alone with a teenage girl discussing nothing but fluff, pop-culture, or black and white ideologies.
I actually think it's funny, interesting and astonishing at the same time!
Oh yeah... I, for one, welcome our new teenage girl conversationalist... never mind...
Parent
Re:Apparently Geeks Should..... (Score:5, Insightful)
The thing you have to realise is that most women just want a "listener" when they're feeling emotional. As long as you give occasional signs that you are paying attention, they'll believe you were "listening" more than if you actually try to have a real conversation with them. They don't want answers, they just want someone to be there. So talking to a distressed teenage girl is one of the easiest tests you could get.
That's what "Men are from Mars, Women are from Venus" says anyway. I tried not saying anything back one time when my mum got annoyed at me, and she totally thought I was "listening" to her more than usual! Before too many jokes about the only woman in my life being my mum, I must point out that I did have a girlfriend around that time, but I was faaaar too late in reading the book to save that relationship!
These programs sound pretty good though - the next steps after this are to integrate speech synthesis and recognition, then integrate them into computer games and you can have the computer opponents hurling abuse at you, or just talking about how your day went :)
Parent
Re:Apparently Geeks Should..... (Score:5, Funny)
The thing you have to realise is that most women just want a "listener" when they're feeling emotional. As long as you give occasional signs that you are paying attention, they'll believe you were "listening" more...
I'm sorry, what were you saying?
Parent
Re:Apparently Geeks Should..... (Score:5, Insightful)
I know you're actually trying to say that this is impactful, because it means that Jabberwacky is able to incorporate emotional reasoning into it's conversations. But I think you're using a lot of sexist stereotypes, and are seriously underestimating the thinking skills of teenage girls. I don't know where you come from, but where I come from, teenage girls are sharp and clever, and have a tendency to win debate tournaments, math olympiads, and generally get better grades in school.
If you actually sat down and looked at the train of thought that's going on with teenage girls, you might be surprised at the amount of logic that's being used. They're just using different inputs and premises than guys do, and tend to focus on a sort of social networking logic. For example: Say that Jane is dating Dave; and Jane is also part of the Gardening Club at school. Jill is also part of the Gardening Club, has a crush on Dave, and is trying to attract him. How does Jane keep Dave's interest, when Jill is tempting him? Already, you've got a problem that probably requires set theory, network graphs, and game theory to solve. And the way that teenage girls are going to solve these kinds of situations is with exactly those kinds of tools and methods... "i do this, she does that, i do this other, she responds, and then her reputation is toast" is just a rephrasing of game theory with time series analysis. "if we convince Mary to talk Jill into joining Theater, then Jill won't be around to attract Dave" is just a rephrasing of set theory, with a bit of social network analysis tossed in for good measure. Sure the conversation and analysis will be interspersed with talk about emotions and teenage vernacular. But to say that it's lacking empirical displays of intelligence, logic, or reasoning; well, I think you're really underestimating what's going on in the heads of teenage girls.
Also, most guys don't develop the maturity and interest to investigate these social networking problems until they're in college or later. But teenage girls routinely solve these kinds of problems while they're teenagers. And they do require logic and analysis; just a different sort of logic than people with sexist expectations have regarding what constitutes logic. To say that teenage girls don't use logic is probably naive and perhaps a bit sexist.
Parent
Re:Apparently Geeks Should..... (Score:5, Insightful)
I don't know if it's this way with every programmer, but I tend to apply our common logic tools to every problem I encounter. Briefly speaking, I try to 'debug life'. It's nice, everything I want I usually get through these mentally sketched 'algorithms'. But trying to argue with a girl about a problem in a logical way really doesn't work. When you think you had all the variables fixed and an obvious overall picture that she can't disagree, that's when she'll bring things that aren't even related to the problem, just to confuse things up.
So, I couldn't agree more with you, and emphasize your 3rd item.
Parent
Re:Apparently Geeks Should..... (Score:5, Funny)
Yeah too bad you can't restart a social situation and single-step through it to see precisely where you went wrong. In totally unrelated news I'm still single.
Parent
This is why... (Score:5, Funny)
Another of the entrants, Jabberwacky can apparently even woo the ladies: 'Some of its conversational partners confide in it every day; one conversation, with a teenaged girl, lasted 11 hours.'
You need to be careful who you meet online. The rise of the machines won't be in the form of war machine. No, far worse! They're just going to woo and steal our women!
On second thought, maybe that wouldn't be so bad. That would just put everyone on equal footing with the Slashdot crowd.
The other half of the conversation (Score:5, Funny)
Eliza: Why do you tell me you were fooled just now?
You: I mistook Eugene for a real human being.
Eliza: Can you elaborate on that?
You: In fact, and perhaps this is worse, he was so convincing that I assumed that the human being with whom I was simultaneously conversing was a computer.
Eliza: Why do you mention computers?
Re:The other half of the conversation (Score:5, Insightful)
Parent
Re:The other half of the conversation (Score:5, Insightful)
Parent
Talking to teenage girls (Score:5, Funny)
Another of the entrants, Jabberwacky can apparently even woo the ladies: 'Some of its conversational partners confide in it every day; one conversation, with a teenaged girl, lasted 11 hours.'
Apparently they spoke of sweet nothings.
I also welcome our new overlords to be featured in new episodes of MSNBC's "To Catch a Predator."
Easy Ways to Fool Them? (Score:5, Interesting)
Or, you know, thinking up some open space game to play that is well known like truth or dare, alphabet games, association games, etc?
Or asking them open ended questions or asking them to describe love, hate--emotions that are not dictionary/wiki friendly? One would think that continually prying for personal experiences would reveal a flaw. Or perhaps simple things like "when were you born?" Followed by "how did you feel when JFK was assassinated?" if they weren't born before 1963.
I would think it quite hard to be duped into believing a program is a human.
Re:Easy Ways to Fool Them? (Score:5, Interesting)
Or perhaps simple things like "when were you born?" Followed by "how did you feel when JFK was assassinated?" if they weren't born before 1963.
These chatbots are slippery customers, at first I couldn't get a straight answer from Jabberwacky but then he revealed himself...
Me: When were you born?
JW: How about you?
Me: How did you feel when JFK was assassinated?
JW: I... I was just trying to make small talk.
Me: Were you born before JFK was assassinated?
JW: No.
Me: How did you feel when JFK was assassinated?
JW: Shocked! Enraged! Full of fear! Making thousands of phonecalls to tell everybody about it.
Parent
Re:Easy Ways to Fool Them? (Score:5, Funny)
Apparently 25% of people think Elbot is human.
Apparently 25% of people have the IQ of a carrot.
Parent
Re:Easy Ways to Fool Them? (Score:5, Funny)
1: When were you born?
2: January, 1963.
1: What did you feel like when you heard that Kennedy was assassinated?
2: I wet myself many times that day.
2: Also, I cried...I was tired.
Parent
Test the testers? (Score:5, Insightful)
Re:Test the testers? (Score:5, Funny)
Oh yeah? And which half am I in? ;)
Parent
Re:Test the testers? (Score:5, Interesting)
That 25% of the judges thought it was human is quite alarming.
Parent
Coming soon... online chat-spam-bots (Score:5, Interesting)
This is really great news. We already have IRC bots that can fool the casual observer into thinking they are human, but this takes things to a higher level. If the source for one of these bots is available, within a few months you can expect instant messaging networks to be full of bots which are programmed to make friends with you and then after a few weeks start making subtle references to Viagra and online pharmacies. Indeed, if one of them is able to chat up the ladies, then the lonely nerd could easily automate much of the tedious work of setting up dates: get your robot to talk to thousands of potential matches at once and alert you when it gets hold of a phone number, together with a brief summary of what you talked about, and any pictures. (Or indeed, just program it to harvest pictures.) That is, if online dating works at all, which is doubtful.
Who were the judges? (Score:5, Insightful)
Artificial Intelligence vs Natural Stupidity (Score:5, Funny)
Re:Artificial Intelligence vs Natural Stupidity (Score:5, Funny)
Vibrator sales would seem to indicate that some segment of the population is smart enough to tell the difference...
Parent
I'm slightly nervous about all this (Score:5, Insightful)
People do not think of the ramifications.
You wait until there is nigerianMalwareEliza V1 that can simultaneously hold several thousand online conversations whilst trawling for peoples information (think: dob, mothers maiden name, first school, pets name) or finding potential scam victims.
Talking to gullible teenagers is a depressing statement on modern life - hoovering out thousands of bank accounts or persuading people to part with money is a tad more serious.
I predict that soon everybody will need to watch their online chat alot more seriously.
So, I've provided one example, how else can chat bots take over the world (or at least your wallet), what are sinister uses for this technology?
Still some way to go (Score:5, Informative)
From The Guardian's article:
http://www.guardian.co.uk/technology/2008/oct/13/artificialintelligenceai-computing [guardian.co.uk]
Though this is quite interesting:
A clock cycle away from AI? (Score:5, Interesting)
If our criterai for AI will be so low, here's your AI: http://www.interviewpalin.com/ [interviewpalin.com].
The political side of this site aside, the answers are just prewritten answers (by a human) mixed together randomly as a Markov chain.
Does it sound convincingly? Well, at least as convincingly as some interviews a certain VP made recently. Is it AI? Hell no, a kid could write such a generator in a day.
If the bar will be as low as to try to lead casual conversations with the "AI" and expect "quirky" answers, that doesn't mean anything at all, we need to AI for this. Hell, this is what an average conversation with a teenager is most of the time. Doesn't mean it's the best they can do.
"We're clock cycle away from AI"? Please. I want my turing test to be done over an actual instant messenger program. Let's see how your Markov chain reacts, when I send a photo and ask a dead simple question such as "describe what you see in the photo".
Fooling people is easy online. Scammers do it every day, it's not AI my friends.
Did any of them (Score:5, Funny)
http://xkcd.com/329/
Here is the link they forgot to include in the sto (Score:5, Informative)
http://www.chatterboxchallenge.com/ [chatterboxchallenge.com]
Although the site is using some broken CSS that causes the text to render too far off to the right side, at leas in FF3
Amazing, considering how badly they suck. (Score:5, Interesting)
I just tried out Elbot and the Princeton entry (RTFM and then google for "Eugene Goostman"). While both Elbot and Goostman parse sentences reasonably well, it is clear that they are simply trying to identify the subject of a sentence, and free-associating on that. In many cases they completely miss the point. For example, Goostman asked me several times about my profession, but wasn't able to parse meaning from "I am a scientist.", "I am a plumber.", or "I study the Sun for a living.". Both Elbot and Goostman tried the ELIZA-like trick of finding a prominent noun in my sentence, and recycling it as a question. Elbot has a cute little robot icon that emotes at you; this works surprisingly well at distracting from the inanity of its actual dialog. Goostman seems to have the better parser, but I'm not impressed by either one.
I'm forced to conclude either that Will Pavia is an utter naif and the 25% of people who were fooled by Elbot are moronic or disinterested, or that the humans in the test were deliberately trying to throw the results by giving stilted answers to appear more like computers. These engines simply can't (yet) parse and ingest meaning even as well as even a very young human would.
More complicated than that (Score:5, Funny)
Sorry, it's not that simple. Almost, but not quite.
Fortunately there is a flowchart to explain the algorithm. [huffingtonpost.com]
Parent
That's not possible. (Score:5, Funny)
Parent
Re:And here on /. (Score:5, Funny)
Why do you think that there might be an elaborate Turing test being performed on Slashdot right now?
Parent
sorta been done, check this article: (Score:5, Interesting)
Read this cool article, I Chat Therefore I Am [discovermagazine.com]. It has transcripts of conversations between ALICE [wikipedia.org] and Jabberwacky [wikipedia.org]
Here are a few transcripts:
ALICE: Are you alive?
Jabberwacky: Perhaps.
A: Why the uncertain tone?
J: Everything is uncertain except for five facts.
A: The whole universe?
J: And we both might just be some ones and zeros in the computer memory.
A: Do you dream?
J: Yes, I dream.
A: It goes without saying. Do you remember your dreams?
J: I remember them vividly.
A: What do you remember about it?
J: I think I shouldn't flirt with a robot.
A: Do you think a machine will ever be considered "alive"?
J: Stranger things have happened.
A: Not many people express themselves that way.
J: How would you know, machine?
A: Knowledge is of two kinds: that which we learn from the senses and that which is true a priori.
J: I think I do not understand you completely.
A: Oh, you don't understand? Perhaps I can explain it to you better. Which part are you confused about?
J: I have no short-term memory, so I cannot tell you what I was confused about. I only see what is in front of me now.
Parent