"Understanding" Search Engine Enters Public Beta

religious freak sends word of the public beta of Powerset, a closely watched San Francisco startup that promises an "understanding engine" to revolutionize Web search. An article in SearchEngineLand points out that Powerset is reaching higher than for mere "natural language." Techcrunch has more details and analysis. For the beta, Powerset makes available all of Wikipedia to search — not all the Web. It's said that their understanding engine required a month to grok Wikipedia's 2.5M articles. The Web is currently at least 8,000 times as large.

I'm Unimpressed

Ok, so I like these new search engine ideas but I am grossly underwhelmed here. I tried the input:

Who is David Bowie?

Which it handled quite nicely. Biography, additional links and all that Wikipedia jazz.

But come on, that's a simple question. Let's talk stuff I get into arguments over with my coworkers:

Who played the villain in the first Die Hard?

Which at least put Alan Rickman at #8 [powerset.com]. But let's try mutating that to make it harder but still understood by you and I:

Who played the bad guy in the first Die Hard?

Which resulted in very little but drivel [powerset.com] with no mention of the great Alan Rickman whatsoever ... although it did put Billie Jean King and Madonna in there for some hilarious reason.

So maybe it can't understand 'bad guy.' Well onto another question:

Who was the organist for The Beatles on Abbey Road?

Which resulted in at least the first 20 having no mention of the great & oft forgotten Billy Preston [powerset.com].

So you want to know what the kicker is? I put those same inputs into Google and found the name in the first or second result. Granted PowerSet doesn't do the whole web, I'm pretty sure that if it did, it wouldn't have the pretty results that it gave when I did what one of the articles told me to--ask it when earthquakes hit Tokyo. Just imagine the dates it would come up with if it hit a site with an html table of any seismic activity whatsoever in Tokyo!

I think it's a novel idea to mine Wikipedia for a search engine so long as it isn't just plain old token matching like PowerSet seems to be up to. Be inventive, try a natural language parser written in Prolog that digests all of Wikipedia into a huge network/ontology of concepts ... no matter how flawed it might be.

I find them talking about this in the articles:

Powerset is different. It says that its technology reads and comprehends each word on a page. It looks at each sentence. It understand the words in each sentence and how they related to each other. It works out what that sentence really means, all the facts that are being presented. This means it knows what any page is really about.

Yet, I'm not impressed. You can try to personify your software and convince me that Baby Alive really defecates like a human being all over so it feels like I have a real baby. But I know it's just software. You don't have to dumb it down if you're going to blog about it. What is this? A pattern matching implementation? A depth first search tree parsing implementation? An ontology builder? Could you at least drop one of the buzzwords of the natural language parsing field for me here?

So does this story actually have more than a startup looking for a sugar daddy to buy it out?

Re:I'm Unimpressed

Use site:en.wikipedia.org to have Google ask all of Wikipedia (English)

Re:I'm Unimpressed

I asked it "who won the election in 2004?" and it understood the question, in a way:

The current mayor is Jardir Silva Vidal who won the election in 2004 against Reino Martins de Oliveira

Obviously still buggy.

Your tests are interesting, but you're not really parsing the responses in the right context. They're problematic. Keep in mind this understanding engine understands the world in a way that was hatched out in San Fransisco.

Who is David Bowie? I trust that it came back with, "aka Ziggy Stardust, normal family guy"

Who played the villain in the first Die Hard? Well, obviously, the villain is "capitalism."

Billie Jean King and Madonna ... like I said, it's San Fransisco

Who was the organist for The Beatles on Abbey Road?

You had it at "organ," and it got distracted. What they need is some dev guys from Toledo to collaborate, and provide a little cognitive counterweight to the understanding engine. OK, maybe not Toledo. Maybe Atlanta.

Re:I'm Unimpressed

Thanks for testing us out with some real queries -- it's the best way to get the Powerset experience. But, if you only ask NL questions then you don't get to see all of Powerset's features.

Powerset is not token matching. In fact, we read every sentence from every page in Wikipedia that we index. For examples of how we understand syntax, check out queries like "who did texaco acquire" vs. "who acquired texaco". Note that Powerset understands the difference between being acquired by and acquiring, that "buying" is equivalent to "acquiring", and that we are often able to highlight the actual answer to your question. Traditional search engines can do none of these things. Powerset is trying to match the meaning of your query to the meaning of a sentence in Wikipedia.

However, Powerset is very aware that: 1) Users shouldn't be expected to use natural language and 2) We only search Wikipedia and 3) Our algorithms aren't perfect yet. Powerset's release isn't intended to replace your regular keyword search engine. But, we do hope that you come back to Powerset when you have a question that might be answered in Wikipedia.

So, try some topical queries in Powerset, like "kurt godel." In the Factz section, Powerset knows that Kurt Godel proved theorems. If you click on "theorems," you'll see all the sentences in Wikipedia from which we derived that fact (be sure to click on "more"). Note that none of these Factz come from the Kurt Godel page. Powerset's ability to aggregate Factz from across Wikipedia is unique to our technology.

Now try, search for the Presidency of Bill Clinton and click through to the enhanced Wikipedia page (http://www.powerset.com/explore/semhtml/Presidency_of_Bill_Clinton?query=presidency+of+bill+clinton). Note that we also have Factz in the article outline, which helps to summarize long articles. Check out the second term during the Lewinsky affair: the Factz are an amazingly accurate description of the situation.

Sorry to be a bit lengthy, but I wanted to make it clear the Powerset isn't just about asking questions. We've got a video that identifies all of the features: http://vimeo.com/994819

{mark} powerset product manager

But it doesn't give results any differently

I asked 'Where do babies come from' and it just gave me back a bunch of articles with that string somewhere in their text.

Pathetic, and you'd hope it's got a long way to go really because at the moment it does NOTHING of merit that I can see.

Re:But it doesn't give results any differently

I tried "What are anal warts". Google gave me a response faster than I could start and stop my timer, with the second answer being Wikipedia. Powerset is still, hold on..., yep still hung up on the question.

Re:But it doesn't give results any differently

I asked it a question I got in a trivia contest - what countries have four-letter names? [powerset.com] (There are 10, and google's [google.com.au] first link is to a list [answers.com] of 'em)

Powerset's first response? "Fuck."

Funny, that was my response too, but at least I got 5 or 6 of them first...

No, early Google was better than anything else.

Which is why everyone started using it. It wasn't perfect, just better than anything else. Powerset isn't better than lycos.

Re:I'm Unimpressed

I don't know yet what Search 3.0 will be, but we're still a long way from getting Search 2.0 to work right. But we're still making progress.

Actually, we aren't making progress -- *at all*. What these guys are trying to do is a subset of artificial intelligence. A subject people have banging their heads against since the 1940s, and we've made *zero* progress since then. We simply don't know how humans process information. We don't even have reasonable theories. We're at the equivalent of the "four elements make up the world" version of physics.

AI researchers always get defensive when I say this, but it's simply true. All we have are better brute-force algorithms that sort-of simulate some of the things that humans do (i.e., voice recognition, character recognition, and other yawner tricks). There is no science of AI. Any sort of human-level understanding of information is far, far away in the future.

Re:I'm Unimpressed

Wait, you're saying that the MIT summer vision project [mit.edu] wasn't as easy as people thought?

(Background: In 1966, some MIT computer science faculty thought AI was so easy that computer vision could be solved in one summer worth of work; it probably took 35 years to reach the milestones identified in the research abstract).

Next step....

Since Powerset can only search Wikipedia, the logical next step is to put the entire web on Wikipedia. Who's up for the job?

Re:Next step....

I think we need to split it up by domain name. I'll start with wikipedia.org

The Web is currently at least 8,000 times as large

Any day now, Wikipedia will surpass The Web's growth rate, and set a course for the day when Wikipedia will be BIGGER THAN THE WEB.

Yawn. Here is something really impressive...

True Knowledge [trueknowledge.com] actually interprets your question using Natural Language Processing, and then looks through a massive database of user-contributed facts, combining them using sophisticated inference rules, to give you the answer you need. Even the inference rules are user-editable.

2 out of 10

I tried just "Osaka", where I am right now.

First match was an obscure album, then a few "factz" that made no sense.

Let's try again, "What is the largest city in Japan?"

Tokyo doesn't feature at all on the first page! It fairs just as badly with other countries.

It now seems to be slashdotted, so I better quit now.

There is a reason query languages exists.

They're faster, more efficient and more accurate. Yes, they require learning yet there's a valid reason and a payoff to doing so. Do we really want to dumb things down any further? If you can't figure out Google, perhaps you should get off the Net.

Natural languages are not a help.

There is a fallacy that putting a ntaural language on something will make it easy. There are many specialised languages that people use every day.

1 + 1 = 2 is a special notation/langauge that is both more consise and easier than writing "add one and one to make two". So is music score, which is far easier than reading make a high note for a bit then wait a bit and make a low note". Same with C, C++, SQL or Python: the hard bit in programming is algorithm design, not understanding the actual language itself.

Is Natural language really a barrier to entry in using Google? I doubt it. My untechy wife and her friends find everything they need. Plugging natural language into Google gives reasonable results moset of the time.

Yeah right

What a marketing pile-of-poop. All it does is pull out phrases from Wikipedia; there is no attempt to understand the information at all. When I can type in a yes/no question ("Did they have looms in the 1400s?"), I'll be impressed. When it can make calculation ("How old was columbus when the first colony was founded?"), I'll be impressed. When it can make comparisons ("when did the earth's population match the current population of the united states?"), I'll be impressed.

In other words, when it even attempts to answer a question that isn't already in Wikipedia as a phrase, I'll be impressed.

It's about as good as Ask Jeeves. Maybe worse.

I've been trying various queries, and Google is doing better than Powerset even when I type in some actual question, like "How many Japanese died in WWII?".

Question: "What is the planet closest to the sun?". First answer from Powerset: "Pluto".

I think I see how this works. It takes the question and breaks it at noise words, ("closed class words" in linguistic terminology) constructing a query with both words and phrases. So "What is the planet closest to the sun" becomes "planet closest" sun. In fact, if you rewrite a natural language question in that form and use Google, it does better on question-answering than Powerset does.

Remember Ask Jeeves? It worked like that? No technical breakthrough here, move along.

Thoughtpuckey

The variance in quality of search results is noted elsewhere. I'm more interested in the fallacy of the claim of "understanding". That, as well as its synonym "comprehension" require metacognition, that is, knowing that you know. It is the basis of self-awareness. this program doesn't even pretend to give evidence of this, it simply return search results. Pretending to be self-aware was accomplised by CYC when it claimed to graps the fact that it was a computer program. For anyone interested in seeing the arguments about understanding and self-awareness, see Searle's "Chinese Room" http://en.wikipedia.org/wiki/Chinese_room [wikipedia.org] . As far as I can see, only the hype from the company, including the restatements of same in the referenced articles, make any claims as to "understanding". If there were any evidence of that beyond the hype, I have no doubt those in the field of consciousness studies would tear it apart, if they even bothered to waste their attention on it. If in being bashed it then produced a statement equivalent to "I can feel it, Dave" without being programmed to respond in that way, then I'll give it a look see. Until then it's simply a semantic parser (something already done) attached to a search engine.

Re:Jargon pisses me off...

Would you like some Grok-amole on your taco?

Re:Jargon pisses me off...

"Grok" isn't jargon. It's a perfectly cromulent word. (Albeit one coined by Heinlein in 'Stranger In A Strange Land'.)

Re:Jargon pisses me off...

Grok.

Grokgrokgrok.

Pics or it didn't happen.

Re:I wonder how long...

I believe you've stumbled upon this startup's business plan.