Stories
Slash Boxes
Comments
typodupeerror delete not in

Hot Comments

Comments: 138 +-   Extracting Meaning From Millions of Pages on Friday June 12, @07:48AM

Posted by kdawson on Friday June 12, @07:48AM
from the data-mining-gone-large dept.
software
business
google
internet
freakshowsam writes "Technology Review has an article on a software engine, developed by researchers at the University of Washington, that pulls together facts by combing through more than 500 million Web pages. TextRunner extracts information from billions of lines of text by analyzing basic relationships between words. 'The significance of TextRunner is that it is scalable because it is unsupervised,' says Peter Norvig, director of research at Google, which donated the database of Web pages that TextRunner analyzes. The prototype still has a fairly simple interface and is not meant for public search so much as to demonstrate the automated extraction of information from 500 million Web pages, says Oren Etzioni, a University of Washington computer scientist leading the project." Try the query "Who has Microsoft acquired?"
story

Related Stories

This discussion has been archived. No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More
Loading... please wait.
  • by Finallyjoined!!! (1158431) on Friday June 12, @07:49AM (#28306709)
    "Who has dumped Vista?"
    • Oh man, it's the new sucks-rules-o-meter for sure. Who hates vista: 55 results. Who loves vista: 11 results. Obviously, vista blows hairy goats. It becomes even more clear when you look at the actual results: somehow "
      Bookmark Islamic Screensaver download-All people (12) love screensaver-Windows Vista Downloads" counts as a hit.. ahh there, a reload with js and the spam disappears leaving 9 :D

    • Better yet: "Why does Windows suck?"

      Retrieved 0 results for Why does Windows suck?.

      Being in Washington, MSFT has obviously paid them off to filter out unpleasant results.

  • I suppose the major problem with this is that it cannot tell the difference between truth and lies or urban legends, it just repeats what other people have said, even if they are conspiracy theorists. The query "Who killed JFK?" suggests the CIA did it.
    • Re: (Score:3, Insightful)

      The major problem is that it assumes the presence of meaning in Web pages in the first place.

    • Actually, just like any other search, it just shows ALL of the likely results and you are still responsible for determining for yourself which of the statements is true. It says "CIA killed JFK" but the first result it returns is "Lee Harvey Oswald killed JFK". It also seems to pare down the results somewhat, because I know I've seen conspiracies also suggesting that the KGB killed JFK, or that the Mafia killed JFK. I'm guessing that more people think the CIA killed JFK than the KGB or the Mafia.

    • by owlnation (858981) on Friday June 12, @08:16AM (#28306991)

      I suppose the major problem with this is that it cannot tell the difference between truth and lies or urban legends, it just repeats what other people have said, even if they are conspiracy theorists. The query "Who killed JFK?" suggests the CIA did it.

      So much like Wikipedia then?

    • shhhh! we aren't supposed to talk about the grassy knoll or the bad men will come again.
    • ... and yet "Who was responsible for the World Trade Centre attacks?" returns no results...

      [/tinfoilhat]
    • Re: (Score:3, Insightful)

      it just repeats what other people have said

      I don't see anything new here, most people have done this since the beginning of time.

      • it just repeats what other people have said

        I don't see anything new here, most people have done this since the beginning of time.

        Yeah, Textrunner just repeats what other people have said, like most people since the beginning of time.

    • Re: (Score:3, Interesting)

      I suppose the major problem with this is that it cannot tell the difference between truth and lies or urban legends

      Most humans can't either, how do you expect a search engine to?

      There will be a lot of false positives and negatives that will be hard to identify as such unless it directly works with something like snopes.com , which kind of defeats the purpose because it means someone has had to research every question anyway.

      If a project like this which simply scoured the whole 'net, you wouldn't really be able to verify anything beyond people's opinions or beliefs, which may or may not be 'true'.

      I think something like t

    • "The query "Who killed JFK?" suggests the CIA did it"

      Hmmm....And now its not responding because its "slashdotted"

    • And you suggest they didn't?

    • Re: (Score:2, Informative)

      Semantic processing systems like this (it's not something new) aren't usually able to determine correctness. The truth of a statement is assumed and the best these NLP [wikipedia.org] engines can do at the moment is identify conflicts and maybe use some reputation metrics to assign a veracity rating to a particular statement, or notify the user that there are differing conclusions. These systems are just really, like the summary states, "information extraction [wikipedia.org]" systems. Just as a regular search engine will return you the r
        • by Colonel Korn (1258968) on Friday June 12, @08:50AM (#28307379)

          That is how Wikipedia was meant to be. A group of statements about subjects, all of which can be referenced to some original source. So that people can look up something quickly and then look at the sources for more definite information....

          Seeing how many people cite Wikipedia directly, use it as the main source for their research and the amount of newspapers that have been reported to directly quote inaccurate facts from Wikipedia... I don't think it is working properly. It requires a lot of optimism to believe "People will use that as a initial source and then verify the information"

          That's not wikipedia's failure. Those same people would just be referencing nothing or a web site with zero public review and commenting without it.

  • Nascent AI? (Score:5, Funny)

    by Drakkenmensch (1255800) on Friday June 12, @08:02AM (#28306841)
    I've always viewed intelligence as the ability to take unrelated facts and create new and original ideas from their synthesis. This project may very well lead to new ideas to create the first true AI.

    I'll start stockpiling food and armor piercing rounds for the moment Skynet goes live.

    • I've always viewed intelligence as the ability to take unrelated facts and create new and original ideas from their synthesis.

      Intelligence, like insanity, is finding links between seemingly unrelated facts. It can also be keen observation and recognition of interactions between things where others see chaos. Either way, truly unrelated things are just that: unrelated.

      • I should add that the distinction between intelligence and insanity blurs as the relationship between the facts becomes weaker. Well, at least to the observer of the intelligent/insane person.
  • by Dunbal (464142) on Friday June 12, @08:06AM (#28306885)

    Yet strangely, I get a result of:

    TextRunner took 9 seconds.
    Retrieved 0 results for what is the airspeed velocity of an unladen swallow?.

    Meh, call me when this stuff can answer the really USEFUL questions in life.

  • Zero results (Score:3, Interesting)

    by John Hasler (414242) on Friday June 12, @08:12AM (#28306945)

    I tried half a dozen queries of the sort I often use Google for (example: "What is the velocity of sound in hydraulic fluid?"). No answers.

  • Concise (Score:2, Interesting)

    Try "Who paid SCO?" Concise, to the point. Nice.
  • But AOL is nothing like Shakespeare.

  • by umundane (1490741) on Friday June 12, @08:38AM (#28307231)

    I learned that

    > smoking (387) causes cancer.

    I was also surprised to learn that

    > girls and women (11) cause most cases of cervical cancer

    This is a great resource if you need to cite a reference for a Wikipedia article.

  • Who is at Area 51
    aliens (3), Carter (2), Colonel Sanders (2), Hi Group (2) is at Area 51

    Who bombed WTC
    Al Qaeda (5), Bush (5), Clinton (2), 4 more... bombed the WTC

    Who built the pyramids (example on site):
    Egyptians (298), aliens (73), Pharaohs (40), 77 more... built the pyramids

    What contains antioxidants (example on site):
    Coffee (17), Recent scientific research (15), food (6), 5 more... contain significant amounts of antioxidants

    -- man, I gotta get me some more recent scientific research.

  • Slashdot isn't
    a professional news site
    a normal news site
    a social news site
    a News Site
    a valid source
    a reputable source
    the right source
    a healthy online community
    a goddamn online community
    a Terrorist Organization

    • Ah, but look at what Slashdot is:

      Slashdot is the single most important english site (8), another extremely sophisticated example (4), another online community (3), 15 more...

  • The significance of TextRunner is that it is scalable because it is unsupervised.

    That's what they said about SkyNet.

  • Apparently Mount Marcy, Mount Elbrus, Mount Kilimanjaro and Mount Etna are all the highest mountain. Then again, I was also informed that "high mountains are the hum of human cities torture", so I think I'll just steer clear of mountains altogether.

  • Try
    "What is Slasdot?"

    Answer
    Digg is Slashdot

  • Correction.... (Score:5, Insightful)

    by wowbagger (69688) on Friday June 12, @09:32AM (#28307983) Homepage Journal

    "...that pulls together facts by combing through more than 500 million Web pages."

    Correction:

    "...that pulls together assertions by combing through more than 500 million Web pages."

    Whether those assertions are correct or even reasonable is a completely different issue.

    It might be interesting to then take those assertions and have some means to validate or invalidate them, but currently that's going to require meat, not metal.

    Now, if you could come up with some form of AI^Walgorithm to do that automatically, then you would have something.

  • So I take it this thing also hates grammar?
  • same fuckers(2) that Framed Roger Rabbit

  • TextRunner gets rid of that manual labor. A user can enter, for example, "kills bacteria," and the engine will come up with of pages that offer the insights that "chlorine kills bacteria" or "ultraviolet light kills bacteria" or "heat kills bacteria"--results called "triples"--and provide ways to preview the text and then visit the Web page that it comes from.

    Wow, incredible. Because doing a search of "kills bacteria" with the quotes on Google won't get you those kind of results. Oh wait, yeah it will.
    • Re: (Score:3, Interesting)

      I think you're missing the point. This is an AI project - it's research. Presumably, the questions you are typing in haven't been processed by a complicated nest of if-thens written by someone who knows English; instead, statistical models of language and meaning were extracted from the internet. Some people claim this is the equivalent of "teaching" a computer.

      The first example, which is what most search engines do, leads to impressive search results but is limited by the logic people can code up. This AI,

      • Re: (Score:2, Interesting)

        by Anonymous Coward

        Allowing a search engine to visit a site and allowing somebody to pass your web page content around are two completely different things.

A bank is a place where they lend you an umbrella in fair weather and ask for it back the when it begins to rain. -- Robert Frost