Is Microsoft Crawling Google?

triplecoil writes "Jason Dowdell over at WebProNews has written a piece questioning a tactic Microsoft might be using to beef up its new search engine. He thinks they might be dipping into Google's results to supplement its own. Dowdell likens it to leaving your garbage on the curb--anyone could conceivably go through it and take whatever is there for their own."

Msn Crawling

[clinko]

If you've been watching the logs to your site lately Microsoft has been RAPING most servers. Most crawlers will pick through pages with large lists 1 at a time, then come back every hour or so.

MSN starting last week has been pulling EVERY LINK in sequence from my site. Even the larger Artist Index pages [clinko.com] of my site.

Seriously, I've had this same spider on my site for about 36 hours now.

Violates Google's TOS

[Anonymous Coward]

From Google's Terms of Service [google.com]

Personal Use Only

The Google Services are made available for your personal, non-commercial use only. You may not use the Google Services to sell a product or service, or to increase traffic to your Web site for commercial reasons, such as advertising sales. You may not take the results from a Google search and reformat and display them, or mirror the Google home page or results pages on your Web site. You may not "meta-search" Google. If you want to make commercial use of the Google Services, you must enter into an agreement with Google to do so in advance. Please contact us for more information.

Re:LOL

[Anonymous Coward]

Habbit = What a priest wears
Habit = A regular behavior for a person/thing

Re:Try this term on MSN search

[fireshipjohn]

Now try it on google and you get articles about the 'more evil that....' debate.

I know which search engine I'm sticking with :)

Spike the results, then sue

[G4from128k]

It would be easy for Google to insert a small fraction of non-sequiturs in the results, look at Microsoft's search results, and then sue for misuse. Even if MSFT uses random proxies to avoid detection, it cannot manually recheck all the hits to make sure they are correct (if they could, they had the resources to check all the sites, then they not need to crawl Google. A few made-up sites or inappropriate search hits would be enough to establish a pattern of abuse.

Re:Try this term on MSN search

[Garion Maki]

pritty funny :)

but it seems like google started it several years ago.

http://www.cnn.com/TECH/computing/9911/15/search.e ngine.ms.idg/ [cnn.com]
and
http://searchenginewatch.com/sereport/article.php/ 2167621 [searchenginewatch.com]
btw, it doesen't seem to work on google anymore...

Re:Microsoft stealing someone elses technology???

[isometrick]

Google Terms of Service [google.com]

" ... You may not take the results from a Google search and reformat and display them, or mirror the Google home page or results pages on your Web site. You may not "meta-search" Google. If you want to make commercial use of the Google Services, you must enter into an agreement with Google to do so in advance ..."

Re:Does it violate Google's Terms of Service

[nick13245]

Yes it does.
From Googles Privacy Center (http://www.google.com/terms_of_service.html):

Personal Use Only

The Google Services are made available for your personal, non-commercial use only. You may not use the Google Services to sell a product or service, or to increase traffic to your Web site for commercial reasons, such as advertising sales. You may not take the results from a Google search and reformat and display them, or mirror the Google home page or results pages on your Web site. You may not "meta-search" Google. If you want to make commercial use of the Google Services, you must enter into an agreement with Google to do so in advance. Please contact us for more information.

Re:This could be entirely natural...

[IIH]

If MSN's bot happened upon a link to a Google search page, is it somehow wrong for the MSN bot to follow that link, and spider as normal?

Find a link, fine
Follow the link, fine
Spider the link, not fine - google's Robots.txt [google.com] does not give them permission to.

Re:Don't concern yourself with this crap...

[Jahf]

IANAL but I would see this as falling under fair use.

1) the LoC is not profitting from your works nor is it re-using them (with the exception of providing an archive to others, see next item).

2) the LoC regularly tells people requesting copies of their information to first obtain permission from the copyright holder (in other words, as with any library, you can browse but you can't copy without permission and copy permission does not equal permission to reuse in a commercial work).

3) Copy protection schemes require active protection to fall under the DMCA, even if it is so simple that anyone can defeat it. Robots.txt is -passive- protection because you have to purposefully search for the file and then purposefully utilize it. To be active protection the document should not come up without the viewer (or blocked viewer) performing some form of action. When someone/something visits an unprotected public web page there is not a way for your web server to invoke the robots.txt file, therefore it is not an active mechanism.

Re:Don't concern yourself with this crap...

[ad0gg]

If don't want your site indexed or cached by google. Go here and follow the directions.

Remove yourself from google [google.com]

"Note: If you believe your request is urgent and cannot wait until the next time Google crawls your site, use our automatic URL removal system. In order for this automated process to work, your webmaster must first insert the appropriate meta tags into the page's HTML code. "

Re:Spike the results, then sue

[Dogun]

Seems you don't understand how search engines work^^

What a normal spider does is generally try different IP's, see if they're running a webserver. Then they do a DNS lookup, fetch http:///robots.txt and read that to decide if indexing is allowed, and where. Then it just walks through the website. A number of places on the website might not be directly accessible, but also not disallowed for indexing by robots.txt.

If some other site has a link to that webserver in some disconnected region of the website, then the crawler generally makes sure it's okay to index that against the robots.txt, and if so, indexes.

The accusation here is that Microsoft isn't finding these adresses on their own, but instead using google's 'site:host.domain' results as a shortcut, which would constitute a violation of google's terms of service.

Re:More lies from cowardly trolls

[Asphalt]

They do profit from your data. However, being that it is publically available on an HTTP server, that's pretty much their right. That's like you handing me $5 for me to tell you which magazines you might like to buy.

And MSN crawling Google's site is really no different. As long as the Google data is on a public server, it is fair game to crawl.

Re:Try this term on MSN search

[StikyPad]

His thought process probably started here [google.com]

google doesn't allow bots to crawl google.com...

[lixlpixel]

see their http://www.google.com/robots.txt [google.com] robots.txt

so if the msn bot does what they say it doesn't do what it's supposed to do.

Re:Don't concern yourself with this crap...

[djcapelis]

To remove all the images on your site from our index, place the following robots.txt file in your server root:
User-agent: Googlebot-Image
Disallow: /

That should work? No?