Is Microsoft Crawling Google? 480
triplecoil writes "Jason Dowdell over at WebProNews has written a piece questioning a tactic Microsoft might be using to beef up its new search engine. He thinks they might be dipping into Google's results to supplement its own. Dowdell likens it to leaving your garbage on the curb--anyone could conceivably go through it and take whatever is there for their own."
Don't concern yourself with this crap... (Score:4, Insightful)
Sure, I see crawlers on my site all the time sometimes hitting the same URL over and over again. Do I understand their repetitive behavior? No. Do I care what they are doing? No, as long as they are obeying my robots.txt.
I have complained before about MSNbot ignoring changes to robots.txt while Google happily changed its habbits (I can't find the link sorry). My recent fighting with Googlebot has come to a head when I had to disallow them access to my gallery completely because they refused to honor anything except Disallow:
Do I care if MSNbot is crawling Google and then finding sites and links to search? No as it's none of OUR concern. What is OUR concern is our own robots.txt and how the spiders interact with our sites through that file. Let Google deal with Microsoft/MSNbot if that's what needs to be done but don't concern yourself with it otherwise.
Difficult to do if Google doesn't want them to (Score:5, Insightful)
Does it violate Google's Terms of Service (Score:5, Insightful)
If not, it's called doing business and gaining an advantage any legitimate way that you can.
I think the interesting bit is in the conclusion. If MS is using this to establish a baseline, they can benchmark their spider against Google's over time.
Why not? (Score:1, Insightful)
I'd do the same thing if I could. This is all "speculation" anyway, but since it feeds the stereotype of the insidious Microsoft, it gets posted front page to this "tech news" site.
Re:Does it violate Google's Terms of Service (Score:3, Insightful)
If not, it's called doing business and gaining an advantage any legitimate way that you can.
I think the interesting bit is in the conclusion. If MS is using this to establish a baseline, they can benchmark their spider against Google's over time.
If I copy your work and take credit or it, does it violate your terms of service? If so, you have legal remedies. If not, it's called doing business and gaining an advantage any legitimate way that I can.
Furthermore, I think the interesting bit is in the conclusion. If MS is using this to establish a baseline, they can benchmark their spider against Google's over time.
Comment removed (Score:4, Insightful)
Re:Don't concern yourself with this crap... (Score:3, Insightful)
Crawling a gallery of images (and all image property links as well) all day for several days might be considered "DoSing" I consider it being rude.
You're right, they don't have to obey the robots.txt but they should when they say they will.
Re:More lies from garcia (Score:3, Insightful)
Absurd (Score:5, Insightful)
1) His whole theory is based on the "fact" that the only way in the world to find his pages is to use site:www.sitename.com in Google, implying that Google has cached the results from an earlier crawl. Of course, there is no way that the Microsoft search couldn't have also cached it.
2) Then, he claims that Microsoft is probably screen-scraping Google's results (for all the millions of sites out there), and using these results to recrawl those sites? This doesn't even make any sense.
3) And last but not least, Microsoft is certainly basing its whole search architecture on the assumption that Google wouldn't ever notice MSN mirroring its whole index. Yeah right.
Re:But will this mean Google can crawl back? (Score:0, Insightful)
If Microsoft is indeed doing this however, they could become real competition a lot sooner than you'd think.
a company I worked for did this once... (Score:3, Insightful)
Terrible article (Score:5, Insightful)
Uh.
Microsoft has been developing their internal search engine for quite a while now. Part of developing a search engine is using it to crawl and creating a large corpus of test data. It's hugely likely that M$ has had a working crawler system for much, much longer than would be indicated by their public announcement. Quite a few people who helped develop Altavista at HP/Compaq/DEC research joined Microsoft Research about two years ago - the kind of people who could write a high-performance crawler in their sleep and wake up feeling refreshed.
That article seems like baseless, uninformed speculation, to put it not-so-politely.
This could be entirely natural... (Score:5, Insightful)
Interesting (Score:2, Insightful)
worthless (Score:1, Insightful)
Bogus article (Score:3, Insightful)
And I'm supposed to take this clown's "friend" seriously? That's not a good start, anyway.
But then there's the real howler: the site can allegedly only be found through site: on Google. How does the friend know that? Has he done a complete crawl of the web to find all forward links to any image in his site -- even broken ones? MSNBot, like all bots, recognizes that many anchors are broken, and tries plausible corrections around the broken links. That's particularly useful with a deep link, where the deep link may have timed out but the shallow link still exists.
Highly unlikely (Score:4, Insightful)
How do I know? Because a friend of mine decided to find out how common all TLAs are (three-letter acronyms) by counting Google hits on each TLA. This was before the Google API, so he did it with good old fashioned HTTP/HTML. It didn't take long for Google to flag him as evil and block access from his IP block.
Sure, Microsoft could find some way around this-- using different enough IP addresses to conceal the source-- but that's more trouble than it's worse. Worse yet, it sets up a cat-and-mouse game and keeps M$ dependent on Google-- when their stated goal is to beat Google at its own game.
I've got a simpler explaination for what the author is seeing. His evidence is based on the fact that some pages being requested exist only in Google's cache. Well, spiders are supposed to do breadth-first searches so they don't hit the same site too often. Microsoft is probably going against data it collected a few weeks ago but hasn't put on its public servers yet. (Why not? Could be lots of things. Maybe they haven't put enough hardware on the front end to support the amount of data they have on the back end. Or maybe they're just slow.)
As much as I'd like to bash M$, there's nothing here that really looks suspicious to me.
Re:Don't concern yourself with this crap... (Score:2, Insightful)
In a sense it's like tourism. The world is full of stuff like Historical buildings and the owners of those places have legal rights against theft/damage etc. But the tour companies can still take people around the streets and show them the places without having to necessarily pay a fee.
Not quite (Score:4, Insightful)
My garbage doesn't have a copyright statement, contain my patented technology, nor does it come with terms of service or licensing agreements.
Re:Don't concern yourself with this crap... (Score:2, Insightful)
I would imagine a metacrawler, which attributes its links back to Google, is probably OK, because it keeps Google's adstream intact when the user clicks on the link to a Google search result.
But MSNSearch (or whatever it's called), taking Google search results as its own without attribution, well, that might be a copyright infringement...
If you were an on-line bookstore and deep-linked to Amazon's reviews while portraying them as your own, well, you're gonna get a C&D from Amazon's lawyers awfully fast.
Re:Don't concern yourself with this crap... (Score:5, Insightful)
I shouldn't need to go and fill out some form for every search engine to protect my rights. One accepted standard way to say "do not index this" should be sufficient. This is an automated system. There is an accepted automated method to stop crawlers indexing your site (robots.txt). If they (Google or anyone else) take your copyrighted content and reproduce it automatically when their automatic system could have automatically respected your explicitly stated and legally protected rights they are knowlingly making a flagrant copyright violation.
Re:Don't concern yourself with this crap... (Score:1, Insightful)
They cached your page with whatever copyright notices you put on it.
What, are you going to tell me that when you put you zipped your website and put it in your p2p shared folder, and someone downloaded it, that they are commitying flagrant copyright violation?
IF YOU DON'T WANT PEOPLE TO SEE IT, DON'T PUT IT ON THE INTERNET NUMBSKULL.
Re:Don't concern yourself with this crap... (Score:2, Insightful)
Google should follow the robots.txt - definitely. But there needs to be some way on confirming on your website that you actually want the pages removing - otherwise what's to stop your competitors "accidentally" entering your URL into the removal form? Meta elements would seem to be the natural choice.
P