Google Opens Up (Some) Search Algorithms

overmars writes "After years of closely guarding the formula for its search algorithms, Google is opening up a little. The search engine company has kept its search formula a closely guarded secret for two reasons: competition and to prevent abuse, said Udi Manber, Google's vice president of engineering, search quality, in a post on the corporate blog. Manber said the blog post is the first part of a renewed effort at the company 'to open up a bit more than we have in the past.' Manber said the most famous part of Google's ranking algorithm is PageRank, an algorithm developed by Google cofounders Larry Page and Sergey Brin. While PageRank is still in use, it is a 'part of a much larger system,' he said. 'Other parts include language models (the ability to handle phrases, synonyms, diacritics, spelling mistakes, and so on), query models (it's not just the language, it's how people use it today), time models (some queries are best answered with a 30-minutes old page, and some are better answered with a page that stood the test of time), and personalized models (not all people want the same thing),' he said."

Dont do it Google!

As long as Microsoft wants to dominate the search engine market at the expense of Google, Yahoo and anyone else that gets in the way (knowing Microsoft's track record of abusive & dirty underhanded methods). I would keep that a secret to protect the in

Re:

As long as we're twisting the lion's tail, you might instead say that the more people and companies who desire common progress share their work, the more people and companies who want to isolate themselves and their work for the sake of competition will be

Re:

In reality, I'm sure Google's leadership has done some heavy analysis on exactly how much openness benefits them.
and
The search engine company has kept its search formula a closely guarded secret for two reasons: competition and to prevent abuse

Secur

Re:Dont do it Google!

I took one course in Information Retrieval, and I could come up with most of these things with an evening or two of brainstorming, at least on a general level like this. Ideas like PageRank gave Google the edge in the early days, but now, their advantage lies in other areas. The have a stunning amount of capital tied up in hardware, giving them amazing speed, and amazing amounts of data. They have code optimized to handle those amounts of data in reasonable time. They have the experience to take simple probability models like the ones described in the article, and make them work with those amounts of data.

This is why it's impossible to beat Google at search and other data-based markets. It's not one simple patented idea anymore. If it was just that, Google would've disappeared years ago. The only way to beat the points described above, is to have the capital to buy the hardware, and knowledge to match Google. Microsoft can do that, but Google has one other thing that Microsoft doesn't. They understand their developers. They understand that if you give these kinds of scientist/developers an interesting problem, a fantastic dataset and the freedom to attack it in their own way, you barely even have to pay them anymore. The interest will take over and completely fuel the project. They will work overtime, and come in on the weekends, without being asked.

That will bring energy to a project and a company, that you can never get through any tactic that Microsoft is likely to employ. I admit I don't precisely know what Microsoft is like on the inside, but I simply cannot conceive of them as a company that understands the joy of programming, or the joy of science (which is a huge big part of information retrieval). In any case, one blog post with some sketchy details isn't going to tell Microsoft anything they don't know already.

Re:

I took one course in Information Retrieval, and I could come up with most of these things with an evening or two of brainstorming, at least on a general level like this.

Coming up with things is easy; implementing them is hard. Any average Joe Sixpack can come up with the idea of a flying car in five seconds, but to actually build one is another matter entirely - and doing so in a commercially viable way is yet another ma

Re:Dont do it Google!

I took one course in Information Retrieval, and I could come up with most of these things with an evening or two of brainstorming, at least on a general level like this.

Of course you could. ;-) You took a course in Information Retrieval, after all.

Re:

I've worked at both, they are more similar than not...

I'll be joining Microsoft full-time this summer; if that says anything.

What exactly is open?

What, exactly, has Google opened up? As far as I can see fron TFA all that is explained is on a very general level, with no detail what so ever. I can't see Google's competion gaining any significant benefit from this.

Re:

Right. And the competitors already know pieces of what Google has, as a result of the inevitable stream of engineers leaving to take new jobs. Particularly at SV startups founded by ex-Googlers.

While Rob Enderle puts the matter trollishly, I agree with th

To be fair, he's a VP

I've never seen a VP who knows anything about what he's overseeing. So he caught some general phrases from his engineers and put them on the blog. Scientists' posts would be much more interesting.

Re:

The engineering VPs at Google are all engineers themselves. Udi himself was hired for his extensive background in web search, at Yahoo and Amazon. He knows a great deal about what he oversees.

No, they _used to be_ engineers

Now they're VPs. He probably hasn't seen any code and hasn't read any whitepapers in the last decade.

Re:

From TFA...

Other parts include language models (the ability to handle phrases, synonyms, diacritics, spelling mistakes, and so on), query models (it's not just the language, it's how people use it today), time models (some queries are best answered with a 30-minutes old page, and some are better answered with a page that stood the test of time), and personalized models (not all people want the same thing).

Obviously, this usage of the word "open" is not related to open source software. It's more like he is willing to talk about it at all.

Deus Ex, anyone?

Pagerank: I am a prototype for a much larger system.
User: What else do you know about me?
Pagerank: Everything that can be known.
User: How about a report on yourself?
Pagerank:I was a prototype for Echelon IV. My instructions are to amuse visitors with
inform

License?

Under which license is the algorithms being released? If it's a BSD-like license, MS will probably be all over it, but if it's a GPL license, it may be harder for them to claim the algorithms as their own, since they'll have to open up their own code.
At l

Re:

I dont think algorithms are typically licensed. source code is licensed, algorithms are patented.

Re:

There's no indication in the article that any code or algorithms will be released. They're just talking about it on a very broad, conceptual level. The headline and summary are quite misleading.

The secret ingredient...

// Disclosed code snippet from
// Google search algorithm

for (int i=0; i <= numResults; i++)
{
    if (results[i].good)
    {
        show(results[i]);
    }
}

// ...

Re:

You forgot to sort the results by goodness. Do you work for Microsoft?

Re:

You forgot to sort the results by goodness.

That's google's secret

consider the Pagerank important

Accordingly, we must still consider the Pagerank important because it is the only part of the algorithm which we know and we know how to raise it. This is for all those who thought they no longer served the Pagerank for positioning in search engines.

Mystified by 'the google"

I have a terrible admission to make. I, among other things, design websites. Yet, when I search for me on the google, I don't come up. I use relevant terms that are all over my site, and in the metadata (although I understand they don't really matter anymore), yet my own personal site does not come up, even though the url has been up and running for 8 years. The final straw was when I did a search for web design, Ottawa, and a newly opened competitor (just around the corner actually) came up on the second page. I spent the last couple of days researching this (again) and I seem to be meeting all of googles requirements. I have never used a sleazy SEO company, my content is consistent and legal. What's up with that?

Much can be determined by using google

I have noticed often I search for a word and get pages the only contain synonyms (or variations on the word). Likewise for the handling of accents search for resumé and you'll find pages with resume.

Re:

Well, they're probably using some kind of hash based document fingerprinting anyway. Ignoring low entropy characteristics of a word when calculating the fingerprint makes sense, because you can always go back and take it into account once you've eliminated

Re:

While I'm pretty sure Google wouldn't be so ignorant as to violate open source licenses for the code they utilize, is there any claim to his "pirate" label, or is he just trying to be inflamitory?

I think what he's saying is that he thinks Google violates the spirit of licenses (particularly the GPL), even though they follow all the requirements of them. Some people get upset that the Internet makes it so that you can separate the using of softwar