Google's Secret Lab

Voelspriet writes "The mystery behind eval.google.com should be solved soon. It's a secret lab of Google. Real people, from all over the world, are paid to finetune the index of Google, reveals Searchbistro, a new weblog of the Dutch reporter Henk van Ess. A Flash-movie shows some screens of Google's 'Rater Hub', Full details about Eval.Google will be published later this week.

Oops!

[MicroBerto]

Looks like all of those people from around the world will be getting paid for doing nothing while the site's slashdotted!

Re:Oops!

[bigbigbison]

Correct me if I'm wrong, but doesn't the article say that if you don't have a password you will get a blank page?

Re:Oops!

[Eric604]

I don't know, I suggest you reread the article.

Lab.

[Poromenos1]

And from this article we learn what, that google has a feedback facility?

Re:Lab.

[ackthpt]

And from this article we learn what, that google has a feedback facility?

Google have secrets you can't even begin to imagine, like this toast [tinyurl.com]!

Eval?...

[Jambon]

...I thought they said they wouldn't do any of that?

Re:Eval?...

[Anonymous Coward]

yeah, i read evil.google.com too...

Re:Eval?...

[Gopal.V]

No, they never said "Do No Eval" in their mission statement...

I think the mystery of google's secret sauce of search engine algorithms has finally crumbled. Now I wouldn't be surprised pigeonrank [google.com] was true or not.

And perhaps they need to employ somebody other than young teenagers with left hand typing skills - see the image search results for RSS [google.co.in].. (mm.. seductively feathered pigeons comment ? ).

Whew!

[TripMaster Monkey]

<sarcasm>
Finally...past noon on the East Coast and still no Google Slashdot story....I was beginning to get worried.
</sarcasm>

From TFA:

What is it? It's a lab of humans from all over the world (from China to The Netherlands, from Korea to Brasil) They are paid to check search results of Google every day. Most of the employers, called international agents by Google, were recruited through universities all over the world. The aim is to avoid spam, to get the right sites at the top of the listing and to test new features, not shown to the public yet.

OK...so the best search engine is people. Well, as long as you have the scratch to pay these international agents (which Google certainly has), it seems like a great idea. If this arrangement can cut some of the spam, hallelujah, although I'm wondering just who chooses which sites are the 'right sites' to ocupy the top of the list...

Anyone know where I can put in an app?

Re:Whew!

[Naikrovek]

They've had these at Yahoo! since the beginning - they're paid surfers. Their title is "Surfer".

This is just a plain-old good idea, I'm not surprised that Google is doing it, or that Yahoo! did it first.

Disclaimer: I used to work for Yahoo!

Also, the majority of the female Surfers at Y! were gorgeous when I worked there. But that's neither here nor there.

Re:Whew!

[twofidyKidd]

"Also, the majority of the female Surfers at Y! were gorgeous when I worked there."

Hiring hot chicks is ALSO just a plain-old good idea.

Re:Whew!

[Patik]

Also, the majority of the female Surfers at Y!

Two out of the three?

Re:Whew!

[pizen]

<Charlton Heston voice>
so the best search engine is people.
</Charlton Heston voice>

Google is the Soylent Green of search engines!

Re:Whew!

[alnjmshntr]

So much for Google being all about geeky search algorithms... I guess that doesn't work.

Re:Whew!

[joepeg]

Det. Thorn: It's people. Google is made out of people. They're making our search engine out of people. Next thing they'll be breeding us like cattle for answers. You've gotta tell them. You've gotta tell them!

Hatcher: I promise, Tiger. I promise. I'll tell the exchange.

Det. Thorn: You tell everybody. Listen to me, Hatcher. You've gotta tell them! Google is people! We've gotta stop them somehow!

Soylent Google

[DuckofDeath87]

Google is made from people!

Re:Whew!

[VAXcat]

What is it? It's a lab of humans from all over the world (from China to The Netherlands, from Korea to Brasil) They are paid to check search results of Google every day. Most of the employers, called international agents by Google, were recruited through universities all over the world. ...in other words, they're all on the Global Frequency.

Obligatory Matrix Tie-in

[Wattsman]

TFA:

What is it? It's a lab of humans from all over the world (from China to The Netherlands, from Korea to Brasil) They are paid to check search results of Google every day. Most of the employers, called international agents by Google, were recruited through universities all over the world. The aim is to avoid spam, to get the right sites at the top of the listing and to test new features, not shown to the public yet.

SEO employee: Our client listings got kicked from the first page of Google. I don't k

Bias?

[Saeed al-Sahaf]

It's going to turn into a Slashdot GroupThink (tm) situation. Indexing must be done without bias, something that MOST (certainly not at /.) are not.

Re:Bias?

[NoTheory]

This isn't a group think issue, because individuals presumably not using a judgement of what they think other people want, they're doing it via faculty to determine what they think is best for them.

On top of that, why isn't this the way you want to do it? Essentially, what this is doing is highering a bunch of low rate librarians, or researchers, figuring out a computational way to amplify their results so that i can scale to a huge number of people (rather than just a few) and then running with it.

Re:Whew!

[ImaLamer]

Well, as long as you have the scratch to pay these international agents

I assume you mean "agent" to read "sweatshop worker"

Asia and South America? Where the wages are "extreme" to say the least.

Re:Whew!

[ryanjensen]

Would you rather make $10 a day surfing the web, $10 a day working in a shoe factory for 14 hours, or $0 a day without a job? I'd say Google is doing them a favor. (wink)

The wages may be extreme, but the cost of living is also "extreme" in the same way.

Re:Whew!

[DrinkingIllini]

Anyone know where I can put in an app?

They'd put these little quiz/application things in the student newspaper here (UIUC) all the time last year. It usually consisted of a bunch of pattern finding / abstract thought questions. I think this must be what they were actually applications for.

Re:Whew!

[markhb]

Chuck Woolery?? Is that you??

Re:Whew!

[bill_kress]

Hmm. I was under the impression that they already monitored the selections people made within the searches and PageRank to adjust future searches.

How is paying them to do so better?

Re:Biased lab.

[wo1verin3]

If you're looking for an exact phrase you can just put the query "in quotes" and it'll look for exact matches.

Re:Biased lab.

[moonbender]

Are you kidding? He obviously knows about the phrase-feature - doesn't everybody? - the problem he describes is that it doesn't really work properly. Try the example queries he quotes. Result 2 when searching for exact matches on "montana mountain ranges" is indeed Montana's Mountain Ranges (put in bold so there is no doubt it's this phrase that triggers the hit).

Of course, this isn't so much a bug as a documented feature - Google automatically searches for morphologically similar keywords. But what typica

Evil

[Anonymous Coward]

Thought it said "evil.google.com" for a sec then instead of "eval.google.com".

Re:Evil

[hosecoat]

and all this time googles been playing with us, when they really ment 'do no eval'

Re:Evil

[mbaudis]

he is just eval.

secret lab

[mindwar]

.. every mad genius need a secred lab

Wouldn't this look better?

[peragrin]

Secretlab.google.com ?

make sure you don't let google search it or it might be come known.

Re:Wouldn't this look better?

[morgan_greywolf]

Hmmm... So, let's say you are a 31337 h4x0r and you are looking to 0wn some b0x. Which host would you try? eval.google.com secretlab.google.com The latter, of course....then you can say 'w00t! 1 0wnz0r3d g00gl3's s3cr37l4b!!!

Don't you love it

[slashrogue]

When the submitter of the story writes the summary as if he were an objective 3rd party when he isn't? Far as the actual thing goes, it's interesting but so far very few details. I want to know how much these guys get paid and how long they work each day, and where do I apply if it's decent? ;)

Do no eval

[tehshen]

This is proof... Google is slowly turning eval!

</bad puns>

Re:Do no eval

[Reverend528]

Or it could just mean they're switching to Lisp...

Proof of the Evil Bit

[sbowles]

This seems to be one of the first documented uses of the Evil Bit (RFC 3514) [faqs.org]:

eval -> 01100101011101100110000101101100
evil -> 01100101011101100110100101101100

Google has obviously turned off the Evil Bit.

Real people?

[Timesprout]

Damn I always imagined Google would have legions of tiny googlenauts or elves to do this sort of thing.

Re:Real people?

[Fred Ferrigno]

No, they're pigeons. [google.com]

Re:Real people?

[pizen]

They used to use pigeons.
http://www.google.com/technology/pigeonrank.html [google.com]

Oh no! Secret labs!

[half_d]

Are the US going to attack Google, like they did with Irac's secret labs??

Re:Oh no! Secret labs!

[chadbailey]

No, and learn how to spell.

There aren't many people evaluating pages

[waynegoode]

There aren't many people evaluating pages. It's page rank [netcraft.com] is only 513,258; right after the ever-talked-about talk.guns.ru.

surprise ?

[cyberfunk2]

Is this really so surprising to people that google has a human feedback loop ?

Their pagerank system, while incredibly good, is not infalliable, and requires some checks and balances.

Did they ever say directly that they didnt use humans to check up on the system?

I dont really see what the big secret is here..

Re:surprise ?

[SolidGround]

It means that Google either retired/fired all of its pigeons (http://www.google.com/technology/pigeonrank.html [google.com]) in favor of humans, or it reveals that the pigeons were actually peons all along.

Gotta love the mainstream media.

[Anonymous Coward]

Real people, from all over the world

As opposed to imaginary people that exist only in my head.

Hey, wait a minute...!

[Junior J. Junior III]

I thought Google was all about NOT being eval. What gives?

Re:Hey, wait a minute...!

[Slime-dogg]

Give a man a fish, he owes you one fish.

Teach a man to fish, you give up your monopoly on fisheries.

Give a man a fish, he owes you one fish.

Teach many men to fish, you create healthy competition, and collect on fishing IP licensing.

Re:Hey, wait a minute...!

[BlackShirt]

mod down

pigeon

[unk1911]

shouldn't that read http://pigeon.google.com/ [google.com] ?

--
http://unk1911.blogspot.com/ [blogspot.com]

Google Slashdotted

[gcatullus]

Looks like Google has finally been slashdotted, or well at least a tiny part of Google.

Secret Lab?

[grassy_knoll]

Don't tell me it's in a supposedly inactive volcano on a south-pacific island, perhaps guarded by large numbers of minions in silver jumpsuits.

'Cause while I'm sure there are many Phd's there... I'd like to think none of them went to evil medical school.

NB: Not responsible for the reactions of the humor impaired.

Re:Secret Lab?

[mbbac]

You're thinking of Google's Secret Lair.

Re:Secret Lab?

[SUB7IME]

True, most PhDs didn't go to any medical school.

Erotic Writing

[mizel]

I especially like the last link in the flash-movie:
"Erotic Writing"
Looks like this google agent is enjoying his job.

Google Rocks!

[coopaq]

Google I love you! Google! I want to have your babies! Gooooooogle!!!

Re:Google Rocks!

[GraZZ]

Insightful? WTFBBQ?!

Re:Google Rocks!

[coopaq]

Insightful? WTFBBQ?!

Dude, I don't know, but shutup about it.

If the people find some infinite wisdom in it then more points for me :)

Re:Google Rocks!

[cmdr_beeftaco]

God I wish I had mod points for you.

Can't get to the link...

[bogaboga]

...because it's already slashdotted! Since the slashdot effect is almost always guaranteed here at slashdot, what can really be done to avoid it? We do really need a solution to this effect.

What about slashdot editors setting up a mirror before posting?

Re:Can't get to the link...

[HyperChicken]

Read the Slashdot FAQ, moron. If users of Slashdot who subscribe (and thus can see the story early) would CoralCache it and then post a link, we'd all be saved.

Re:Can't get to the link...

[General Wesc]

> Since the slashdot effect is almost always guaranteed here at slashdot, what can really be done to avoid it? We do really need a solution to this effect.
Google, as always, has the answer. [google.com]

proof of their pur eval

[th0mas.sixbit.org]

http://toolbar.netcraft.com/stats/hosters [netcraft.com]

proof of their eval - they are #4 most phishiest hosting company!

(anyone have an explanation for this?)

Re:proof of their pur eval

[topper24hours]

Maybe they consider companies than have toolbar products that take over the standard msn search and switch it to google for example to be phishing products somehow?

This doesn't surprise me at all.

[bigwayne]

It doesn't surprise me how they have this set up.

The system is widely distributed (read: international), low cost (read: $20 an hour), with no single point of failure, every person can be replaced if they break down, because it's been factored into the design! They apply the same Zerg mentality to everything!

Mirror!!!

[Cavalkaf]

Managed to get the files just before it got slashdoted :-D http://cavalkaf.frihost.net/mirror/secret_google/ [frihost.net] (Entire Page) http://cavalkaf.frihost.net/mirror/secret_google/s ecretlab.swf [frihost.net] (Flash Movie) Oh, and visit my forum http://forum.scuzzstuff.org/ [scuzzstuff.org] while you are at it ;-)

Unfortunately...

[spacefight]

... neither eval.google.com nor searchbistro.com were available for comment :P

Re:Unfortunately...

[Morgahastu]

Eval.google.com isn't down, it just doens't respond to you unless you're a known ip.

Oh My God!

[tfaust]

It's made out of people! eval.google.com is people! It's people!

Re:Oh My God!

[Muhammar]

Yes, the secret is out and Eval is dealing with the situation...And yes, the MIBs are mostly people.

Raters

[RealProgrammer]

I wonder if they use meta-moderation?

[rant]

Do the folks who meta-mod the raters actually get more rater points, or is it like a certain well-known site (hint: rhymes with "cashbot") where you can meta-mod all the time and never get to mod?

Every time I ignore that certain site (rhymes with "'stache rot") for a few weeks and come back, voila, I'm rewarded with mod points. I don't like that. When I come back after being away, I want to spout off, not critique. Gimme mod points after I've been active ever

Re:Raters

[geekoid]

"... I want to spout off, not critique...."

So it works just fine.

Re:Raters

[dubiousmike]

my sig says it all

A few years old

[m00nun1t]

http://www.webmasterworld.com/forum3/11972.htm [webmasterworld.com] discusses people getting referrals from eval.google.com back in 2003, so it seems this has been around for a while.

Google, google google, google

[EpsCylonB]

Someone should create a slash google site.

Re:Google, google google, google

[teorth]

Someone should create a slash google site.

Someone already has. [googledot.org]

Terry

$20 an hour

[Virtual Karma]

Many people are asking as to how much Google pays those international agents. The complete page of the blog didn't load (obviously /. ed) and the alternate text for the image read "... $20 per hour.." so thats how much they are paying

Am I the only one...

[Shadow Wrought]

Who read eval.google.com as evil.google.com? I figured they had taken the "be good" option to a new level and had locked all their evil stuff away to its own site;-)

Slashdotted.

[SharpFang]

Slashdotted already. Google cache anyone?

manual review...not a 'secret lab'

[mcguyver]

At first glance this looks like a portal for people to manually review serps and provide feedback for the serps that are in beta. Yahoo does this. Google has recruited people over the past year to review results (first indication of this came from a craiglist ad). If this is true then it's cool to witness how this process is accomplished. While on the subject of monitoring serps, here's a way to search on Yahoo and tune [yahoo.com] your results.

MOD PARENT UP

[inKubus]

That's cool. Real cool. I love Yahoo.

Agggh!!!

[MightyMartian]

Google searches are made out of people! They're people!

I bet their labs have more than that!

[Man in Spandex]

I bet they have a thousand monkeys working on a thousand typewriters.

Burns: It was the best of times, it was the blurst of times!

Re:I bet their labs have more than that!

[bennomatic]

Naw, all it takes is a hundred [100monkeys.org].

Google now advertising with Aurora

[bbrockit]

I was searching Google to find out how to remove this obnoxious Aurora adware client when I got this pop-up [banzailabs.com] ad for Google Desktop Search.

Secret lab discovered? Time to call in...

[IronChefMorimoto]

...Brock Samson (http://www.adultswim.com/shows/venturebros/index. html [adultswim.com] ) to kick the sh*t out of this guy running the blog.

IronChefMorimoto

Where's the beef in that?

[broothal]

Uhm... either you're all getting this wrong, or I am. Of course google has people evaluating their search index. You guys aren't really buying the pigeon thingy [google.com], right?

Of course they're evaluating their search and ranking technology all the time. It's also evolving all the time. And of course there's a site where you can log into from all over the world (google is global you know) and of course this site is restricted to those people they hired to do the job.

Way to toot your own horn.

[Anonymous Coward]

Voelspriet -> [e-mail]

Henk van Ess

Domain Name: SEARCHBISTRO.COM
Registrant:
voelspriet.nl
regenboog 36
amersfoort 3824ED
Netherlands

Registered through: GoDaddy.com
Domain Name: SEARCHBISTRO.COM
Created on: 22-May-05
Expires on: 22-May-07
Last Updated on: 22-May-05

Administrative Contact:
van ess, henk [e-mail]
voelspriet.nl
regenboog 36
amersfoort 3824ED
Netherlands
[phone] Fax -- [phone]
Tec

Flash animation on second server

[Voelspriet]

I've put the Flash-animation of eval.google.com on a second [zoekzone.com] server.

I know what this is.....

[JaF893]

Google has made a Beowulf cluster of old Korean people and its based in Soviet Russia! But it will be in Beta until Duke Nukem Forever comes out in Japan.

(eval is the Russian for Beowulf cluster of Old Korean people)

Google REPL?

[sinserve]

For a second there I thought I was gonna get a google repl;

(define google
(lambda ()
(and (print (eval (read)))
(google)))

Huh?

[Rorschach1]

I don't get it. Where do the pigeons fit into this scheme?

Huh?

[Mac Degger]

I thought their corporate charter specifically said that Google should 'do no eval'?

Google sells out

[Ed Avis]

Surely they've broken one of the founding rules of Google: Don't Be Eval.

Re:Eval ~ Evil???

[deutschemonte]

That would have been one of the best google jokes ever.

Re:Eval ~ Evil???

[ReverendLoki]

...or maybe the should create a site evil.google.com, and have it resolve to either a blank page, or in spite, MSN search. Throw in good.google.com that resolves to Google's main page, and you're set.

Re:YAGS

[c0d3h4x0r]

I'm sorry, but "Jesus Christ on a Stick" has already been patented by my company. Prepare to be sued.

Re:YAGS

[geekoid]

Who new Pontius Pilate posted to slashdot?

Re:YAGS

[Leebert]

I'm sorry, but "Jesus Christ on a Stick" has already been patented by my company.

Your website says you work for Microsoft.

Somehow, I'm unsurprised. :)

Re:Like /. Moderation??

[downsize]

right, that's what we need to make this world a better place, a bunch of idiots unable to separate their feelings or opinions on the results - or simply not intelligent enough to figure out if the search result is helpful or not to the searcher.

how do you know I don't want to see baseball links when google for '...midget and with a bat'