What's It Like to be Google's Boss Techie? 671
We'd like to welcome Google Director of
Technology Craig Silverstein as our next Slashdot
interview victim... err... guest. You think you
run a big Linux server farm? Craig's is bigger.
Think your Web site gets a lot of traffic and
creates a lot of headaches? Just think what Craig
must face! Post whatever you'd like to ask Craig
below, one question per post. About 24 hours after
this runs we'll email Craig 10 of the
highest-moderated questions, and we'll post his
answers shortly after he gets them back to us.
technetcast.ddj.com/ (Score:3, Informative)
The Technology Behind Google 2000-10-19 (1hr 13min) By Jim Reese, Chief Operations Engineer, Google. How to build an internet search engine that indexes 1-2 terabytes of data 200 million web pages- and serves it up at a rate of 1000 requests/second. (Hint: Start with a farm of 10,000+ Linux servers). The technology behind Google: company overview, search parameters and results, hardware and query load balancing, Linux cluster topology, scalability, fault tolerance, and more. [420]
http://technetcast.ddj.com/tnc_search.html?key=
Re:Question 2 of 2: Browser Stats (Score:3, Informative)
http://www.google.com/press/zeitgeist.html
Re:Logo work? (Score:4, Informative)
If you want to know more about the special logos (referred to as "Google doodles"), as well as see an archive of the Google doodles over the years, go here [google.com].
Nathan
Re:Google cache (Score:3, Informative)
Re:Statistics (Score:5, Informative)
And the Sergey says.... (Score:4, Informative)
From one interview...
Jason: What led to Google's decision to use Linux? When did that start?
Sergey: Well, Larry Page and I were in the Stanford PhD program in Computer Science. And we developed Google there. The way the computer science program worked is there was a hodgepodge of computer equipment lying around, and we would grab whatever scraps we could. We had all kinds of computers: HPs, Suns, Alphas and Intel's running Linux. So, we gained a lot of experience with all of those platforms.
When we started Google, we had to make the decision of what we wanted to use. Of course we chose Linux, because it is the most cost effective solution.
PCs are not only much cheaper these days, but we can also get them very quickly, because they're such a commodity item. That's an incredible benefit. We just installed another 1,000 computers and we got that done in a few weeks. That's really hard to do with any other kind of workstation. I think that's an advantage that people don't entirely realize.
Jason: Did you view it as being better, or was cost the main reason?
Sergey: It was better in some ways. Certainly for our purposes, we felt the support was better. For example, the actual kernel authors will respond to problems pretty quickly. They are especially responsive to Google nowadays, since we're so widely used. We can have a 15 minute turnaround. You can't really beat that for support.
That was an important factor, but frankly, the cost was a bigger issue. PCs are so cheap, which is very important. Sun's Solaris is probably more stable than Linux on PCs. It's hard to determine the blame, whether it's the hardware or the operating system. But, it's a minor difference.
Jason: Then, does all of your support come from newsgroups or do you actually pay for it through Red Hat?
Sergey: We have an operations team of about ten people, which helps a lot. And other than that we check newsgroups and e-mail the authors of the code. Usually, if it's a problem we can't figure out, we go straight to the authors.
Jason: Is Linux used on desktops at Google?
Sergey: It depends. Engineering mostly runs Linux. Business development/marketing runs Windows. Actually, I use Linux with VMWare running Windows. Some people have two computers, particularly some people in engineering who do UI development and need to test things out on Windows platforms. I find it better to just use VmWare and have one computer.
Jason: In a technical sense, what does Linux lack? What does it not provide?
Sergey: The 64-bit file system, which I know they are working on. It's slowly coming around. I think there are still occasionally some stability issues. I'm not saying Linux is unique in that respect, but you definitely want to have reliability. There are some issues dealing with higher memory systems. If you get to 2GB, and you try to push it past that, we encounter various problems. I know we've had some trouble with the network stack when we really push it hard. In terms of having lost most connections from lots of different machines.
And from another...
How is Linux used at the Google Projects? Why was Linux choose to improve Google search engine?
Sergey Brin: Actually, we currently run over 6,000 RedHat servers.
Linux is used everywhere...on the 6,000+ servers themselves, as well as desktop machines for all of our technical employees. We chose Linux because if offers us the price for performance ratio. It's so nice to be able to customize any part of the operating system that we like, at anytime. We have a large degree of in-house Linux expertise, too.
Most of our administrative tools were developed in-house, as well.
Re:Google toolbar for open source browsers? (Score:3, Informative)
Re:Regexp Support Someday? (Score:2, Informative)
Re:Slashdot effect? (Score:2, Informative)
One commercial website I look after has been slashdotted quite a few times on some quite big stories, sometimes resulting in an extra 20k pages/hr, though normally quite a bit less.
On an average day, we serve about 2m pages (so say 150k/hr during the popular daytime hours) and we've quite a lot of spare capacity, so an extra 20k pages/hr doesn't worry us too much.
Google say they're serving 150m searches/day so an extra 20k/hr will go practically unnoticed I imagine.
(see http://www.google.com/press/highlights.html [google.com])
While I'm on the subject, does anyone else have experience with running a fairly big site while it's being
Re:Why did you chose Linux? (Score:3, Informative)
fFollow me a moment. it is certain to be cheaper to setup a thousand computers of linux, rather than, lets say, a dozen computers of windows. licensing is just out of this world. but what about now? 100,000 machines later, several years down the road, heavily entrenched in linux, do you, Craig Silverstein, wish you had a different platform? would it, at this point, be more convenient to have something else, if you could change it all in an instant?