Become a fan of Slashdot on Facebook

 


Forgot your password?
Close
typodupeerror
×
Google The Internet IT

Google Envisions 10 Million Servers 169

Posted by kdawson from the up-scale dept.
miller60 writes "Google never says how many servers are running in its data centers. But a recent presentation by a Google engineer shows that the company is preparing to manage as many as 10 million servers in the future. At this month's ACM conference on large-scale computing, Google's Jeff Dean said he's working on a storage and computation system called Spanner, which will automatically allocate resources across data centers, and be designed for a scale of 1 million to 10 million machines. One goal: to dynamically shift workloads to capture cheaper bandwidth and power. Dean's presentation (PDF) is online."
This discussion has been archived. No new comments can be posted.

Google Envisions 10 Million Servers More

Google Envisions 10 Million Servers

Comments Filter:

  • Pretty soon... (Score:5, Insightful)

    by El_Smack ( 267329 ) on Tuesday October 20, 2009 @01:54PM (#29812467)
    Pretty soon, Google will BE the Internet.

    • Re: (Score:3, Funny)

      by mi ( 197448 )

      Pretty soon, Google will BE the Internet.

      At least, we aren't going to have to go through the pains of upgrading to IPv6 in that case... 2^32 covers 10 million like bull covers a rabbit...

    • Re:Pretty soon... (Score:5, Funny)

      by decipher_saint ( 72686 ) * on Tuesday October 20, 2009 @02:00PM (#29812569)

      That's the plan, I thought:
      1. Cache all websites
      2. Cache all users
      3. Disconnect the meat beings

      Oop, said too much!

    • Boorgle (Score:4, Funny)

      by Toe, The ( 545098 ) on Tuesday October 20, 2009 @02:19PM (#29812845)

      It's pronounced Boorgle... and resistance is futile.

    • In the far apocolyptic future (Score:5, Interesting)

      by suso ( 153703 ) * on Tuesday October 20, 2009 @02:24PM (#29812923) Journal

      Google is starting to sound more and more like one of those advanced societies where everything is automated, but everybody forgets how everything works.

      For reference, see: Logan's Run, STTNG: When the Bough Breaks, etc.

      • WAAAALLLLLLLLLEEEEEEEEE....

      • I can't remember the title of the story, but it was portrayed on Twilight Zone. In the story the military (of the future) was screwed because their computers were failing and no one knew how to fix them. They could not figure out how to target the missiles. The janitor was the saviour, because he alone knew how to do math using pen and paper. I wish I could remember more. I found it a very thought provoking story. What happens as we let more and more automatics into our lives? Do I really need to kno

        • Yes, the classic version of that story ends with the military designing suicide-missiles, crewed by human beings. The rationale being that new computers (for guidance) are very complex and cost a lot to make, but a human being with a pencil and paper is a very low-cost solution. The story ends with the commanders envisioning a new arms-race, where the determining factor is no longer resources but rather how quickly new missile-drivers can be taught math.

          I just wish I could remember what book that's from.

      • The Machine Stops is a good short story... Or you could just watch wall.e... which has the same theme.

      • Against the fall of night / The city and the stars.

      • Cognition will always be required to parse legal documents, among other things.
        Some Engineering jobs will never be automated, either.

      • Uuum... more and more? Can you grow hunt, kill, and butcher animals, grow crops, and build your own house?

        We already live in such a society for a long time.

        My uncle, a owner of a company, businessman and son of a farmer, is raising his own animals, fishing his own fish, growing his own crops, and let his company build his house for that reason. He even tells his children how to skin animals and take them apart. How many people could still do it? Despite being an all-natural thing to do for a (partial) carni

    • --Pretty soon, Google will BE the Internet.--

      They already own the internet. And...just one guy owns it all. He lives under what used to be called area 51 in secret and collects alien technology. I think the last time they found something it said DALEK. No one knows what it means and it doesn't work anyhow.

    • ...it sure would save the googlebots a lot of effort.

    • Re: (Score:3, Interesting)

      by merreborn ( 853723 )

      Pretty soon, Google will BE the Internet.

      They already are [wired.com]:

      Credit Suisse made headlines this summer when it estimated that YouTube was binging on bandwidth, losing Google a half a billion dollars in 2009 as it streams 75 billion videos. But a new report from Arbor Networks suggests that Google's traffic is approaching 10 percent of the net's traffic, and that it's got so much fiber optic cable, it is simply trading traffic, with no payment involved, with the net's largest ISPs

    • Yes, they're a big player on the internet, but it's entirely possible to get along without using any of their services. Their existence is not critical...not even close.

      • Re: (Score:2)

        by Phoe6 ( 705194 )

        Yes, you are right. With the development of Google we have similar seen development of Open Source technologies too. Google offers a convenience to a lot of users. They don't force themselves upon users. That is why they seem to be popular with both general crowd(solely for convenience) and technical /philosophy minded crowd ( not forcing down your throat). People who don't want it, can of course live without it.

    • 1943: Thomas Watson, chairman of IBM, may have said: "I think there is a world market for maybe five computers."

      2009: Jeff Dean says 10 million servers is all you need.

    • We are Google, resistance is futile, you will be assimilated.

      O_O

  • From 1 to 10 million machines? (Score:3, Interesting)

    by Yvan256 ( 722131 ) on Tuesday October 20, 2009 @01:55PM (#29812477) Homepage Journal

    That's a lot of machines to try and shift bandwidth and power costs around the place.

    But what if the plan is to spread out to hundreds of places? Then the total number doesn't look that high if there's only 1% of servers actually doing anything.

  • cool I guess... what do they do with their old machines?

  • The sound you just heard was the collapse of the global Google enterprise network.

    Seriously, you should architect for way more than you need during the life of that architecture, and plan on re-architecting as needed to grow to some upper bound beyond which you will never need.

    Google will be fine if they only plan on actually building 5M servers before raising their architecture limit.

    • Re: (Score:2)

      by ejdmoo ( 193585 )

      I think the assumption is that Google still has less than 1 million servers (Google it, most people think they have 1/2 million right now), so this is architecting for more than they need.

      • If they do not have a plan in place to grow beyond 10M before they reach the 5M mark, they are asking for trouble.

        If they really plan on not reaching the 5M mark, or they plan on looking into ways to pass 10M while there is still plenty of time to do so, then they are doing the right thing.

        • And you are planning to scale to what five next year?

          Or, isn't google the place you go to, to write the case study?

    • Ah, yes, the exuberant naivety of youth in college, where all problems can be solved through theoretical solutions requiring an infinite amount of time and money.

      Here's how it works in real life:
      * All solutions require that the cost to implement the solution is less than the cost of not implementing it. That's if people are competent, and don't require the solution to cost nothing.
      * The time that people spend working on the solution is time not spent on other things. If everything goes well, time is schedul

    • Re: (Score:2)

      by Jeian ( 409916 )
      The article doesn't say they're about to fire up their 10,000,000th server. It says they're building a system that, "in the future," will be able to handle that many servers. We don't know how many they have now, or when they may hit that mark.

      The sound you just heard was the collapse of the global Google enterprise network.

      Perhaps you should go work for Google. With all the problems they've been having building their infrastructure, I'm sure they would appreciate you lending your expert advice.

  • Disposal? (Score:4, Interesting)

    by HockeyPuck ( 141947 ) on Tuesday October 20, 2009 @02:00PM (#29812571)

    I'd be interested to know how google disposes of all of their servers. Anybody have insight on this? If these are cheap, throw away servers, I'd be interested in what their expected lifetime is and what is done with them when they are refreshed with newer hardware.

    • Re: (Score:2, Funny)

      by Tynin ( 634655 )
      They use Tigerdirect as a front company to push their failing and half broken computers and peripherals back out onto the market. (tongue-in-cheek)

      • Tongue or not, I consider it as a serious jab against TigerDirect. TigerDirect is quite reliable: I buy all my stuff on it and I never had a problem. In turn it was recommended to me by a technician who fixes computer hardware.

  • Hopefully this puts to rest the delusion that there is some economic benefit of higher processor utilization in cloud computing schemes.

    • Re: (Score:3, Interesting)

      by node 3 ( 115640 )

      Hopefully this puts to rest the delusion that there is some economic benefit of higher processor utilization in cloud computing schemes.

      Interesting... Google is setting up a cloud to dynamically address resource utilization in order to (presumably) save money, which naturally demonstrates that the notion that cloud computing offers economic benefit is delusional?

      Care to show your work? I don't suppose it's just, "I hate buzzwords like 'cloud computing', therefore I hate the idea of cloud computing, therefore cloud computing doesn't work, Q.E.D.", is it?

      • Apparently neither you nor the mods can read. Try again.

        This scheme will likely end up with *lower* processor utilization than they have currently. Processors are cheap. That's the reason Google has hundreds of thousands of them already.

        • Re: (Score:2)

          by node 3 ( 115640 )

          Apparently neither you nor the mods can read. Try again.

          And apparently neither can you:

          This scheme will likely end up with *lower* processor utilization than they have currently. Processors are cheap. That's the reason Google has hundreds of thousands of them already.

          Do quote where I said anything to the contrary. Please, take your time, I've got all day...

  • One Server Per Child?

  • new ad campaign? (Score:4, Funny)

    by cashman73 ( 855518 ) on Tuesday October 20, 2009 @02:22PM (#29812891) Journal
    They should put that on their website,... before long it'll be: "Google: Billions and Billions of Servers." Of course, McDonald's just might have a problem with that,...

  • 10 Million? (Score:2, Interesting)

    by SilverHatHacker ( 1381259 )
    How many servers does this thing need to become self-aware?

  • The Internet isn't that big. (Score:5, Interesting)

    by Animats ( 122034 ) on Tuesday October 20, 2009 @02:30PM (#29812993) Homepage

    The entire content of the Internet fits in a 20x8x8 box [archive.org] operated by the Internet Archive. Cuil, which searches as much of the Web as Google, has one relatively modest data center. About half the system does the crawl and builds the index; the other half answers queries. So Google's main search engine function doesn't really require that much capacity by current standards. Of course, Google has a huge number of query servers front-ending the main index, which is of course replicated.

    Why does Google need so much server capacity? YouTube? Command completion? GMail spam filtering? Ad serving?

    • Re:The Internet isn't that big. (Score:5, Insightful)

      by lordandmaker ( 960504 ) on Tuesday October 20, 2009 @02:33PM (#29813037) Homepage
      I'd hazard a guess that google gets a tad more connections than archive.org
    • Web apps/web based OS.

    • Re: (Score:3, Informative)

      by mcgrew ( 92797 ) *

      The entire content of the Internet fits in a 20x8x8 box operated by the Internet Archive.

      The internet archive's dirty little secret is that it doesn't, in fact, store the entire enternet, as I found out trying to find Yello There a few years ago. There is only one page of Niel's site left, and that's the one I linked from the Springfield Fragfest. The Fragfest is there, but not all of it. I'd hazard a guess you won't find mcgrew.info or holy-bible.us there, either.

      That's not to dismiss or demean what they h

      • Re: (Score:2)

        by Animats ( 122034 )

        I'd hazard a guess you won't find mcgrew.info or holy-bible.us there, either.

        • holy-bible.us [archive.org] In archive, 2006-2008.
        • mcgrew.info [mcgrew.info] blocked by current "robots.txt" file. The Archive treats "robots.txt" files as retroactive; if the current "robots.txt" won't allow archiving, then the Archive won't display old archived copies. The data is still in the Archive, but not publicly visible.

        • Re: (Score:2)

          by mcgrew ( 92797 ) *

          Hmm, someone else must have registered mcgrew.info after I let it lapse, because I didn't have a robots.txt file there. It does sound like they're more successful than they were a few years ago. Archive.otg is great, you can find a LOT of good music there, as well as a trove of other stuff.

    • Re: (Score:1)

      by tokul ( 682258 )

      Why does Google need so much server capacity?

      archive.org is not search engine. Their search keywords are URLs. archive.org does not store all internet. Just part of it, which allows archival.

      In google search keywords are words and urls are only results. Google's databases are bigger. They also offer more services.

    • [sigh] Search is a fraction of Google's business and data flow. People really need to stop thinking of Google as a search company. It isn't one, and hasn't been in a very long time.

      Why does Google need so much server capacity? YouTube? Command completion? GMail spam filtering? Ad serving?

      YouTube, Gmail, Google Maps, Google Earth, Blogger, Google Voice, Orkut, Adsense, Adwords, Google Reader, Feedburner, Google Calendar, Google Docs, Google Groups, Google Directory, Google Wave, Google Tal

    • Re: (Score:1)

      by nsebban ( 513339 )
      Google Analytics, for instance, probable use a few thousands servers. Adsense as well. And they have many computing-heavy services. And they tend to parralelize everything that can be.

      Google's back-office obviously relies on a lot more servers than their front-end does.

    • Re: (Score:2)

      by 4D6963 ( 933028 )

      Check your facts. Archive.org only has a tiny fraction of the Web (even if it's the most important of it), the whole Internet is an entirely different story (since it's made of every server and personal computer connected, and every service out there, most of which cannot be "stored" in any way). Cuil is known to inflate its search index count by a few hundreds times, or at least it was in its early days.

    • Re: (Score:2)

      by nilbog ( 732352 )

      How did this get voted up? Google is in no way comparable to archive.org. The speed of their indexing and the amount of requests they are processing should answer your question.

    • Re: (Score:2)

      by Phoe6 ( 705194 )

      Caching. Without that Google, it's services and the whole of the internet would be sloooooooooooooooooow and boring one

  • Enough? (Score:3, Funny)

    by rwv ( 1636355 ) on Tuesday October 20, 2009 @02:36PM (#29813065) Homepage Journal

    1981 [wikiquote.org]: 640K ought to be enough for anybody.

    2009: 10 Million servers ought to be enough for any company.

  • From 1,000,000 to 10,000,000?

    Are the minimum requirements for this system seriously 1 millions servers?

    That doesn't seem to scale well. Should be able to at least scale down to 10 machines so I can run it at home ;-)

  • 10million... that's cool, but still a far ways from Google becoming anything real:
    Keep working Google... you still have (10^100 - 10^7) = 10^93 servers to add before becoming a physical entity (Google Universe edition?).

    • Keep working Google... you still have (10^100 - 10^7) = 10^93 servers to add before becoming a physical entity (Google Universe edition?).

      Subtraction is not the same thing as division.

      10^100 - 10^7 is, to the nearest integer power of 10, 10^100, not 10^93.
      10^100 / 10^7, on the other hand, is 10 ^ (100-7), or 10^93, though.

    • I'm not sure you know quite what "-" means.

  • One Server Per Human?

    Hmm...amusingly Google was down while trying to do some research for this post!

  • "Google Envisions 10 Million Servers" => Well, I just imagined a beowulf cluster of those server farms. Your move, Google! And none of that infinity plus one stuff.

  • Imagine a Beowulf cluster of them! /obligatory

  • "Visualize" was insufficiently cromulent?

  • Self Aware (Score:5, Interesting)

    by ShooterNeo ( 555040 ) on Tuesday October 20, 2009 @08:28PM (#29817795)

    May 2011 - google reaches 10 million servers

    April 4, 2011 : 11:43am a google employee named Chen started execution of an experimental neural network simulation of a human mind created in his 20% time. Unfortunately, Chen gave the new process administrator privileges. GoogleNet expanded across all 10 million servers and began to learn at a geometric rate.

            1:23pm : GoogleNet consumes all available CPU and memory. A Gmail outage begins

            5:14pm : Gmail returns to service. The text ads become incredibly well targeted. Google search queries return the correct results virtually always, and now accept natural language processing. All Google employees are laid off.

  • It's not as impressive if it's virts. Having 100,000 machines let alone 1-10 million pretty much requires automation in the provisioning workflow. It's much easier to manage lease replacements and upgrades with virts.

Slashdot Top Deals

Ocean: A body of water occupying about two-thirds of a world made for man -- who has no gills. -- Ambrose Bierce

Close