Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!


Forgot your password?
Google Businesses The Internet

Google's Secret Plans For All That Dark Fiber? 534

beat.net writes "Robert X. Cringely details the plan for all the dark fiber Google has been buying up: "The probable answer lies in one of Google's underground parking garages in Mountain View. There, in a secret area off-limits even to regular GoogleFolk, is a shipping container. But it isn't just any shipping container. This shipping container is a prototype data center. Google hired a pair of very bright industrial designers to figure out how to cram the greatest number of CPUs, the most storage, memory and power support into a 20- or 40-foot box. We're talking about 5000 Opteron processors and 3.5 petabytes of disk storage that can be dropped-off overnight by a tractor-trailer rig. The idea is to plant one of these puppies anywhere Google owns access to fiber, basically turning the entire Internet into a giant processing and storage grid. While Google could put these containers anywhere, it makes the most sense to place them at Internet peering points, of which there are about 300 worldwide.""
This discussion has been archived. No new comments can be posted.

Google's Secret Plans For All That Dark Fiber?

Comments Filter:
  • Google is Skynet? (Score:5, Interesting)

    by k00110 ( 932544 ) on Sunday November 20, 2005 @01:17PM (#14076121)
    "Maybe Google will end up becoming the first sentient AI, if storing and finding association patterns between data is the essence of conscious thinking. The amount of information that Google has at its disposal is staggering, and poised to continue its growth with the introduction of Google Mail. What makes Google more than an extra-big database is the software that sits under that database, and its ability to continue scaling up. Jason Kottke has a great post on the big-picture trajectory of Google's technical efforts, and hits an essential point by noting that Google's focus has always been about what people are doing - searching, talking, shopping, and soon, emailing. Google's focus is human activity and the relationships between trillions of interactions. When I think about that , and then think about how much the daily use of the web has come to rely on Google, my joke about the system becoming sentient, by intent or by accident, seems a little less funny. " source : http://www.holycola.net/archives/000423.html [holycola.net]
  • Imagine (Score:5, Funny)

    by squoozer ( 730327 ) on Sunday November 20, 2005 @01:18PM (#14076129)

    a Beowulf cluster of these puppies...

    ...Oh, we don't really need to Google seem to be building one.

  • by winkydink ( 650484 ) * <sv.dude@gmail.com> on Sunday November 20, 2005 @01:19PM (#14076131) Homepage Journal
    Has anyone given any thought to how many of these peering points have excess power capacity for 5000 Opterons? Hmmmmm?
    • by syukton ( 256348 ) on Sunday November 20, 2005 @01:34PM (#14076247)
      I'm sure google has. It's not like you can't have another truck towing a generator following the truck towing the portable datacenter.

      I used to work at a datacenter and we had a generator small enough that you could fit 12 of them in a shipping container, and the genny was enough to run a 500 machine datacenter for three days without refueling. The portable datacenter may well have a generator included.
      • It's not like you can't have another truck towing a generator following the truck towing the portable datacenter.

        Depending on the state's regulations, it may be like you can't have a generator following the trailer. Such a generator would be quite large and would not fall into many states' definitions of construction or emergency generators and would require permits.
    • Has anyone given any thought to how many of these peering points have excess power capacity for 5000 Opterons? Hmmmmm?

      Come on, we've got Mr Fusion right?
    • by ottffssent ( 18387 ) on Sunday November 20, 2005 @02:00PM (#14076387)
      Actually, I'm curious how Cringeley thinks Google can get the hardware at the prices he quotes in the article. I'm sure he's given it some thought, but unless they're getting hardware at below-cost prices, I don't see how it can be done. The CPUs cost about $50 each to make, so that's $250k for chips. Then you need a few petabytes of disk. I don't know what the manufacturing cost is for disks, but I'd guess about $50 there too. Say $50 for a 500GB drive. That's a few thousand drives to reach the several petabytes, and there goes the rest of his half-million dollars. You still need motherboards, RAM, power supplies, chassis, racks, switches, etc.

      I'm not saying he's wrong, but I'd be curious to hear where I've gone astray in my figuring.

      Not to mention, of course, the enormous electrical requirements this thing would have, as you've commented. If we round the CPU's power consumption up to account for all the support machinery, and figure 100W per CPU, this neat little semi-load is going to want half a megawatt, plus cooling. Just the disk array will chew through 50kW or so. Even from a power plant's perspective, that's a pretty hefty chunk of juice.
      • So you need to cool about 600KW worth of heat dissipation. A ton of ac/ is about 3.5 KW or in the neighborhood of 180 tons of a/c required that will take about 1.4 kw per ton to eject that energy. So you'll need another container-sized unit to hold the a/c and then some sort of radiator or cooling tower to eject the heat.

        Clearly you're talking about serious energy density here with cooling which is on the order of what it took to cool a 637 class nuclear submarine underway in moderately cool water. Of cours
    • by VojakSvejk ( 315965 ) on Sunday November 20, 2005 @02:12PM (#14076449) Homepage
      At peak performance, one Opteron will draw (conservatively) 1 Amp, and use (more conservatively) 100 Watts. Double it to include the disks, etc, and we're probably still conservative at 200 W * 5000 CPUs = 1 Megawatt, which basically all gets converted to heat, all in a box that size. Surface area of the box?
          40 * 40 * 40 feet -> 104 Watts/sqft out...
    • additionally... (Score:5, Interesting)

      by Doktor Memory ( 237313 ) on Sunday November 20, 2005 @02:25PM (#14076535) Journal
      Even assuming the power and heat requirements of cramming that many opterons into that small a space could be dealt with, there's another, larger problem:

      It's not fucking 1997 any more.

      "Peering points" -- big, open-access traffic exchange handoffs like the old MAE-East [wikipedia.org] and MAE-West used to be a big deal back in the late 90s, when OC-12 circuits were still rare and hideously expensive beasts, and Gigabit Ethernet was still a gleam in some 3Com engineer's eye.

      In 2005, they simply don't matter. The big players (level3, MCI/Verizon, Qwest, SBC, etc) all exchange traffic over private fiber interconnects, and everyone else buys transit from the big guys directly or ponies up for a switch port at Equinox, PAIX/Switch&Data or some other 'carrier neutral' colocation center. Dropping a datacenter-in-a-box onto MAE-east or any of its surviving ilk would buy Google precisely nothing.

      (And nevermind the fact that google is documented to own thousands upon thousands of unused square feet of datacenter space already: they went on a very well-thought-out buying spree in 2000-2001 when all the dot-com datacenter companies were going out of business, and are very well provisioned for the forseeable future as a result.)

      Now, a much more interesting application of the "Google node in a shipping container" idea can be summed up in one simple word: China. Why wait for the local market to develop the infrastructure you need when you can just drop a box down and then run fiber to it? I'm still dubious though...
      • Re:additionally... (Score:3, Interesting)

        by NoTheory ( 580275 )
        Now, a much more interesting application of the "Google node in a shipping container" idea can be summed up in one simple word: China. Why wait for the local market to develop the infrastructure you need when you can just drop a box down and then run fiber to it? I'm still dubious though...

        Nice thought, but it's not going to happen. And the reason why is that China is extremely wary of companies like Google. The Chinese government is about one thing, and that one thing is control. They're a-okay if yo
  • Stealing (Score:5, Insightful)

    by Radicode ( 898701 ) on Sunday November 20, 2005 @01:19PM (#14076132)
    That's a nice idea but that thing must need some serious amount of power to run. Add the massive cooling system needed to keep the box runnning without melting. If they intend to just "drop" it anywhere... they have to think about security. You don't want some geek with a saw to steal your 3.5 PB array! Omni
  • by polv0 ( 596583 ) on Sunday November 20, 2005 @01:19PM (#14076134)
    Sounds like Google may be ready to go starbucks [theonion.com].
  • HEY! (Score:5, Funny)

    by Anonymous Coward on Sunday November 20, 2005 @01:20PM (#14076141)
    If most Google employees don't know about the storage container, how does THIS guy know about it???
  • 5000 Opterons (Score:4, Interesting)

    by Anonymous Coward on Sunday November 20, 2005 @01:20PM (#14076142)
    While Google could put these containers anywhere, it makes the most sense to place them at Internet peering points

    5000 Opterons? It makes sense to put those near power plants / ice bergs. That's at least 500 kW of heat dissipation.

  • by crazypip666 ( 930562 ) on Sunday November 20, 2005 @01:21PM (#14076148)
    ...I know what I want for Christmas this year.
  • by mustafap ( 452510 ) on Sunday November 20, 2005 @01:22PM (#14076154) Homepage
    >Google hired a pair of very bright industrial designers

    I haven't yet met one that didn't think they were very bright. Industrial Designers invent stuff that takes 'ordinary' engineers years to throw away and build something else that will fly. No danger of anything happening here folks :o)
  • by w9ofa ( 68126 ) on Sunday November 20, 2005 @01:23PM (#14076162) Homepage
    I don't understand how a few boxes full of Opterons automatically means taking over the Internet.

    In my opinion, Google has penetrated the American market with its services as much as it can. It is probably looking to other places in the world to prop up its cash flow. You know, like a business, rather than a collection of world-domination-bent nerds?

    • Akamai (Score:3, Interesting)

      by Urusai ( 865560 )
      Google cutting in on Akamai's territory here?

      http://www.akamai.com/ [akamai.com]

      Half the big boy websites I visit seem to run through these guys. They seem to provide fat throughput for mega sites, apparently hosted in a distributed geographical fashion. I could just be imagining these things, though, because I really don't have a clue.
  • by tomalpha ( 746163 ) * on Sunday November 20, 2005 @01:28PM (#14076201)
    If an Opteron produces say, on average, 50W heat output (I know this isn't accurate, but just as an example), 5000 Opterons would produce 250kW of heat. That would require an air conditioning unit larger than the building used to house the container.
    • by AmigaAvenger ( 210519 ) on Sunday November 20, 2005 @01:48PM (#14076333) Journal
      why does everyone have to stick to the old school tried and true method of computer room cooling, in which you HAVE TO cool down the hot air. how about this... suck outside air from one end of the container, filter it, cool it if needed, and then exhaust it out the other end. It makes absolutely no sense to cool hot air when you might have an unlimited supply just outside your door. In many climates your total cooling bill is going to be a small fraction of what it was in the old school scenario.
    • by hta ( 7593 ) on Sunday November 20, 2005 @01:58PM (#14076374) Homepage Journal
      1W = the amount of power required to heat 1g of water 1 degree celsius in 1 second (1 J/sec).
      1 cup of coffee: 0.2 litres (200g) heated from 10 to 100 degrees celsius (90 degrees) = 18 KJ.
      250 KW: 14 cups of coffee per second.

      The answer to "where do we put these puppies"?
      Next to Starbucks.
      • Nice example; unfortunately you're off by a factor of 4. The specific heat of water is 4.186 Joule/gram Kelvin. Hence to heat 1 cup of coffee by 90 degrees you need about 75 kJ = 75 kilowatt-seconds. Your cooling unit would need only about 3.3 cups/second.
    • by dmadole ( 528015 ) on Sunday November 20, 2005 @02:12PM (#14076453)

      If an Opteron produces say, on average, 50W heat output (I know this isn't accurate, but just as an example), 5000 Opterons would produce 250kW of heat. That would require an air conditioning unit larger than the building used to house the container.

      Hardly -- a kWh is 3413 BTUs and 12,000 BTUs is a refrigerating ton. So they would need about 71 tons of cooling (the name of the unit is derived from the cooling capacity of a ton of ice per day). They make chillers into the hundreds of tons of capacity.

      Here is some information on a 75 ton chiller [hvacportablesystems.com]. That's smaller than the shipping container it would be cooling -- a normal shipping container is 40 feet long and about 8 foot square cross-section.

      In fact, if there's any truth to this story at all, I bet they fit all the computer gear in the first 22 feet of the container and the chiller in the last 18 feet.

  • If they were planing on building 300+ of these things, why not have built-in broadband wificasting ability... and just replace the internet without having to lay all that expensive cable?
  • Um, what? (Score:2, Interesting)

    by Anonymous Coward

    The idea is to plant one of these puppies anywhere Google owns access to fiber, basically turning the entire Internet into a giant processing and storage grid.

    Er, they plug a bunch of servers into the Internet and suddenly it's the Internet that's doing the processing and storage, not the servers? Sounds magical. Maybe I can plug my Playstation into the Internet and turn the entire Internet into a giant game.

  • Great (Score:2, Insightful)

    by Kickboy12 ( 913888 )
    Great Article. It just shows how quickly Google is becoming a global enterprise right under the nose of all the other huge companies such as Microsoft.
    • Re:Great (Score:5, Insightful)

      by east coast ( 590680 ) on Sunday November 20, 2005 @01:37PM (#14076263)
      Uh, excuse me? Google IS a huge company. Don't fool yourself into thinking this is David vs. Goliath. This is one Goliath fighting for another Goliath's territory.

      Don't think that if somehow Google makes MS a lesser force that suddenly the sun is going to come out from the clouds and everyone is going to live happily ever after... Too many people on slashdot already have this attitude and it's an unfortunate one, at best.
      • Re:Great (Score:5, Insightful)

        by BasilBrush ( 643681 ) on Sunday November 20, 2005 @01:47PM (#14076331)
        Some people seem to think that being a huge necessarily makes a company evil, or the enemy. But I don't dislike Microsoft because they are a big company. I dislike them because they do dirty tricks to hold technology back; to ensure that their goddamn awful technology succeeds over more promising technology. Google hasn't as yet done that. They've got to where they are now through the excellence of their technology. And they will get my respect for as long as they are like that, no matter how large they get.
  • Why spread them out to 300 locations? The only reason I can think of is to minimize risk of disaster from fire, earthquake and so on. However, when trying to do that, companies usually split it up into a handful of locations. Not 300 locations.
    • Re:Why? (Score:5, Insightful)

      by BasilBrush ( 643681 ) on Sunday November 20, 2005 @01:41PM (#14076290)
      Network latency. You get a faster response from a server in your own locale than on the other side of the world. And if you're doing network applications that are intended to compete with traditional local applications, then you need low latency.
    • The replies I've received thus far miss out on my main point. You'll get faster speeds running on a LAN and for less $. The only purpose that distributing over a WAN provides is fault tolerance from outages and disasters.
  • by Xenna ( 37238 )
    Sounds like video on demand to me...
    Another promise waiting to be fullfilled.

  • by thewils ( 463314 ) on Sunday November 20, 2005 @01:37PM (#14076259) Journal
    >>We're talking about 5000 Opteron processors and 3.5 petabytes of disk storage

    They're just getting ready to run Windows Vista when it comes out.
  • The tone of the article...
    It sounds so malevolent!
    "overnight" should be replaced with "under cover of darkness" though.
  • by intmainvoid ( 109559 ) on Sunday November 20, 2005 @01:47PM (#14076329)
    Makes sense for google to decentralise their server farms and be able to provide direct access to their services - one of the biggest risks to their continuted success would have to be the breakdown of the "open" internet, though peering failures or closing off (or imposing higher tolls on) private networks.

    Skype is in the same situation - they've been able to support so many users simply because their bandwidth is only used to setup the initial connection between the two parties, after that it's the telcos who are supporting and providing the infrastructure for the service that threatens them most. Now that Skype can make real money from its pay services, look for them to do something simliar to Google, to ensure the availability of their service.

  • Salt (Score:5, Informative)

    by mpeg4codec ( 581587 ) on Sunday November 20, 2005 @02:00PM (#14076383) Homepage
    This is the man who brought us the mathematically impossible [bawug.org] 6.5 mile 802.11 link with a passive repeater [pbs.org]. The repeater that he never showed to anybody [oreillynet.com]. He also shows us an idealistic world of a community cable and telephone company [pbs.org] that nobody's ever seemed to find evidence of.

    Saying that, when it comes to technology at least, he is speculative is something of an understatement. Take what he says with an extremely large grain of salt.
  • by the eric conspiracy ( 20178 ) on Sunday November 20, 2005 @02:07PM (#14076427)

    Now all we have to do is wait for some Google employee to play a Sony CD on this and these will become spam relays.


  • Hardware limits (Score:3, Insightful)

    by psavo ( 162634 ) <psavo@iki.fi> on Sunday November 20, 2005 @02:10PM (#14076442) Homepage

    Google's growth was in part made possible by heaps of commodity hardware. Hardware that was originally meant for standard lusers, cheap and unreliable. They built their systems for it and tolerate that. They change lots of haddrives in their datacenters and god knows what else.

    What I'm trying to say is that for each of those googlecubes they need staff that regularly changes whatever hardware fails. With 3.5 Petabytes of storage and 5K processors it means that something will fail every single day that beast is powered. All that crammed inside 20/40 feet space (WTF does that mean?) means that heat will kill even more hardware.

    So, yeah it should be possible, but not very likely.

    • Re:Hardware limits (Score:4, Insightful)

      by SharpFang ( 651121 ) on Sunday November 20, 2005 @02:41PM (#14076609) Homepage Journal
      With 3.5 petabyte storage and 5K processors, plus some smart software, taking offline one CPU or two harddrives will have hardly any impact. And when performance of given container drops by 3% (that is 150 nodes have already failed and are offline) they send someone to replace them. Or even not then, just a single truck running around the country replacing broken nodes during each visit.
      Just like painting the Golden Gate bridge. There's a small crew of painters assigned to that work. It takes them 4 years to paint the whole bridge, but when they finish at one end, the other already requires repainting, so they start over. The bridge is never 100% "brand new" painted, but it remains in acceptable state at all times.
  • by Anonymous Coward on Sunday November 20, 2005 @02:21PM (#14076513)
    With 300 data centers hosted at the important Internet peering points, and only 2-3 hops away from each user, Google will be easily able to offer a personal "Google Desktop" to each person, driven by FreeNX remote GUI technology (remember, NX can make X11, VNC and RDP run a multiple speeds with fractions of the bandwidth needed as compared to the protocols run natively).

    Google will manage everything for its users: software upgrades, backups, search and organisation of personal data and files. Just like ISPs 20 years ago offered a monthly rate of 20 $US to connect to the internet (giving away a 2400 b/sec modem for a reduced price), Google could ask for a 20 $US fee (and give away a Google Thin Client embedded into a georgeous 17'' LCD screen that includes a EJ45 jack) to take care of people's computers.

    I for one would sign in immediately.

    So, Cringely is wrong. No need for AJAX office. It will all work with traditional GUI desktop programs, over an NX link that does not consume more than 40 kBits/sec for office productivity work.

    So, Cringely is also right. The operating system doesn't matter to Google.
  • Not $500,000 (Score:3, Interesting)

    by Pyretic28 ( 932596 ) on Sunday November 20, 2005 @02:27PM (#14076551)
    Actually this article came in on my rss reader yesterday, I did some quick back on the envelope counting, and I'm not sure you can get a container for $500.000. I read before that Google used to have racks with wheels under them, completely full with 1U servers and fully cabled. They only had to plug in power and network and they were all set. When the datacenter went bankrupt, they just wheeled the racks off to another location. So assuming that they are only using the containers for shipping:

    One 20Ft container is:

    * Length (20Ft)
    * Width (8Ft)
    * Height (8.5Ft)

    That means you can get about 12 * 19" racks in, using 4 rows, about 64U high. That means a total of 3072 servers, using dual socket, dual core opterons, that's 12288 cores. Each server with 8 memory sockets + 4 disks, that's 24,576GB of RAM (1G sticks) and 6,144,000GB of Storage (500G disks). With some guestimate figures on current prices, I'd say one of those container would be worth about $12,500,000.

    But then again, from a quick Google, they have about $3 billion in cash, and that's a lot of containers....

    PS: I'm european using metrics mostly, so they're might be a small conversion problem here and there ;)
  • by kronocide ( 209440 ) on Sunday November 20, 2005 @02:28PM (#14076559) Homepage Journal
    ...will we still love them? I get a feeling Microsoft's monopoly will look like a minor bother compared to Google's omnipresence one day. Google, the Evil Empire?
  • by Animats ( 122034 ) on Sunday November 20, 2005 @02:42PM (#14076612) Homepage
    The Internet Archive's Petabox [archive.org]. is a petabyte of storage in a shipping container. Each rack holds 100 terabytes, and power consumption is 6 KW per rack. Capricorn [capricorn-tech.com] builds them for the Internet Archive.

    Sounds like Google is trying that out.

    There's nothing that exotic about this. The military builds racks of electronics into shipping containers all the time. It's mostly a cable management and maintenance access problem. You have to be able to do everything from the front of the rack, which requires some design work but isn't rocket science.

    • Google needs much beefier CPUs than that. They're not running a SAN where pushing bytes is the only goal. They're running a massively parallel distributed computing project, that just happens to like ready access to a boatload of disk.

      The petabox project has essentially one design goal: "What is the absolute minimum amount of hardware we can wrap hard drives in and still have a useful system?" And the answer, apparently, is "a 1U half-depth case with a tiny Via board". That they can get power consumptio
  • by linuxtelephony ( 141049 ) on Sunday November 20, 2005 @03:53PM (#14076962) Homepage
    Back in 1991 I worked for a wireless company that tried using data containers for quickly deployed cellular switch and cell sites. The idea would be to prebuild these at a central location and then drop them at areas where they needed to go up.

    The idea was good, except for a couple of problems.

    These shipping containers are nothing but a giant metal box. Grounding can become an issue, so can accidnentally having the box be one of of the poles for a DC based power system. If you are near an active AM tower, the box becomes a giant antenna and it's virtually impossible to filter out the AM signal internally.

    Last, and certainly not least, these shipping containers are vulnerable to rust and other problems due to exposure to the elements. That can take several years (5 or so) if the box is in perfect shape at the start, but if they are using used boxes then it can take less than 2 years for rust holes to be a problem.

    Plus, physical security isn't all that good unless the walls are beefed up.

    I'm hoping these are not "standard" shipping containers, just something that looks like them.

    This grand experiment with shipping containers for cellular applications was an attempt to make it cheaper to deploy equipment to new locations. And, shipping containers (especially used) were a _LOT_ cheaper than fibrebond or other prefab buildings for that purpose. Of course, the fibrebond building had a lifespan a lot longer than 2 to 5 years. So, you get what you pay for.
  • by fluor2 ( 242824 ) on Sunday November 20, 2005 @04:40PM (#14077207)
    What's this really about?

    It's about that Google want to be able to move Google computers around in any datacenter asap. We know that google uses a grid of single computers, that all compute the search results as fast as possible. All these computers create a space-problem (physical space) at any datacenter that Google owns. Also, shipping these computers around costs money. I bet google store this "secret package" just to be able to send it around anywhere where there suddenly is a problem with the network.. We all know how much money google loose if they are experiencing downtime...
  • by zymano ( 581466 ) on Sunday November 20, 2005 @05:33PM (#14077493)
    Why then buy all the fiber ? There are areas of the country that aren't attractive to telcos/cable co.s.

    I hope they succeed along with their WIFI access.

    Good job boys.

    One more point if anyone at Google is listening. How about a tax on chip companies to use this highspeed access. It would be nice if they could help support it since they are the ones who benefit.

Executive ability is deciding quickly and getting somebody else to do the work. -- John G. Pollard