Google's Secret Plans For All That Dark Fiber? 534
beat.net writes "Robert X. Cringely details the plan for all the dark fiber Google has been buying up: "The probable answer lies in one of Google's underground parking garages in Mountain View. There, in a secret area off-limits even to regular GoogleFolk, is a shipping container. But it isn't just any shipping container. This shipping container is a prototype data center. Google hired a pair of very bright industrial designers to figure out how to cram the greatest number of CPUs, the most storage, memory and power support into a 20- or 40-foot box. We're talking about 5000 Opteron processors and 3.5 petabytes of disk storage that can be dropped-off overnight by a tractor-trailer rig. The idea is to plant one of these puppies anywhere Google owns access to fiber, basically turning the entire Internet into a giant processing and storage grid. While Google could put these containers anywhere, it makes the most sense to place them at Internet peering points, of which there are about 300 worldwide.""
Google is Skynet? (Score:5, Interesting)
Re:Google is Skynet? (Score:5, Interesting)
The Alternative Re:Google is Skynet? (Score:4, Insightful)
Re:The Alternative Re:Google is Skynet? (Score:3, Informative)
Re:The Alternative Re:Google is Skynet? (Score:4, Insightful)
Re:The Alternative Re:Google is Skynet? (Score:4, Interesting)
Great distances of rural farmland .
http://www.locustworld.com/ [locustworld.com] is a awesome idea, and they have done some
great things, but there are many places off the coast where distances
between cities is greater than the range of WiFi unless u use the ballon trick .
But long shots in the midwest are going to have to route via
conventional telecom unless we setup telecommunications WiFi ballons .
www.21stcenturyairships.com
Someone has to pay for them
high enough to avoid all wind .
Ex-MislTech
Re:The Alternative Re:Google is Skynet? (Score:3, Funny)
Went to an Oracle "development shop" (or so they said), where they called me in because they thought they had an Oracle tuning issue. Turns out that their entire office of 65 people were plugged into a series of daisy-chained LinkSys 10MB hubs, and they were all accessing this Oracle DB with some rather high traffic requests.
I went in and did some investigation, and it was the first time I've EVER seen the actual network connection time out like that.
I raised this to th
Re:Google is Skynet? (Score:3, Insightful)
Google at least APPEARS to be trying to improve technology FOR its customers.
Whether they will be successful at that, and how they will use that in the future is an open question.
Re:Google is Skynet? (Score:5, Funny)
Re:Google is Skynet? (Score:5, Funny)
I hadn't heard of Crzmblski's Limit, so naturally I went to Google to find out what it was
Re:Google is Skynet? (Score:4, Funny)
The limit's definition is posted on everything2 (Score:5, Funny)
Re:Is this a joke? (Score:4, Informative)
Re:Is this a joke? (Score:3, Informative)
Here's a link to the entire text of the novel, for anyone that's interested:
The Great Time Machine Hoax [webscription.net]
Crmblznski's Limit -- Definition (Score:4, Informative)
Crmblznski's Limit, sometimes spelled Crizmblski's Limit, has its origins in Keith Laumer's novel "The Great Time Machine Hoax" [1].
The basic theorem is that there is a finite limit to the complexity of any given machine, which specifically precludes the operation of "a machine with sufficiently extensive memory banks, adequately cross-connected and supplied with a vast store of data, [that by its very essence] would be capable of performing prodigious intellectual feats simply by discovering and exploring relationships among apparently unrelated facts." The Limit is an irrational number, much like Pi, in that the total complexity of machine is wholy dependent upon both hardware and software designs.
Re:Google is Skynet? So is Wikipedia now Google? (Score:4, Interesting)
Re:Google is Skynet? So is Wikipedia now Google? (Score:3, Interesting)
In addition, I've noticed that any time I don't find much help on Google, invariably I can easily rephrase what I want as a series of very general questions, and then feeding a few of
Re:So is Wikipedia now Google? Nope, too slow... (Score:3, Insightful)
Grown up with TV and playstation our attention span has degraded to..Oh, something beeped, hold'on a sec.
Re:Google is Skynet? (Score:3, Funny)
Why not? (Score:3, Interesting)
Is doing what we're told bad altogether or are people going to open their eyes?
Re:Why not? (Score:3, Insightful)
Heh, good one. I guess it depends on how you define Christianity -- are you referring to the teachings of Mr. Christ, or to the actual beliefs and practices of contemporary people who call themselves Christians? Because the two are widely separated these days... (not that that's anything new [wikipedia.org] of course)
Re:Google is Skynet? (Score:4, Insightful)
all the buzzwords like "dark fiber" "secret underground garages" and "dropped off overnight" and even "peering points" make this sound like a recipe for a very red future.
Re:Google is Skynet? (Score:4, Funny)
How about a nice game of chess?
Re:Google is TV ?? (Score:3, Interesting)
Maybe Google will end up becoming the first sentient AI,...
Not AI, try TV and broadcasting on demand. What else could you do with that much storage, CPU and memory?
My guess, and it is only a guess, GoogleTV is geting a lot bigger and going to carry some 500,000 to 1,000,000 full length movies and shows or something. And it will be so kewl for us to watch what we want and not what some clown wants us to watch. I suspect it will change ratings too as your not stuck picking from the least boring shows be
Re:Google is Skynet? (Score:3, Funny)
More importantly, who on Slashdot will be the first to welcome it as their new overlord?
Re:Google is Skynet? (Score:3, Funny)
Does the name Pavlov ring a bell?
Imagine (Score:5, Funny)
a Beowulf cluster of these puppies...
...Oh, we don't really need to Google seem to be building one.
Re:Imagine (Score:2)
Now which one, 80 or 443?
Obviously... (Score:5, Interesting)
...the puppy's on fire.
How are they going to cool these things?
Re:Obviously... (Score:5, Funny)
Nice work of fiction (Score:5, Insightful)
Re:Nice work of fiction (Score:5, Informative)
I used to work at a datacenter and we had a generator small enough that you could fit 12 of them in a shipping container, and the genny was enough to run a 500 machine datacenter for three days without refueling. The portable datacenter may well have a generator included.
Re:Nice work of fiction (Score:2)
Depending on the state's regulations, it may be like you can't have a generator following the trailer. Such a generator would be quite large and would not fall into many states' definitions of construction or emergency generators and would require permits.
Re:Nice work of fiction (Score:2)
</sarcasm>
Re:Nice work of fiction (Score:2)
Come on, we've got Mr Fusion right?
Re:Nice work of fiction (Score:5, Interesting)
I'm not saying he's wrong, but I'd be curious to hear where I've gone astray in my figuring.
Not to mention, of course, the enormous electrical requirements this thing would have, as you've commented. If we round the CPU's power consumption up to account for all the support machinery, and figure 100W per CPU, this neat little semi-load is going to want half a megawatt, plus cooling. Just the disk array will chew through 50kW or so. Even from a power plant's perspective, that's a pretty hefty chunk of juice.
Not all that much of a heat load to deal with (Score:3, Informative)
Clearly you're talking about serious energy density here with cooling which is on the order of what it took to cool a 637 class nuclear submarine underway in moderately cool water. Of cours
Re:Lots of heat, lots of power (Score:5, Informative)
The sort of temperature-differential energy recovery you speak of is technically possible but isn't efficient enough to substantially reduce the cluster's power requirements, and thus its need to vent waste heat.
Re:Lots of heat, lots of power (Score:3, Informative)
That doesn't happen yet, and liquid nitrogen conducts electricity so you don't want to immerse things in it. High temperature superconductors that will work in liquid nitrogen exist and can be made fairly easily (the BiSiCuYt superconductor has been made in a lot of high school labs) but you don't see any of them in electronic components. Also think about what you are using the things for - you want your semiconductors to be semiconductors an
Re:Nice work of fiction (Score:4, Interesting)
40 * 40 * 40 feet -> 104 Watts/sqft out...
Your numbers are off... (Score:4, Informative)
(40 * 8) * 4 + (8 * 8) * 2 ==
1408 sq. ft.
which, for 1 megawatt, is more like 710 watts/ sq. ft.
additionally... (Score:5, Interesting)
It's not fucking 1997 any more.
"Peering points" -- big, open-access traffic exchange handoffs like the old MAE-East [wikipedia.org] and MAE-West used to be a big deal back in the late 90s, when OC-12 circuits were still rare and hideously expensive beasts, and Gigabit Ethernet was still a gleam in some 3Com engineer's eye.
In 2005, they simply don't matter. The big players (level3, MCI/Verizon, Qwest, SBC, etc) all exchange traffic over private fiber interconnects, and everyone else buys transit from the big guys directly or ponies up for a switch port at Equinox, PAIX/Switch&Data or some other 'carrier neutral' colocation center. Dropping a datacenter-in-a-box onto MAE-east or any of its surviving ilk would buy Google precisely nothing.
(And nevermind the fact that google is documented to own thousands upon thousands of unused square feet of datacenter space already: they went on a very well-thought-out buying spree in 2000-2001 when all the dot-com datacenter companies were going out of business, and are very well provisioned for the forseeable future as a result.)
Now, a much more interesting application of the "Google node in a shipping container" idea can be summed up in one simple word: China. Why wait for the local market to develop the infrastructure you need when you can just drop a box down and then run fiber to it? I'm still dubious though...
Re:additionally... (Score:3, Interesting)
Nice thought, but it's not going to happen. And the reason why is that China is extremely wary of companies like Google. The Chinese government is about one thing, and that one thing is control. They're a-okay if yo
Re:additionally... (Score:3, Interesting)
(And I find the idea of Google parking a tractor trailer with their hypothetical container-box node on the street next to 111 8th ave particularly amusing. Nevermind heat dissip
Re:Nice work of fiction (Score:5, Funny)
Stealing (Score:5, Insightful)
Re:Stealing (Score:5, Interesting)
Re:Stealing (Score:2)
Re:Stealing (Score:5, Funny)
Knowing Google, I would think that these shipping container computer things would be covered with sensing devices. It's probably scanning the face, gait, apparent weight, and shoe size of anyone that gets near it, and googling for their name, their address, their family and children, employer, and all other relations. As it prepares to activate the lightning sprocket, it's probably composing emails, editing video footage, and notifying the newspapers of an impending obituary.
I'd sooner touch the Ark of the Covenant of the Lord your God, than touch one of these here Google Skynet Singularity Machines.
When will sinister phase two begin? (Score:5, Funny)
Re:When will sinister phase two begin? (Score:3, Funny)
As it turns out, the same news source has recently revealed Google's "phase two" plans [theonion.com].
Cheers,
IT
HEY! (Score:5, Funny)
Re:HEY! (Score:5, Funny)
Re:HEY! (Score:3, Funny)
5000 Opterons (Score:4, Interesting)
5000 Opterons? It makes sense to put those near power plants / ice bergs. That's at least 500 kW of heat dissipation.
Re:5000 Opterons (Score:3, Informative)
Mommy Mommy... (Score:4, Funny)
aren't they all? (Score:5, Funny)
I haven't yet met one that didn't think they were very bright. Industrial Designers invent stuff that takes 'ordinary' engineers years to throw away and build something else that will fly. No danger of anything happening here folks
Missing something here... (Score:3, Insightful)
In my opinion, Google has penetrated the American market with its services as much as it can. It is probably looking to other places in the world to prop up its cash flow. You know, like a business, rather than a collection of world-domination-bent nerds?
Akamai (Score:3, Interesting)
http://www.akamai.com/ [akamai.com]
Half the big boy websites I visit seem to run through these guys. They seem to provide fat throughput for mega sites, apparently hosted in a distributed geographical fashion. I could just be imagining these things, though, because I really don't have a clue.
Cooling 5000 Opterons? (Score:4, Interesting)
Re:Cooling 5000 Opterons? (Score:5, Interesting)
Re:Cooling 5000 Opterons? (Score:4, Insightful)
Re:Cooling 5000 Opterons? (Score:5, Funny)
1 cup of coffee: 0.2 litres (200g) heated from 10 to 100 degrees celsius (90 degrees) = 18 KJ.
250 KW: 14 cups of coffee per second.
The answer to "where do we put these puppies"?
Next to Starbucks.
Re:Cooling 5000 Opterons? (Score:3, Informative)
Re:Cooling 5000 Opterons? (Score:5, Insightful)
If an Opteron produces say, on average, 50W heat output (I know this isn't accurate, but just as an example), 5000 Opterons would produce 250kW of heat. That would require an air conditioning unit larger than the building used to house the container.
Hardly -- a kWh is 3413 BTUs and 12,000 BTUs is a refrigerating ton. So they would need about 71 tons of cooling (the name of the unit is derived from the cooling capacity of a ton of ice per day). They make chillers into the hundreds of tons of capacity.
Here is some information on a 75 ton chiller [hvacportablesystems.com]. That's smaller than the shipping container it would be cooling -- a normal shipping container is 40 feet long and about 8 foot square cross-section.
In fact, if there's any truth to this story at all, I bet they fit all the computer gear in the first 22 feet of the container and the chiller in the last 18 feet.
Why not wirelessly replace the internet? (Score:2, Interesting)
Um, what? (Score:2, Interesting)
The idea is to plant one of these puppies anywhere Google owns access to fiber, basically turning the entire Internet into a giant processing and storage grid.
Er, they plug a bunch of servers into the Internet and suddenly it's the Internet that's doing the processing and storage, not the servers? Sounds magical. Maybe I can plug my Playstation into the Internet and turn the entire Internet into a giant game.
Re:Um, what? (Score:3, Funny)
You mean it isn't already a giant game?
Great (Score:2, Insightful)
Re:Great (Score:5, Insightful)
Don't think that if somehow Google makes MS a lesser force that suddenly the sun is going to come out from the clouds and everyone is going to live happily ever after... Too many people on slashdot already have this attitude and it's an unfortunate one, at best.
Re:Great (Score:5, Insightful)
Why? (Score:2)
Re:Why? (Score:5, Insightful)
Re:Why? (Score:2)
VOD (Score:2)
Another promise waiting to be fullfilled.
X.
Nothing to worry about folks... (Score:5, Funny)
They're just getting ready to run Windows Vista when it comes out.
Oh My. (Score:2)
It sounds so malevolent!
"overnight" should be replaced with "under cover of darkness" though.
Same applies to Skype (Score:3, Interesting)
Skype is in the same situation - they've been able to support so many users simply because their bandwidth is only used to setup the initial connection between the two parties, after that it's the telcos who are supporting and providing the infrastructure for the service that threatens them most. Now that Skype can make real money from its pay services, look for them to do something simliar to Google, to ensure the availability of their service.
Salt (Score:5, Informative)
Saying that, when it comes to technology at least, he is speculative is something of an understatement. Take what he says with an extremely large grain of salt.
So when does this become self-aware? (Score:5, Funny)
Now all we have to do is wait for some Google employee to play a Sony CD on this and these will become spam relays.
Perfect.
Hardware limits (Score:3, Insightful)
Google's growth was in part made possible by heaps of commodity hardware. Hardware that was originally meant for standard lusers, cheap and unreliable. They built their systems for it and tolerate that. They change lots of haddrives in their datacenters and god knows what else.
What I'm trying to say is that for each of those googlecubes they need staff that regularly changes whatever hardware fails. With 3.5 Petabytes of storage and 5K processors it means that something will fail every single day that beast is powered. All that crammed inside 20/40 feet space (WTF does that mean?) means that heat will kill even more hardware.
So, yeah it should be possible, but not very likely.
Re:Hardware limits (Score:4, Insightful)
Just like painting the Golden Gate bridge. There's a small crew of painters assigned to that work. It takes them 4 years to paint the whole bridge, but when they finish at one end, the other already requires repainting, so they start over. The bridge is never 100% "brand new" painted, but it remains in acceptable state at all times.
"Google Desktop" delivered via FreeNX? (Score:5, Interesting)
Google will manage everything for its users: software upgrades, backups, search and organisation of personal data and files. Just like ISPs 20 years ago offered a monthly rate of 20 $US to connect to the internet (giving away a 2400 b/sec modem for a reduced price), Google could ask for a 20 $US fee (and give away a Google Thin Client embedded into a georgeous 17'' LCD screen that includes a EJ45 jack) to take care of people's computers.
I for one would sign in immediately.
So, Cringely is wrong. No need for AJAX office. It will all work with traditional GUI desktop programs, over an NX link that does not consume more than 40 kBits/sec for office productivity work.
So, Cringely is also right. The operating system doesn't matter to Google.
Not $500,000 (Score:3, Interesting)
One 20Ft container is:
* Length (20Ft)
* Width (8Ft)
* Height (8.5Ft)
That means you can get about 12 * 19" racks in, using 4 rows, about 64U high. That means a total of 3072 servers, using dual socket, dual core opterons, that's 12288 cores. Each server with 8 memory sockets + 4 disks, that's 24,576GB of RAM (1G sticks) and 6,144,000GB of Storage (500G disks). With some guestimate figures on current prices, I'd say one of those container would be worth about $12,500,000.
But then again, from a quick Google, they have about $3 billion in cash, and that's a lot of containers....
PS: I'm european using metrics mostly, so they're might be a small conversion problem here and there
When Google owns all our data access... (Score:3, Interesting)
Sounds like Cringely saw a Petabox (Score:5, Insightful)
Sounds like Google is trying that out.
There's nothing that exotic about this. The military builds racks of electronics into shipping containers all the time. It's mostly a cable management and maintenance access problem. You have to be able to do everything from the front of the rack, which requires some design work but isn't rocket science.
Re:Sounds like Cringely saw a Petabox (Score:3, Interesting)
The petabox project has essentially one design goal: "What is the absolute minimum amount of hardware we can wrap hard drives in and still have a useful system?" And the answer, apparently, is "a 1U half-depth case with a tiny Via board". That they can get power consumptio
Shipping Containers can be problematic (Score:5, Interesting)
The idea was good, except for a couple of problems.
These shipping containers are nothing but a giant metal box. Grounding can become an issue, so can accidnentally having the box be one of of the poles for a DC based power system. If you are near an active AM tower, the box becomes a giant antenna and it's virtually impossible to filter out the AM signal internally.
Last, and certainly not least, these shipping containers are vulnerable to rust and other problems due to exposure to the elements. That can take several years (5 or so) if the box is in perfect shape at the start, but if they are using used boxes then it can take less than 2 years for rust holes to be a problem.
Plus, physical security isn't all that good unless the walls are beefed up.
I'm hoping these are not "standard" shipping containers, just something that looks like them.
This grand experiment with shipping containers for cellular applications was an attempt to make it cheaper to deploy equipment to new locations. And, shipping containers (especially used) were a _LOT_ cheaper than fibrebond or other prefab buildings for that purpose. Of course, the fibrebond building had a lifespan a lot longer than 2 to 5 years. So, you get what you pay for.
This is pure bullshit (Score:4, Interesting)
It's about that Google want to be able to move Google computers around in any datacenter asap. We know that google uses a grid of single computers, that all compute the search results as fast as possible. All these computers create a space-problem (physical space) at any datacenter that Google owns. Also, shipping these computers around costs money. I bet google store this "secret package" just to be able to send it around anywhere where there suddenly is a problem with the network.. We all know how much money google loose if they are experiencing downtime...
Cringley is wrong. Google is going ISP (Score:3, Interesting)
I hope they succeed along with their WIFI access.
Good job boys.
One more point if anyone at Google is listening. How about a tax on chip companies to use this highspeed access. It would be nice if they could help support it since they are the ones who benefit.
Re:Stop Google!! (Score:3, Insightful)
Can't buy latency... (Score:5, Interesting)
Re:Can't buy latency... (Score:3, Insightful)
The speed of light is just too low for Google's AJAX applications to take over the world.
No, you are wrong. Even if Google had only 2 datacenters on the surface of the Earth located at 2 antipodal points, a path from any location to one of the datacenters would always be shorter than 10010 km (mean Earth's circumference divided by 4) and it would take 33 ms for the light to cover this distance. So the RTT would be 66 ms, which is sufficient for Ajax applications.
Though the speed of light contribute
Re:Can't buy latency... (Score:3, Interesting)
So, they send trunks of servers to near you to reduce the latency, and when you need more responsive systems, they send servers to your building, and if you need to reduce the latency even more, they send the servers to your desk.
Yes, I see where it goes, very inovative. But ignore the rant, I don't want to stay on the way of a nice buzzword.
Re:article doesn't explain network (Score:5, Informative)
To remove the network latency between them and you.
They're not being used "for computing" in the sense you're envisioning. For one thing, 5000 Opterons is enough to tackle pretty much any problem you'd care to throw at it, so there's no need to talk to anyone else. For another thing, they wouldn't be doing big computations, they'd be doing massive numbers of small ones. Think Gmail. 3.5PB is enough to store an awful lot of email, and a few thousand Opterons can run rather a lot of simultaneous HTTP connections from people accessing the mail. Add in a fast network link (for talking to all those many people accessing the mail, and for replicating everything offsite), and you're set.
Cringeley's penchant for sensationalism aside, it's pretty clear that Google's got the expertise and the mindset to deal with problems that start with "if we had 10,000 fast CPUs, 10,000 hard disks, and 10,000 GB of RAM...". Google's rapidly expanding, and has been ever since they started. Back when Google fit in a closet, a new server constituted a big expansion. I'm not surprised that these days their unit of expansion is a tractor trailer with a few dozen racks in it. And if you've got something that packages up that nicely, it only makes sense to pepper the globe with capacity.
Re:article doesn't explain network (Score:5, Interesting)
The specifications [bigbruin.com] [see footnote for a few other sites] state 146 mm) x (101.6 mm) x (25.4 mm) x 7 340 = 2.76551705 m^3 [google.com],
and, running with the article's numbers, let's see how much of 20 feet cubed that is... (article: the most storage, memory and power support into a 20...foot box -- note that a BOX of course is less cubic area than a 20-foot cube)....
((146 mm) x (101.6 mm) x (25.4 mm) x 7 340) / (20 (feet^3)) = 4.88316565 [google.com]...
WHAT? it's not a fraction, but larger by a factor of 4+??? Just for the hard-drives? Even when we assumed a CUBE???
Man, I want some of the shit that guy's smoking. I was expecting to debunk with just the hard-drives taking an impossibly large percentage of the proposed 20-foot "box". But....man. Cringely must not have done even a basic sanity check. (And remember, I'm pretty sure he didn't have a 20 foot high, 20 foot wide box in mind, or he would have said cube. To a writer, a "20-foot box" sounds like an elongated storage container [uniteam.org], e.g. 8x8x20 feet.... BTW that's the first hit for 20 foot storage container [google.com], I can only assume a writer would have such a thing in mind...)
English and math, people, English AND math.
Footnote:
Other sources for specifications:
Re:5,000 opterons? That'd make a fine... (Score:4, Insightful)
Re:Scary (Score:3, Insightful)
Re:Scary (Score:3, Interesting)
In fact, I can see this working because the media companies themselves (the labels, the networks, the studios), the very ones who can't afford to have all those redundant servers and data and managing their own damn network, will be the ones to finance it all by buying local c