Google Reveals "Secret" Server Designs 386
Hugh Pickens writes "Most companies buy servers from the likes of Dell, Hewlett-Packard, IBM or Sun Microsystems, but Google, which has hundreds of thousands of servers and considers running them part of its core expertise, designs and builds its own. For the first time, Google revealed the hardware at the core of its Internet might at a conference this week about data center efficiency. Google's big surprise: each server has its own 12-volt battery to supply power if there's a problem with the main source of electricity. 'This is much cheaper than huge centralized UPS,' says Google server designer Ben Jai. 'Therefore no wasted capacity.' Efficiency is a major financial factor. Large UPSs can reach 92 to 95 percent efficiency, meaning that a large amount of power is squandered. The server-mounted batteries do better, Jai said: 'We were able to measure our actual usage to greater than 99.9 percent efficiency.' Google has patents on the built-in battery design, 'but I think we'd be willing to license them to vendors,' says Urs Hoelzle, Google's vice president of operations. Google has an obsessive focus on energy efficiency. 'Early on, there was an emphasis on the dollar per (search) query,' says Hoelzle. 'We were forced to focus. Revenue per query is very low.'"
The New Mainframe (Score:5, Insightful)
I think Google may be selling themselves short. Once you start building standardized data centers in shipping containers with singular hookups between the container and the outside world, you've stopped building individual rack-mounted machines. Instead, you've begun building a much larger machine with thousands of networked components. In effect, Google is building the mainframes of the 21st century. No longer are we talking about dozens of mainboards hooked up via multi-gigabit backplanes. We're talking about complete computing elements wired up via a self-contained, high speed network with a combined computing power that far exceeds anything currently identified as a mainframe.
The industry needs to stop thinking of these systems as portable data centers, and start recognizing them for what they are: Incredibly advanced machines with massive, distributed computing power. And since high-end computing has been headed toward multiprocessing for some time now, the market is ripe for these sorts of solutions. It's not a "cloud". It's the new mainframe.
Re:The New Mainframe (Score:5, Interesting)
Re:The New Mainframe (Score:5, Interesting)
I don't know which 80's you lived through, but mainframe processing was alive and well in the 80's I lived through. Minicomputers were a joke back then, and were seen as mostly a way to play video games. (With a smattering of spreadsheet and word processing here and there.) In the 90's, PCs started to take hold. They took over the word processing and spreadsheet functionality of the mainframe helper systems. (Anybody here remember BTOS? No? Damn. I'm getting old.)
Note that this didn't retire the mainframe despite public impressions. It only caused a number of bridge solutions to pop up. It was the rise of the World Wide Web that led to a general shift toward PC server systems over mainframes. All we're doing now is reinventing the mainframe concept in a more modern fashion that supports multimedia and interactivity.
Welcome to Web 2.0. It's not thin-client, it's rich terminal. The mainframe is sitting in a cargo container somewhere far away and we're all communicating with it over a worldwide telecom infrastructure known as the "internet". MULTICS, eat your heart out.
Re:The New Mainframe (Score:5, Funny)
Derr... minicomputers should say microcomputers. My old brain is failing me. Help! Help! Help! He-- wait. What was I screaming for help for again?
Re:The New Mainframe (Score:5, Funny)
It's ok.
We have a nice table with an integrated NEC 8000 for you to sit at. We even sprung for the sound dampening box for the daisy wheel printer for you.
Re:The New Mainframe (Score:5, Funny)
When I first skimmed your post, I saw the words "daisy wheel printer" and my first reaction was, "Put it in the other room! Those fuckers are LOUD!" But it seems you've thought of everything.
And that's what I'm talking about! WWII levels of efficiency. Not this namby, pamby, "I didn't know that slotting DIMMs of different sizes into the motherboard would disable dual-channel access" BS. Somebody give this boy a raise!
Re: (Score:3, Funny)
Actually, yes. I just can't remember where I got to play with the B20s.
Re:The New Mainframe (Score:5, Insightful)
Not quite. While these server farms in a box are fault-tolerant they are not fault-tolerant in the same way as at least some mainframes where the calculations are duplicated. With mainframes you'd have wasted resources (doing every calculation twice) with lower latency. With server farms in a box you get, arguably, better resource utilization (route around something that is broken but wait till it breaks before doing so) but higher latency. The difference is incorporating the way the internet works into "mainframe" design.
Re:The New Mainframe (Score:5, Informative)
Re: (Score:3, Insightful)
Disclaimer: I work at Google, though the stuff below is something anyone from a large web company could tell you.
Actually the argument depends on the application, and Google does have some applications that make different tradeoffs. For search, availability is more important than consistency: A search on 99% of the data is still better than a 404 any time you don't have all of your servers available. However, for something such as billing (which occurs on every single ad click for pay-per-click ads), you'
Re: (Score:3, Informative)
Actually google does everything thrice (not unlike the Ramans). And returns the result that reaches it first. So in effect it is even more fault tolerant than the Mainframe. And it does them at different locations not on a single Facility (as opposed to a server or a 1AAA sized Container).
You are underestimating Google.
Re: (Score:3, Insightful)
What we've mostly been busy doing for the last 10 years is reinventing CICS. The same old business applications that generated bazillions in revenue and worked well under CICS have now been (painfully) rewritten to work on hopelessly buggy Web browsers across the public net.
Congratulations, but... whoo hoo.
Re:The New Mainframe (Score:5, Funny)
Dude, seriously. Would it have hurt you to read the FIRST *BLEEPING* REPLY
Give him a break. He probably typed that on an IBM 029.
Re:The New Mainframe (Score:5, Interesting)
Technology sways back and forth, and there is nothing wrong with that.
1980s 2400/9600 bps Serial connections displayed the data that the people wanted fast enough for them to get their work done. And the computer had a lot of processing that can handle a lot of people for such simple tasks. And computers were expensive heck it was a few thousand bucks for a VT terminal.
1990s More graphic intensive programs are coming out, Color Displays, Serial didn't cut it, way to slow. Cheaper hardware made it possible for people to have individual computers and networks were mostly for file sharing. So you are better off processing locally and allowed more load per demmand
2000s Now people have high speed networks across wide distances Security and stability issues begin to happen so it is better to have your data and a lot of the processing done in one spot. So we go back to the thin client and server where the client actually still does a lot of work but the server does too to give us the correct data.
Re: (Score:3, Interesting)
We used to run a small company off of a single 2400 baud link with an 8 port statmux (statistical multiplexor) to a remote VAX minicomputer.
It worked fine.
If I remember correctly, a VT100 was something like $1,200 or $1,600. After a while, there were third party VT100 compatibles that were much cheaper.
I bought a brand new out of t
Re:The New Mainframe (Score:4, Funny)
Re:The New Mainframe (Score:4, Funny)
As one of the priests, I sincerely wished that the congregation wouldn't return.
Re:The New Mainframe (Score:4, Informative)
By some measurements they exceed the computing power of a mainframe, by others they don't.
Re:The New Mainframe (Score:5, Insightful)
A fair point. However, I should probably point out that mainframe systems are always purpose built with a specific goal in mind. No one invests in a hugely expensive machine unless they already have clear and specific intentions for its usage. When used for the purpose this machine was built for, these cargo containers outperform a traditional mainframe tasked for the same purpose.
Re:The New Mainframe (Score:4, Insightful)
Well, I think it goes without saying that machine A (designed for a specific type of computing) will outperform machine B (not so designed) - and this will remain true whether A is a server cluster and B is a mainframe, or vice versa. And you need to keep in mind there are significant design differences between a server cluster and a mainframe, even when the mainframe is itself a clustered machine.
Huh? Here in the real world, mainframes are as generic as desktops - what determines what they can do is the OS and the applications. People buy mainframes because they need a mainframe's capability. (And container data centers aren't exactly cheap either - nobody is going to buy them without a use in mind either.)
Re:The New Mainframe (Score:5, Insightful)
by others they don't.
Seriously, I've fairly recently gone through every single benchmark, comparison, inference, etc, that I've been able to find on the subject (they're not exactly sprinkled all over the place) and I can't find any indications anywhere that mainframe hardware can surpass modern commodity hardware on any measurement. On price/performance variants it's not rare to see it outclassed more than an order of magnitude, and in absolute performance, well, there's very little magic hardware in the mainframe either anymore, it's pretty much the same silicon as anywhere else; Power CPU's, DDR infiniband, CPU to SC bandwidth almost equivalent to Hypertransport, same SAN as is used anywhere else, and as far as I can tell, to my horror, DDR2 533 memory(??). Please, correct me if I'm wrong and I very well may be, because actual specs aren't exactly flaunted. I mean, it's nice enough, but it's hardly magic.
Sure, there's the old trick of moving system and IO load into extra dedicated CPUs, but that's becoming less and less relevant as pretty much any significant IO load has long since moved to dedicated ASICs that do DMA on their own without any CPU cost, and things like encryption accelerators aren't that hard to find. And it's not like you're not paying for the assist processors.
Two or three years ago it might have been conceivable that it could have had at least a possibility of being superior in consolidation capabilities like being able to have the most unused OS instances running at a time, but with paravirtualized xen-derived tech commodity x86 hardware can accomplish the same or higher density. I can't say I've tried running 1500 instances, but for fun I did try running 100 instances on 5 years old junked x86 hardware which went fine until I ran out of memory at 6GB on the (like I said, junk) hardware in question. No significant performance degradation in relation to load versus what could be expected of the hardware, all 100 instances fully loaded both IO and CPU for a week to test for any throughput issues or over-time degradation, but that worked as well.
IE, no practical limit for any non-contrived consolidation situation, and I have no doubt that it scales fine up to 1500 instances on reasonably modern hardware as well as it did on that hardware (and if you need higher density than that you should seriously be considering why you're using that number of OS instances that don't appear to actually be doing anything or consider moving to system-level virtualization like vserver or openvz)).
So have you found any measurements that I couldn't find that you could point out that demonstrate lingering categories in which a mainframe might consistently outperform commodity hardware (ie, any measurement that is or can be compared to another at least somewhat related measurement on commodity hardware which demonstrates an advantage for the mainframe)?
Outside pure performance there is the in-system redundancy which is nice in theory but which in practice seems to rarely result in higher actual uptime (mainframes appear to require an inordinate amount of scheduled service time and admins often engage in a disturbingly high IPL frequency).
There is also the consistent load levels they tend to get (which seems to be largely due to culture, load selection and ROI requirements, rather than any inherent capacity), but beyond that it seems that the remaining aura of capability doesn't have much basis in reality anymore.
Of course. That's why APC is a mainframe vendor (Score:3, Insightful)
Arguably, APC has become a mainframe vendor. [apc.com] They sell rack systems with integrated power, cooling, and cable management. Add commodity motherboards, CPU parts, disk drives, and software, and you have a mainframe. It's not that different from what HP or SGI or IBM or Sun will sell you. Especially since the "mainframe" vendors have mostly moved to commodity CPU parts.
I've pointed out before that computing is becoming more like stationary engineering. [local39.org] Stationary engineers run and maintain all the equip
Re:They are computers, no more advanced than befor (Score:5, Funny)
Google is basically re-implementing the efficiency that already exists in a laptop.
You have a laptop with >1000 processors, consisting of several times that many cores, with its own built-in gigabit ethernet running on built-in gigabit switches?
I'd hate to sit next to you on an airplane!
Re:They are computers, no more advanced than befor (Score:5, Funny)
It's ok, appearently he stores it in his middle finger.
Re:They are computers, no more advanced than befor (Score:4, Interesting)
Actually, they're not.
Laptops run slower than their PC counterparts.
Laptop drives run slower than their PC counterparts.
Laptops run hotter under load than their PC counterparts.
If you look carefully at the picture, they've found a 12v motherboard, tied a 12v battery directly into it, and used otherwise commodity parts. That's been the mantra for Google for as long as I can remember. Oddly enough, that was my mantra when I built up a big network. Lots and lots (and lots and lots) of cheap servers are better than a handful of really expensive ones. That saved our cumulative posteriors on more than one occasion.
I've spoken with some people who have personal knowledge of Google's equipment. They were setting up with RAID 01 or 10. I suspect with the two drive configuration, they're only setting up with RAID 0 now, and the redundancy is across multiple servers. I can confirm that they are using this open tray system for it's superior cooling.
I had considered open trays like this, except there's one huge downfall. You would have to be amazingly careful of what happens near the rack. If you are screwing something in, and the screw or screwdriver falls, that can become very bad very quickly. Did you see any fuses or breakers from the battery to the power supply?
Short of making the area around the rack a metal-free zone (no screws, screwdrivers, rings, keys, watches, etc), you'd seriously run the risk of shorting something out. I know I've been working up in the higher areas of a rack, and dropped screws. You listen to it rattle it's way down across several machines until it finally hits the floor. Since I use closed servers cases, it's never a problem. Maybe they don't have a big problem with it at Google, but I'd be terrified of it. Anyone who says they've spent any substantial time working in and around racks, and haven't ever dropped anything, are lying. I do love the idea for free airflow and better cooling, but ... well ... I like to keep magic smoke in it's place. :)
The one-battery-per-server is a nice idea though. I may look into that for future builds. Most PC's have 5v and 12v output. That power supply only indicated a 12v output, and didn't have any wires that indicated anything different.
Re:They are computers, no more advanced than befor (Score:4, Interesting)
The problem I have with running a motherboard directly from a 12V battery is that most batteries are 12V nominal; actual voltage varies quite a bit (10.5-13.8 for a Pb-Acid). So the question is how well do the 12V components cope with the lower/high voltage? Most of the logic should be OK; that's all 5/3.3/1.xV. I'm guessing the only stuff that really uses 12V anymore is actually disk drive circuitry(not technically on the MB).
I have a suspicion that you really don't want to be running a hard drive off a voltage supply that varies by up to 25%. They must have solved this somehow (step up + step down converter? But that is not efficient) but I really see no point in using 12V motherboards unless everything else can reliably run off the battery first. The home consumer may as well stick with getting 5V from the PSU and letting that dissipate the heat from the step down conversion until we're all using 5V disk drives. In which case, we can probably move to lower voltages (and lower voltage batteries); ~8V seems about right to get a stable 5V.
Re:They are computers, no more advanced than befor (Score:4, Insightful)
Modern high speed chips (which draw the bulk of the power in a typical PC) run thier core logic at much lower voltages. Typically somewhere between 1V and 2V though I think some may have gone below a volt now. Theese very low voltages have to be produced very close to the chip that uses them to avoid huge losses.
This means that modern PC motherboards take most of thier power at 12V anyway. The 5V and 3.3V lines really only serve to power the low speed chips and some of the interfaces between chips.
Given that I doubt there would be too much efficiancy loss from making a 12V only board. You could probablly even design it to hapilly deal with an input that was only approximately 12V without losing too much (since most of that 12V power is going to the input of switchers anyway).
Patents & Catch-22 (Score:5, Informative)
Ah, the catch--22 of the patent--being forced to reveal your hand in order to protect it while underpaid workers at Baidu figure out how to integrate your ideas into their hardware.
Re:Patents & Catch-22 (Score:5, Informative)
considering some of the mini's I worked on had similar setups in additions to external UPS.
then again, we achieve all sorts of power, cooling, and reliability, when we consolidated many "pc" style servers into minis which do the same work. (the heat change alone was staggering)
Re:Patents & Catch-22 (Score:5, Informative)
Ah, the catch--22 of the patent--being forced to reveal your hand in order to protect it while underpaid workers at Baidu figure out how to integrate your ideas into their hardware.
That's not a catch-22, that's the point. In exchange for everyone learning from what you've done, you get society's protection for a limited number of years.
Also, the workers at Baidu are not underpaid- if they where, they'd leave for better oppurtunities. The workers in question have obviously decided they're better off making stuff for google- they don't need your 'superior' judgement to tell them they should go back to subsistenance farming or melting hazardous materials for precious metals in their homes.
A decision to work, or not to work, and to hire, or not to hire, are based on realistic alternatives, not what some westerner sitting at a keyboard 9,000 miles away thinks is best.
Re:Patents & Catch-22 (Score:5, Interesting)
Wow, you missed the point. Poster is contending that the patent FAILS to protect IP, BY MAKING AVAILABLE the instructions to REPLICATE said IP.
Yeah, it may work against Yahoo!, but it doesn't save you from companies in China and India, who can undercut you on labor costs, and have a much more rapidly expanding market.
Re:Patents & Catch-22 (Score:5, Informative)
Please see the Patent Cooperation Treaty [wikipedia.org] which covers this situation; China acceded in 1993, India in 1998. [uspto.gov]
But how could it not be obvious? (Score:3, Interesting)
There's only so many places you can connect a battery to a PC and all of them have already been implemented by someone at some point. There's been motherboards with second power connectors, motherboards with battery connectors, power supplies with batteries, power supplies with battery connectors, DC power supplies connected to external batteries, integrated UPS systems which take in and put out AC and which are basically just hooked up in line with the power supply... Off the top of my head I immediately t
Kidding Me? (Score:5, Funny)
Get that man a beer.
Re: (Score:3, Funny)
You parsed it wrong: he worked 14 'one hour days' over two and a half years.
On the other hand... getting away with that deserves a beer too...
Re:Kidding Me? (Score:5, Funny)
You parsed it wrong: he worked 14 'one hour days' over two and a half years.
On the other hand... getting away with that deserves a beer too...
Somebody who can get away with that has probably had many, many free beers.
Pretty cool stuff (Score:3, Interesting)
Nothing to do but wait for a finished product at this point though.
Re:Pretty cool stuff (Score:4, Informative)
then you need to move your offices to the middle of a desert. Space problem solved :)
SMEs often get themselves a small server room, and don't plan for expansion. When the time comes to stick more servers in, they usually have to put them in an office instead, with non-redundant power, little cooling. You're not alone there, but it doesn't necessarily apply to datacentres.
Space at datacentres is often the least of their worries nowadays, (it used to be different), but power is the big problem. Even the DCs in the middle of the metropolis has enough space to fit a few servers, but they can't get the power to them if they did.
Re: (Score:3, Insightful)
googles pretty sure about it...
do you also run a multibillion dollar server farm?
No way (Score:2, Redundant)
Re:No way (Score:4, Insightful)
Greater than 99.9% efficiency? They likely made a mistake in their measurements.
Maybe they measured 99.92% efficiency.
That is greater than 99.9% efficiency and they aren't breaking any laws of thermodynamics.
Re:No way (Score:4, Informative)
Stop the lies (Score:5, Funny)
We all know the searches are actually being done by a large amount of people in suspended animation, being fed the corpses of the previous people.
The thing about each server having its own battery is a cruel joke.
Re:Stop the lies (Score:4, Funny)
Don't you remember in the Matrix where Morpheus holds up the Duracell battery to describe what the people are being used for? Google just managed to actually do it.
Don't worry. (Score:5, Funny)
I'm working on a solution. If only I can contact Oracle.
Re: (Score:3, Funny)
General inquiries
+1.800.ORACLE1
http://www.oracle.com/corporate/contact/index.html [oracle.com]
*ducks*
Re:Don't worry. (Score:5, Funny)
I'm working on a solution. If only I can contact Oracle.
"Thank you for calling Oracle. For English press 1, para en Español marque el numero dos.
*beep*
You have reached the Oracle Help Line. Please hold for the current Oracle. All calls are answered in the order received. There are currently [1,983,457] callers ahead of you. Estimated wait time is [5,347,987] minutes.
Have you tried knowing thyself? Try checking our website at thereisnospoon.oracle.com.
Thank you for holding."
Re: (Score:3, Funny)
Onboard UPS not new (Score:5, Informative)
The in-computer onboard UPS is not a new idea. I don't see how they could have gotten any patents on it since I used it have one of these (my day might still). The device I saw had a gel cell mounted on an 8-bit ISA card, full length. It had +5/12v pass through connectors for powering the drives and it powered the computer through the main bus. There was more logic to it, as it had some monitoring capabilities too.
What's next, patenting a hard drive on a plugin board? Been there, it was called the Hard Card and put a 20mb HDD in an 8 bit full length ISA slot, a truly neat idea for upgrading old XT computers back in the day. You could make them work with AT computers too by putting a regular disk controller, without a drive connected, on the bus too and the BIOS would see the XT controller and boot from it.
Re:Onboard UPS not new (Score:5, Funny)
Indeed. (Stares at laptop).
Re:Onboard UPS not new (Score:5, Insightful)
A patent is an implementation of an idea.
You can have the idea of how to put an UPS in a computer one way, and I can do it another way and both be valid patents.
I do know this gets abused, and companies try to sue becasue it's there 'idea', but that's ot how it works.
If you find a different way to do a hard drive plugin board, then yes you can patent it. I would advise you only do it if it's better in some way, and there is a demand.
Re:Onboard UPS not new (Score:4, Informative)
Agreed, the onboard UPS is not new. I have a ~10 year old (I believe the CPU is a K6-233) device meant as a SOHO file/print/webserver from IBM that has a built-in gel-cell battery for UPS power just like this server does. Google is 5+ years too late.
Anyone want my prior art to invalidate the patent?
Re:Onboard UPS not new (Score:5, Informative)
I agree!
http://www.globtek.com/html/ups.html [globtek.com]
No shit? (Score:5, Funny)
When the weather gets warmer, Google notices is that it's harder to keep servers cool.
Brilliant journalistic work there.
OMG!!! Google patented laptop (Score:3, Funny)
Googles secret is that all there computers have battery.
I think, it is called a laptop.
Always wondered.... (Score:3, Insightful)
...why desktops didn't have a built in battery deal that lived in an expansion bay. If you could even keep RAM alive for extended periods even with the machine shut down that would be spiffy as an option, let alone as a little general UPS.
Oh, for God's sake people... (Score:4, Informative)
Re: (Score:3, Informative)
Fair point -- I never thought of that. However, there are some other links to this:
Maybe this is legit...
99.9% efficiency (Score:4, Insightful)
This is a questionable number. The best DC-DC conversion is around 95% so they aren't including voltage conversions from the battery to what the system is actually using.
Re:99.9% efficiency (Score:4, Informative)
The article says that they use special motherboards that require 12V only, which is what the batteries put out. No conversion needed.
Date centre fire risk? (Score:5, Interesting)
Many data centres expressly forbid UPSes or batteries bigger than a CMOS battery in installed systems - because when the fire department hits the Big Red Button, the power is meant to go OFF. IMMEDIATELY.
So while this is a nice idea, applying it outside Google may produce interesting negotiation problems ...
Re:Date centre fire risk? (Score:4, Interesting)
Pretty sure if the fire department is coming in to throw water lines around, they are going to cut the power to the building and not to just the circuit on the datacenter floor.
I could be mistaken, but I don't think a 12 volt battery backup in these applications are going to pose much of a "life" risk. Obviously you don't want to put your tongue on the terminals, but I don't think they pose the same threat that the power lines under the floor do.
Re: (Score:3, Insightful)
Pretty sure if the fire department is coming in to throw water lines around, they are going to cut the power to the building and not to just the circuit on the datacenter floor.
Yes, but if they cut the power to the building the server room will still be fully energized thanks to all those huge batteries running the place. That's why they have the big red buttons - they kill all the power in the room so that there is no electrocution danger.
As another posted indicated, commercial UPS systems typically have
Re:Date centre fire risk? (Score:4, Insightful)
But anyway. A rack mount HP UPS I installed in the past year has a stand-off that you can hook into the "Big Red Button System". I'm guessing such hookups are either standard on rack mount units, or at least it wouldnt be hard to find models with that feature.
yeah, but does it run Ubuntu? (Score:2)
You're probably thinking "man, these things are just too big, no one will want one for their home" but in a few years these things will be on everyone's desktop. Sure, the first few desks will be crushed, but I'm 100% sure they will make them fit nicely into your cubicle.
From the diagram it looks like they just need to put a chair in there and you're good to go. Now to compile Counter Strike for this thing.
Outgassing hydrogen? (Score:2)
Anyone concerned that when a SLA batter is charged, hydrogen is one of the by-products?
Re: (Score:3, Informative)
You're Doing It Wrong(tm). A sealed cell will only vent hydrogen if overcharged (at the cost of increasingly reduced cell capacity - you're not filling it back up with water!). An intelligent charger will eliminate any routine hydrogen venting, leaving only the occasional bad battery or battery hooked to a broken charger venting. Google is probably OK with that.
Who swaps out all those dead batteries? (Score:3, Interesting)
Hundreds of thousands of servers == thousands of dead batteries each month, since those batteries don't last more than a few years.
Now I'd think their design could be gentle on the 12V batteries, since it's possible to design UPSes that don't murder batteries at the rate cheap store-bought UPSes do. But still, they must have an army of droids swapping out batteries on a continuous basis.
Or maybe they are more selective, and only swap out batteries on hosts that have suffered one or two outages. It only takes one or two instances of draining a gel cell to exhaustion before it is unusable.
Re:Who swaps out all those dead batteries? (Score:5, Insightful)
Or maybe they think bigger...
They're deploying containers of servers. Maybe when a container gets a to a certain age or a certain failure rate, they replace/refurbish the entire container.
I doubt they care if some of their nodes go down in a power outage as long as some percentage of them stay up.
Re:Who swaps out all those dead batteries? (Score:5, Insightful)
Hundreds of thousands of servers == thousands of dead batteries each month, since those batteries don't last more than a few years.
I would imagine that the battery replacement schedule mimics the server obsolescence perfectly.
LOL, when the battery catches fire, time to replace the server.
Re: (Score:3, Insightful)
Given what has been said about Google's maintenance policies in the past, probably not. Google doesn't do detail maintenance - they wait till an entire rack (or now probably container) falls below a certain performance level, and then replace it with a
A quick peek at the picutres says a lot (Score:4, Insightful)
This is composed purely of commodity parts. The power supply is the same thing you'd buy for your desktop, those are SATA disks (not SAS), and that looks like a desktop motherboard (see the profile view where all the ports on the "back" are lined up in the same manner they would need for a standard desktop enclosure).
Only the battery is custom (or even non-consumer grade), and you can note that since the power goes through the PSU first, that's DC power. DC is significantly better than AC, since the PSU then has to convert AC-to-DC (which wastes power and generates needless heat). While you can get DC battery supplies for server-grade systems, these are not server-grade systems. Built-in DC battery backup therefore affords them the ability to keep the motherboards cheaper. Very smart.
Also, if you recall from a few months ago, Google has applied pressure on its suppliers (I'm not sure why Dell comes to mind...) to develop servers that can tolerate a significantly higher operating temperature (IIRC, they wanted at 20 degree (Fahrenheit?) boost). I wouldn't be surprised if the higher temperature cuts down on operating expenses more than smarter battery placement.
Re:A quick peek at the picutres says a lot (Score:4, Informative)
Actually, looking at the battery, ir looks like the same exact type of battery as you'd find in an APC small (450-800VA) UPS. We also used the same batteries for emergency power in our door access systems to power the controller when I was managing those at a small college. That type of battery is widely used to compensate for short term power outages.
I presume, given the amount of hardware shown (2 drives, 2 processors, motherboard, RAM) that the battery would probably last that given system about 7-10 minutes... plenty of time for the electric system to failover to the generator farm (you know they have more than 2 for redundancy.
As to the lifetime on those batteries... I was replacing them every 3-3.5 years, maybe 4 if I was lucky. It's a standard generic battery, and the failure rate on them is quite low.
I'd echo another user... If Google wanted to be smart, they wouldn't bother repairing a server when a component fails. Server obselescence at a company that can afford it is about 3-4 years... pretty close to the time for these batteries. They'd probably just pull the main power on it, and when a threshold of servers is "dead" in the container, they pull it offline for renovation... Either to repair the bad servers, or just retire everything.
Too Bad, that they do not carry it further (Score:3, Insightful)
mass servers = "21st century energy refineries"? (Score:3, Informative)
Keyboard, Mouse and two USBs? And slots? (Score:4, Interesting)
I'm a little surprised by the keyboard and mouse port and the two USB ports. If it uses USB, why not just use that for the keyboard and mouse? And why the second USB port? I suspect the second port doesn't consume extra energy directly, but it causes air resistance where they'd like to clear path to drag air across the RAM and CPUs.
And why the slots which will never get used? In quantities like Google buys, you'd think those would be left off.
Maybe they don't make any demands on Gigabyte (the manufacturer) and just buy a commodity board? When they're buying this many, you'd think Gigabyte would be happy to make a simpler board for them. On a trivial search, I don't see the ga-9ivdp for sale anywhere, but maybe it's just old.
Video: Inside the Container Data Center (Score:5, Informative)
Sigh... (Score:3, Interesting)
Ah, too bad I kept no notes, no logs, could have made a fortune suing Google.
Not too difficult, but impressive nonetheless (Score:5, Interesting)
I could design this PSU configuration, and I do electronics only as a hobby.
First, your main PSU delivers 12V in this scheme. Then this is stepped down to 5V and 3.3V for mainboard use, a design that is already employed by some Enermax PSUs, for example. For the 12V line, remember that +/-10% lower is acceptable. The lead-acid battery delivers up to 14V, so you need a step-down converter to 12V. In fact, you can design a switching regulator that steps the input voltage down to 13.2V (12V+10%), if it is larger, and just passes it through for 13.2V...10.8V with very, very low losses. A similar design can be done for 5% tolerances. Modern switching FETs go down to 4mR per transistor and you can do the transition from switching mode to pass-through mode very easily, e.g. with a small microcontroller that can then also do numerous monitoring and safety things. I had actually considerd such a design (purely analog though) for a lower-power, 12V external supply system myself some years ago, but a single UPS was so cheap that I did not went through with it.
I do not mean to belittle the what the Google folks do, though. The real ingeniuity is relaizing you can do it this way on a datacenter scale when nobody else does it. The engineering is then not too demanding, at least for folks that know what they are doing.
Re: (Score:3, Informative)
Found the patent:
Application number: 11/756,939
Publication number: US 2008/0030078 A1
Filing date: Jun 1, 2007
Inventors: William Whitted, Montgomery Sykora, Ken Krieger, Benchiao Jai, William Hamburgen, Jimmy Clidaras, Donald L. Beaty, Gerald Aigner
Assignee: Exaflop LLC [Note that Exaflop LLC's mailing address is the same as Google's.]
U.S. Classification
307066000
The most interesting parts, to me:
Figure 1, which shows an AC-DC converter, a battery, and a motherboard on a tray, in parallel.
And the following e
Very innovative, but... (Score:3, Insightful)
Wait, my laptop has one of those too...
In other news, is anyone else surprised that a built-in UPS is so slow to catch on for the desktop when notebooks have had it by definition for years? Sure, powerful batteries are expensive, but you'll wish you had one when a power blackout destroys half a day's work. It's one reason why I hesitate to get a desktop PC.
Re:Hey google, want to save some money? (Score:5, Insightful)
Re: (Score:3, Informative)
Plus you also have to account for gradual scaling up & geographical distribution. Easy to do with additio
Re: (Score:3, Interesting)
Form a quick once over of from IBM ftp://ftp.software.ibm.com/common/ssi/pm/sp/n/zsd03005usen/ZSD03005USEN.PDF [ibm.com]
There z10 can hold a max of 1.5 tb of Ram
Lets say for the load that is processed by each server, Google needs 8 gigs of memory. Which they can supply for $2,000 each.
No lets be generous to IBM and its reliability and say that we need twice as many google servers per the equivalent ibm reliability.
We can replace 375 (1.5 TB of ram /8gigs per google mache * 2googlemachines/ibm equvalence) google serv
Re:Hey google, want to save some money? (Score:4, Informative)
When I worked for a University, we bought a few of the largest IBM pSeries machines (power4 at the time). These were powerhouse machines 5 years ago. Each one had a dedicated 24" oversized rack cabinet, and then we had a couple racks just for disk. The 4 machines, and about 40T of Fibre channel disk (or was it DASD), I think it was a total of 128 core and 256GB of ram. I think we paid about a million for that setup.
As was mentioned elsewhere on the webs, the machine shown off by Google was based on Nocona CPUs.. those are atleast 4 years old now. Not likely what they're buying new now.
I bet you could get a base z10 for a few hundred thousand, but a fully loaded one? With a disk array of 750 drives? I bet 4 racks of disk from IBM would cost most of that 950k budget.
Re:Hey google, want to save some money? (Score:4, Insightful)
Have you ever even seen Mainframe pricing? No really have you?
It will cost you at least 10000$ to match the power of a single quad core intel/amd cpu.
And you do not want to run a mainframe(Or other computer that have a cpu bound task) for a decade. I think my current desktop computer have more power then avg mainframe
from a decade ago, and when I buy a new development workstation in then next decade, it will most likely have more cpu power then a 1 million $ mainframe you could buy today.
Just to set things in perspective: I am pretty sure, that google have more cpu power, more ram, more hd space and more aggregate io, then all mainframes in USA combined.
Re: (Score:3, Insightful)
And you do not want to run a mainframe(Or other computer that have a cpu bound task) for a decade. I think my current desktop computer have more power then avg mainframe from a decade ago
You can run a mainframe for a decade or more because every part except the steel frame is hot-replacable. You upgrade the processors, memory, everything really every few years, without ever interrupting service. There's a reason they aren't cheap.
Even good minicomputers (or expensive servers, if you like that term berret) let you swap processors, I/O processors, memory, and sometimes motherboards while the machine is running. High-end mainframes just take that to the next level, by ensuring that every bo
Re: (Score:3, Insightful)
There is no possible way their solution is cheaper than a real mainframe (created for the task) when all costs are considered.
Nor is there any possible way their solution is more reliable, or more "green".
That depends on how you're measuring cost, reliability and "green"itude. Cost-wise, there's an enormous opportunity cost associated with going with a single mainframe vendor. Reliability... well, they've made the choice of having small, frequent failures that are cheap and easy to deal with rather than single large uncommon events that might put a division out of action all at once. Green credentials? Again, it's a trade-off. They've traded physical resource cost against energy cost.
Also, by doing it this
Re: (Score:3, Informative)
A 12v battery. I never knew DC was more efficient than AC! WOW GOOGLE IS SO COOL!
I'm not sure if you are being sarcastic. A 12v battery in the power supply is more efficient than taking DC -> AC -> DC. That is what a UPS does, each conversion introduces loss. Having the battery in the power supply means there is no conversion so less power loss.
Re: (Score:3, Informative)
A switch mode PSU takes AC, converts it to DC, switches it at a high frequency and then filters it back to DC at each rail voltage. They have obviously modded this PSU so that it can take DC directly in at a much lower voltage and still work so the PSU and UPS are combined. I find this neat.
The whole server
Re:Hey google, want to save some money? (Score:5, Insightful)
* Latency, if you have all your data centre's located in essentially a single part of the USA (lets ignore the rest of the world for this.. regardless that there are no deserts in Europe for example) won't that increase latency quite a bit to the more further away places that want the search results?
* Bandwidth/redundancy, if you have all your eggs in one basket as it were aren't you going to have to pay extra to have lots of extra fibre laid down to be able to handle all that extra traffic? What about natural disasters, if you have all your data centres in a single location then surely you run the risk of things going pear shaped if it burns down, suffers earthquakes, aliens destroy the building etc.
* Cooling, because it's in the desert isn't a lot of the electricity that is generated going to be cooling not only the building because of the outside heat, but also the heat generated by the servers? Surely it makes more logical sense to build in a colder climate say further north and use hydroelectricity? (if you're talking of using exclusively non active polluting (and non radioactive) natural electricity solutions)
Re:Hey google, want to save some money? (Score:4, Informative)
A desert does not describe the temperature of a region but the (lack of) rainfall/moisture.
http://desertgardens.suite101.com/article.cfm/definition_of_a_desert [suite101.com] (link found using Google).
And besides, put the containers underground and I'm pretty sure that "hot" you refer to becomes a non-issue as well.
Re: (Score:3, Insightful)
How did this get marked informative?
I mean it's certainly true that Deserts are defined by lack of rainfall but since the GP said
"Build your data center in the desert and build 150 MW industrial solar thermal system to power it."
I think it's fair to assume they were talking about the stereotypical sunny and hot desert.
Secondly the reason it's cool underground is because soil is generally a very good insulator. I would suggest that it's a really bad idea to put things that are going to get hot inside a huge
Re: (Score:3, Interesting)
Fly the data center above the arctic circle in the northern hemisphere's summer, and fly it down below the antarctic circle in the southern hemisphere's summer, and you could do the solar thing 24 hours a day with cheaper cooling.
Re:Hey google, want to save some money? (Score:5, Interesting)
A google mainframe would be stupid.
If you take the price of a mainframe, and compare that to what google can get for the same money using their current solution, their current solution offers at least 10 times as much cpu performance, and much much more aggregate io(Both hard disk and memory) bandwidth.
There are only 2 reasons to use mainframes now.
1: Development cost. Building software that can scale on commodity hardware is expensive and difficult. It require top notch software developers and project managers. It make sense for Google to do it, because they use so much hardware(>100000 computers at last count).
2: Legacy support.
Re:Hey google, want to save some money? (Score:4, Informative)
"If you take the price of a mainframe, and compare that to what google can get for the same money using their current solution, their current solution offers at least 10 times as much cpu performance, and much much more aggregate io(Both hard disk and memory) bandwidth."
no it doesn't.
Plus they are cheaper to maintain, require less power per cycle, require less square feet to house.
Yeah, I actually know about these things.
Re: (Score:3, Insightful)
Ok. So, your load fits onto 5 mainframes. Now your requirement increases. What do you do? Do you buy number 6 now, and have it running at less than capacity for the next 18 months (or whatever)? That's a huge waste. Do you degrade your service for the next 9 months until number 6 would be at half capacity, then install? Again, you've wasted an opportunity, and number 6 is *still* not going to be at capacity.
Smaller computational units means better matching of demand to supply.
Re: (Score:3, Funny)
wouldn't trust them any further than I can throw them
Given the reliability, it's likely that someone has already measured that particular parameter for you. Have you checked the data sheets?
Re: (Score:3, Insightful)
Just because something is published on April 1st doesn't mean it's an April Fools joke. In the case of this article, it's clearly not.
You're seeing a connection where there is none. The two SATA cables run back behind the plate the drives are mounted on. Presumably, the mainboard connectors are back there as they're not visible on the rest of the mainboard.
Re:FCC? UL? (Score:4, Informative)