Forgot your password?
typodupeerror
Supercomputing Data Storage Media Movies

One Data Center To Rule Them All 112

Posted by CmdrTaco
from the my-preciousss dept.
1sockchuck writes "Weta Digital, the New Zealand studio that created the visual effects for the 'Lord of the Rings' movie trilogy, has launched a new "extreme density" data center to provide the computing horsepower to power its digital renderings. Weta is running four clusters that are each equipped with 156 of HP's new 2-in-1 blade servers, and use liquid cooling to manage the heat loads. The Weta render farms currently hold spots 219 through 222 on the current Top 500 list of the world's fastest supercomputers."
This discussion has been archived. No new comments can be posted.

One Data Center To Rule Them All

Comments Filter:
  • by DamienNightbane (768702) * on Monday September 08, 2008 @10:50AM (#24919933)
    Imagine a Beowulf cluster of these...
  • by starglider29a (719559) on Monday September 08, 2008 @10:54AM (#24919983)
    How do they release the heat in the hot water?

    Two Towers?
    • Re: (Score:3, Funny)

      by MindKata (957167)
      "How do they release the heat in the hot water?"

      They don't, the Orc's working there, have been waiting for a hot water bath for years.
    • The hot air passes over an air to water heat exchanger where an energy transfer takes place The Rittal unit can draw 30kW of heat out of each rack
  • I'll show you heat loads [wikipedia.org].

  • by eln (21727) on Monday September 08, 2008 @10:57AM (#24920009) Homepage

    I can't mention who I work for for obvious reasons, but we did some experimenting with "extreme density" computing some time ago as part of a black ops project for the government. We achieved densities previously unheard of by man.

    Unfortunately, we got greedy. We increased the density so far that the entire facility ended up collapsing into a black hole, wiping out much of the state of North Dakota. We were able to contain the damage, and we've managed to keep it a secret by replacing the state with a hologram projection, but eventually someone is going to go there and figure out that something is amiss.

    • So that's why the homesteading programs i read about in the back of Popular Science all those years ago were shut down.
    • We increased the density so far that the entire facility ended up collapsing into a black hole, wiping out much of the state of North Dakota.

      By North Dakota, I assume you mean Idaho [google.com].

    • Note the vague phonetic similarity between "North Dakota" and "Mordor."

      Did this project happen to hand out t-shirts... or, hm, something less obvious like... rings?

    • by RobBebop (947356)

      wiping out much of the state of North Dakota. We were able to contain the damage, and we've managed to keep it a secret by replacing the state with a hologram projection, but eventually someone is going to go there and figure out that something is amiss.

      Wiping out North Dakota [wikipedia.org] shouldn't be too hard. There's only about 600,000 people in the state [wikipedia.org] and half of them live on the eastern border within 4 miles of Minnesota.

    • by Pascoea (968200)
      I could only have hoped for such a thing this morning. I really didn't want to get up for work this morning. Getting rid of my whole state would have solved that problem quite sufficiently.

      It does kind of remind me of Austin Powers though:
      (http://www.imdb.com/title/tt0145660/quotes)
      Commander Gilmour: Are you suggesting that we blow up the moon?
      The President: Would you miss it?
      [looks around the table]
      The President: Would you miss it?

    • Amazing! So what was North Dakota like before it became the vast, desolate wasteland devoid of any trace of humanity it is now?

    • North Dakota is 2-dimensional, you insensitive clod!

    • And here I always thought the Eureka Advanced Research Facility was in Oregon! I am gonna go waaay out on a limb here and guess your name "eln" is a clever reference to either VAXeln or electronic lab notebooks, either one suggests you are in fact Henry Deacon.....

      Just a sec, someone's landing a unmarked chopper in my back-----------------

      >>>connection reset by peer

    • by ikeman32 (1333971)
      Sssssshhhhhh. You are coming awfully close to violating the Prime Directive and the Temporal Prime Directive. These pre-warp flight primatives must not know the truth lest they panic. *Puts on rediculously dark sun glasses* Now everyone if you would plese look here --------->(*)-----------(flashy thing) *Bright flash of light, removes rediculously dark sungalsses* "Ok folks that was not a UFO you saw, swamp gas off of a weather ballon refracted the light from venus. No that's not it. Oh well I return you
  • by Bandman (86149) <bandman@gmai[ ]om ['l.c' in gap]> on Monday September 08, 2008 @11:13AM (#24920205) Homepage

    I'm of a mixed mindset when it comes to water-cooled datacenters.

    On one hand, you've got the makings of a biblical scale disaster with all that water and electricity mixing.

    On the other hand, you can't argue with it's effectiveness.

    I'll stick to non-catastrophic issues when my
    air conditioner breaks down.

    • I agree. Plus, even though they mention the Rittal units [rittal.com], no one seems to be able to answer how the water cooled units plan for redundancy. It might not be pretty, but at least in a "traditional" raised floor/forced air solution, you can deploy spot coolers and/or fans as you need to mitigate an outage or allow for maintenance. What do you do when you burst a cooling water pipe as it distributes to a row of equipment?
    • by davesag (140186) on Monday September 08, 2008 @11:31AM (#24920421) Homepage

      Liquid cooled, not water cooled. They cool it in an inert liquid, rather than water.

      • by Bandman (86149)

        Ouch. I've seen the prices of some of the 3M inert liquids, and those are incredibly expensive. I can't imagine buying the hundreds of gallons it would take to cool an entire datacenter.

        Also, like Critical Facilities above mentioned, what do you do for heat dissipation in the event of a leak?

        • Re: (Score:2, Informative)

          by Anonymous Coward

          I can't speak for anyone else, but in our data center we have two sets of chilled water pipes and two sets of return pipes, in case one does break. They are valved all over the place so any one set of pipe can go down and the others continue working.

          As well, the chilled water never enters the data center. Our CRAC units sit outside of the data center and are ducted in overhead (because we can reach higher space velocities than with a raised floor). Thus, no chilled water ever gets inside of hte data cent

      • by mowall (865642)

        Liquid cooled, not water cooled. They cool it in an inert liquid, rather than water.

        No. From TFA: "One of our most important decisions was to invest in water-cooled racks from Rittal, which allows us to precisely control the amount of cooling that a specific rack requires," said Shand.

      • From the fine article:

        "That kind of density creates a cooling challenge. âoeOne of our most important decisions was to invest in water-cooled racks from Rittal, which allows us to precisely control the amount of cooling that a specific rack requires,â said Shand."

      • by Firehed (942385)

        Pure water is electrically inert. Though given how dusty servers tend to get, I doubt it really matters.

        Regardless, I've never had a watercooling leak that wasn't my own fault (not connecting the fitting properly), and obviously in this kind of scenario you test things first. Then again, I'm talking about a couple of boxes, not a datacenter.

        Fluorinert or something similar would be very effective and a whole hell of a lot safer for this. It's certainly not cheap, but given the fact that you can effectivel

        • by dbIII (701233)
          Pure water is pretty hard to achieve and manage - check out the water treatment areas at any thermal power station. Boiler water is pretty close to pure with often some fairly nasty additives to get the dissolved oxygen out.
      • by Brigadier (12956)

        it's only inert until it is mixed with some dirt/dust/grime/paint. :(

      • the system uses pure water with an anti freeze agent for winter
    • by MrMunkey (1039894) on Monday September 08, 2008 @11:35AM (#24920463) Homepage
      Would it be a good idea to replace the water with mineral oil? I'm sure a lot of you have seen the computers submerged in mineral oil, so it would probably just cause a mess if there was a leak. The problem with mineral oil (based on my limited knowledge from searching just now) is that it's not as efficient at removing heat as water, and after time the oil breaks down and needs replacing to remain at its most efficient. I'm not sure if the heat levels from a server would be high enough to degrade the oil though. I'm also uncertain if the oil would cause any damage to the pumps (or whatever pushes the liquid around) in the water cooling system.
      • by funaho (42567)

        Throw in some bikini-clad babes and you've got a supercomputer cluster that makes its own porn!

      • One other problem, if I'm not mistaken, is that it has a way of chemically altering the materials used in electrical insulation and rubber. Meaning, increased chances of shorts in cables that contain the stuff.

        Another that the potting/submerging discussion touched on not long ago is the fact that mineral oil climbs anything and everything when in motion, leaching out of even the tightest seals, which is the biggest cause of the "big mess" you mentioned.

      • by houghi (78078)

        Why not motor oil?

    • by plover (150551) * on Monday September 08, 2008 @12:00PM (#24920775) Homepage Journal

      I'm of a mixed mindset when it comes to water-cooled datacenters.

      On one hand, you've got the makings of a biblical scale disaster with all that water and electricity mixing.

      We had such a disaster many years ago.

      The coolant lines themselves circled the walls near the ceiling of the dinosaur pen. Beneath each line was a drip tray that was alarmed to sound in case moisture was detected. These drip trays ran the entire length of the coolant lines. That is, the entire length except for about six inches in the very corner, where it was too hard for the tray installers to get to because they were behind the conduit leading down to the main power transformer. I can only assume that all that conduit also made it difficult for the plumbers to properly solder the elbow in the cooling line as well.

      Of course there was only one place for the pipe to leak. The version of the tale I heard implies an impressive display of fireworks was seen by all present as the coolant entered the transformer.

      Fortunately, the loss of power did not damage the mainframes (except for their ability to run.) The rooftop generator was fired up, and in short order the mainframes were back on line.

      And in short order the generator engines stopped because the fuel tanks were kept almost empty, the plan being to fully fuel them only in case of need.

      Once the replacement fuel was delivered, the generators ran for only a few minutes before dying again, this time for good. The ancient fuel had congealed in the injectors once the engines had stopped running.

      We ran our data center for two weeks powered from a truck-based generator parked in the street while the electricians replaced the transformers and repaired the generators.

      I now hear the diesel generator being started every month or two, and run for a few hours.

      Rather than a biblical disaster, I'd say it was more like a Marx Brothers' movie.

      • by Bandman (86149)

        wow, that's rough, but thanks for sharing!

        Our generator test-runs every Thursday for a few minutes to make sure everything will be ok. You know, theoretically

        It doesn't always work out like that [blogspot.com]

      • by ginbot462 (626023)

        Reminds me of those pressure valves on water heaters. Periodically, one turns it on to check the line still flows. But, I've never met anybody that does. Not that it's likely to come up in a conversation.

        • by plover (150551) *

          Now that you mention it, I've actually done that. I discovered (the hard way) that mineral deposits built up inside the valve get dislodged when it's triggered, and sometimes those chunks then get in the way of sealing the valve again. Which meant a stream of hot water that couldn't be shut off until I shut off the house valve and couldn't be turned on again until I bought a new pressure relief valve.

          So now I don't do that anymore.

      • by conureman (748753)

        Big, empty fuel tanks tend to condense water from the atmosphere, creating crud and corrosion. Bad for the fuel system. I try to store my fuel tanks all topped-up.

        • by plover (150551) *
          Thanks, that explains why there was damage after the refueling. Someone told me "congealed fuel" but it didn't quite make sense. (I now assume that was the "mechanic-to-manager explanation" that made it out to the rest of us.) An near empty tank providing a concentrated mix of rust particles, impure water, and diesel fuel explains it much better.
    • by felipekk (1007591)

      On one hand, you've got the makings of a biblical scale disaster (...)

      On the other hand, you can't argue with it's effectiveness.

      Sounds like commercial aviation. It's very effective and considered one of the safest means of transportation. Yet, every once in a while you have a "biblical" disaster...

    • Re: (Score:3, Informative)

      by LWATCDR (28044)

      Well they may be using a different liquid than water. Mainframes have been using water cooling for years.
      But even if they are using water we are talking about professionals here. They have been water cooling electrical devices that are far more dangerous than CPUs for many years.
      http://www.cobermuegge.com/details.asp?id=90 [cobermuegge.com]
      And here http://en.wikipedia.org/wiki/Vacuum_tube#Cooling [wikipedia.org]

    • by zx-15 (926808)
      And the liquid is usually glycol, not water.
  • by darkmeridian (119044) <william@chuang.gmail@com> on Monday September 08, 2008 @11:16AM (#24920237) Homepage

    I think it's great that they are using liquid coolants for their system. Whenever I see a traditional server farm, I just can't help but think that air conditioners are so inefficient for the task of cooling computers. Not only do you have to cool the air, you also have to blow it around. The floors in some data centers are raised just to allow better airflow. And if you think about it, only the insides of the computers have to be cooled, not the entire freaking room. I hope this ushers in a new age of more power-efficient computing.

    I also think it's pretty funny that a supercomputer is used to make movies.

    • by Bandman (86149)

      I know what you mean, but I get nervous about that much water around my computers. A leak would be catastrophic.

      When my air conditioner breaks down [blogspot.com], I don't have a life threatening situation.

    • That darn Vinge... (Score:4, Informative)

      by jeffb (2.718) (1189693) on Monday September 08, 2008 @12:26PM (#24921037)

      I also think it's pretty funny that a supercomputer is used to make movies.

      It was pretty funny forty years ago [wikipedia.org], too.

    • by Kingrames (858416)
      "Not only do you have to cool the air, you also have to blow it around." It should be noted, air is a fluid that is more fluid than water. blowing it around isn't the problem. it's that the air doesn't carry heat away from the computer as fast, because liquids have more substance and can absorb more heat in the process. at least that would be my guess, I'm sure there are far more qualified quantum fluidicists in the discussion.
    • by aaarrrgggh (9205)

      The Rittal racks just have a coil in them; the servers are cooled with air.

      On a power basis, there isn't much of an efficiency improvement. Your biggest gain is if you take the filters out since the cabinet is sealed.

      As for a leak hitting the computers, the coil is in a sidecar which is designed pretty well to segregate leaks. Biggest concern is usually what happens when you lose water or fans. Most of the cabinets open the doors automatically.

    • What I don't get is why they cool the entire room. Why not have duct work leading from the AC to the bottom of a server shelving unit, with duct work on the top leading outside, and sucking the hot air out, then pulling cool air in.

    • by houghi (78078)

      So how much more efficient is it? The heat of the processors still needs to be removed.

  • by saikou (211301) on Monday September 08, 2008 @11:29AM (#24920391) Homepage

    I wonder how much they have to pay for external bandwidth. I always thought that "super data centers" are used to help split the job between multiple special effects studios, so, say, group in London can work on part of the shot and still have all data in the same place.
    Except in New Zealand there are no "unlimited" plans, and there are severe bandwidth caps in place.

    • by OttoM (467655) on Monday September 08, 2008 @11:59AM (#24920767)
      As if a datacenter like this would use a capped ADSL line... You do not have to use the public internet. At some point, it becomes cheaper to use dedicated connections between your offices and datacenters.
      • by Bandman (86149)

        Leased OC192 lines are probably available there. I'm willing to bet they're not on "broadband" too.

        • by jggimi (1279324)
          Yes, they were using "fat pipe" connections between Wellington and London during ROTK post-processing.
          • by Bandman (86149)

            I'm kind of surprised. It doesn't look [cnet.com] like NZ has any major bandwidth, compared to the rest of the world. Maybe they can get Google to hook them up [blogspot.com] to the pan-Asian cable going in soon. ;-)

            • by Boricle (652297)
              Southern Cross Cable Network [southerncrosscables.com]

              (as opposed to Southern Cross Network Cabling - in Golden, Colorado of all places! :)

              The Southern Cross Cable Network provides the fastest, most direct, and most secure international bandwidth from Australia, New Zealand and Hawaii to the heart of the Internet in the USA.

              Southern Cross is currently delivering 295 gigabit/s of fully protected bandwidth and has the potential as demand growth requires to increase to 1.2 Tbps using the existing 10 Gbps technology or 4.8 Gbps using 40 Gbps.

              • by Bandman (86149)

                Nice, thanks. I've heard of the Southern Cross network, but never researched it. Good link.

        • I don't know the specifics of how fast their connection is but they paid [stuff.co.nz] for 10 km of fibre to be laid out to their offices so no they wouldn't be on ADSL.
    • by felipekk (1007591)

      With all those tubes around you still think they are going to have bandwidth problems?

  • Just curious, but what does ILM run?
  • The LHCb experiment has a large processor farm for their online data analysis, all water cooled. Apparently it makes the computer scientists very nervous. OTOH, the main computing centre just uses air cooling, so we've got a real mix of technology.
  • I was working on a show which had a bunch of SGI Boxes similar to this. Two servers (2 dual core processors per server) in a 1U case. Only problem with it was that if something went wrong with one of two servers in the box you had to send the whole box back to SGI to get fixed.

    Now you would think that normally it wouldn't be a problem if 8 CPUs out of 200 go down on a farm but the way they usually set up the farm (in my experience) is by assigning some processors to each department using a priority s
    • by dbIII (701233)
      The old NASA pbs queuing system is still being updated as "torque". That is the sort of system you can use to distribute jobs in a variety of ways with a variety of priorities. You can define a virtual cluster just by adding the machines you want in it to a queue. A queue can be parallel as well as sequential - so you can just submit a pile of jobs to a queue and they will run all machines at once and step through the jobs as more cpu capacity becomes available.
  • The Weta render farms currently hold spots 219 through 222 on the current Top 500 list of the world's fastest supercomputers.

    That's a lot of orcs.

  • Richard showed up a couple years ago at the Wairarapa Railway Modellers annual BBQ with a 7 1/2" scale locomotive that was built at Weta Workshops for a film. In the photo located at http://www.certsoft.com/NZ2/richard_taylor.jpg [certsoft.com] Richard is the one in the middle.
  • I can't seem to find the video I'm looking for but I'm a fan of the new Sun Data Center cooling design. Works on the idea of cooling each cab individually and not all the rest of the rubbish air floating around. I know its been done before but this time its done right. Info somewhere here [sun.com]

To err is human -- to blame it on a computer is even more so.

Working...