Forgot your password?
typodupeerror
Businesses IT Technology

NYC Data Center Needs Focus On Fuel 162

Posted by timothy
from the now-let's-talk-economic-fallacies dept.
Nerval's Lobster writes "Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel? In the wake of Hurricane Sandy, the challenges facing data center operators in the affected zones include pumping water from basements, waiting for utility power to be restored, and managing fuel-truck deliveries. And it's become increasingly clear which companies had the resources and foresight to plan for a disaster like Sandy, and which are simply reacting. Here's the latest on providers around the New York area." And remember, having fuel for machines sometimes only means it's time to start the manual labor.
This discussion has been archived. No new comments can be posted.

NYC Data Center Needs Focus On Fuel

Comments Filter:
  • Idiots (Score:2, Funny)

    by Anonymous Coward

    Data center on the coast - and they're surprised by what happened?

    Good God! I saw this coming for years - I guess I AM GOD!

    Listen to me my sheep, if you're on the coast, you will be flooded and wiped out by hurricanes!

    And more, my sheep, you will be knocked out by tornadoes in the Mid-West!

    And yet, my sheep - oh, fuck it! - my morons - you will be taken out by Earthquakes on the West Coast!

    And for you in the Middle East... You'll be taken out by terrorists!

    And in a few years by the Omicron Persei 7 peoples

  • ... you would know. The blogs about keeping data centers running in New Orleans during Katrina were incredible.

  • How can you not have a multi-day supply of diesel on hand?

    • Re: (Score:2, Insightful)

      by Anonymous Coward

      How can you not have a multi-day supply of diesel on hand?

      Some places did, but in the city, you can't store it above ground - which means that that week's supply of diesel is now mixed in with the ocean.

      • by rickb928 (945187)

        In the Northeast, diesel and #2 are pretty similar. Why not switch the valve and send some furnace oil to the gensets?

        Or do they not use oil in NYC?

        • #2 is diesel.

        • You need a little more compression to light #2 fuel oil, the engine in the duece and a half [wikipedia.org], LDS-427 Turbo Multifuel I-6; LDS-465 Multifuel I-6; 127 to 170 hp (95 to 130 kW) will burn it OK, but they are small for data center usage. I'm sure there are marine engines off the shelf that are both large enough and able to run on both.

      • by mcgrew (92797) *

        I don't understand why they're using deisel rather than natural gas. With natural gas they could have the generators on the roof. Illinois Secretary of State mainframes have two backup generators, both natural gas. I've lost electricity a whole lot of times, but I've never had a natural gas disruption in my six decades. Not even in 2006 when we were hit with two tornados and electricity was out for a week; I heated my apartment with the oven (the gas furnace needs electricity for blowers and theromostats).

  • by Sarten-X (1102295) on Thursday November 01, 2012 @03:26PM (#41846313) Homepage

    I was under the impression that a fuel supply was a standard part of the contingency plan for any data center.

    I recently visited a new data center opening near me. The operator had contracts with several fuel suppliers that in the event of a power outage, the first one to get a full tank truck through their front gate got paid, and would keep getting paid for each additional truck that was needed. Any latecomers would be turned away, effectively making it an exclusive contract upon arrival.

    • by berashith (222128)

      You also have to hope that the trucks are allowed to travel. I worked at a DR company that sold contracts on datacenter in a trailer solutions. Many companies that got hit by Katrina declared, and not one single trailer was sent. The contract had provisions that fuel for the generators had to be available. There were going to be problems, so nothing was sent to assist them. Basically free money for the potential use of these trailers.

    • by rickb928 (945187)

      Driving tank trucks around any of the Boroughs seems like a pretty tough proposition even today. In a real disaster, plan to not be able to even call the fuel dealers and ask if they are coming, much less getting them through the main gate and down the service road.

    • Only works if the fuel truck operators have nowhere else to be.
      Otherwise, why woukd you turn up without a guarantee of payment.
      Unless a massive premium is offered.

    • by Ogive17 (691899)
      That's a silly solution considering how many people/companies would be trying to procure that same fuel.

      If the fuel isn't already stored on-site, you're screwed.
  • by gestalt_n_pepper (991155) on Thursday November 01, 2012 @03:29PM (#41846345)

    Once refined, it degrades. Oxidation, bacteria. Algae. Amazing things grow in diesel. You can add preservatives, but these only go so far. Keep it too long, and it's unusable. If you don't use it, you have to dispose of it. There's a large disincentive to keep diesel around.

    • by Guppy (12314)

      Once refined, it degrades. Oxidation, bacteria. Algae. Amazing things grow in diesel. You can add preservatives, but these only go so far

      I wonder if it would be economical to manufacture a grade of diesel-compatible fuel that doesn't degrade, for purposes of long-term storage. It would consist purely of highly refined aliphatics, leaving no unsaturated bonds available for oxidative attack. Heavy metal content would need to be near-zero as well, as they could function as catalysts for breakdown, or nutrients for microbes. The storage air-space above it would be purged with inert gas or at least desiccated, again to avoid microbial growth.

      E

      • by adolf (21054)

        If a liquid paraffin similar to medical grade mineral oil (but a lighter fraction) will keep a modern Diesel (yes, with a capital D) happy and survive long-term storage, then the problem is you: Go forth and sell.

  • What's the point in trying to keep these servers operating, and risking another fire hazard. Not to mention all the pollution and contamination. I'd be renting a bunch of cloud servers on Amazon, or move what you can to another location.
  • by hawguy (1600213) on Thursday November 01, 2012 @03:32PM (#41846377)

    Everyone already knew that the on-site fuel supply is the limiting factor of power availability in a disaster. Even fuel delivery contracts mean nothing in a disaster or wide-spread outages - hospitals, EMS and other government services will trump the fuel delivery contract, if a hospital needs fuel, they are going to get the fuel that's been "guaranteed" for your datacenter.

    There's no reason to spend big $$ creating a flood proof, earthquake proof, tornado proof, airplane crash proof datacenter in the middle of a city when you can have a disaster recovery site 1000 miles away that's not subject to the same type of disaster. (except maybe an asteroid strike, but there are few datacenters on the moon). No matter how disaster-proof you make your datacenter, mother nature (or man) will always find a way to create a disaster you didn't plan for -- even if that "disaster" is a typo in a router configuration file that takes down the network, or a contractor accidentally shorting out the emergency power cutoff switch wiring when bolting a rack to the wall.

    • by tnk1 (899206)

      If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York. Now, of course, you generally want a backup site that is more than 100 miles away, in the event of actual disasters, but you definitely want your primary facilities as close to your users as you can get.

      • by hawguy (1600213)

        If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York. Now, of course, you generally want a backup site that is more than 100 miles away, in the event of actual disasters, but you definitely want your primary facilities as close to your users as you can get.

        Sure, there will always be datacenters in NYC, but that doesn't change the fact that instead of putting all your money into trying to build a datacenter impervious to all hazards, you're better off having a second site far from your primary site. Real-world constraints mean that you can't build that perfectly impervious datacenter when you're subject to real estate prices, building codes, fire codes, and Murphy's Law -- there will always be a disaster that the facility can't handle.

      • If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York.

        I don't think OP was saying that you shouldn't have datacenters in NYC, persay, but rather that you'd be better off spending most of your "disaster-proofing" budget on off-site disaster recovery equipment/services.

        Don't put all your eggs in one basket, and such.

    • by Hillgiant (916436)

      ...there are few datacenters on the moon...

      The latency is a real bitch.

    • by girlintraining (1395911) on Thursday November 01, 2012 @04:39PM (#41847065)

      I would have thought it was obvious, but the middle of the city is where the telecommunications infrastructure is. It doesn't exist in a barn a hundred miles from any major city. And fuel has a shelf life. Diesel will slowly oxidize over time, and so the time you can keep it in the tank is about 12 months. You'll burn about 72 gallons an hour per megawatt (as a rough average). So a 2 megawatt data center will need about 3,500 gallons of diesel per day. A gallon takes up 231 cubic inches of space, so a single day's worth of fuel would need a tank with a capacity of 67,375 cubic feet. The average height of a floor in a skyscraper is 12.5 feet. In New York city, the average city block is 264 feet. That means that even if you filled an entire floor of a skyscraper with nothing but diesel fuel, you'd still get less than a week's worth of fuel guys. At a rate of perhaps $100 per month per square foot... you're talking about $83 million a year for a floor of an outlying area just to store that diesel fuel. Mind you, near Wall St., that price is probably going to be double or triple. The price of fuel is peanuts compared to this; 7 days of fuel for a 2MW plant would cost you only $94,000, plus transport costs.

      So as you can see, this isn't a question of them not stocking enough fuel -- the cost of storing that fuel is prohibitively expensive.

      And that, people, is why they didn't load up on fuel ahead of the storm. You can't simply pay $83 million a year for your data center to protect against a threat that might only materialize once a decade, and be severe enough to deny fuel deliveries or power restoration for that long of a time frame. Generator backup is a short term solution. There is no long term solution for disaster recovery, at least not one that's cost effective. Not in an urban setting.

      The only time you can justify spending that kind of cash is if you're supporting critical infrastructure like phones, hospitals, and emergency services. Everyone else plans for a couple day supply and leaves it at that.

      • by hawguy (1600213)

        I would have thought it was obvious, but the middle of the city is where the telecommunications infrastructure is. It doesn't exist in a barn a hundred miles from any major city. And fuel has a shelf life. Diesel will slowly oxidize over time, and so the time you can keep it in the tank is about 12 months. You'll burn about 72 gallons an hour per megawatt (as a rough average). So a 2 megawatt data center will need about 3,500 gallons of diesel per day. A gallon takes up 231 cubic inches of space, so a single day's worth of fuel would need a tank with a capacity of 67,375 cubic feet. The average height of a floor in a skyscraper is 12.5 feet. In New York city, the average city block is 264 feet. That means that even if you filled an entire floor of a skyscraper with nothing but diesel fuel, you'd still get less than a week's worth of fuel guys. At a rate of perhaps $100 per month per square foot... you're talking about $83 million a year for a floor of an outlying area just to store that diesel fuel. Mind you, near Wall St., that price is probably going to be double or triple. The price of fuel is peanuts compared to this; 7 days of fuel for a 2MW plant would cost you only $94,000, plus transport costs.

        While I agree with your point in general, your math is off. I can see our own 1MW generator and 3-day fuel tank from my office, which is no where near the size of tank you calculated -- it should consume around 1/4 of a city block using your figures.

        A week of fuel (24,000 gallons) is 3200 cubic feet [google.com], with a 10 foot high tank, that's only 320 square feet, at $100/month that's still a pricey $384,000/year. But since that 2MW of power is enough to power 1000 servers (assuming 1000 watts/server, half the powe

        • I just googled to source the numbers to get a ballpark figure. So yeah, undoubtedly they're off -- people spend months preparing reports for their data center on fuel consumption, storage costs, location, etc. I spent 20 minutes. But putting the numbers together shows that even if you lowball all the numbers by 50%, storage cost is still massively eclipsing fuel cost.

          People on slashdot here were being 'armchair CTOs' and saying how if it was their data center, they'd have bought all that extra fuel. "Well

          • by hawguy (1600213)

            I just googled to source the numbers to get a ballpark figure. So yeah, undoubtedly they're off -- people spend months preparing reports for their data center on fuel consumption, storage costs, location, etc. I spent 20 minutes. But putting the numbers together shows that even if you lowball all the numbers by 50%, storage cost is still massively eclipsing fuel cost.

            Sure, I understand the fuel usage numbers were an estimate, but your calculations weren't off by 50% - you were claiming that it would take an entire 264x264 foot office floor (70,000 square feet) to store a week's worth of fuel. I was pointing out that it only takes about 300 square feet - you were off by a factor of 233, not by 50%. Instead of an unaffordable $83M in rent and an entire office floor to store a week's worth of fuel, it only takes an 18 foot by 18 foot room and $400K/year. That's a huge di

  • by Builder (103701) on Thursday November 01, 2012 @03:36PM (#41846419)

    Cell sites I worked in Africa run a pair of Cummins generators as their main power. In the unlikely event that both of these fail at the same time, there's a chance that the main grid might be working well enough to take over. But the fuel is the biggest priority on these sites.

    At the moment, we have tankers on site at all of our NY DCs. 2 of them are on generator only, so we're topping up the tanks every 4 hours. The generators will need full overhauls when we're back on the grid properly, but for now we're keeping our clients serviced which is what matters.

    • by BigDish (636009)

      I've heard this before, but I've never seen the reasoning - why do the generators need full overhauls after running for a few days? Aren't these the same engines that are in trucks/construction equipment and run for thousands of overalls without an overhaul?

      • by hawguy (1600213)

        I've heard this before, but I've never seen the reasoning - why do the generators need full overhauls after running for a few days? Aren't these the same engines that are in trucks/construction equipment and run for thousands of overalls without an overhaul?

        When i worked at some remote construction sites, they had 1MW+ generators that went for over a year of nearly continuous use between overhauls. As I recall, they did oil changes and other routine maintenance monthly.

        An emergency power generator might be run at a higher load than a continuous duty generator, but I can't imagine any large diesel generator needing an overhaul after a few hundred hours of operation.

      • by Builder (103701)

        In this case, it's because the generators got a little damp due to some rain in the area just before going into service :D

        Given the amount of things that have happened, it's a 'better safe than sorry' move to do the overhaul.

  • by NinjaTekNeeks (817385) on Thursday November 01, 2012 @03:37PM (#41846423)
    If this was their first time, or it had been a few years, I would expect there to be mass hysteria and a general failure of DR plans. Mostly because DR plans are theoretical and costly to test and not tested very often. However I think Irene gleaned much useful information for those developing these DR plans which led to NY data centers being better prepared. I'd love to read the DR plans in 6 months and compare the new changes from lessons learned during and after this storm.
  • Ok, so this is a geeknet story submitted by a geeknet employee.

    Why make it look like a user submitted the article?

    Not that it actually matters.

  • (Please keep in mind that I write this with a somewhat cruddy memory as to historical events.) I'm on the fence with how to feel on this one. Being SO dependent on computers and data centers is a relatively recent phenomenon. Not saying they haven't been important before, but it's really blossomed over the last 10 to 15 years. Prior to Katrina in the gulf region, I can't recall a storm of this magnitude in the last decade. (Although the same northeast region got hit with "Hazel" back in 1954) And this
    • by na1led (1030470)
      Don't put all your eggs in one basket, that's the best plan.
      • But there again, budget comes into play. Can be pretty costly to maintain a backup data center in another state though. Infrastructure, hardware, good data lines connecting the two, etc.
        • Don't put all your eggs in one basket, that's the best plan.

          But there again, budget comes into play. Can be pretty costly to maintain a backup data center in another state though. Infrastructure, hardware, good data lines connecting the two, etc.

          Then you'll get what you (are willing to) pay for.

          • And colo providers get driven down on cost as customers forget the previous catastrophic event. Every dollar spent enhancing a data center is passed along to the customer in some fashion. When your per-cab or per-foot pricing is higher than the competition--even if you can demonstrate more resiliency--it is a harder sell.

            You can talk flood plains, gen sets, fuel locations and delivery schedules, CRAH and chiller redundancy, etc. As long as the SSAE16 or SOC 2 is current and there are no lingering memories o

      • by rickb928 (945187)

        There is only one Wall Street.

    • by berashith (222128)

      This is very easy to plan for. Simply decide how far away your DR location needs to be. If the problem is fuel, then the DR plan was a shelter in place plan, and is going to suffer miserably for many reasons. Long term power outage caused by floods is very obvious, and will bring the exact issues that are being seen here every time. This is likely in a plan for all of these places. A fire would knock these completely offline, so there always has to be consideration of failing to an alternate location. The m

  • Natural Gas (Score:2, Insightful)

    by Anonymous Coward

    I was visiting 1 summer st once, and I asked the guy who set the place up "why don't you use natural gas? You can just get it from your local utilities, or have it delivered by truck." and he looked at me with the "you poor idiot" look people give me some times. Then he said "Nobody uses natural gas because you can't tell if the supply upstream has been disrupted."

    I didn't feel like explaining "no, dude, you pump it into a pressurized tank and monitor the flow, *and* you have a diesel generator as a backu

    • Water pipes are also under ground and those break.
    • by heypete (60671)

      Having large quantities of pressurized natural gas can be an explosion risk. Diesel's just a fire risk. Storing pressurized natural gas likely involves a lot of regulatory issues that diesel doesn't.

      Also, natural gas is not very efficient volumetrically: diesel has 37.3 MJ/liter. Natural gas at normal pressure has 0.0364 MJ/liter. Even compressed to 250 bar (3,600 psi), a non-trivial pressure, it only has an energy density of 9 MJ/liter. Liquified natural gas at -160C has an energy density of 22.2 MJ/liter.

    • Lastly -- why the heck did these guys put the pumps *outside* the tank? Why not put them in the tank?

      ever tried to service a pump that's inside a tank? real pig to replace them... far easier if they're outside the tank as you can have shut-off valves either side and isolate them while replacing them... you don't have to drain the tank and purge it of fumes first before accessing the pump...

      • by adolf (21054)

        SOP on gasolineautomotive fuel pump replacements is, if possible, to do so with a full tank to minimize the risk of badness (gas itself doesn't burn, while gasoline vapors both burn and can be explosive).

        What is it about a large volume of Diesel fuel that makes this different in any meaningful way?

  • Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel?

    I assume everybody knew. I knew and I am not even in the data center business.
    This is not just some afterthought that I suddenly thought: well that seems logical.
    I talked to people who worked in data centers and when we talked about contingency, they all had examples where there was an issue with the fuel supply. Most where the same story, probably as it never happened to them, yet

  • by WayfinderSteve (2659663) on Thursday November 01, 2012 @04:03PM (#41846701)

    One of my buds is IT director for a company that resupplies generators. The logisitics for it are crazy as you route trucks on available streets, deal with priority of the customers (hospitals front of the line) and optimize resupply into mostly empty tanks before they actually empty, etc... And you have >10 days of this 24/7 after a storm.

    After Hurricane Allison here, some companies in this sector went out of business before the power came back on. We were better prepared for Ike, but I think that's because facilities are more willing to sign contracts with the pricey but reputable businesses.

    This is the most expensive way to get fuel: you need a massive amount for a short time, and you need it consistently during that time.

    • by tlhIngan (30335)

      This is the most expensive way to get fuel: you need a massive amount for a short time, and you need it consistently during that time.

      The problem has been getting it from where you fuel up the tanker, to where it's needed. The fuel's cheap comparatively, but the trucks aren't really off-road vehicles. When you've got flooded streets, streets lined with debris, or streets that are completely gone, the trucks really can't make it in to resupply.

      So the massive amount of fuel needed is needed right when the ro

      • by knarf (34928)

        but the trucks aren't really off-road vehicles.

        That's not a fuel truck. This [badhaven.com] is a fuel truck.

    • After Hurricane Allison here

      I think you mean Tropical Storm Allison. Man that thing flooded the hell out of Houston. Then went on to travel over land all the way to the mid-Atlantic States flooding pretty much everywhere it went. Following this the name Allison was retired from the Hurricane name list. It remains the only name to be retired from the list for a named storm that never reached Hurricane strength.

  • Natural Gas diesels (Score:5, Interesting)

    by fleebait (1432569) on Thursday November 01, 2012 @05:20PM (#41847511)

    30 years ago, I retired from the Navy, along with a friend who was a Machinist Mate on Navy submarines. I didn't see him for a couple years. Turns out that he had spent 10 of those years servicing batteries on diesel submarines, and ended up servicing large batteries for the phone company -- about the same size batteries that had been on the boats. He switched to servicing the diesels (because of his prior experience on submarines with their diesel generators), and one of the first to deploy natural gas fired diesels for sustained power, first for the phone company, and then for a local utility.

    There's the solution. Don't pipe flammable liquid to the top floor, pipe the natural gas. Contrary to popular belief, it is a safer fuel, and requires less maintenance. Automatic shutoff valves work better, less explosive volume released on a tubing breakage, and it doesn't rot the pipes like liquid diesel fuel, disperses to atmosphere in the case of a leak, and doesn't make everything around it flammable, when it does leak.

    Engines run at least 4 times as long between between overhaul cycles, and it doesn't dilute the lube oil.

    Natural gas is going to be a HUGE change for this countries infrastructure, both in common usage, as well as emergency failover services.

    This guy has made tons of money, by the way. Has a condo and car in San Diego, LA, Seattle, New York, and Atlanta -- cheaper than hotels and taxis, and doesn't know what else to do with his money.

    • by evilviper (135110)

      Natural gas is going to be a HUGE change for this countries infrastructure, both in common usage, as well as emergency failover services.

      I'm generally enthusiastic about natural gas, primarily because they can safely run unmanned for extended periods, but let's be honest, there are plenty of problems... Not only do the generators need to be bigger and more expensive, but you've gotta have a back-up plan in case those natural gas lines are either damaged or shut-down.

      Propane isn't a direct substitute, but d

  • At my work we had our generators on the roof above 6th floor. It had a small tank that can run for a couple of hours. In the basement, a larger tank that had fuel for 2 days.
    Sadly when the power went away, so did the power to the pump that were supposed to lift it to the root. So the our electrician all of the suddon got busy.

    At another workplace we had two huge generators with flywheels for starting them, plenty of fuel and even our own redundant line to the powerplant some miles away. Sadly when the insta

  • Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel?

    Uhm, anyone that was conscious for Katrina...

  • We found these heaps of backup tapes down in the basement. We just shoveled them into the boilers and kept the generators running.

1 Mole = 007 Secret Agents

Working...