Forgot your password?
typodupeerror
The Internet

NYC Data Centers Struggle To Recover After Sandy 231

Posted by Unknown Lamer
from the bucket-brigade dept.
Nerval's Lobster writes "Problems in New York's data centers persisted through Wednesday morning, with hosting companies and other facilities racing against time to keep generators humming as water was pumped out of their facility basements. The fight now is to keep those generators fueled while pumps clear the basement areas, allowing the standard backup generators to begin operating. It's also unclear whether the critical elements of infrastructure (power and communications) will both be up and running in time to restore services. The following is a list of some of the data centers and services in the area, and how they're faring." I'm responsible for a few servers at Peer1, and their efforts are interesting: "Peer1’s operations at 75 Broad are operating on sheer manpower: a bucket brigade. According to a blog post from Fog Creek Software, one of the clients at the building, about 30 customers are lifting buckets (or cans) of diesel fuel up 18 flights of stairs."
This discussion has been archived. No new comments can be posted.

NYC Data Centers Struggle To Recover After Sandy

Comments Filter:
  • The Cloud (Score:5, Funny)

    by azalin (67640) on Wednesday October 31, 2012 @12:04PM (#41831871)
    When it's wet, the clouds go down
    • On my netbook, I sinced all my work
      to the Cloud, the promised best way
      No storms over here, only sunny pleasant day
      my work is gone anyway.
      • by tlhIngan (30335)

        On my netbook, I sinced all my work
          to the Cloud, the promised best way
          No storms over here, only sunny pleasant day
          my work is gone anyway.

        There's your problem. A sunny pleasant day doesn't have clouds! So by definition your cloud is gone.

        Or put another way - cloud gone due to stretch of good weather.

  • Add to that, NYI... (Score:5, Informative)

    by malakai (136531) on Wednesday October 31, 2012 @12:08PM (#41831923) Journal

    At 100 Williams Street, http://www.nyistatus.com/ [nyistatus.com]
    My server and connections have been up non stop.

    I know it's cynical of me, but I find it a bit sad that we can better plan data centers then medical factilities [go.com].

    I know all the colocation facilities I've been to in Manhattan have generators above the 6th floor ( sometimes in addition to generators in the basement). A few had them on the roof with some special setup that allows fuel to be flown by helicopter for worse case scenarios.

    • by afidel (530433) on Wednesday October 31, 2012 @12:20PM (#41832107)

      NYU's generators were fine, it was their fuel supply that got fouled. Fire regs don't allow them to have thousands of gallons of diesel anywhere but underground holding tanks and those were overcome by seawater. Bellevue lost two of their primary generators due to water in the basement but was still running on another on the 13th floor but they had the same limited fuel problem Peer1 is running into. They considered having the national guard bucket brigade fuel up to the 13th floor but after some analysis it was decided it would be better to transfer folks to other hospitals (I'm not sure how many generators were on the 13th floor but it was probably only a single one and so they were down to a SPOF so better to transfer people in an orderly manner while you still have working facilities than to try it after the generator went down).

    • by Bronster (13157)

      Been really impressed with NYI - we haven't had a single glitch at FastMail either.

      We have an emergency backup plan (Iceland) - but it's nice not to have to use it.

    • by twisted_pare (1714106) on Wednesday October 31, 2012 @01:25PM (#41832915) Homepage
      Poor planning, plain and simple.

      I work for a major financial institution on the street. Various facilities were swamped, and we never missed a beat. What, were we just "lucky?" I don't think so.

      Starting a week ago we had disaster crisis centers setup.
      * Every few hours all East coast facilities reported in any issues
      * Inspection and testing of all critical systems ahead of time
      * Stockpiles of supplies on hand
      * Prefail over to DR where possible
      * All hands on deck to respond


      Sadly, if you want to be prepared, you can be. If tons of money is on the line, then the price of being prepared is well worth it. We test our systems continuously year round. We have disaster recovery drills at all facilities multiple times a year. Departments' rating depend on how well prepared they are for things like this.

      And don't throw that "1888," "worth storm ever" crap around. This is Wall Street. Manhattan. Terrorists have tried to blow it off the map multiple times. Several hurricanes have hit this spit of land that sits a mere few feet above sea level in the last decades. A hurricane hit and flooded parts last year even! If you did not prepare for this including flooding and sealed underground tanks and sandbag walls, it was your own fault.
      • by tibit (1762298)

        Ta-ta-ta-tah! We have a winner :) Couldn't agree more.

      • by jbmartin6 (1232050) on Wednesday October 31, 2012 @02:36PM (#41833967)
        Thank you! I thought the same thing about all the hospitals. I worked for a hospital for a while and believe me, they don't do disaster planning or even equipment life cycle planning. You would think with all the lip service paid to 'patient care' that continuity and disaster planning would be more of a priority, but no. Usually these aren't real businesses and they have no incentive to behave like one. As another poster mentioned, evacuation is usually their continuity plan.
      • by DerekLyons (302214) <fairwater@gmFREEBSDail.com minus bsd> on Wednesday October 31, 2012 @02:38PM (#41833993) Homepage

        And don't throw that "1888," "worth storm ever" crap around.

        Well, it's a fact. If you can't deal in facts, you shouldn't be working where you claim to be.
         

        I work for a major financial institution on the street. Various facilities were swamped, and we never missed a beat.

        You're lucky enough to be able to afford to have a complex defense in depth - this isn't true of everyone.

        So take your "I'm big bad dude" attitude, and stuff it. You're only big because daddy has money.

        • Re: (Score:2, Troll)

          by Ltap (1572175)

          Well, it's a fact. If you can't deal in facts, you shouldn't be working where you claim to be.

          I disagree. Dealing with facts would probably be an impediment to his job.

      • by artg (24127)

        This is Wall Street. Manhattan. Terrorists have tried to blow it off the map multiple times. Several hurricanes have hit this spit of land that sits a mere few feet above sea level in the last decades. A hurricane hit and flooded parts last year even!

        And that's the best site you can find for a datacentre ? Doesn't give me much confidence in your financial organisation's acumen.

    • by alphatel (1450715) *
      100 William and 111 Eighth Ave have both been planned for the worst quite appropriately. All the equipment at Telx and Level3 datacenters in both buildings are operational, though I did see some minor outages during the storm as systems switched over.
  • How much in fines from OSHA or the NY EPA are these companies looking at for the bucket brigade?
    • by vlm (69642)

      Usually the epic fail of "fuel in non-fuel rated areas" is the fire chief flips his lid, especially if the sprinkler system is down and/or you're transporting slippery oil via the emergency evac route.

      This is just "no cans of gasoline (for scooters, mopeds, etc) allowed in the dorms" writ large.

    • by sunking2 (521698)
      Stop being so literal, it says (or cans)
    • by CastrTroy (595695)
      I was wondering why they don't carry it up the side of a the building using a winch or window washing lift.
      • by bws111 (1216812)

        A winch or window washing lift powered by what?

        • by houghi (78078)

          human power, a rope and http://en.wikipedia.org/wiki/Pulley [wikipedia.org]

        • by CastrTroy (595695)
          As somebody else already said, human power. There's also the option of tapping into the generator. I don't know the specifics, but if it takes less than a gallon of fuel to lift a gallon of fuel up the side of the building, you could just tap into the generator you're trying to refuel. The other option is to have a second smaller generator on the ground that powers just the lift/winch. This could be easily refueled. Seems like bucket brigade was the first thing that came to mind, and once they had that goi
          • by hawguy (1600213)

            As somebody else already said, human power. There's also the option of tapping into the generator. I don't know the specifics, but if it takes less than a gallon of fuel to lift a gallon of fuel up the side of the building, you could just tap into the generator you're trying to refuel. The other option is to have a second smaller generator on the ground that powers just the lift/winch. This could be easily refueled. Seems like bucket brigade was the first thing that came to mind, and once they had that going, people stopped thinking of better ideas.

            Or maybe they've seen enough Bugs Bunny cartoons to know what happens when a makeshift winch fails - whatever it's carrying falls to the ground in a big splat, flattening whoever it falls onto.

            Hoisting a 30 pound gas can 200 feet in the air during a disaster is not the time for amateurs to jury rig something together.

          • by bws111 (1216812)

            You are making a lot of assumptions there, starting with the assumption that what you are suggesting is even legal. Next is that the people in charge of the building are also the people in charge of the datacenter (unlikely), and that the people in charge of the building rank your datacenter as a higher priority than everything and everyone else in the building.

            This is not some sort of heroic lifesaving operation where 'do anything possible' applies. It is just very bad disaster preparedness on the part

          • I don't know if you have ever driven a car, but you can move your gallon of fuel through many stories of elevation change (along with literally tons of steel) for maybe a teaspoon or two of combusted fuel.

            This is why everything still uses combustion engines (including series-hybrid trains which are driven by purely electric motors connected to diesel generators).

  • by Nyder (754090) on Wednesday October 31, 2012 @12:12PM (#41831991) Journal

    Seems sort of stupid to me to put generators in a basement, considering that your on the coast, surrounded by water, and hurricanes like to come thru every now and then. Maybe this doesn't happen all the time, I don't know. I live on the west coast. I just have to worry about volcano's. (and I don't worry about volcano's).

    • by war4peace (1628283) on Wednesday October 31, 2012 @12:29PM (#41832241)

      Volcano's what?

      • by mcgrew (92797) *

        Well, one of two things:
        1. Volcano's hot ash and lava, or
        2. He's a semi-literate who hasn't graduated junior high yet.

    • by aaarrrgggh (9205)

      If I recall correctly, NYC requires emergency generators to be at roof level so that a mid-rise fire would not cut off power to the upper floors. Most generators in flood prone areas are well protected-- we jumped through some odd little hoops to make generators work in Florida.

      HOWEVER, there is a limit to what you can economically protect against. Usually, you are looking at 100-year events as a basis, not 500 year events or 500 years plus 10%.

    • > Seems sort of stupid to me to put generators in a basement

      "Up north", basements are PRECISELY where you store things like gunky, messy generators (not necessarily with the full blessing of local officials) if you're in a big city like New York, in an old building that was built before elevators were mandatory and people still used coal for heat. They can't go on the roof, because they'd get damaged by the wind and rain. They can't go on the top floor, because it's the expensive penthouse. A newer build

    • by bobcat7677 (561727) on Wednesday October 31, 2012 @02:16PM (#41833671) Homepage
      They don't have a choice. Fire codes require the fuel tanks to be underground. It doesn't have to make sense, it's the law.
  • Why would anyone in their right mind place generators and tanks below ground where flooding would be an issue?
    • Why would anyone in their right mind place generators and tanks below ground where flooding would be an issue?

      Yeah, when you can put them on the roof... where... there is rain and wind...

      I suppose there is no winning here.

    • by Zeromous (668365)

      I'll field this one, slashdot.

      Managers. Lots of them.

    • Re:Poor Planning? (Score:5, Insightful)

      by Anonymous Coward on Wednesday October 31, 2012 @12:23PM (#41832139)

      Because having large tanks of diesel fuel dozens of stories above ground isn't a good solution either? Lightning...wind...spills...leaks...fires... all probably more statistically relevant than major flooding, and the consequences of failure far more disastrous than simply losing power in a flood. Even storing the tanks underground and the generators above ground has 2 problems: 1) you need power to pump the fuel up to the generator, which kind of defeats the purpose, and 2) high pressure fuel lines running through a building isn't exactly safe or desirable either.

    • by berashith (222128)

      I am hoping that the "hasn't happened since the 1880's" bit is partly to blame, but Japan's little issue last year definitely comes to mind. I thought the same thing. I also question hauling tanks of diesel up the fire escape. That should have been better thought out. I guess the firewalls work both ways as long as you dont plan on leaving.

    • by sunking2 (521698)
      Because below ground is convenient, the least expensive cost per square foot of floor space, and it's where all your crap you want to keep out of the way goes.
    • Re:Poor Planning? (Score:4, Insightful)

      by Baloroth (2370816) on Wednesday October 31, 2012 @12:38PM (#41832333)

      Why would anyone in their right mind place generators and tanks below ground where flooding would be an issue?

      Lets see how you feel with a few thousand gallons of highly flammable liquid suspended above your head, in a building with lots of electricity running through it, where an earthquake is more likely than flooding in the basement. And that is ignoring the possibility of deliberate sabotage. A building with fuel stored above ground level where something went wrong would turn rather quickly into a giant pillar of flame. If one of the tanks gets ruptured, all it takes is a single spark to kill hundreds or thousands.

      Below ground, however, fire-fighters can deal with it relatively easily, and the flames won't descend to engulf the entire building in a matter of a few minutes.

      • Re: (Score:2, Interesting)

        by Anonymous Coward

        Since when has Diesel been highly flammable? You can actually use it to put fires out. It takes quite some heat to get it going so really poses less risk than you average stationary cupboard.

        • by hawguy (1600213)

          Since when has Diesel been highly flammable? You can actually use it to put fires out. It takes quite some heat to get it going so really poses less risk than you average stationary cupboard.

          Until it leaks out, saturates wood framing and other building materials, then comes into contact with an ignition source (like a pilot light, or candles that the residents are using since the power is out).

          Then this hard to ignite fuel quickly turns an office into an inferno.

          I can knock a candle over in my stationary cupboard and as long as I pick it up quickly, I wouldn't expect a fire. Soak that same cabinet and knock a candle over and it's a different story. Kerosene (very similar to diesel) makes a good

      • Is an earthquake a more likely event than a flood in Manhattan?

        • by Zeromous (668365)

          Nope, once again +5 Insightful is meaningless.

          Generators should not be below ground, and at least 2-12 ft above ground if possible (or higher depending on risk).

          They should not be too high, lest you have trouble fueling them.,

          As with anything this likely came down to a retrofit risk assessment. The risk of fire or not making code is far higher than floods in manhattan, but being an island and subject to the ocean, still a high risk and should have been planned for. To me, it comes down to money at the e

    • by barc0001 (173002)

      Because they're required to by building code? That's what happened at the hospital that lost power. Tanks underground, water covers them, fuel gets contaminated, and poof, no more power.

    • The tanks are required to be below ground by the fire code. I think the better question is why put big data centers on a low lying coastline/island and/or a city with a giant target painted on it by every anti-american, anti-establishment, anti-whatever whacko the world over. Data centers belong in places with low risk of natural disaster, war, terrorist attack, riot or really anything that brings the police out.
      • by hawguy (1600213)

        The tanks are required to be below ground by the fire code. I think the better question is why put big data centers on a low lying coastline/island and/or a city with a giant target painted on it by every anti-american, anti-establishment, anti-whatever whacko the world over. Data centers belong in places with low risk of natural disaster, war, terrorist attack, riot or really anything that brings the police out.

        Because some companies want to be physically close to their datacenter. Virtualization and cloud computing is changing this... slowly.

  • by sl4shd0rk (755837) on Wednesday October 31, 2012 @12:32PM (#41832277)

    Why not move them to the roof? And while we're at it, do the same for all the nuke plants? A simple f*cking appliance that needs air and fuel to run and somehow they manage to spend life at the bottom of a potential indoor swimming pool.

  • by ThatsNotPudding (1045640) on Wednesday October 31, 2012 @12:36PM (#41832321)

    According to a blog post from Fog Creek Software, one of the clients at the building, about 30 customers are lifting buckets (or cans) of diesel fuel up 18 flights of stairs.

    Ahh: Diesel Control Protocol

  • Gas stations bury their gas tanks underground with few problems of water seepage. Granted, they aren't surrounded by water and there is little water pressure (i.e. covered by water). However, you would think that they would have at least waterproofed the fuel tanks.

    • by h4rr4r (612664)

      Even if you did, if the top of the tank is under water, how do you refuel it?

      You have X dollars and Y risks, you can never be prepared for everything.

      • That ones fairly easily dealt with after the fact. In fact many modern refueling trucks come equipped with air-tight locking mechanisms on the refueling lines. Diesel generators are inherently pretty fucking tough so if a gallon or two of water gets into a sealed tank while they're getting the refueling line hooked up, who cares? A few gallons won't make it out of the fuel tank all at once anyways and with the compression ratio on diesel turbines and engines it'll probably just cause some sputtering and van

    • Re:Waterproof... (Score:4, Interesting)

      by Remus Shepherd (32833) <remus@panix.com> on Wednesday October 31, 2012 @12:43PM (#41832407) Homepage

      The problem is that 'waterproofing' is a short-term guarantee. Water is insidious, it dissolves almost anything (although some things like metals very slowly), and it will eventually creep inside of any 'waterproof' container. That's why there's such a problem designing radioactive dumps like Yucca Mountain -- water would eventually eat its way into the vitrified radioactive cask.

      Gas station underground tanks can survive for 10-20 years and still be waterproof. But most of the infrastructure in NYC is a hundred years old. There isn't anything waterproof in that city. Even brand new structures are probably permeable to water, if the designers just never thought it would be an issue.

    • Re:Waterproof... (Score:5, Informative)

      by aaarrrgggh (9205) on Wednesday October 31, 2012 @01:15PM (#41832815)

      The bigger problem is usually the pumps. You generally try to use turbine-type submersible pumps with the motor above the tank and the inlet down low to avoid problems with priming the pump. If the place where the pump motor floods, you are pretty much SOL.

      If you place a suction pump 25' above the bottom of the tank to avoid flooding risks, you have the problem of priming the thing and maintaining suction. You could do a submersible pump with a really long shaft so the motor is high enough... but that would look really stupid.

      Ultimately, you have backups on backups in most data centers (and hospitals), but you often have a limited window to respond. We have an (illegal) 15-gallon gas can in one facility up by the generator. That can will give them about 9 minutes extra run-time if the day tank runs dry. There is a hand pump in the basement that can be used to manually pump the fuel up 50' to the generators, but if the room it is in is flooded what can you do?

      Big enough problems need disaster recovery plans; you will go down, the issue is how quickly you can return to normal operations.

    • Gas stations bury their gas tanks underground with few problems of water seepage.

      Are you sure? Because when I managed a gas station 20 years ago, I had to check the tanks daily for water, and generally had to pump the water out at least twice a month before seepage and condensation built up enough to reach the uptake pipe. Using a hand pump.

      And that's with gasoline, which floats on water - diesel absorbs water so you can't just pump it off the bottom of the tank. Look up "hygroscopic" and "diesel fuel dr

  • by Samuraid (824799) on Wednesday October 31, 2012 @12:41PM (#41832379)
    There's a detailed list of downed datacenters as well as a good discussion of status over at webhostingtalk: http://www.webhostingtalk.com/showthread.php?t=1205042 [webhostingtalk.com]
    Per the topic, the following locations are experiencing or have experienced outages:
    • 75 Broad Street
    • 33 Whitehall
    • nLayer at 882 3rd Ave
    • Voxel/Internap at 111 8th Avenue
    • XO, nLayer, Cogent, Verizon, Sidera Networks and AT&T at 882 3rd Ave
    • 121 Varick
  • by GeoGreg (631708) on Wednesday October 31, 2012 @01:03PM (#41832655)
    Like many other posters, my first question was why were the generators on upper floors but fuel (and pumps) in the basement? And as soon as I read the answer, it was completely obvious: fire codes. Duh. Thinking of how fuel is stored elsewhere, the only other option I can think of would be storing the fuel outside the building but above potential floodwaters. Not in a place like Manhattan. The price of real estate is much too high for tank farms on stilts. And the earthquake risk in New York is non-zero, so that solution might have the same problems as the current solution. So maybe the answer is that flood-prone urban areas are just not a good place for critical data infrastructure. Is relocating major data centers out of flood-prone areas of Manhattan (and other similarly risky areas) feasible? The potential of a major flood event in Manhattan has been well-known for a long time. Much of lower Manhattan is built on landfill. Did the builders of these data centers include basement flooding + extended power outage in their risk forecasts and just decide to deal with it if it happened?
    • The data centers are located downtown because that's where the banks and exchanges are. The banks and the exchanges originally built their data center close to them (this started in the 70's). Customers wanted to be as close to the servers as possible (and still do - high frequency trading) and it just kind of organically grew into what it is now. It also didn't hurt that AT&T and Verizon both have massive switch stations downtown and when these things were being built out high speed connections were
      • They are downtown because all the fiber terminates there (111 8th, 60 Hudson, 75 Broad). They originate from the teletype days. Closest point to the main meeting places is a good place to build data centers. Hence Ashburn, London, Frankfurt, etc.

    • by tibit (1762298)

      So maybe the answer is that flood-prone urban areas are just not a good place for critical data infrastructure.

      That's aggrandizing the problem, really. The solutions are quite simple.The fuel pumps and other fuel delivery and refill systems for the generators should be in waterproof enclosures and rated to operate while the basement is submerged. Alternatively, there should be provisions for feeding fuel from a tanker parked out on the street to a riser going to the upper floor generator(s). That's all there's to it. Why nobody thought of it, I wouldn't know.

    • by Guppy06 (410832)

      why were the generators on upper floors but fuel (and pumps) in the basement? And as soon as I read the answer, it was completely obvious:

      Because diesel fuel weighs around 55 lb/ft^3?

  • source: http://twitter.com/linode [twitter.com]

    Let's hope power is restored soon...

  • by concealment (2447304) on Wednesday October 31, 2012 @01:10PM (#41832745) Homepage Journal

    It looks like Google was ahead of the curve [nytimes.com] after all with their idea for floating server farms.

  • In*ter*es*ting - adj.
    1. capable of holding one's attention.
    2. arousing a feeling of interest.
    3. oh God, oh God, we're all going to die.

  • by Anonymous Coward

    Those of you with systems up and running with Peer1 should be thankful to the data center folks as well as the Squarespace staff who're pitching in big-time! If you want to know what's really going on, along with some pictures, check out their status page: http://status.squarespace.com

    Basically, to address other folks' various questions: Yes, the generators are on the roof (along with a small start-up + several hour supply of fuel), but most of the fuel supply is in the basement due to fire codes, etc.. T

    • by couchslug (175151)

      If you are involved in improving systems after the recovery, consider using pumps driven by hydrostatic (hydraulic) drive which would allow you to place pump drive motors well above flood zones. No electric motors below water level means nothing to short out. Hydraulic driven fuel pumps are often located IN aircraft fuel tanks, and it may be an aircraft pump or pumps would solve your problem. They move a LOT of fuel. The parts would be easy to get (tough hydrostatic pumps and motors have been standard farm

There is no royal road to geometry. -- Euclid

Working...