NYC Data Centers Struggle To Recover After Sandy 231
Nerval's Lobster writes "Problems in New York's data centers persisted through Wednesday morning, with hosting companies and other facilities racing against time to keep generators humming as water was pumped out of their facility basements. The fight now is to keep those generators fueled while pumps clear the basement areas, allowing the standard backup generators to begin operating. It's also unclear whether the critical elements of infrastructure (power and communications) will both be up and running in time to restore services. The following is a list of some of the data centers and services in the area, and how they're faring."
I'm responsible for a few servers at Peer1, and their efforts are interesting: "Peer1’s operations at 75 Broad are operating on sheer manpower: a bucket brigade. According to a blog post from Fog Creek Software, one of the clients at the building, about 30 customers are lifting buckets (or cans) of diesel fuel up 18 flights of stairs."
Re:generators in basements, smart or not? (Score:0, Insightful)
Seems sort of stupid to me to put generators in a basement, considering that your on the coast, surrounded by water, and hurricanes like to come thru every now and then. Maybe this doesn't happen all the time, I don't know. I live on the west coast. I just have to worry about volcano's. (and I don't worry about volcano's).
Maybe you'd understand better if you learned where to use a fucking apostrophe.
Re:Poor Planning? (Score:5, Insightful)
Because having large tanks of diesel fuel dozens of stories above ground isn't a good solution either? Lightning...wind...spills...leaks...fires... all probably more statistically relevant than major flooding, and the consequences of failure far more disastrous than simply losing power in a flood. Even storing the tanks underground and the generators above ground has 2 problems: 1) you need power to pump the fuel up to the generator, which kind of defeats the purpose, and 2) high pressure fuel lines running through a building isn't exactly safe or desirable either.
Why are generators in the basement? (Score:4, Insightful)
Why not move them to the roof? And while we're at it, do the same for all the nuke plants? A simple f*cking appliance that needs air and fuel to run and somehow they manage to spend life at the bottom of a potential indoor swimming pool.
Re:Poor Planning? (Score:4, Insightful)
Why would anyone in their right mind place generators and tanks below ground where flooding would be an issue?
Lets see how you feel with a few thousand gallons of highly flammable liquid suspended above your head, in a building with lots of electricity running through it, where an earthquake is more likely than flooding in the basement. And that is ignoring the possibility of deliberate sabotage. A building with fuel stored above ground level where something went wrong would turn rather quickly into a giant pillar of flame. If one of the tanks gets ruptured, all it takes is a single spark to kill hundreds or thousands.
Below ground, however, fire-fighters can deal with it relatively easily, and the flames won't descend to engulf the entire building in a matter of a few minutes.
Re:Why are generators in the basement? (Score:5, Insightful)
Depending on the generator size, there's the weight issue. There's also fire regulations. 500 gallons of Diesel on the roof + fire == cascading burning diesel fuel.
Manhattan unsuitable for data centers? (Score:5, Insightful)
Floating server farms (Score:5, Insightful)
It looks like Google was ahead of the curve [nytimes.com] after all with their idea for floating server farms.
Re:Add to that, NYI... (Score:5, Insightful)
I work for a major financial institution on the street. Various facilities were swamped, and we never missed a beat. What, were we just "lucky?" I don't think so.
Starting a week ago we had disaster crisis centers setup.
* Every few hours all East coast facilities reported in any issues
* Inspection and testing of all critical systems ahead of time
* Stockpiles of supplies on hand
* Prefail over to DR where possible
* All hands on deck to respond
Sadly, if you want to be prepared, you can be. If tons of money is on the line, then the price of being prepared is well worth it. We test our systems continuously year round. We have disaster recovery drills at all facilities multiple times a year. Departments' rating depend on how well prepared they are for things like this.
And don't throw that "1888," "worth storm ever" crap around. This is Wall Street. Manhattan. Terrorists have tried to blow it off the map multiple times. Several hurricanes have hit this spit of land that sits a mere few feet above sea level in the last decades. A hurricane hit and flooded parts last year even! If you did not prepare for this including flooding and sealed underground tanks and sandbag walls, it was your own fault.
Re:Your priorities are NOT all messed up!! (Score:5, Insightful)
Re:Add to that, NYI... (Score:5, Insightful)
A First-Hand, On-Site Ongoing Account... (Score:2, Insightful)
Those of you with systems up and running with Peer1 should be thankful to the data center folks as well as the Squarespace staff who're pitching in big-time! If you want to know what's really going on, along with some pictures, check out their status page: http://status.squarespace.com
Basically, to address other folks' various questions: Yes, the generators are on the roof (along with a small start-up + several hour supply of fuel), but most of the fuel supply is in the basement due to fire codes, etc.. They switched over smoothly to generator power but the pumps to get fuel up from the basement went offline. They've been doing the bucket brigade since, and they did have various backup plans in place (and even a basic plan for a flooded basement & no grid power), but everything is basically in worst-case-scenario-mode right now. That is, difficulty getting fuel trucks to re-supply (though that's OK now), no air-lift available for the refueling on the roof, accessible backup fuel pumps not powerful enough and the right equipment can't get to them because of conditions throughout the rest of the city, and they've been trying to clear the basement of water for a good while now but it appears that in addition to the flooding there may be a water main break that's pushing water back into their basement as fast as the pumps take it out.
Given the situation, they're doing a hell of a job!
Re:Add to that, NYI... (Score:4, Insightful)
Well, it's a fact. If you can't deal in facts, you shouldn't be working where you claim to be.
You're lucky enough to be able to afford to have a complex defense in depth - this isn't true of everyone.
So take your "I'm big bad dude" attitude, and stuff it. You're only big because daddy has money.
Re:Add to that, NYI... (Score:4, Insightful)