NYC Data Centers Struggle To Recover After Sandy 231
Nerval's Lobster writes "Problems in New York's data centers persisted through Wednesday morning, with hosting companies and other facilities racing against time to keep generators humming as water was pumped out of their facility basements. The fight now is to keep those generators fueled while pumps clear the basement areas, allowing the standard backup generators to begin operating. It's also unclear whether the critical elements of infrastructure (power and communications) will both be up and running in time to restore services. The following is a list of some of the data centers and services in the area, and how they're faring."
I'm responsible for a few servers at Peer1, and their efforts are interesting: "Peer1’s operations at 75 Broad are operating on sheer manpower: a bucket brigade. According to a blog post from Fog Creek Software, one of the clients at the building, about 30 customers are lifting buckets (or cans) of diesel fuel up 18 flights of stairs."
The Cloud (Score:5, Funny)
Re: (Score:3)
to the Cloud, the promised best way
No storms over here, only sunny pleasant day
my work is gone anyway.
Re: (Score:3)
There's your problem. A sunny pleasant day doesn't have clouds! So by definition your cloud is gone.
Or put another way - cloud gone due to stretch of good weather.
Re: (Score:2)
Sandy blows through
Generators in flooded basement
The Cloud
Add to that, NYI... (Score:5, Informative)
At 100 Williams Street, http://www.nyistatus.com/ [nyistatus.com]
My server and connections have been up non stop.
I know it's cynical of me, but I find it a bit sad that we can better plan data centers then medical factilities [go.com].
I know all the colocation facilities I've been to in Manhattan have generators above the 6th floor ( sometimes in addition to generators in the basement). A few had them on the roof with some special setup that allows fuel to be flown by helicopter for worse case scenarios.
Re:Add to that, NYI... (Score:5, Interesting)
NYU's generators were fine, it was their fuel supply that got fouled. Fire regs don't allow them to have thousands of gallons of diesel anywhere but underground holding tanks and those were overcome by seawater. Bellevue lost two of their primary generators due to water in the basement but was still running on another on the 13th floor but they had the same limited fuel problem Peer1 is running into. They considered having the national guard bucket brigade fuel up to the 13th floor but after some analysis it was decided it would be better to transfer folks to other hospitals (I'm not sure how many generators were on the 13th floor but it was probably only a single one and so they were down to a SPOF so better to transfer people in an orderly manner while you still have working facilities than to try it after the generator went down).
Re:Add to that, NYI... (Score:5, Informative)
in the case that con-ed shuts off nat gas flow due to fires (which is happening all over the place right now...), do you want your generator to choke out?
that leaves lp. diesel packs about a 50% higher energy density than lp. which would you prefer to use? further, diesel is much more commoditized than lp, and is more readily available during a crisis.
My favorite generator (Score:4, Informative)
There are commercial generators which run on a variable mix of natural gas and diesel. With such a set, you can greatly extend your runtime by reducing the diesel percentage to a minimum when natural gas is available. Then if or when the natural gas goes out you can run them on 100% diesel and you're no worse off.
Re:Add to that, NYI... (Score:5, Informative)
why diesel? any generator worth it's salt is running on Natural Gas or Propane.
Obvious answer: Natural gas lines are usually cut during emergencies like flooding, fires, etc. Storing a liquid fuel like diesel allows you use the generator when external energy sources have been severed.
Re: (Score:2)
Because during a disaster utilities including natural gas aren't guaranteed to be up, plus if diesel is considered a fire hazard then natural gas is probably verboten. Oh, and high pressure natural gas of the type needed for 100kw+ generators isn't available everywhere. Diesel generators also tend to be cheaper per rated and true kw.
Re: (Score:2)
Hand pump? What kind of an idiot thought that up? You need a pump with a foot crank drive. A.k.a. a pickup off an exercise bike or somesuch.
Re: (Score:3)
You are never allowed to carry fuel in an elevator. The fire department would red tag the building if you were caught doing it.
Spilling fuel in an elevator shaft makes for a very effective chimney.
Re: (Score:3)
So instead you risk spilling it down a stairwell?
Re: (Score:3)
Been really impressed with NYI - we haven't had a single glitch at FastMail either.
We have an emergency backup plan (Iceland) - but it's nice not to have to use it.
Re:Add to that, NYI... (Score:5, Insightful)
I work for a major financial institution on the street. Various facilities were swamped, and we never missed a beat. What, were we just "lucky?" I don't think so.
Starting a week ago we had disaster crisis centers setup.
* Every few hours all East coast facilities reported in any issues
* Inspection and testing of all critical systems ahead of time
* Stockpiles of supplies on hand
* Prefail over to DR where possible
* All hands on deck to respond
Sadly, if you want to be prepared, you can be. If tons of money is on the line, then the price of being prepared is well worth it. We test our systems continuously year round. We have disaster recovery drills at all facilities multiple times a year. Departments' rating depend on how well prepared they are for things like this.
And don't throw that "1888," "worth storm ever" crap around. This is Wall Street. Manhattan. Terrorists have tried to blow it off the map multiple times. Several hurricanes have hit this spit of land that sits a mere few feet above sea level in the last decades. A hurricane hit and flooded parts last year even! If you did not prepare for this including flooding and sealed underground tanks and sandbag walls, it was your own fault.
Re: (Score:2)
Ta-ta-ta-tah! We have a winner :) Couldn't agree more.
Re:Add to that, NYI... (Score:5, Insightful)
Re:Add to that, NYI... (Score:4, Insightful)
Re: (Score:3)
My experience of multiple contracting gigs for hospitals is that unless the expenditure can be linked directly to day-to-day patient care you pretty much have no chance of getting funding approved for any initiative, even when it will reduce operational expenses.
Re:Add to that, NYI... (Score:4, Insightful)
Well, it's a fact. If you can't deal in facts, you shouldn't be working where you claim to be.
You're lucky enough to be able to afford to have a complex defense in depth - this isn't true of everyone.
So take your "I'm big bad dude" attitude, and stuff it. You're only big because daddy has money.
Re: (Score:2, Troll)
Well, it's a fact. If you can't deal in facts, you shouldn't be working where you claim to be.
I disagree. Dealing with facts would probably be an impediment to his job.
Re: (Score:2)
This is Wall Street. Manhattan. Terrorists have tried to blow it off the map multiple times. Several hurricanes have hit this spit of land that sits a mere few feet above sea level in the last decades. A hurricane hit and flooded parts last year even!
And that's the best site you can find for a datacentre ? Doesn't give me much confidence in your financial organisation's acumen.
Re: (Score:2)
Re: (Score:3)
Good luck convincing the fire department you meet the fire codes with your five thousand gallon tank of fuel on floor 45.
Re: (Score:3)
extraordinary effort = extraordinary cost? (Score:2)
Re: (Score:2)
Usually the epic fail of "fuel in non-fuel rated areas" is the fire chief flips his lid, especially if the sprinkler system is down and/or you're transporting slippery oil via the emergency evac route.
This is just "no cans of gasoline (for scooters, mopeds, etc) allowed in the dorms" writ large.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
A winch or window washing lift powered by what?
Re: (Score:2)
Re: (Score:3)
Re: (Score:2)
As somebody else already said, human power. There's also the option of tapping into the generator. I don't know the specifics, but if it takes less than a gallon of fuel to lift a gallon of fuel up the side of the building, you could just tap into the generator you're trying to refuel. The other option is to have a second smaller generator on the ground that powers just the lift/winch. This could be easily refueled. Seems like bucket brigade was the first thing that came to mind, and once they had that going, people stopped thinking of better ideas.
Or maybe they've seen enough Bugs Bunny cartoons to know what happens when a makeshift winch fails - whatever it's carrying falls to the ground in a big splat, flattening whoever it falls onto.
Hoisting a 30 pound gas can 200 feet in the air during a disaster is not the time for amateurs to jury rig something together.
Re: (Score:3)
You are making a lot of assumptions there, starting with the assumption that what you are suggesting is even legal. Next is that the people in charge of the building are also the people in charge of the datacenter (unlikely), and that the people in charge of the building rank your datacenter as a higher priority than everything and everyone else in the building.
This is not some sort of heroic lifesaving operation where 'do anything possible' applies. It is just very bad disaster preparedness on the part
Re: (Score:2)
This is why everything still uses combustion engines (including series-hybrid trains which are driven by purely electric motors connected to diesel generators).
generators in basements, smart or not? (Score:3)
Seems sort of stupid to me to put generators in a basement, considering that your on the coast, surrounded by water, and hurricanes like to come thru every now and then. Maybe this doesn't happen all the time, I don't know. I live on the west coast. I just have to worry about volcano's. (and I don't worry about volcano's).
Re:generators in basements, smart or not? (Score:4, Funny)
Volcano's what?
Re: (Score:2)
Well, one of two things:
1. Volcano's hot ash and lava, or
2. He's a semi-literate who hasn't graduated junior high yet.
Re: (Score:2)
If I recall correctly, NYC requires emergency generators to be at roof level so that a mid-rise fire would not cut off power to the upper floors. Most generators in flood prone areas are well protected-- we jumped through some odd little hoops to make generators work in Florida.
HOWEVER, there is a limit to what you can economically protect against. Usually, you are looking at 100-year events as a basis, not 500 year events or 500 years plus 10%.
Re: (Score:3)
> Seems sort of stupid to me to put generators in a basement
"Up north", basements are PRECISELY where you store things like gunky, messy generators (not necessarily with the full blessing of local officials) if you're in a big city like New York, in an old building that was built before elevators were mandatory and people still used coal for heat. They can't go on the roof, because they'd get damaged by the wind and rain. They can't go on the top floor, because it's the expensive penthouse. A newer build
Re:generators in basements, smart or not? (Score:4, Informative)
Re: (Score:2)
Seems sort of stupid to me to put generators in a basement, considering that your on the coast, surrounded by water, and hurricanes like to come thru every now and then. Maybe this doesn't happen all the time, I don't know. I live on the west coast. I just have to worry about volcano's. (and I don't worry about volcano's).
Maybe you'd understand better if you learned where to use a fucking apostrophe.
I did it to bother you.
Poor Planning? (Score:2)
Re: (Score:2)
Why would anyone in their right mind place generators and tanks below ground where flooding would be an issue?
Yeah, when you can put them on the roof... where... there is rain and wind...
I suppose there is no winning here.
Re: (Score:2)
I'll field this one, slashdot.
Managers. Lots of them.
Re: (Score:2)
And to bungle risk assessments.
Seems to me Manhattan has been a true flood risk, forever.
Now, what happens when we balance that assessment with other risks, such as fire risk of storing a generator on the 2nd floor. Say the risk survives to the next phase, where Fire Risk A and Flood Risk A are both on the table.
Then you get a room full of managers or politicians that say: We have X dollars and Risk A costs Y to mitigate and risk B costs Y+1 to mitigate. X is already Y. And that's how you have your ge
Re:Poor Planning? (Score:5, Insightful)
Because having large tanks of diesel fuel dozens of stories above ground isn't a good solution either? Lightning...wind...spills...leaks...fires... all probably more statistically relevant than major flooding, and the consequences of failure far more disastrous than simply losing power in a flood. Even storing the tanks underground and the generators above ground has 2 problems: 1) you need power to pump the fuel up to the generator, which kind of defeats the purpose, and 2) high pressure fuel lines running through a building isn't exactly safe or desirable either.
Re: (Score:2)
I am hoping that the "hasn't happened since the 1880's" bit is partly to blame, but Japan's little issue last year definitely comes to mind. I thought the same thing. I also question hauling tanks of diesel up the fire escape. That should have been better thought out. I guess the firewalls work both ways as long as you dont plan on leaving.
Re: (Score:3)
Re:Poor Planning? (Score:4, Insightful)
Why would anyone in their right mind place generators and tanks below ground where flooding would be an issue?
Lets see how you feel with a few thousand gallons of highly flammable liquid suspended above your head, in a building with lots of electricity running through it, where an earthquake is more likely than flooding in the basement. And that is ignoring the possibility of deliberate sabotage. A building with fuel stored above ground level where something went wrong would turn rather quickly into a giant pillar of flame. If one of the tanks gets ruptured, all it takes is a single spark to kill hundreds or thousands.
Below ground, however, fire-fighters can deal with it relatively easily, and the flames won't descend to engulf the entire building in a matter of a few minutes.
Re: (Score:2, Interesting)
Since when has Diesel been highly flammable? You can actually use it to put fires out. It takes quite some heat to get it going so really poses less risk than you average stationary cupboard.
Re: (Score:3)
Since when has Diesel been highly flammable? You can actually use it to put fires out. It takes quite some heat to get it going so really poses less risk than you average stationary cupboard.
Until it leaks out, saturates wood framing and other building materials, then comes into contact with an ignition source (like a pilot light, or candles that the residents are using since the power is out).
Then this hard to ignite fuel quickly turns an office into an inferno.
I can knock a candle over in my stationary cupboard and as long as I pick it up quickly, I wouldn't expect a fire. Soak that same cabinet and knock a candle over and it's a different story. Kerosene (very similar to diesel) makes a good
Re: (Score:2)
Is an earthquake a more likely event than a flood in Manhattan?
Re: (Score:2)
Nope, once again +5 Insightful is meaningless.
Generators should not be below ground, and at least 2-12 ft above ground if possible (or higher depending on risk).
They should not be too high, lest you have trouble fueling them.,
As with anything this likely came down to a retrofit risk assessment. The risk of fire or not making code is far higher than floods in manhattan, but being an island and subject to the ocean, still a high risk and should have been planned for. To me, it comes down to money at the e
Re: (Score:2)
Because they're required to by building code? That's what happened at the hospital that lost power. Tanks underground, water covers them, fuel gets contaminated, and poof, no more power.
Re: (Score:2)
Re: (Score:2)
The tanks are required to be below ground by the fire code. I think the better question is why put big data centers on a low lying coastline/island and/or a city with a giant target painted on it by every anti-american, anti-establishment, anti-whatever whacko the world over. Data centers belong in places with low risk of natural disaster, war, terrorist attack, riot or really anything that brings the police out.
Because some companies want to be physically close to their datacenter. Virtualization and cloud computing is changing this... slowly.
Why are generators in the basement? (Score:4, Insightful)
Why not move them to the roof? And while we're at it, do the same for all the nuke plants? A simple f*cking appliance that needs air and fuel to run and somehow they manage to spend life at the bottom of a potential indoor swimming pool.
Re:Why are generators in the basement? (Score:5, Insightful)
Depending on the generator size, there's the weight issue. There's also fire regulations. 500 gallons of Diesel on the roof + fire == cascading burning diesel fuel.
Re: (Score:2)
The fuel is stored underground due to fire safety regulations. The problem is fuel pumps don't work very well when submerged underwater.
DCP (Score:3)
Ahh: Diesel Control Protocol
Waterproof... (Score:2)
Gas stations bury their gas tanks underground with few problems of water seepage. Granted, they aren't surrounded by water and there is little water pressure (i.e. covered by water). However, you would think that they would have at least waterproofed the fuel tanks.
Re: (Score:2)
Even if you did, if the top of the tank is under water, how do you refuel it?
You have X dollars and Y risks, you can never be prepared for everything.
Re: (Score:2)
That ones fairly easily dealt with after the fact. In fact many modern refueling trucks come equipped with air-tight locking mechanisms on the refueling lines. Diesel generators are inherently pretty fucking tough so if a gallon or two of water gets into a sealed tank while they're getting the refueling line hooked up, who cares? A few gallons won't make it out of the fuel tank all at once anyways and with the compression ratio on diesel turbines and engines it'll probably just cause some sputtering and van
Re:Waterproof... (Score:4, Interesting)
The problem is that 'waterproofing' is a short-term guarantee. Water is insidious, it dissolves almost anything (although some things like metals very slowly), and it will eventually creep inside of any 'waterproof' container. That's why there's such a problem designing radioactive dumps like Yucca Mountain -- water would eventually eat its way into the vitrified radioactive cask.
Gas station underground tanks can survive for 10-20 years and still be waterproof. But most of the infrastructure in NYC is a hundred years old. There isn't anything waterproof in that city. Even brand new structures are probably permeable to water, if the designers just never thought it would be an issue.
Re:Waterproof... (Score:5, Informative)
The bigger problem is usually the pumps. You generally try to use turbine-type submersible pumps with the motor above the tank and the inlet down low to avoid problems with priming the pump. If the place where the pump motor floods, you are pretty much SOL.
If you place a suction pump 25' above the bottom of the tank to avoid flooding risks, you have the problem of priming the thing and maintaining suction. You could do a submersible pump with a really long shaft so the motor is high enough... but that would look really stupid.
Ultimately, you have backups on backups in most data centers (and hospitals), but you often have a limited window to respond. We have an (illegal) 15-gallon gas can in one facility up by the generator. That can will give them about 9 minutes extra run-time if the day tank runs dry. There is a hand pump in the basement that can be used to manually pump the fuel up 50' to the generators, but if the room it is in is flooded what can you do?
Big enough problems need disaster recovery plans; you will go down, the issue is how quickly you can return to normal operations.
Re: (Score:2)
Are you sure? Because when I managed a gas station 20 years ago, I had to check the tanks daily for water, and generally had to pump the water out at least twice a month before seepage and condensation built up enough to reach the uptake pipe. Using a hand pump.
And that's with gasoline, which floats on water - diesel absorbs water so you can't just pump it off the bottom of the tank. Look up "hygroscopic" and "diesel fuel dr
Status of datacenters affected (Score:5, Informative)
Per the topic, the following locations are experiencing or have experienced outages:
Re: (Score:2)
111 8th apparently has common fuel from the landlord (no individual tanks). This morning a valve failure starved out some gen sets.
It is still wild and wooly in downtown....
Manhattan unsuitable for data centers? (Score:5, Insightful)
Re: (Score:3)
Re: (Score:2)
They are downtown because all the fiber terminates there (111 8th, 60 Hudson, 75 Broad). They originate from the teletype days. Closest point to the main meeting places is a good place to build data centers. Hence Ashburn, London, Frankfurt, etc.
Re: (Score:2)
So maybe the answer is that flood-prone urban areas are just not a good place for critical data infrastructure.
That's aggrandizing the problem, really. The solutions are quite simple.The fuel pumps and other fuel delivery and refill systems for the generators should be in waterproof enclosures and rated to operate while the basement is submerged. Alternatively, there should be provisions for feeding fuel from a tanker parked out on the street to a riser going to the upper floor generator(s). That's all there's to it. Why nobody thought of it, I wouldn't know.
Re: (Score:2)
why were the generators on upper floors but fuel (and pumps) in the basement? And as soon as I read the answer, it was completely obvious:
Because diesel fuel weighs around 55 lb/ft^3?
Linode NJ datacenter use power gen (Score:2)
source: http://twitter.com/linode [twitter.com]
Let's hope power is restored soon...
Floating server farms (Score:5, Insightful)
It looks like Google was ahead of the curve [nytimes.com] after all with their idea for floating server farms.
"interesting" bucket brigade (Score:2)
In*ter*es*ting - adj.
1. capable of holding one's attention.
2. arousing a feeling of interest.
3. oh God, oh God, we're all going to die.
A First-Hand, On-Site Ongoing Account... (Score:2, Insightful)
Those of you with systems up and running with Peer1 should be thankful to the data center folks as well as the Squarespace staff who're pitching in big-time! If you want to know what's really going on, along with some pictures, check out their status page: http://status.squarespace.com
Basically, to address other folks' various questions: Yes, the generators are on the roof (along with a small start-up + several hour supply of fuel), but most of the fuel supply is in the basement due to fire codes, etc.. T
Re: (Score:3)
If you are involved in improving systems after the recovery, consider using pumps driven by hydrostatic (hydraulic) drive which would allow you to place pump drive motors well above flood zones. No electric motors below water level means nothing to short out. Hydraulic driven fuel pumps are often located IN aircraft fuel tanks, and it may be an aircraft pump or pumps would solve your problem. They move a LOT of fuel. The parts would be easy to get (tough hydrostatic pumps and motors have been standard farm
Re:Your priorities are all messed up!! (Score:5, Funny)
People can be replaced. Uptime, on the other hand..
Re:Your priorities are all messed up!! (Score:4, Funny)
To close the sarcasm tag, here's also an obligatory xkcd [xkcd.com].
On a side note, this reminded me of Cory Doctorow's "When Sysadmins Ruled The Earth [craphound.com]".
Re: (Score:3, Funny)
But, you see, without the Internet, it will be impossible to get all of those less essential services back up and running! After all, how is anyone going to obtain official updates from the Administration if they cannot access ready.gov, since that is apparently the only place to get them?
So, let's make sure you have the priorities straight in your head:
1) Internet
2) Power
3) Facebook Access
4) Starbucks
5) Running Water and Sewer
Re: (Score:3)
What, no cold Pizza slices? You inconsiderate clod!
Re: (Score:3)
No, Starbucks is NOT good coffee!
Re:Your priorities are all messed up!! (Score:5, Funny)
Re: (Score:3)
Re: (Score:3)
That's nonsense. Do you even know what lockout/tagout is? How did those "suits" remove your coworker's lock? Did they cut it off or did he not put it on? Every person working on that circuit should have had their own lock and tag on that breaker.
I assume if they had cut his lock off you would have mentioned it since that's a much more serious offense than simply flipping a breaker that isn't locked/tagged.
Your coworker definitely should have been fired if he was working on an industrial power circuit withou
Re:Your priorities are NOT all messed up!! (Score:5, Interesting)
Whatever part of the city you can keep operating is good.
Don't criticize what you know nothing about.
Re: (Score:2)
If any of these data centers are really as important as you claim, they should have been switched to their alternate sites (far from NYC) days ago. If these sites are not important enough to have real disaster recovery plans including alternate sites then they simply are not critical. Just being a datacenter does not make them any more important than the thousands of other businesses (stores, restaurants, you name it) that are also 'offline' because they are flooded.
Re:Your priorities are NOT all messed up!! (Score:5, Insightful)
Re: (Score:2)
Certainly there are criticial datacenters that have been impacted. Exactly what criticial services are on Peer1 (which is where they are carrying fuel through emergency stairwells)?
Re: (Score:3, Informative)
I work for the city. Happy to report that while we lost a lot of nodes and circuits downtown, none of our datacenters took a hit. We lost Mainframe DR replication for about 12 hours. No impact. 911 and 311 obviously experienced high call volume. 911 took no weather related hits, 311 saw about 2 minutes of weather related outage during a brief period where our two COs were flooded and we were in the process of switching to other trunks.
EXCELLENT work by the Cities workers throughout this event.
Re:Your priorities are all messed up!! (Score:5, Interesting)
Fewer people died than the people who didn't die due to not being in a car crash due to not being able to drive to work. So on the life/death front it's a win!
Re: (Score:2)
I don't think anyone died in NYC. I heard someone got electrocuted in Queens, but haven't heard of any fatalities in the city proper.
It really was just a flood event. The wind didn't do much to the infrastructure ( other than increase the surge ). I saw a pretty large tree in half out back of the Natural History Museum, but not much else around me in the city.
Re: (Score:2)
Don;t you all understand!? People _died_ during this storm.
Yup, a damn shame, especially the avoidable deaths (not sure that there were any, but you know how people can be when they panic).
Lots of Newyorkers are still without power and water and here your all woried about data centers!? Get a grip nerds!
We are not paramedics; we are not firemen, or police, or any other sort of emergency personnel. What would you have us do, other than get in the way of the professionals?
We let the emergency personnel do their jobs by doing our jobs and getting the systems back up and running. Believe me, you wouldn't want a bunch of sysadmins doing triage anymore than you would want a bunch o
Re: (Score:2)
Just a guess, but there is likely a fuel line...but the input is currently under water. Other comments have said that the fuel tanks must be at/below ground per city regulations.
If you say put the redundant input higher, how high? the 17th floor?
Slightly above water level is the correct location, but that might not be easy to figure out until flooding occurs.
Re: (Score:2)
Re: (Score:2)
Think that falls under did you actually get a business class SLA that's backed up by a deep pocket insurance company? Nope you got the cheap guy that works fine 99% of the time. I work primarily with the hosting industry and a decent subset of that is voip hosting the vast majority of these guys talk a good redundancy plan but fail to test it and/or execute it. Being that a huge chunk of the industry's costs are gear not people having to have n+1 gear and the tech to do site to site redundancy makes huge
Re: (Score:2)
Definition of "ideologically crazed"- a group of people who systematically deny reality. In this case, human civilization threatening reality.
What reality are they denying? The reality that either we will take world-wide coordinated action against carbon-induced climate change or we will extinguish human civilization.
That's pretty much the definition of a terrorist. Think the al Queda true believers didn't think they had a point to what they were doing? Who cares WHY a terrorist acts the way a terror
Re: (Score:3)
Your state office would in that case go offline roughly 90 minutes after virtually every disaster possible. Natural Gas, being a utility, is virtually guaranteed to be taken offline in cases where power goes out, ESPECIALLY when the power is deliberately cut off by the provider as a preventative against cataclysmic explosions where a gas pipeline explosion would be even worse.