Explosion At ThePlanet Datacenter Drops 9,000 Servers

An anonymous reader writes "Customers hosting with ThePlanet, a major Texas hosting provider, are going through some tough times. Yesterday evening at 5:45 pm local time an electrical short caused a fire and explosion in the power room, knocking out walls and taking the entire facility offline. No one was hurt and no servers were damaged. Estimates suggest 9,000 servers are offline, affecting 7,500 customers, with ETAs for repair of at least 24 hours from onset. While they claim redundant power, because of the nature of the problem they had to go completely dark. This goes to show that no matter how much planning you do, Murphy's Law still applies." Here's a Coral CDN link to ThePlanet's forum where staff are posting updates on the outage. At this writing almost 2,400 people are trying to read it.

9 Volts of Love

Electricity is a fickle mistress, one moment she's gently caressing your genitals through gingerly applied electrodes the next she's blowing up your data centers.

It must have been HACKERS

Because Hackers can turn your computer into a BOMB [ytmnd.com]!

Kudo to their support team

... for posting frequent updates to the status of the outage.

Re:Kudo to their support team

It's probably less effort to spend a few minutes updating a forum than it would be to man the phones against irate customers demanding their servers be brought back online.

Re:Kudo to their support team

It's little known mostly because it's not actually true. I think you're confusing theplanet with the world, aka world.std.com.

explosion?

Lesson learned: don't store dynamite in the power room.

Re:explosion?

Lesson learned: don't store dynamite in the power room.

But they told me to take it out of the room with the fuel for the generators, the management offices, and HR department...

Printer ignition source

Last message on the linux console before the explosion:

        lp0 printer on fire!

trying to read it

At this writing almost 2,400 pelople are trying to read it. Posting it on slashdot should help speed it up.

Re:trying to read it

Here's a new update from Urvish Vashi: To keep you up-to-date, some idiot posted this forum page on slashdot. Expect some slowdowns and interruptions trying to access this page. ps: **** you slashdot.

Recovery costs

I wonder what the dollar value of the repairs will run? I'm sure insurance covers this kind of thing, but I'd love to see hard figures like in one of those mastercard commercials: Structural damage: $15000 Melted hardware: $70000 Halon refill: $however much halon costs Real-Life Slashdot effect: Priceless

Re:Recovery costs

Not to mention the cost of pulling all those consultants in, overnight, on a weekend...

Also, only the electrical equipment (and structural stuff) was damaged - networking and customer servers are intact (but without power, obviously).

I read that they pulled in vendors. Those types would be more than happy to show up at the drop of a hat for some un-negotiated products that insurance will pay for anyway, and they'll even throw in their time for "free" so long as you don't dent their commission.

This is BAD KARMA!!

Clearly this is bad karma resulting from all their years of human rights violations....especially Tiananmen Square...oh wait--

Helpful Slashdot!

At this writing almost 2,400 people are trying to read it

and as of this posting, make that 152,476.

Coral cached LOFI status page

Kudos to them for their timely updates as to system status. Having their status page listed on /. doesn't help them much, but I was encouraged to see a Coral Cache link to their status page. In that light, here's: a link to the Coral Cache lofiversion of their status page:

• http://forums.theplanet.com.nyud.net:8080/lofiversion/index.php/t90185.html

Monty Python

electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding their electrical equipment room.

But the fourth wall stayed up! And that's what you're getting, son - the strongest data centre in all of Texas!

Re:More planning could have prevented this

It is often the case that transformers are kept apart from all other components

And that appears to have been the case here. Had you read the article, or even the unusually accurate headline, you would know that the 9,000 servers were 'dropped' rather than 'blown apart'. They are still physically with us, they are just dropped from service because they don't have any power because the power supply blew up.

Further, the 9,000 servers were physically, geographically, isolated enough from the power supply (which is what exploded) to be protected. We know this to be the case because we read the article and headline and understood them and they indicate that the 9,000 servers were not blown up.

To put it another way, only the power supply was damaged by the explosion, the servers were not. Probably there was no way to isolate the power from its own explosion. The servers, however, we protected.

So, in summary, the 9,000 servers were not blown up. Only the power.

The power is off due to the explosion but there servers themselves are A-OK.

Re:More planning could have prevented this

so you're agreeing with me. The servers getting blown up was a huge mistake, one that certainly could have been avoided with a little proper planning.

you are a fucking moron

Re:More planning could have prevented this

I wish I had mod points...I think this is the first time I ever wanted to mod those 5 words up.

Correction

Sorry for replying to myself, I don't think I made my post clear; the backup power is not on (the mains was blown to bits), because the fire department told them to shut it off.

Re:More planning could have prevented this

ThePlanet has 5 or more datacenters. The cost and complexity of doing a full blown physically separated 2N power system at every datacenter is far more expensive than taking the chance of having to issue a credit against an SLA. Not to mention that when a fire is involved, the fire department has full authority and may instruct you to cut all power anyway - they are coming in to an unknown situation and won't risk their own people just because you say the other power system is isolated.

Another issue is the complexity of a full blown 2N power system is likely to cause more outages due to human error during routine maintenance over an N+1 system. Complete 2N power systems from grid and backup sources all the way to the servers with no single point of failure (transformers, wiring, switching, PDUs, UPSs, etc.) are enormously complex and expensive, so it's not "the only thing that makes sense". I assure you issuing a one-day pro-rated credit to all your customers is cheaper.

Re:Server/customer ratio?

ThePlanet is a popular host for hosting resellers. Many of the no-name shared hosting providers out there host at ThePlanet, amongst other places. So... Many of these customers would be individuals (or very small companies), who in turn dole out space/bandwidth to their own clients. The total number of customers affected can be 10-20x the number reported because of this.

Re:Server/customer ratio?

Wouldn't people who want such redundancy consider putting the other server in another DC?

Re:Photos or informaton on building?

I'm a mechanical/electrical engineer by training, and what you're saying makes no sense to us. Mistakes are made in the laboratory, where things are allowed to blow up and start fires. Once you hit the real world the considerations are *very different*. While it's possible that this fire could be caused by something entirely unforeseeable (unlikely given our experience in this field), it's also possible that this was due to improperly designed systems.

I don't suppose you'd be singing the same tune if this was a bridge collapse that killed hundreds. There's a reason why engineering costs a lot, and that's directly correlated to how little failure we can tolerate.

Re:Explosion?

Actually, modern batteries should be sealed valve or Absorbed Glass Mat (AGM) that don't vent (too much) hydrogen. During a thermal runaway, they vent a tiny bit before killing themselves, but hydrogen doesn't become explosive until the concentration in an enclosed environment is ~4%. 4% of a data center is a fairly large area. I've heard of this happening in one data center where the primary and fail over (IIRC) HVAC units failed and no one had been on site for well over a month. IOW, every battery in the place started venting and it took over a month without any air circulation for it to get to 4%.