Europe's OVH Web Hosting Provider Knocked Offline Following Fire (techradar.com) 43
Kelerei writes: A major fire has destroyed a data center of European cloud provider OVH in Strasbourg, France. The SBG2 data center is completely destroyed, while the blaze caused some damage to SBG1 before being contained. SBG3 and SBG4 were also taken offline, but a plan is underway to restart them once the firefighters give the all-clear.
All OVH staff at the site are accounted for and unhurt, but it is unlikely that the data in SBG2 is recoverable. On OVH's status page, an ominous note states "if your production is in Strasbourg, we recommend to activate your Disaster Recovery Plan." Among the sites affected is the WordPress image optimization site Imagify and the encryption utility VeraCrypt.
(Submitter's note: this is why any disaster recovery plan should include offsite backups...)
All OVH staff at the site are accounted for and unhurt, but it is unlikely that the data in SBG2 is recoverable. On OVH's status page, an ominous note states "if your production is in Strasbourg, we recommend to activate your Disaster Recovery Plan." Among the sites affected is the WordPress image optimization site Imagify and the encryption utility VeraCrypt.
(Submitter's note: this is why any disaster recovery plan should include offsite backups...)
Looks really bad. :( (Score:2)
My personal server (VM) is with OVH, but luckily in Montreal (it's really the iWeb data centre). I feel really sorry for OVH and its customers. That really sucks.
Re: (Score:2)
Not sure. I think they have a couple of data centres. I think it might be north of the 40 somewhere.
Re: (Score:2)
iWeb? OVH has an office/pop in Montreal proper, but their actual datacentres in Montreal are just south of the island in Beauharnois in an old aluminum smelter building. iWeb is their competitor. Are they colocating with iWeb or something?
Re: (Score:1)
LOL! I wrote just below that Beauharnois was near Trois-Rivieres! I was confused with Becancour. You are correct!
Re: (Score:2)
Aaaargh, my bad. I was confusing OVH with Luna Node, which *does* piggyback off iWeb's infrastructure. Never mind.
Re: (Score:2)
And again, I'm wrong. Luna Node in Montreal piggybacks off of OVH. Sheesh.
Re: (Score:2)
I am not aware of any "OVH VM" at iWeb data center in Montreal. Different companies IMHO. OVH data center is in Beauharnois, Quebec (close to Trois-Rivieres).
As well TFS title is misleading; only Strasbourg in France data center is impacted, OVH isn't totally knocked offline! Other OVH European data centers are up and running.
They have four data centers in Strasbourg and SBG2 is totally destroyed (all data lost) while there might be hope for 1-3-4.
I have a buddy who is bad-lucky enough to have been in SBG2.
Blasht it! (Score:3)
Re: (Score:3)
Looks like they went cheap all the way, because any real DC has automated, no-hardware-damage fire suppression systems. This one obviously does not.
Re: (Score:2)
My thoughts exactly. I had to stop and make sure I wasn't seeing things when I saw the first pictures and thought they looked like shipping containers.
It doesn't take a genius to figure out why OVH used shipping containers - I'm sure they were cheap - but man oh man, what a terrible way to build a datacenter. Even looking past the fire suppression issues, the la
OVH's DR Plan? (Score:2)
I would expect OVH has offsite backups and can bring up services at one of their other data centers.
Right?
Granted that's not likely to be instantaneous but we should have heard something about it by now.
It's like they're saying you should have other hosting providers if you choose them. Many of us here are sufficiently paranoid to do that but it's not what most businesses think they're paying for.
Re:OVH's DR Plan? (Score:4, Informative)
If the customer paid for backup services, sure, the data is intact. VPS/server providers don't normally provide free backups for customers.
Re: (Score:2)
Exactly correct, no backups unless you pay for it. I do it myself, it's cheaper and you have more control that way. 2 differently geo-located data centers with data replication and backups to my home server.
A buddy of mine lost everything he had in SBG2, data + backups. He had several servers in SGB2 and backing up in the same data center but different servers :(
Re: (Score:2)
I have my hosting through a low cost VPS but have everything snapshotted on a 4 hourly cycle off site, I never feel comfortable about my data being at somebody elses mercy for the exact reason outlined in the original article.
Re: (Score:2)
I just use two providers in different locations. If you do DNS and email right, you need that anyways.
Re: (Score:2)
Re: (Score:2)
Given that they apparently did not even have a working fire suppression system (standard in any real DC), I doubt they have anything.
Re:OVH's DR Plan? (Score:4, Informative)
I'd say the only thing not working properly here is your ridiculous expectation that a fire suppression system would have any chance of putting out a blaze that took 43 fire trucks and 100 firefighters using them to do
Different from you, I have a minimal clue about DC design. I also understand that a fire starts small, which you seem to be unaware of. A proper DC fire suppression system may, for example, fill the complete DC with CO2 in a matter of minutes and it is automatized. In a real DC you get about a minute to leave before that happens and you get warned that you really, really need to run when the warning horn goes off because you will stop being able to breathe in there very soon after. That the fire even got to this size means they had nothing.
The standard case for a DC fire is that there is minimal damage, the fire suppression system needs to be re-loaded and most of the hardware is completely undamaged.
Re: (Score:2)
I agree with what you said in principle, but there is far more to this story than some basic assumption about lack of functioning fire suppression. OVH isn't some fly by night company. They have been in the web hosting / cloud business for longer than Amazon, and back when Zuckerberg was still dealing with pimple outbreaks. They are Europe's biggest datacentre operator and own and operate multiple datacentres.
It's highly convenient that their largest datacentre burns to the ground very quickly only two days
Re: (Score:2)
Hmm. You have a point. It is highly unusual for a DC to not have good fire safety. The question is who profits.
Re: (Score:2)
Oh actually more interesting would be EncroChat. A company offering secure chat services to the criminal underworld of Europe hosted at OVH. The French Gendarmerie and the Dutch police digital crimes unit managed to infiltrate the servers and decrypt months of chats in part OVH were the ones who provided the authorities access to the server in question.
So it's a drug bust, nothing big about that right? Well as it turns out in the past 6 months the EncroChat breach has been cited as being a contributor to 30
Re: (Score:2)
That does not make a lot of sense. When the police is at the door with a warrant, any DC operator provides access. Maybe the ones doing the hypothetical arson do not understand how this works. On the other hand, sabotaging a DC fire suppression system is not amateur level.
Re: (Score:2)
Hey, don't let sense get in the way of a good conspiracy theory :-)
Re: (Score:2)
My apologies ;-)
Don't worry (Score:2)
Your data are safe in the cloud now!
Re: (Score:2)
Well till it rains.
Rust Game Servers (Score:2)
This also seem to be where 25 Rust Game servers and player data were located and seems to have been lost in the fire.
Guess all those people will have to restart. Not sure how many players that is, but quite a few.
Yes this is why you backup date off site.
How often do datacenters go up in flames? (Score:2)
OVH was forced by a French court order to allow law enforcement and intelligence agencies access to a server hosting an encrypted messaging service called "EncroChat". This allowed officials from multiple European countries to listen in on all communications going through that server and exposed drug shipments, kill orders, corruption, etc. The intel acquired resulted in a lot of indictments and caused all sorts of damage to a lot of criminal organisations. Now, this may be far fetched but it wouldn't surpr
Re: (Score:2)
Re: (Score:2)
OVH announced on Monday that they were going for an IPO as well. There's a lot of convenient things here. That said they if this was true they would have burnt down the wrong datacentre. EncroChat was hosted in Roubaix which is the opposite side of France to the Strasburg datacentre that went up. Mind you the Strasburg one was the biggest so if damage was the main goal it would certainly make sense.
Don't worry (Score:2)
Looks like a cladding fire (Score:2)
OVH being offline is a good thing. (Score:4, Funny)
In related news, world-wide spam flows decreased 10% while OVH was offline.
you get what you pay for (Score:2)
Great to pay a few Euros a month less for hosting, but it comes with a few corners cut. The last time I spoke to them, they were literally stamping their own server enclosures, building custom cooling solutions, hosting gear in old industrial buildings rather than datacenters and there was a great emphasis on cost cutting. That eventually comes back to bite you even if it is cheaper in the short-run.
Re: (Score:2)
Very true. We have servers in a cheaper DC, but it has full fire suppression and assurances that if power fails they will buffer/filter it for a few minutes and then do a clean power off. Of course, that costs probably 2x as much than this ElCheapo crap did cost. Well until you factor in the additional risk-cost which has just realized (as some friend in insurance like to call it).
Re: (Score:2)
It depends, you can probably get 2 servers at OVH at 2 different DCs (SBG and Montreal) for the same price as one at some other places. Of course, you have to sync both and have disaster planning and that's a lot of administration overhead. But in the end, it might even be more reliable than 1 server at a more expensive provider.
Re: (Score:1)
Impressive photos and video (Score:3, Informative)
Here is a nice thread full of information. It wasn't easy to know which DC my server was in... but sadly, it was right in the fire... My data is now indeed in the cloud!
https://lafibre.info/datacente... [lafibre.info]