Lightning Wipes Storage Disks At Google Data Center 141
An anonymous reader writes: Lightning struck a Google data center in Belgium four times in rapid succession last week, permanently erasing a small amount of users' data from the cloud. The affected disks were part of Google Computer Engine (GCE), a utility that lets people run virtual computers in the cloud on Google's servers. Despite the uncontrollable nature of the incident, Google has accepted full responsibility for the blackout and promises to upgrade its data center storage hardware, increasing its resilience against power outages.
Whaaa? (Score:5, Funny)
Re:Whaaa? (Score:5, Interesting)
From what I read elsewhere it was new/current data, not even an hour old, and the lightening may have caused things to run off batteries for a bit too long due to the multiple strikes. Seems not unreasonable as an explanation, might be entirely wrong though. Articles implied that users can also backup on their own sites to ensure that they are not behoved to anyone.
Re:Whaaa? (Score:4, Insightful)
All datacenter class storage devices should be backed by battery units with enough capacity to flush all pending writes to disk.
I have never bought a server that didn't have battery-backed hardware RAID.
Google, however, runs the cheapest, commodity parts, often refurbished / purchased used, and relies on software RAID and massive replication schemes. Such schemes don't work for new data, as they've found out.
I wouldn't blame them if their shit got directly hit by lightning and that caused damage (you can't expect anything to survive that), but if we're saying the extended power outage caused data loss, then it's absolutely Google's fault.
Re: Whaaa? (Score:3, Informative)
In this case it was disks the customer had specifically requested to be un replicated for performance. So all the data could be gone in a flood and Google would still be within its obligations.
Re: (Score:3)
Except you don't have to blame them, they took responsibility.
Pretty sure what they have 'found out' is that paying for the fallout from the occasional freak occurrence and minor data loss is cheaper in the long run than buying more expensive hardware to gaurd against occurrences so rare that they end up on news sites.
Re:Whaaa? (Score:5, Funny)
Darkening reduces the output of solar panels, so you can't win either way.
Re: (Score:2)
Re: Whaaa? (Score:1)
Our datacenter generator fail to start at the last powermoutage we had, the culprit was a squirel
Re: (Score:2)
Brief is apparently long enough to cause data loss. But power disruptions to running systems don't have to be very long to shut them down.
However, Google apparently hasn't really engineered a proper solution here. In the data center, batteries are there only to bridge you over until the onsite power generator can come on line or until you can perform a safe and orderly shutdown. Plus, you *DON'T* shutdown the generator or reboot shutdown systems after power comes back until you have enough battery capac
Re: (Score:2)
Very likely. Cloud providers are routinely lying about their performance, or you need to dig very deep to find what the actual technical assurances are.
Re:Whaaa? (Score:5, Interesting)
Re: (Score:1, Troll)
No, the problem is that replication isn't instantaneous. If you can't accept reality, please kill yourself.
Re: (Score:1)
It was on the VAXcluster I administered in the '80s. The "reality" is that until the replication has happened, the data shouldn't in principle be regarded as ready for processing. In practice, a few seconds' write caching to ALL nodes is the limit of acceptability.
I've come to learn that these days most people doing stuff "with computers" are doing nothing of value, and coincidentally this means that the fucking atrocious reliability and security of outsourcing are considered good enough.
Re: (Score:1)
I demand my data is backed up before I've sent it to google!
first bump should have taken the center offline (Score:2)
and all data flushed to disk, and resources examined, before they opened for input again.
no generators? (Score:2)
come ON, google eyes... slack-writing disks is always a danger. you don't have generator backup for the battery backup, and you're slopping data all over the floor as a result?
losers.
Re: (Score:2)
But but ... it's the cloud. Why would you need a backup in the cloud?
You mean we've been lied to?
Re: (Score:2)
It's a good thing that users routinely make local backups before uploading to the cloud.
Re: Whaaa? (Score:1)
Re:Cannot be trusted (Score:5, Interesting)
Just use Amazon like everyone else. Google cannot be trusted, and I have said that many times. They 1) frequently decide to shut down services users rely on. One of the persistence mechanisms we depended on recently got the head shot, costing us so much money that we decided to move to Amazon, which has a standardized stack, and 2) data loss, and 3) non-existant customer service. Try contacting Google with a pressing issue.... you'll eventually give up.
Re: Cannot be trusted (Score:5, Insightful)
I know that's what I base my critical data storage choices on - how fast a tangentially-related service's static front page loaded 15 years ago on dialup.
Re: (Score:2)
I base all my decisions on ACs responding to themselves. Splitting a single post up into three posts all acting like different people agreeing makes an AC look that much more trustworthy.
Re: (Score:2)
Re: (Score:2)
I'm afraid that had nothing to do with write speed for storage. It had a great deal to do with deliberately keeping the design very light, using very effective proxies for the very light and consistent images, and keeping tight reins on web designers who might want to front load the Google pages with exciting content that had no relationship to the service.
I'm confused why you mention it, unless you think that the wise practices and designs that led to such effective and quick interfaces affected their stor
Re:Cannot be trusted (Score:5, Informative)
The announcement is about Google Cloud Engine. Not about Google's own services (gmail, search, photos, docs, that sort of things). AFAIK, none of Google's own service announced any loss - presumably because they don't rely on a single location.
From the post:
> In particular, it was possible at all times to recreate new Persistent Disks from existing snapshots.
i.e. snapshots were fine.
> This outage is wholly Google's responsibility. However, we would like to take this opportunity to highlight an important reminder for our customers: GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters. Customers who need maximum availability should be prepared to switch their operations to another GCE zone. For maximum durability we recommend GCE snapshots and Google Cloud Storage as resilient, geographically replicated repositories for your data.
So, if some poor users of GCE thought a single geographical location can withstand disasters, they now know.
Oh realy? (Score:5, Interesting)
Lightning struck the same place not twice, but four times?
Re:Oh realy? (Score:5, Funny)
Cloud to Cloud lightning is about 3 times more common than cloud to ground, so its not that crazy.
Re:Oh realy? (Score:5, Funny)
Another reason not to store your important data in the cloud.
Re: (Score:2, Funny)
Another reason not to store your important data in the cloud.
Arrrgh, somebody please mod this up!!! You made me snort my coffee.
Re: (Score:1)
I hate it when I miss an obvious pun. I shall now steal this for every argument against cloud computing, along with all the serious ones.
Re:Oh realy? (Score:5, Informative)
Contrary to popular belief, it's common that ligthning strikes several times at the same place. Something that was attracting lightning, if not destructed by the first strike, will still be a major target standing for the next ones.
Re: (Score:1)
Agreed ... I have a tree that my city wont let me cut down because it is an old growth tree and it has been hit 3x and every time it happens the EMP from the bold nukes half of the running electronics in that side of the house.
Comment removed (Score:5, Interesting)
Re:Oh realy? (Score:4, Funny)
It's cloud computing after all. Of course there will be frequent lighting.
If my calculations are correct... (Score:3, Funny)
Re: (Score:2)
Great Scott!
Re: (Score:2)
Re:Oh realy? (Score:5, Interesting)
Lightning striking in the same spot repeatedly is a lot more likely than people think. The reason lightning may have struck a spot is due to there being a good path. Thus lightning is likely to strike that easy path again.
We have that. We live on a mountain where there is a large copper vein running under us. I have watched lightning strike repeatedly in the same spot.
There are videos of lightning repeatedly striking tall buildings during a single storm.
More over, lightning does not need to be very close to do a lot of damage. In a recent storm we had nine nearby strikes - not all in the same spot but spread out over at least a square mile of our land. We lost many miles of wire because of the EMP that the lightning strikes generated got picked up by the wires and overloaded them causing the wires to melt. Some sections of fence wire simply vanished. Google could have had a few nearby strikes that did that. This happens.
See:
http://sugarmtnfarm.com/2015/0... [sugarmtnfarm.com]
and
http://sugarmtnfarm.com/2015/0... [sugarmtnfarm.com]
Re: (Score:2)
If you don't get directly hit by lightning, it can still do a lot of damage to things... There are two ways this happens...
1. Induced currents into wires close to the strike - This is where the huge amounts of current flowing into the ground from the lighting strike induces currents in other wires. It's basically cross talk, but with a really large impulse input in one wire, you get some pretty impressive signals in parallel wires. This is what fries your corded phones and electronics which are plugged
you've never been near high places, then, son (Score:2)
at some point in your life, have you ever listened to AM radio or broadcast television? their stuff gets hit with lightning all the time. if a really good thunderstorm happens to float over their tower, AM radio tends to sound like "Weather Desk Radar shows the //SPLATbzzHummm// county, with //SPLATbzzHummm// t's the latest repor //SPLATbzzHummm//" with the transmitter knocked off the air every few seconds by a direct hit. in NTSC television, the return would be about 3-4 seconds of variable contrast an
Re: lightning strike (Score:5, Insightful)
Bullshit. I used to design high voltage connections, and tested using a 300kV impulse generator. I've seen a lot of crazy stuff analyzing field failures. You can greatly reduce the risk, but you cannot remove all risk in an above ground facility, as a practical matter.
I do see lots of silly stuff done, based on myth and lack of knowledge.
Re: (Score:1)
Or maybe they should start digging, copy the NSA and have everything underground would solve most issues no ?
instead, check out polyphaser.com (Score:3)
you left out about 99-44/100 percent of the technology and art of lightning protection.
HIllary (Score:1)
...is now requiring all of her email servers be hosted by Google.
In other news... (Score:5, Funny)
... Alphabet's new "personal re-vivification" project is making good progress. The project leader, V.Frankenstein was unavailable for comment however.
Re: (Score:2)
Around here, there is a guy that runs for sheriff (never really cared if he won) named Moran, it is entirely possible that the poster is a joke.
http://www.washingtonpost.com/... [washingtonpost.com]
All his signs say "Moran for sheriff", I always read it as Moron for sheriff and wondered why anyone would vote for a moron for sheriff.
Re: (Score:2)
There's a Senator, and Representative by the same name. Speaks volumes for Congress.
What do you mean permenantly erased...only 1 DC? (Score:1)
You mean to tell me users data is stored in a SINGLE DC and not redundant elsewhere?
I find that easy to believe for most "cloud" providers but google? Really?
Ummmmmmmm?
So presumably if someone took out the right, big big DC in a huge fasion, many users would be fucked by it?
Re: (Score:2, Informative)
If you RTFA you'll see they mention it only affected "recently written data" that had not yet made it to persistent storage. So probably only a few hours old at most.
Re: (Score:3)
Thats no excuse. It should be distributed amongst seperate machines in seperate centres instantaniously.
Re:What do you mean permenantly erased...only 1 DC (Score:5, Insightful)
Thats no excuse. It should be distributed amongst seperate machines in seperate centres instantaniously.
So faster than the speed of light using the infinitely-wide infinite improbability data bus?
Re:What do you mean permenantly erased...only 1 DC (Score:4, Informative)
*GUFFAW*
You're wasted here, you should do stand up.
Re: (Score:2)
You're wasted here, you should do stand up.
And get hit by lightning?
Re: (Score:3)
Thats no excuse. It should be distributed amongst seperate machines in seperate centres instantaniously.
You can have that services if you want to pay for it. You get that, right?
Re: (Score:3)
Re: (Score:1)
GCE is not a backup solution.
Google cloud data is geographically replicated.
Another Flash Vulnerability ? (Score:5, Funny)
n/t
Re: (Score:1)
Re: (Score:2)
No I think this was using Thunderbolt
Re: (Score:2)
Oh wait, I never realised my Lightning connector was for charging my iPhone outdoors.
Re: (Score:1)
Witness said "it was like a silver light"
But google said... (Score:2)
Re: (Score:2)
Re: (Score:2)
Also, adding a couple of lightning rods (Score:2)
And a mad scientist with a cobbled together corpse.
Strike one for the cloud! (Score:2)
Nuf said.
Re: (Score:2)
From the summary, it was at least strike four...
location location location (Score:2)
Headline Failure (Score:3)
Come on people. This has the potential to be legend... ary. What a complete failure.
Even just form a quick punt we could glimpse such lyrical word play as:
"Lightning strike inside Cloud"
"Cloud damaged by lightning"
"Cloud not lightning-proof"
Please read the fucking Register until you gets it.
like lightning on your wedding day (Score:2)
Are you saying Google users lost their data from the cloud due to lightning?
Whoever scripts your reality down there is the most inept of hacks.
Offsite backups (Score:2)
Re: (Score:1)
Administered by a skilled storage, hardware, and network team. Training up such a team and keeping them employed, without managerial or technical errors destroying the whole system, for years, is far beyond the skills of most IT departments.
I've built such offsite storage: I've helped people recover from the _inevitable_ sitewide disasters that overwhelm reasonable protections. Much like running your own mail server, Google has consistently outperformed and outlasted single company systems. Bulk data storag
What did you expect? (Score:1)
Did you wipe the server clean? (Score:1)
Oxymoron (Score:2)
Despite the uncontrollable nature of the incident, Google has accepted full responsibility for the blackout and promises to upgrade its data center storage hardware, increasing its resilience against power outages.
If it is uncontrollable, then any changes Google makes won't matter. On the other hand, if using other equipment, hardening the system, installing better grounding, etc. would have kept the loss from happening, then it is controllable. Maybe what they meant to say was unpredictable. Of course, then they would have had to explain why they didn't plan for the possibility.
Re: (Score:2)
No, not really. As I understand this, what happened is they experienced multiple failures of the AC power coming into the facility which happened to be lighting induced. When the power went out the first few times, the UPS's switched to battery power and everything kept going as the batteries provided the necessary power. Eventually, after repeated dependence on the UPS batteries during the multiple failures, the batteries had no more power left and the UPS output power stopped.
This is really a process
So... (Score:2)
If I understand this correctly, to get your personal data removed from Google search engines it requires 4 lightning strikes to the exact same location?
Must have missed that part in the EULA...
Really? 4 times? Court ordered data? (Score:1)
Re: (Score:2)
Actually, this seems to be a problem with their power redundancy systems not really a lighting protection problem.
After three lighting induced power outages, the UPS ran out of reserve capacity so on the fourth power outage the UPS dropped the system's power.
Really, what they have is a process failure... They should have their standby AC generator running until the UPS batteries where charged enough to safely handle another fail over process. Either that, or they should have quickly moved the processing
depends on who lost data in slack-write centers (Score:2)
if it's yours, I could give a shit.
if it's mine, it's war.