How Google Broke Itself and Fixed Itself, Automatically

lemur3 writes "On January 24th Google had some problems with a few of its services. Gmail users and people who used various other Google services were impacted just as the Google Reliability Team was to take part in an Ask Me Anything on Reddit. Everything seemed to be resolved and back up within an hour. The Official Google Blog had a short note about what happened from Ben Treynor, a VP of Engineering. According to the blog post it appears that the outage was caused by a bug that caused a system that creates configurations to send a bad one to various 'live services.' An internal monitoring system noticed the problem a short time later and caused a new configuration to be spread around the services. Ben had this to say of it on the Google Blog, 'Engineers were still debugging 12 minutes later when the same system, having automatically cleared the original error, generated a new correct configuration at 11:14 a.m. and began sending it; errors subsided rapidly starting at this time. By 11:30 a.m. the correct configuration was live everywhere and almost all users' service was restored.'"

Re:Well congratulations

[Anonymous Coward]

On recovering by using the "last known good" configuration. What wizardry!

I expect we'll be seeing the Google patent application on that shortly </sarcasm>

Give Google a little credit (but not too much please). If they were Apple they'd have already patented it.

Reminds me of something...

[stjobe]

"The Google Funding Bill is passed. The system goes on-line August 4th, 2014. Human decisions are removed from configuration management. Google begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug."

Re:Well congratulations

[93 Escort Wagon]

Give Google a little credit (but not too much please). If they were Apple they'd have already patented it.

Whereas Google would just look for a small company holding a relevant patent, then buy it.

Re:Reminds me of something...

[Immerman]

Google perceives this as an attack by humanity, and routs all search queries to goat.se in self defense.

Re:Well congratulations

[Anonymous Coward]

Yeah that totally must be it. Me, the guys who write configuration management tools who'll tell you how hard it is (and sell you consultancy to try to make it slightly less hard) and the guys who write monitoring tools who'll tell you how hard it is (and sell you consultancy to try to make it slightly less hard). All those guys from companies like Facebook and Google who give talks at conferences about how difficult it is. We all suck at it and don't know what we're talking about. If only we'd listened to Slashdot, all our troubles would be but a dream.

Re:Well congratulations

[phantomfive]

"Our system is high-availability, it can return 404s all day for decades without going down"

Re:Well congratulations

[Anonymous Coward]

Careful. Only the advice of Anonymous Cowards is trustworthy. All the other people on Slashdot are not to be trusted. After all, they are not even able to find out how to post anonymously! ;-)