Google Takes Blame For Internet Disruption Across Japan (theregister.co.uk) 59
An anonymous reader shares a report: Google on Saturday accepted responsibility for the widespread internet disruptions Japan experienced the previous day. The search engine giant apologized for the trouble, saying it was caused by an errant network setting that was corrected within eight minutes of its discovery. Google did not say whether human error or a technical malfunction was to blame. The disrupted services used internet connections provided by NTT Communications Corp. and KDDI Corp., both of which said Friday that the issues were caused by a change in the flow of data traffic. From a report on The Register: The trouble began when Google 'leaked' a big route table to Verizon, the result of which was traffic from Japanese giants like NTT and KDDI was sent to Google on the expectation it would be treated as transit. Since Google doesn't provide transit services, as BGP Mon explains, that traffic either filled a link beyond its capacity, or hit an access control list, and disappeared. The outage in Japan only lasted a couple of hours, but was so severe that Japan Times reports the country's Internal Affairs and Communications ministries want carriers to report on what went wrong.
You know you've reached the summit (Score:3, Funny)
if you can claim responsibility for everything bad that happens on the net and everyone believes you immediately.
Re:You know you've reached the summit (Score:5, Informative)
Hmm, actually we *knew* it was them in the first place the moment it started, most non-joke internet network engineers refuse to fly blind, so there are probes and monitors everywhere in the Internet control plane (DFZ BGP routing). E.g. read: https://bgpmon.net/bgp-leak-causing-internet-outages-in-japan-and-beyond/
Also, most internet network engineers will place a lot of the blame on *Verizon*, not Google. "route leaks" are a *fact of life*, they will happen at least once to everyone. You *MUST* filter the routing plane to not accept crap from other autonomous systems you peer with, and Verizon *utterly failed* at doing it. Had they filtered, they'd have rejected the bogus routing from Google and avoided most, if not all of the damage.
So, Google might have publicly taken the blame since it was their operational error that triggered the damage in the first place, but they are at most responsible for half of it... and *everyone in the field* knows it, Google included.
Re: You know you've reached the summit (Score:1)
Re: (Score:2)
"You know you've reached the summit when you spend energy burning white males at the stake of diversity rather than ..."
It's not about whites or males, it was about someone sending a disruptive email to everyone in the company.
Re: (Score:2)
That's absurd. First, did you read it before saying that it was a "disruptive" email, or are you just one of those phonies who repeat things they don't know shit about? People who repeat their shallow understanding of biased news reports remind me of that coward security guard in The Stand who runs away from the lab and carries the disease in the wild.
Second, the guy was fired because the document was made public, not because it was sent to everyone in the company. He was basically thrown under the bus by h
Re: You know you've reached the summit (Score:2)
Japan accepted 1, out of more than 8000 requests, refugee. That's actually better than the year before.
Google was just punishing them for their lack of diversity! /s
Re: (Score:2)
"First, did you read it ... "
Yes, I felt it was mostly hand waving about how he blamed others for his attitude.
"Second, the guy was fired because the document was made public, not because it was sent to everyone in the company ... "
Thanks for the correction. I don't approve of his firing, at the same time it's what companies usually do, and blaming it on the public is a cop out.
Re: (Score:2)
Have you watched the movie Trumbo? Because that's exactly what's happening.
Re: (Score:2)
"Because that's exactly what's happening"
Not by a long shot. For one, Google is a private company, while Trumbo is about the US government.
Re: (Score:2)
First, it's not just Google. It's most tech companies.
Second, Trumbo wasn't fired by the US government. He was fired by private companies who didn't want to look bad.
Same shit.
Re: (Score:2)
"Second, Trumbo wasn't fired by the US government. He was fired by private companies who didn't want to look bad."
Trumbo was investigative by a government committee of the United States House of Representatives because of his beliefs, he was visited by the FBI, and he was jailed. The MPAA also issued a statement that Trumbo and people like him would not be permitted to work in the industry. That is a long shot from what happened at Google.
"First, it's not just Google. It's most tech companies."
I didn't say
Re: (Score:2)
It's always exhausting to discuss with aspies but you're clearly not a troll so let me summarize things for you.
The climate of political correctness in the USA has crossed an important threshold: people being fired publicly not for incompetence or malice, and not even for badmouthing their employer, but merely for voicing opinions internally about subjects that are not even relevant to the business.
This is similar, although not factually identical (yet) with what happened during the witch trials in Salem a
Re: (Score:2)
"This is similar, although not factually identical (yet) with what happened... "
No it's not. I think you are pushing the relation way too far. Damore was fired by Google. Trumbo was investigative by a government committee of the United States House of Representatives, he was visited by the FBI, he was jailed, and the MPAA stated that Trumbo and people like him would not be permitted to work in the industry.
"people being fired ... merely for voicing opinions internally about subjects that are not even releva
Re: (Score:2)
Thanks for the correction and info.
How... (Score:3)
did a company that does no provide transit services even manage to send a route table that was accepted for use? Just seems like a very exploitable issue there. Does google have authority or permissions that allows this, even though they don't actually have the capabilities?
Re: (Score:1)
Well, in this scenario where talking about Google breaking the internet.
They with the help of the ISPs did it unintentionally this time.
But if Google decided to orchestrate an attack on the internet, they could bring it to its knees in so may ways.
Simply disabling all there local caches would overload most transit lines in the world. That would make everything suck.
It they turned them into a zombie network, everything would break until they where disabled.
The internet of the past is gone, it is no longer a
Re:How... (Score:5, Informative)
Re: (Score:1)
Thanks for letting me be lazy today
Re: (Score:1)
Very well known and it is fixed. But not in the manner stated.
The receivers just happen to trust Google to much and did not apply normal filters and checks. It is more like they did not enable the firewall because they could not be bothered to set up the necessary rules.
That is why there is no fix. BGP is not broken, imperfect for sure. But it does not need to be replaced, the tweaks that are added almost always work with the internal usage side. Like inside an ISP where it is used for all kinds of informat
Re:How... (Score:5, Interesting)
Multihoming. I'd imagine Google provides transit service, but only for their own IP blocks. Each Google datacenter almost certainly has multiple Internet connections to the world. As a result, they have multiple netblocks provided by multiple ISPs.
If, through some unlucky DNS accident, a client on ISP A looks up google.com and gets an IP address provided by ISP B, it would take many more hops to reach that server via public Internet routes than by sending traffic to that datacenter's nearest router (on ISP A) and asking that router to forward traffic through the datacenter to the other set of IPs.
Re: How... (Score:2)
"Multihoming. I'd imagine Google provides transit service, but only for their own IP blocks. Each Google datacenter almost certainly has multiple Internet connections to the world. As a result, they have multiple netblocks provided by multiple ISPs."
No, multi-homing doesn't require netblocks from multiple ISPs. Doing this is a hack and only feasible for very trivial networks. A much better approach is BGP multi-homing.
Google (along with thousands of smaller "Enterprise" businesses) have their own autonomous
Re: (Score:2)
Fair enough, but whether it's a single block owned by Google or one or more blocks owned by one or more ISPs, they still have to use BGP to advertise routes for a block of IPs that doesn't belong to their upstream ISP, so the distinction is largely academic.
Re: (Score:1)
This happens because their peers were too lazy or otherwise unable to set up and maintain proper route filters for Google's ASNs. Problem is that with Google Cloud and Google Fiber (for business), google actually sometimes *does* provide transit, so it would be a difficult thing for all of their upstream providers to keep continuously updated -- instead they think "Well, they are Google. They are probably doing that all on their end and won't ever fuck this up."
Re: How... (Score:3)
Google fibre is definitely a separate AS from Google (Search, Youtube etc.) AS15169, not sure about GCP.
AS15169 doesn't provide transit, so Verizon could have easily prevented this with just as-path filters (although prefix filters in addition would be better, but may incur more maintenance).
Re:How... (Score:5, Insightful)
BGP routing has no authority mechanism. Anyone can publish any route. This is not [bgpmon.net] the [bgpmon.net] first [networkworld.com] time [pcworld.com] this has happened, nor will it be the last.
Re: (Score:1)
Why would you want one? All that would result from it is Yet Another Centralized Bureaucracy that would provide security theater to the public, create another barrier to entry for ISPs, and be a huge target for the likes of the NSA, FBI, etc. Nevermind all of those morons wanting to implement an "internet killswitch" or mandate a federal porn filter / Great Internet Wall of the USA.
Long story short, given the current online political climate, adding new ways to screw up
Re: (Score:1)
Authority is given by every individual peer, by contract and implementation.
They are all autonomous from each other like nations.
There is no one table being sent, there is a bunch on routes in one or more updates. This can be added to the routing database on the receiving side IF THEY CHOOSE. They should have had filters and safe guards like a max amount of prefixes accepted over this peer.
I would say that Google is taking the high road but the receiving ISPs are really to blame. They did not protect there
Sad but true... (Score:2)
Re: (Score:1)
creimer is the new Bigfoot, stomping on everyone's Slashdot user experience.
Re: (Score:2)
Google is the new Godzilla, stomping on everyone's Internet connections.
Does that make Bing "Mothra"?
Duck Duck Go is "Megalon"?
and Yahoo is "Tickle Me Elmo"?
Re: (Score:2)
So much for "routing around" (Score:3)
So much for the fabulous promise [wikiquote.org] of "routing around damage"...
Re: (Score:1)
Yeah, that does not work when the routing is not the problem.
Finally a Google story makes it through. (Score:1)
/. has been strangely silent about Google shenanigans.
Was this Espresso? (Score:2)
Google has been touting how Espresso [sdn-lab.com] "moves BGP to a custom stack running on servers, enabling finer-grained partitioning and much more computation power than available in any Internet router." Theoretically providing real-time changes in edge routing to maximize video streaming performance based on client metrics. Unless there is a bug...