Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
Google Japan The Internet IT Technology

Google Takes Blame For Internet Disruption Across Japan (theregister.co.uk) 59

An anonymous reader shares a report: Google on Saturday accepted responsibility for the widespread internet disruptions Japan experienced the previous day. The search engine giant apologized for the trouble, saying it was caused by an errant network setting that was corrected within eight minutes of its discovery. Google did not say whether human error or a technical malfunction was to blame. The disrupted services used internet connections provided by NTT Communications Corp. and KDDI Corp., both of which said Friday that the issues were caused by a change in the flow of data traffic. From a report on The Register: The trouble began when Google 'leaked' a big route table to Verizon, the result of which was traffic from Japanese giants like NTT and KDDI was sent to Google on the expectation it would be treated as transit. Since Google doesn't provide transit services, as BGP Mon explains, that traffic either filled a link beyond its capacity, or hit an access control list, and disappeared. The outage in Japan only lasted a couple of hours, but was so severe that Japan Times reports the country's Internal Affairs and Communications ministries want carriers to report on what went wrong.
This discussion has been archived. No new comments can be posted.

Google Takes Blame For Internet Disruption Across Japan

Comments Filter:
  • by Dirk Becher ( 1061828 ) on Monday August 28, 2017 @11:15AM (#55097349)

    if you can claim responsibility for everything bad that happens on the net and everyone believes you immediately.

    • by Anonymous Coward on Monday August 28, 2017 @11:37AM (#55097491)

      Hmm, actually we *knew* it was them in the first place the moment it started, most non-joke internet network engineers refuse to fly blind, so there are probes and monitors everywhere in the Internet control plane (DFZ BGP routing). E.g. read: https://bgpmon.net/bgp-leak-causing-internet-outages-in-japan-and-beyond/

      Also, most internet network engineers will place a lot of the blame on *Verizon*, not Google. "route leaks" are a *fact of life*, they will happen at least once to everyone. You *MUST* filter the routing plane to not accept crap from other autonomous systems you peer with, and Verizon *utterly failed* at doing it. Had they filtered, they'd have rejected the bogus routing from Google and avoided most, if not all of the damage.

      So, Google might have publicly taken the blame since it was their operational error that triggered the damage in the first place, but they are at most responsible for half of it... and *everyone in the field* knows it, Google included.

  • by Drethon ( 1445051 ) on Monday August 28, 2017 @11:16AM (#55097351)

    did a company that does no provide transit services even manage to send a route table that was accepted for use? Just seems like a very exploitable issue there. Does google have authority or permissions that allows this, even though they don't actually have the capabilities?

    • Re:How... (Score:5, Informative)

      by Infiniti2000 ( 1720222 ) on Monday August 28, 2017 @11:29AM (#55097433)
      TFA explains how, and at the end of the TFA: "There are various proposals to tweak BGP to stop this sort of thing happening, but as is so often the case, implementation is lagging far behind requirement." In short, it's a known problem and eventually maybe someone will care enough to fix it.
      • Thanks for letting me be lazy today

      • by GESUS ( 557517 )

        Very well known and it is fixed. But not in the manner stated.

        The receivers just happen to trust Google to much and did not apply normal filters and checks. It is more like they did not enable the firewall because they could not be bothered to set up the necessary rules.

        That is why there is no fix. BGP is not broken, imperfect for sure. But it does not need to be replaced, the tweaks that are added almost always work with the internal usage side. Like inside an ISP where it is used for all kinds of informat

    • Re:How... (Score:5, Interesting)

      by dgatwood ( 11270 ) on Monday August 28, 2017 @11:36AM (#55097473) Homepage Journal

      How did a company that does no provide transit services even manage to send a route table that was accepted for use? Just seems like a very exploitable issue there

      Multihoming. I'd imagine Google provides transit service, but only for their own IP blocks. Each Google datacenter almost certainly has multiple Internet connections to the world. As a result, they have multiple netblocks provided by multiple ISPs.

      If, through some unlucky DNS accident, a client on ISP A looks up google.com and gets an IP address provided by ISP B, it would take many more hops to reach that server via public Internet routes than by sending traffic to that datacenter's nearest router (on ISP A) and asking that router to forward traffic through the datacenter to the other set of IPs.

      • "Multihoming. I'd imagine Google provides transit service, but only for their own IP blocks. Each Google datacenter almost certainly has multiple Internet connections to the world. As a result, they have multiple netblocks provided by multiple ISPs."

        No, multi-homing doesn't require netblocks from multiple ISPs. Doing this is a hack and only feasible for very trivial networks. A much better approach is BGP multi-homing.

        Google (along with thousands of smaller "Enterprise" businesses) have their own autonomous

        • by dgatwood ( 11270 )

          No, multi-homing doesn't require netblocks from multiple ISPs. Doing this is a hack and only feasible for very trivial networks. A much better approach is BGP multi-homing.

          Fair enough, but whether it's a single block owned by Google or one or more blocks owned by one or more ISPs, they still have to use BGP to advertise routes for a block of IPs that doesn't belong to their upstream ISP, so the distinction is largely academic.

    • by Anonymous Coward

      This happens because their peers were too lazy or otherwise unable to set up and maintain proper route filters for Google's ASNs. Problem is that with Google Cloud and Google Fiber (for business), google actually sometimes *does* provide transit, so it would be a difficult thing for all of their upstream providers to keep continuously updated -- instead they think "Well, they are Google. They are probably doing that all on their end and won't ever fuck this up."

      • Google fibre is definitely a separate AS from Google (Search, Youtube etc.) AS15169, not sure about GCP.

        AS15169 doesn't provide transit, so Verizon could have easily prevented this with just as-path filters (although prefix filters in addition would be better, but may incur more maintenance).

    • Re:How... (Score:5, Insightful)

      by The Raven ( 30575 ) on Monday August 28, 2017 @11:39AM (#55097499) Homepage

      BGP routing has no authority mechanism. Anyone can publish any route. This is not [bgpmon.net] the [bgpmon.net] first [networkworld.com] time [pcworld.com] this has happened, nor will it be the last.

      • by Anonymous Coward

        BGP routing has no authority mechanism

        Why would you want one? All that would result from it is Yet Another Centralized Bureaucracy that would provide security theater to the public, create another barrier to entry for ISPs, and be a huge target for the likes of the NSA, FBI, etc. Nevermind all of those morons wanting to implement an "internet killswitch" or mandate a federal porn filter / Great Internet Wall of the USA.

        Long story short, given the current online political climate, adding new ways to screw up

    • by GESUS ( 557517 )

      Authority is given by every individual peer, by contract and implementation.

      They are all autonomous from each other like nations.

      There is no one table being sent, there is a bunch on routes in one or more updates. This can be added to the routing database on the receiving side IF THEY CHOOSE. They should have had filters and safe guards like a max amount of prefixes accepted over this peer.
      I would say that Google is taking the high road but the receiving ISPs are really to blame. They did not protect there

  • Google is the new Godzilla, stomping on everyone's Internet connections.
    • by Anonymous Coward

      creimer is the new Bigfoot, stomping on everyone's Slashdot user experience.

    • Google is the new Godzilla, stomping on everyone's Internet connections.

      Does that make Bing "Mothra"?
      Duck Duck Go is "Megalon"?

       

       

      and Yahoo is "Tickle Me Elmo"?

  • widespread internet disruptions Japan experienced the previous day.

    So much for the fabulous promise [wikiquote.org] of "routing around damage"...

  • by Anonymous Coward

    /. has been strangely silent about Google shenanigans.

  • Google has been touting how Espresso [sdn-lab.com] "moves BGP to a custom stack running on servers, enabling finer-grained partitioning and much more computation power than available in any Internet router." Theoretically providing real-time changes in edge routing to maximize video streaming performance based on client metrics. Unless there is a bug...

panic: kernel trap (ignored)

Working...