Why Google Went Offline Today 110
New submitter mc10 points out a post on the CloudFlare blog about the circumstances behind Google's services being inaccessible for a brief time earlier today. Quoting:
"To understand what went wrong you need to understand a bit about how networking on the Internet works. The Internet is a collection of networks, known as "Autonomous Systems" (AS). Each network has a unique number to identify it known as AS number. CloudFlare's AS number is 13335, Google's is 15169. The networks are connected together by what is known as Border Gateway Protocol (BGP). BGP is the glue of the Internet — announcing what IP addresses belong to each network and establishing the routes from one AS to another. An Internet "route" is exactly what it sounds like: a path from the IP address on one AS to an IP address on another AS. ... Unfortunately, if a network starts to send out an announcement of a particular IP address or network behind it, when in fact it is not, if that network is trusted by its upstreams and peers then packets can end up misrouted. That is what was happening here.
I looked at the BGP Routes for a Google IP Address. The route traversed Moratel (23947), an Indonesian ISP. Given that I'm looking at the routing from California and Google is operating Data Centre's not far from our office, packets should never be routed via Indonesia."
All your packets are belong to... (Score:5, Interesting)
... Network Admins who have no clue. Like when just 4 years ago, Pakistan took down Youtube...
http://securitywatch.pcmag.com/dns/285152-pakistan-takes-youtube-down [pcmag.com]
Clearly this should be on the agenda for the new "Cyber Reserves" of the department of Homeland Security. If Google can be taken down by accident in parts of the world, then it certainly can be taken down on purpose. Route filters are your friends!
CYBER RESERVES: http://www.techradar.com/news/internet/department-of-homeland-security-recruiting-for-cyber-reserve-1109906 [techradar.com]
Root cause was PCCW, not Moratel (Score:5, Interesting)
From TFA:
Someone at Moratel likely "fat fingered" an Internet route. PCCW, who was Moratel's upstream provider, trusted the routes Moratel was sending to them. And, quickly, the bad routes spread.
Yes, someone at Moratel screwed up, but this is exactly why upstream ISPs should never allow advertisements from their customers for networks that their customer does not control.
PCCW is to blame for allowing this to happen. Never trust customers with things that don't belong to them.
On the other hand: escape from bad ISP (Score:5, Interesting)
This sort of 'feature' did allow me once to escape from a misbehaving ISP holding me hostage and preventing me getting my mail to, for example, change my DNS glue records many many years ago. A helpful friendly new ISP managed to reroute traffic to me via them with a "bogus" routing announcement long enough for me to fix those records and then escape the old ISP when the new records propagated.
Rgds
Damon
China already did this in 2010 (Score:5, Interesting)
Re:Root cause was PCCW, not Moratel (Score:5, Interesting)
PCCW is to blame for allowing this to happen.
Again. They were also the upstream for the Pakistan-takes-down-YouTube fiasco.
Re:Filtering (Score:4, Interesting)
I get the feeling that upstreams should start to not completely trust BGP announcements from peers.
Start? This was BAU at respectable ISPs a decade ago. Guess what I was doing at that time, endless Fing around with filtering. Bureaucratic level varied a lot over time but when I left that part of the biz it was crystallizing around something like the 800 number letter of agency process, where you need a company officer to fax a signed sheet verifying thats really your space and yes we really do have permission to advertise it. At least at that time ARIN did not do dun and bradstreet numbers and there's no way to verify via whois and everyones merging, so we needed that signed letter to protect us legally just as much as the internet needed it so we could protect the internet from them. At least as I recall.
Basically if you are "Ford dealer of chicago" I have no legal idea if you're allowed to advertise ARIN's ford.com space, but if we have a LOA then at least if it all hits the legal fan we have a signed letter from a corporate officer at the dealership to get us off the hook (at least partially) when the real ford goes after us, or at least we can tell the "real ford" who to add to the lawsuit. Many a time I had to call the ARIN registered owners to verify an apparently unrelated minion should be advertising some of their space. Sometimes yes, sometimes no. It was always an entertaining conversation. Except for when the ARIN contact info was invalid. Then the swearing began.
Most of the time, obviously, its just a dude advertising additional space with identical ARIN contact info as the old space, so it doesn't come to this level of paperwork.
I don't know if the situation has gotten better or worse since the mid 00s.
but the closer to the core you get the larger the list of potentially valid ASes
Ah but that's not where you need it. At least not for black hole events like this. If I'm properly filtering at the border, I don't need to filter in the middle, in fact it shouldn't ever be even theoretically necessary and its none of the cores business what business deal I've signed at the border anyway. Also god help us there were people trying to what amounts to dynamically load balance and disaster recovery using BGP, not necessarily a "stable" situation anyway. Route flap dampening is enough of a PITA.