One Broken Router Takes Out Half the Internet? 412
Silent Stephus writes "I work for a smallish hosting provider, and this morning we experienced a networking event with one of our upstreams. What is interesting about this, is it's being caused by a mis-configured router in Europe — and it appears to be affecting a significant portion of the transit providers across the Internet. In other words, a single mis-configured router is apparently able to cause a DOS for a huge chunk of the Net. And people don't believe me when I tell them all this new-fangled technology is held together by duct-tape and baling wire!"
Half the internet? Are you serious? (Score:5, Informative)
A router takes out 'half the internet' and I learn this from Slashdot?
Seriously, what is/was the impact? I work for a large e-commerce provider and haven't seen a thing that would indicate a problem today.
BGP (Score:5, Informative)
The internet's dirty little secret. It's amazing it works at all.
Yep, Its true (Score:5, Informative)
AS 47868 (Score:5, Informative)
There is a post in nanog and on isc.sans.org.
AS 47868 causing AS paths to become too long...
http://www.merit.edu/mail.archives/nanog/msg15472.html
Ditto the A.C. (Score:5, Informative)
It must have been the "half the Internet" that I don't use. Which would be an interesting half because many of the sites I visit regularly are based in Europe.
From the thread, it looks like AS 47868 was the route being lost.
http://en.wikipedia.org/wiki/Autonomous_System_Number [wikipedia.org]
baling wire, not bailing wire (Score:5, Informative)
http://en.wikipedia.org/wiki/Baling_wire [wikipedia.org]
I think you mean baling wire. One uses buckets for bailing.
Outage Cause: Old software (Score:5, Informative)
The AS 47868 decided that they wanted to prepend their ASN about 75 or so times to their BGP announcements. When this got re-populated throughout the rest of the world, a bug in older versions of Cisco IOS still in use on many ISP/NSP networks does not like paths this long. As soon as they saw the prefix with that long of a path, the software terminated the BGP session, resulting in the doorway being closed between the two networks -- So on and so forth throughout the rest of the web.
Re:Ditto the A.C. (Score:3, Informative)
I think AS47868 was causing the routes to be lost.
It was making mass BGP announcements about really long incorrect routes.
Ye olde versions of IOS (Score:5, Informative)
This only broke BGP implementations that are getting pretty long in the tooth now, on a moderately recent version of IOS all we saw is:
Feb 17 05:25:03.731 nzdt: %BGP-6-ASPATH: Long AS path 10026 3356 29113 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 received from xxx.xxx.xxx.xxx: More than configured MAXAS-LIMIT
It was definitely an insane path, our routers were configured to drop anything with an AS path longer than 75, old versions of IOS would often just drop the BGP session ( or even crash with some _really_ old versions ).
I'm sure there will be some red faced network engineers updating IOS or even doing forklift upgrades of old boxes at their edges in the near future.
BGP Misconfigurations (Score:2, Informative)
Re:Ditto the A.C. (Score:5, Informative)
It wasn't just AS47868, it was kicked off by AS47868 sending real long routes like you can get to a by going through b, c, d, e, f ,g, h... and so on and so forth. Older versions of IOS wack out with the crazy long routes and lose their BGP sessions so it is possible that he lost half of the internet while you were on a network segment which was not seeing the issue. If the OP were to post the ASN or IP block he was on we could run BGP play and see just how much of the net he really lost. I'm going to guess about .5%.
Tens of thousands of web sites were affected (Score:3, Informative)
Only some old versions of IOS broke (Score:5, Informative)
This only took down people running fairly old versions of IOS that didn't patch a known bug.
Did not affect non-cisco.
Did not affect modern versions of IOS
Did not affect old versions of IOS that set the knob to limit the max as-path.
Re:Trust (Score:3, Informative)
This only hit people running old unpatched versions of IOS. Known and patched long ago.
Re:I'm not sure I follow (Score:5, Informative)
If I'm understanding this 'router' thing correctly, its like a faucet connected to the series of tubes?
If not, exactly what role does this router thing play in tube interaction?
Your understanding is rather accurate but what your missing is the manifolds. You see, all the tubes connect to big manifolds with valves to control what gets sent where. At each manifold room there is some poor admin who is in charge of opening and closing valves in order to make sure that the right AOL gets sent down the right tube. In order to keep track of what tube to send your AOL down, the admin keeps a list of all the other manifold rooms and how to get to them. Some of the manifold room operators didn't have a wide enough notebook to write down the new directions so they just closed all of their valves and went home.
Re:baling wire, not bailing wire (Score:3, Informative)
Could I use baling wire to make a new bail for my bailing bucket? If so, would my wire bail made from baling wire become bailing wire?
Concerned and puzzled.
No, because the noun "bail" in the sense you use it means "handle in the shape of an arc". There is no verb form in reference to that noun, therefore there can be no "bailing wire". It's still just "a baling wire bail on your bailing bucket".
Re:Only some old versions of IOS broke (Score:3, Informative)
Gee, you only described about half the mistakes that incompetent network admins could possibly make (buying a fucking Cisco, not updating their NOS, and not limiting AS paths).
That covers half the ISPs in Texas (including mine - these fuckwits can barely configure their routers correctly on a good day, let alone deal with a crisis brought about their own incompetence). YMMV.
I'd mod you up but I already posted.
Re:Intelligence Op (Score:5, Informative)
They need to replace it with a network that is designed to survive a nuclear attack. Oh wait, hang on....
Wish I had mod points today. Parent should already be SCORE:5 Funny. Apparently not enough Slashdotters know the history/evolution of the net.
If you're referring to the myth that the Internet was "designed to withstand nuclear attack", perhaps Slashdotters know more than you think.
The Internet was designed to allow distributed control, and to withstand telephone company malice and incompetence. This was a much more useful goal than withstanding nuclear attack.
One of the early arguments made by DARPA folks to politicians, in order to secure continued federal funding for packet switched network development, was the ability of the network to route around failed or destroyed nodes. They made this argument in the context of the cold war, of nuclear war.
It reality, as you state, this argument had little practical impact on the technical development or evolution of the the network. However, it most certainly did have an impact on the commitment of federal/military funding. This is the origin of the "surviving nuclear attack" lore of the development of DARPANET. It's not a myth. It's real.
Take Obama's current stimulus package as a parallel example. It's not going to solve the recession, but it's being sold as such. And the congress bought into it. Just as this stimulus bill isn't what it's being sold as, most likely DARPANET wouldn't have really given us what it was sold as at one point. Nonetheless, it was sold as such, thus creating the lore that you call myth.
Mod parent up (Score:5, Informative)
Mod the parent up - this is the real cause of the problem.
bgp maxas-limit 75 [cisco.com]
would stop this on most routers.
Re:Intelligence Op (Score:5, Informative)
Aw heck, someone in Nebraska is going to trip over one power cord, and shut down the Interweb. :)
In addition to using public maps, I did a lot more research. I had my own little project going for a little while. The project was intended to monitor for faults between datacenters we had equipment in. I added the root nameservers. I also had a few other points, such as friends houses and places they had virtual hostings at.
Simply enough, it was running traceroutes from everywhere I had control to all points in my "network". I stored what router attached to each hop in a database.
I located each hop simply by the city it was located in. Some were easy. Some weren't so easy.
It was fun and games with 100 routers. I was manually setting city and state locations.
It was a little less fun when it grew to 500 routers. I wrote regular expressions to take known naming conventions and make them into city names. That sounds easy, but it gets pretty hard pretty quick.
It was a lot less fun when the list grew to several thousand routers.
Basically, ever time there was a routing change, I found new routers.
I had a lot of fun using both Google Maps to show the routes (for routers that I could place in a city), and a Graphviz model of the Internet as we observed it. It was a very big map. That was only what we had observed. I doubt we even saw a very small percentage (probably less than 0.01%) of the routes.
The map got very very very complicated. I could point out choke points. They existed, but there were also alternative routes.
Hell, even on a single good provider, there are no good choke points. On one Tier 1 provider that I used, in a non-core city, they had 6 diverse routes with OC192's. It wasn't a matter of me trusting them when they told me. I saw the routes showing up.
There are 4 cities in the US, where if say a big nuke hit each one, ya, the Internet would be hurting. You may not get from Provider A to Provider B, but you'd still have some connectivity within your own provider, and other peerings would start working fairly quickly. More obviously, you'd find that some sites that are hosted in one city would be inaccessible. That's why geographic and topological diversity is very important for anyone who wants to keep their stuff up and running.
Google puts stuff out all over the place for a reason. If a route, or a dozen routes, go funky, you'll very likely still be able to reach some datacenter.
My office is connected by 3 uplinks. They're all with different providers. The odds of a provider outage killing the office is pretty slim. Other things can happen though. Lightning hit a transformer across the street, which serviced our building. From what people on that side of the building said, it was very pretty. :) Was our Internet connection dead? No. Well, not totally. We still had 2 uplinks working. We didn't have power for the desktops though. The UPS (a big one, not the little desktop ones) provides for the server room and a very few workstations.
The biggest effect we saw from that outage was that cell phone service became minimal. The top of our building is also used for cell phone coverage. Without those antennas working, we only had service from the surrounding towers. It probably didn't help that there was now an office building full of people who were evacuated to the ground floor (it tripped the fire alarm), so almost everyone were on their cell phones making calls to customers, friends, family, etc.
The most upset people were stuck in the elevator. They were already going downstairs for a smoke break, when it got stuck because those aren't backed up with anything at all.
Re:TAG THIS ARTICLE KDAWSONSUCKS (Score:1, Informative)
Er, yes, it does? Amsterdam and London are two of the worlds largest internet hubs for trans-continental traffic. Traffic from Europe, Africa, Russia the middle east and large parts of Asia route through Europe to the US. Look at a map sometime and you'll notice that North & South America are a very large island bordered by two very large oceans.
Re:Half the internet? Are you serious? (Score:3, Informative)
That's not actually quite true. Depending on where you are, you might be able to use it in certain circumstances.
For example, in British English, you would use an apostrophe for plurals of single letters (there were 10 C's). You can also use it to create plurals of abbreviations, especially where there would be ambiguity (Four IOU's), as a slightly old-fashioned plural of figures (in the 1930's, but 1930s is becoming predominant) and where short words would be odd if you simply added an 's' - for example, the Oxford English Dictionary gives both "yesses" and "yes's" as plurals of "yes".
Whether he was right or not in this case is debatable, but I can certainly see the logic in writing "T1's" rather than "T1s", to avoid the appearance of it being a different abbreviation - and it isn't without precedent.
Re:Few stories back... (Score:3, Informative)
In the interest of completeness USS Ohio is no longer SSBN-726, it is now SSGN-726. It was converted a few years back from a Trident-carrying boomer to a "slow attack" capable of carrying 100+ Tomahawks, plus some SEAL capabilities.