Forgot your password?
typodupeerror
The Internet Networking

One Broken Router Takes Out Half the Internet? 412

Posted by kdawson
from the brain-gone-punky dept.
Silent Stephus writes "I work for a smallish hosting provider, and this morning we experienced a networking event with one of our upstreams. What is interesting about this, is it's being caused by a mis-configured router in Europe — and it appears to be affecting a significant portion of the transit providers across the Internet. In other words, a single mis-configured router is apparently able to cause a DOS for a huge chunk of the Net. And people don't believe me when I tell them all this new-fangled technology is held together by duct-tape and baling wire!"
This discussion has been archived. No new comments can be posted.

One Broken Router Takes Out Half the Internet?

Comments Filter:
  • Looking to make the big blackout, when needed.

    See Also: Severed Mediterranean Cables.

    • BGP (Score:5, Informative)

      by winkydink (650484) * <sv.dude@gmail.com> on Monday February 16, 2009 @07:39PM (#26879451) Homepage Journal

      The internet's dirty little secret. It's amazing it works at all.

    • by agm (467017) on Monday February 16, 2009 @08:15PM (#26879943)

      They need to replace it with a network that is designed to survive a nuclear attack. Oh wait, hang on....

    • by kenj0418 (230916) on Monday February 16, 2009 @09:15PM (#26880617)

      Don't worry, it wasn't a DOS attack. That was just the Internet becoming self-aware.

      OK, on second thought, maybe worrying is in order.

    • by CarpetShark (865376) on Monday February 16, 2009 @09:23PM (#26880717)

      Yeah, this was my first thought as well. It seems clear that the internet, while designed to route traffic through all sorts of alternate links, is almost certainly being routed through single, centralised listening posts at various intervals.

  • by Anonymous Coward on Monday February 16, 2009 @07:34PM (#26879387)
    A couple of Nuclear Subs probably cut an underwater cable...
  • by Anonymous Coward on Monday February 16, 2009 @07:38PM (#26879423)

    A router takes out 'half the internet' and I learn this from Slashdot?

    Seriously, what is/was the impact? I work for a large e-commerce provider and haven't seen a thing that would indicate a problem today.

    • Ditto the A.C. (Score:5, Informative)

      by khasim (1285) <brandioch.conner@gmail.com> on Monday February 16, 2009 @07:43PM (#26879527)

      It must have been the "half the Internet" that I don't use. Which would be an interesting half because many of the sites I visit regularly are based in Europe.

      From the thread, it looks like AS 47868 was the route being lost.

      http://en.wikipedia.org/wiki/Autonomous_System_Number [wikipedia.org]

    • by Frosty Piss (770223) on Monday February 16, 2009 @07:49PM (#26879631)

      A router takes out 'half the internet' and I learn this from Slashdot?

      Non, no, no. You messed up the troll and got modded "Insightful". Let me fix that for you:

      A router takes out 'half the internet' and this is front page news at Slashdot? Slow news day?

      Thank you, I'll be here all week...

    • by Anonymous Coward on Monday February 16, 2009 @08:42PM (#26880259)

      A router takes out 'half the internet' and I learn this from Slashdot?

      Seriously, what is/was the impact? I work for a large e-commerce provider and haven't seen a thing that would indicate a problem today.

      Well I'm not sure about you.

      Personally, I have BIGGER news! A single router in a remote rural US state managed to take down the ENTIRE INTERNETS!!!!

      Yes, indeed when I noticed my cat had unplugged the power adapter, I replaced it. Then the ENTIRE internet came back! It was amazing how I single-handedly brought back the whole internets. Al Gore would be proud.

  • Sorry (Score:5, Funny)

    by Alcoholist (160427) on Monday February 16, 2009 @07:38PM (#26879425) Homepage

    My bad. I never should have cut that tape.

  • Hmm... (Score:3, Interesting)

    by Vectronic (1221470) on Monday February 16, 2009 @07:39PM (#26879445)

    I suppose that a networking event with one of our upstreams [merit.edu] was behind that router?

    3/11 (invalid or corrupt AS path)

    Or maybe I'm behind that router?

  • Pre-FUD propaganda (Score:3, Interesting)

    by marco.antonio.costa (937534) on Monday February 16, 2009 @07:40PM (#26879475)

    No, we DON'T NEED A NEW INTERNET! Stop pitching it, statist drones.

    The internet works fine, and that's what the RIAA/MPAA/etc are trying to fix.

  • by Forge (2456) <kevinforge AT gmail DOT com> on Monday February 16, 2009 @07:41PM (#26879483) Homepage Journal
    Lucky Yankees with all your fancy technology. If I told you what we use, nobody would respond for fear that in attempting to respond I would cause a few fatalities.
  • Yep, Its true (Score:5, Informative)

    by Bryansix (761547) on Monday February 16, 2009 @07:41PM (#26879485) Homepage
    Our Hosted VOIP service took a dump today at 8:40 AM PST. Supposedly it was a server in the Czech Republic. From the Carrier

    2009-02-16 0945 PST CP experienced a core network connectivity issue due to a world wide BGP issue that affected all BGP interconnected networks. A rouge machine in the Czech Republic was making bad AS advertisements that caused systems world wide to fail. We have worked with our providers as well as our internal Engineering department to effectively block this node and restore service to our network. This is an ongoing issue that is still being worked to get a 100% correction. There is a workaround currently in place until a complete fix is available.

    • by radish (98371) on Monday February 16, 2009 @07:48PM (#26879619) Homepage

      A rouge machine in the Czech Republic was making bad AS advertisements that caused systems world wide to fail.

      Now I really don't know all that much about large-scale networking so maybe someone could explain this to me. What difference does it make if the router is rouge, versus say, green? or black?

      Thanks for any insight :)

    • by Jurily (900488)

      A rouge machine in the Czech Republic was making bad AS advertisements that caused systems world wide to fail.

      Wasn't the internet designed to withstand WW3? And now a single machine takes it down?

      Please enlighten me, how is that possible?

      • by Zironic (1112127)

        Internet is meant to be able to route around broken nodes and it seems to do this rather well, just not as quickly as some would like.

      • Re: (Score:3, Insightful)

        by mail2345 (1201389)
        WW3 is an external problem.
        A misconfigured machine is an internal problem.
        The internet can survive cut cables, provided that there are other routes.
        But if it can't find said routes, then there is a problem.
      • Re:Yep, Its true (Score:4, Insightful)

        by v1 (525388) on Monday February 16, 2009 @09:13PM (#26880599) Homepage Journal

        It was meant more to stop the network from failing due to LOST nodes, not malfunctioning nodes. But that doesn't say much for its ability to withstand sabotage which is expected in wartime.

      • Re: (Score:3, Funny)

        by Zwicky (702757)

        Wasn't the internet designed to withstand WW3?

        Well originally yes. Over the years other needs have dictated a different direction to take it in. These days it is designed not so much to withstand WW3, but to withstand WoW3 (and up).

    • A rouge machine in the Czech Republic

      That's the problem. You shouldn't use rouge on your routers.

  • AS 47868 (Score:5, Informative)

    by Anonymous Coward on Monday February 16, 2009 @07:42PM (#26879495)

    There is a post in nanog and on isc.sans.org.

    AS 47868 causing AS paths to become too long...

    http://www.merit.edu/mail.archives/nanog/msg15472.html

  • Oops (Score:4, Funny)

    by kbob88 (951258) on Monday February 16, 2009 @07:42PM (#26879509)

    Sorry, I *told* Mustafa not to drop the anchor there! But does he listen to me? No...

  • Trust (Score:3, Insightful)

    by TubeSteak (669689) on Monday February 16, 2009 @07:43PM (#26879535) Journal

    Until the internet evolves away from its trust-everyone roots,
    one well placed server will be able to cause massive damage.

    There would be a lot more impetus to force the change if hackers were nuking things from orbit for lulz instead of infiltrating systems for business reasons (spamming, bot herds, etc).

    • Re: (Score:3, Informative)

      by lotaris (34307)

      This only hit people running old unpatched versions of IOS. Known and patched long ago.

      • Re: (Score:3, Insightful)

        by lambent (234167)

        Precisely. It wasn't "one broken router" that took out half the net, it was thousands of substandard routers using obsolete code run by incompetent admins that took out half the net.

        the people who actually know what they're doing were unaffected by this.

  • by bugi (8479) on Monday February 16, 2009 @07:43PM (#26879539)

    http://en.wikipedia.org/wiki/Baling_wire [wikipedia.org]

    I think you mean baling wire. One uses buckets for bailing.

    • by Tomun (144651)

      No only that, but leave duct tape alone. It's an excellent way of holding things together. I'd trust your life to the stuff.

    • Could I use baling wire to make a new bail for my bailing bucket? If so, would my wire bail made from baling wire become bailing wire?

      Concerned and puzzled.

      • Re: (Score:3, Informative)

        by Dun Malg (230075)

        Could I use baling wire to make a new bail for my bailing bucket? If so, would my wire bail made from baling wire become bailing wire?

        Concerned and puzzled.

        No, because the noun "bail" in the sense you use it means "handle in the shape of an arc". There is no verb form in reference to that noun, therefore there can be no "bailing wire". It's still just "a baling wire bail on your bailing bucket".

  • by XanC (644172) on Monday February 16, 2009 @07:45PM (#26879563)

    What is Jen doing with The Internet??

    • Re: (Score:3, Funny)

      by Anonymous Coward

      Googling Google on google?

  • If you can memorize this, you'll be the life of any cocktail party:

    "We're seeing them from AS 48438, coming across to us as an Optional Transitive Attribute which our force-10s are not parsing (but cheerfully passing along to our clients, who are then flapping their peers because of it.)"

    Uh-huh-huh-uh! They've been "flapping their peers".

  • by fm6 (162816) on Monday February 16, 2009 @07:50PM (#26879643) Homepage Journal

    Well, do, you're right to be concerned. The thing is, our technology infrastructure has always been a nasty kludge. In 1965, some coincidental misconfigurations at two minor power plants took out the power grid for an area in the northeast U.S. and eastern Canada where 25 million people lived. It was 14 hours before the grid was fully restored. Our inability to keep our technical house in order is a very old problem.

    • Even before that... (Score:3, Interesting)

      by tjstork (137384)

      The ancient egyptians

      http://www.flickr.com/photos/hdonat/2422108343/ [flickr.com]

      had their engineering problems too.

      As soon as we humans invented technology, we humans began screwing it up.

  • by Anonymous Coward on Monday February 16, 2009 @07:50PM (#26879649)

    The AS 47868 decided that they wanted to prepend their ASN about 75 or so times to their BGP announcements. When this got re-populated throughout the rest of the world, a bug in older versions of Cisco IOS still in use on many ISP/NSP networks does not like paths this long. As soon as they saw the prefix with that long of a path, the software terminated the BGP session, resulting in the doorway being closed between the two networks -- So on and so forth throughout the rest of the web.

  • by tlambert (566799) on Monday February 16, 2009 @07:52PM (#26879665)

    Make sure you are using cat 5 bailing wire.

    -- Terry

  • by need4mospd (1146215) on Monday February 16, 2009 @07:53PM (#26879669)

    In other words, a single mis-configured router is apparently able to cause a DOS for a huge chunk of the Net.

    This means the router was able to take out over 9000 internets. Quite impressive.

  • A lot of things, as it turns out, have these single points of failure that propogate.

    I got to experience this one. [wikipedia.org]

    Drove down Route 76/80 to NYC while it was happening. One city would be on, another off. No rhyme or reason to it at all.

  • by shaitand (626655) on Monday February 16, 2009 @08:02PM (#26879793) Journal

    If I'm understanding this 'router' thing correctly, its like a faucet connected to the series of tubes?

    If not, exactly what role does this router thing play in tube interaction?

    • by petecarlson (457202) on Monday February 16, 2009 @08:58PM (#26880431) Homepage Journal

      If I'm understanding this 'router' thing correctly, its like a faucet connected to the series of tubes?

      If not, exactly what role does this router thing play in tube interaction?

      Your understanding is rather accurate but what your missing is the manifolds. You see, all the tubes connect to big manifolds with valves to control what gets sent where. At each manifold room there is some poor admin who is in charge of opening and closing valves in order to make sure that the right AOL gets sent down the right tube. In order to keep track of what tube to send your AOL down, the admin keeps a list of all the other manifold rooms and how to get to them. Some of the manifold room operators didn't have a wide enough notebook to write down the new directions so they just closed all of their valves and went home.

  • And people don't believe me when I tell them all this new-fangled technology is held together by duct-tape and bailing wire!

    And chewing gum. Don't forget the chewing gum.

  • I wish... (Score:2, Insightful)

    by egcagrac0 (1410377)

    people don't believe me when I tell them all this new-fangled technology is held together by duct-tape and bailing wire

    If only it were that reliable... my duct tape patches and bailing wire repairs typically hold for a decade.

  • by DeadBeef (15) on Monday February 16, 2009 @08:21PM (#26880005) Homepage

    This only broke BGP implementations that are getting pretty long in the tooth now, on a moderately recent version of IOS all we saw is:

    Feb 17 05:25:03.731 nzdt: %BGP-6-ASPATH: Long AS path 10026 3356 29113 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 received from xxx.xxx.xxx.xxx: More than configured MAXAS-LIMIT

    It was definitely an insane path, our routers were configured to drop anything with an AS path longer than 75, old versions of IOS would often just drop the BGP session ( or even crash with some _really_ old versions ).

    I'm sure there will be some red faced network engineers updating IOS or even doing forklift upgrades of old boxes at their edges in the near future.

  • by miller60 (554835) * on Monday February 16, 2009 @08:32PM (#26880141) Homepage
    This incident knocked several major hosting providers [datacenterknowledge.com] offline, including Media Temple in Los Angeles and Canada's iWeb.
  • by lotaris (34307) on Monday February 16, 2009 @08:35PM (#26880175)

    This only took down people running fairly old versions of IOS that didn't patch a known bug.

    Did not affect non-cisco.
    Did not affect modern versions of IOS
    Did not affect old versions of IOS that set the knob to limit the max as-path.

    • Re: (Score:3, Informative)

      by painehope (580569)

      Gee, you only described about half the mistakes that incompetent network admins could possibly make (buying a fucking Cisco, not updating their NOS, and not limiting AS paths).

      That covers half the ISPs in Texas (including mine - these fuckwits can barely configure their routers correctly on a good day, let alone deal with a crisis brought about their own incompetence). YMMV.

      I'd mod you up but I already posted.

  • by drolli (522659) on Monday February 16, 2009 @08:53PM (#26880363) Journal

    Sounds like our lab where we try to make a quantum bit.

  • by adavies42 (746183) on Monday February 16, 2009 @08:53PM (#26880369)
    is that more like a "severe weather event" or an "extreme savings event"?
  • by Genda (560240) <mariet@got.nERDOSet minus math_god> on Monday February 16, 2009 @09:07PM (#26880517) Journal

    Welcome to Sauronet... One Router to Rule them ALL!!!!

"I got everybody to pay up front...then I blew up their planet." "Now why didn't I think of that?" -- Post Bros. Comics

Working...