Blackout Shows Net's Fragility 287
It doesn't come easy wrote to mention a ZDNet article discussing a recent outage between Level 3 Communications and Cogent Communication. A business feud inadvertently highlighted the fragility of the Internet's skeleton. From the article: "In theory, this kind of blackout is precisely the kind of problem the Internet was designed to withstand. The complicated, interlocking nature of networks means that data traffic is supposed to be able to find an alternate route to its destination, even if a critical link is broken. In practice, obscure contract disputes between the big network companies can make all these redundancies moot. At issue is a type of network connection called 'peering.' Most of the biggest network companies, such as AT&T, Sprint and MCI, as well as companies including Cogent and Level 3, strike "peering agreements" in which they agree to establish direct connections between their networks. "
Re:When did this blackout happen (Score:3, Informative)
Internet can route against natural calamities (Score:5, Informative)
Nothing can protect you in this case.
If on the other hand there was a natural calamity and every one was trying to get you access
then you would get it. Like it happened during Katarina.
This is not a natural calamity.
The best option is to ditch your provider if they are not a monopoly and if they are lobby to your government to create multiple providers.
Re:Didn't notice at all. (Score:3, Informative)
The fragility of the net (Score:5, Informative)
Based on expected traffic routed thru each network (Score:1, Informative)
Re:It always will be fragile (Score:4, Informative)
This isn't a good example, because in this case most traffic would automatically be re-routed to go through Asia and the trans-Pacific cables. And if those went down it would go over South America Oceana.
It would get much slower, sure, but would not cause an outage.
There is no *technical* reason this peering relationship breaking down should be causing an outage either. If the both also peered with some third party that could service them both, like MCI or something, then the traffic would still get through. The companies are just being bull-headed.
Re:The small should pay for the big? (Score:5, Informative)
http://www.merit.edu/mail.archives/nanog/msg12302
http://www.merit.edu/mail.archives/nanog/msg12350
Re:The small should pay for the big? (Score:5, Informative)
Re:The small should pay for the big? (mod this up) (Score:5, Informative)
Re:The small should pay for the big? (Score:3, Informative)
Am I missing something here?
I only read about this very briefly, but my understanding is it went beyond that. Just cutting the peering connection is fine and proper and packets then are rerouted through other peers, possibly costing more money, possibly not. Then the internet goes on as before and everyone is happy and the peers involved can negotiate a new link if they want and figure it will save them money by avoiding other routes where they have to pay for traffic.
My understanding is that in this case they not only cut the link, but they advertised routes to their other peers for traffic from the first peer, which they then maliciously, and probably in breach of those other contracts, filtered out, resulting in failed traffic routing. Basically they intentionally lied (to the routers) and said sure we'll route those and then did not.
I don't think this highlights the fragility of the internet, so much as the fact that end users usually rely upon a single peer (ISP) and if they can't trust them to not intentionally break traffic they had better find a new ISP.
Re:The small should pay for the big? (Score:3, Informative)
what we are seeing here is a pissing contest between two "tier1". so there literally is no other route the packets can take to reach each other network (contractually speaking, not technically). each of these networks have peering contracts with other companies, not transit. a peer is only used to reach other's network, a transit lets you reach networks beyond the network you are transiting through.
Re:A New Approach (Score:5, Informative)
If by "work in populated areas" you mean "slow the network to a crawl" then yes, it would work. Mesh networking is cool stuff, but you aren't going to build a backbone out of it. Wireless is really fast compared to your DSL line or cable modem. But it isn't even in the same ballpark as what you can do on fiber. Backbone links are running at 10Gbps or even 40Gbps. Full duplex, so that is 20Gbps or 80Gbps of "marketing bandwidth". Compared to what, 22Mbps or 54Mbps half-duplex for your wireless? You aren't going to build a comparable backbone out of wireless links running at roughly 1/1000th of the speed. Physics pretty much guarantees that fiber links will always be faster than wireless.
The problem here is conflicting business models (Score:2, Informative)
At the fringes there are really two types of internet service offered: upstream and downstream. Most consumers (individuals) need a lot of downstream and very little upstream. They typically are sold assymetric service that is heavily biased in this direction. My cable connection, for example, gives me ~5Mbps down and 768kbps up. On the flip side are the content providers who typically need a lot of upstream bandwidth and less upstream bandwidth. ISPs have found that these customer are willing/able to pay quite a bit more for their internet connections. Therefore, the law of supply and demand has increased the cost of connections with higher upstream capacity.
Several levels up the ISP heirarchy, however, there are mostly only symmetric lines (T3, OCx, ...) providing equal upstream and downstream bandwith. In order to maximize the use of this bandwidth, many providers try to balance the number of content providers with content consumers in order to use the upstream and downstream capacity equally. In theory, this usage should be well balanced by the time it reaches the Teir 1 providers [keynote.com].
The problem we are having right now is caused by Cogent not subscribing to that business model. They have found that the cost to support content consumers is much higher than the cost to support providers. (If for no other reason than there are far more of them.) So, their business model skews heavily towards the provider customers, reducing their operational costs. This, in turn, means that they are able to offer lower costs to those content providers -- in many cases undercutting the other big service providers such as Level 3
This, of course, makes the other providers unhappy because it cuts into their high-yield business. So, occasionally, one of them demands compensation for "transit" instead of providing free peering. They do this because they feel (rightly IMO) that Cogent is able to make more money on these high paying content providers by using an asset owned by the other service providers -- the online customer/consumer base. Basically, Level 3 is telling Cogent that because Cogent is making money by using that virtual asset owned by Level 3, Cogent owes Level 3 some sort of compensation. It is worth noting that several other Teir 1 providers already take this approach with Cogent and Cogent is forced to pay for "transit" service to those providers' customers.
As long as all the Teir 1 providers cooperate, the system works reasonably well. However, in this case, Cogent is trying to take advantage of that informal cooperation to make some extra money. So, they are being capatalists. In this case, capatalism is at odds with cooperation and the system is not working well.
Many people are calling for government regulation to prevent this sort of situation. I expect this to cause some major problems. The issue could be resolved if all the Teir 1 providers would realize that there is a different market value for ingress and outgress traffic. In a free market, I expect that the ingress traffic (corresponding to upstream traffic of content providers from the lower levels) would have substantially more value than the outgress traffic (downstream traffic to consumers). The outgress traffic might even have negative value (meaning that a service provider would charge to take care of it). In the case that two peers balance their traffic well (the ideal cooperative solution) no money needs to change hands. In the other cases (like this one) the ISP with excess outgress usage should probably be charging the one with excess ingress.
Unfortunately, there is no fluidity to the system between the true market (the upstream and downstream bandwidth consumers) and the core market (the Teir 1 providers). If there were, Level 3 could justify their demand for more money based on the value of the traffic they were accepting from lower down the food chain.
Baloney. Its just bad companies. (Score:3, Informative)
The reason these two jokers are having this problem is that they made a business decision to only move traffic with reciprocal peering and then failed to keep that peering alive. That's because they're both cheap-ass bastards; peering costs a heck of a lot less than buying transit.
Go buy from someone else who who isn't a cheap-ass. Someone who buys transit for anything they can't peer. You won't have a problem.
The only lesson here is that most time honored of lessons: you get what you pay for.
It has probably been said... (Score:3, Informative)
The problem isn't soley with the business arrangements between the "big providers" - oh, certain, that does have impact, but the internet would be as robust as ever, if every participant on it could be a peer.
This is how the network was meant to be, a mesh comprised of stupid interconnects and smart nodes. Every node on the internet, from the largest colo to the smallest wireless handheld, should have the ability to be a true peer on the internet. In practice, this isn't really possible, but imagine a mesh network with a distributed p2p DNS system which many people could run if they wanted to - if only a fraction were running it, and were distributed enough, such outages might not occur (the traffic could continue to be routed, albeit at a slower pace).
Everyone should be able to be a peer on the network, everyone should be able to get at least one static IP, everyone should be able to run their own server(s) if they want to. Right now, the only way you can do it is by paying huge amounts of $$$ so you can get a garden hose instead of a straw. I am not saying access to the internet should be or could be free, but peering should be a natural right of being a part of the internet, not something you have to pay extra (a LOT extra) for.
Roadrunner affected (Score:2, Informative)
Re:A solution can be... (Score:3, Informative)
So you claim there are no Internet Exchange Points [wikipedia.org] ?
pray tell, what is this thing [mae.net] ? Or that one, not to mention the middle one [mae.net].
Oh, and what do you think those Guys do [switchanddata.com] for a living ?
Nobody expects you to be a fucking genius or know everything. But why are some folks constantly touting stupid nonsense instead of keeping their mouths shut and learning something ?
Re:It's Nobody's Fault (Score:4, Informative)
If the peering point had been taken out by a bomb, the re-routing would have been performed in fairly short order. However, this is not the case here.
Level3 think that Cogent is taking the piss and is not a real peer. Level3 want Cogent to buy transit to reach Level3, either directly from them (or from someone else) because at the moment the peering is very lopsided, and costing Level3 a bucketload of money and giving Cogent a boatload of free bandwidth.
Cogent on the other hand doesn't want to pay for transit to Level3.
Right now, Cogent could route all their traffic for Level3 over transit they pay for. They don't want to do that because it won't force Level3 back into the peering agreement. So what they do is leave the link severed and do not re-route so that Level3 customers cannot get to sites hosted by Cogent. This means Level3 customers will grumble at Level3. Additionally, they offer a year's free transit to single homed Level3 customers just to raise the brinkmanship with Level3 a notch higher. Basically it's war between L3 and Cogent.
If Cogent re-routes their traffic, they are defeated and L3 will never re-peer. What Cogent are hoping is that enough angry customers on the L3 end will whine at L3 so L3 will be forced to re-peer.
For the rest of us in the peanut gallery (i.e. those of us who aren't single homed customers of Cogent or Level3) we can just watch the fun and games and throw peanut shells at the squabbling combatants because we don't see any black hole at all.
Re:Roadrunner affected (Score:3, Informative)
The Unix traceroute command can be used to do this:
$ traceroute slashdot.org
Re:Didn't notice at all. (Score:3, Informative)
FYI, smaller ISPs pay larger ISPs for bandwidth all the time. The larger ISPs have huge costs. Switches costing hundreds of thousands of dollars, filled with a bunch of cards in it that each cost hundreds of thousands. Lots of them. Lots of fiber and other costs. It gets real easy to have billions invested just in hardware. They offset those costs in part by selling bandwidth to smaller ISPs. That's the way the net works.
Try telling some small ISP that they should stop paying their upstream provider. That the upstream provider should give them bandwidth free so that the larger ISPs customers can access websites hosted by the smaller ISP. They will tell you you are living in a dream world. That's not the way the net works.
Re:It always will be fragile (Score:2, Informative)
Re:The small should pay for the big? (Score:3, Informative)
If Cogent users can get to Qwest and L3 users can get to Qwest, but cogent users can't talk to L3 users, then cogent and L3 are doing something intentionally bad and screwing everyone on the internet.
The peering is back up (Score:2, Informative)
Fixed now? (Score:4, Informative)
That's not how peering works - here's the diff (Score:4, Informative)
Peering arrangements are different. Two networks that have a lot of traffic for each other will set up direct connections, split the direct costs of the connections, and not charge for accepting packets from the other carrier. But they'll only advertise the routes for their *own* customers. If two small ISPs peer with each other, typically they're each also buying transit service from big ISPs, but it's cheaper for them to dedicate a connection or put bits on a public peering point like MAE-West than to both pay their upstream ISPs.
The biggest ISPs in the US are called "Tier 1" ISPs, and they all peer with each other rather than buying transit, though they might buy transit for international connections, if they can't get the other side to buy transit from them. It seems flaky, but it makes business sense, or at least it did for a while. In some sense, being big enough that all the other Tier 1s will peer with you is what defines Tier 1, and aside from technical issues, it's a marketing thing - "See, we're one of the big players!" Peering and Transit don't mix very well - you either connect to a given carrier by peering, or by transit, or else you spend a long time hammering out custom arrangements about exactly which routes you'll accept and tweaking routing tables.
Cogent is a Wannabe-Tier-1. Their main business model is to put fiber into big multi-tenant office buildings and sell everybody 100-meg Ethernet for about the price other carriers charge for one or two T1s. If I were a customer, I wouldn't expect there to be enough upstream to really get that much bandwidth all the time, but I'd expect to get more than a T1 all the time, and a lot more than a T1 almost all the time. Level 3 has apparently decided they're not getting enough value out of the relationship (i.e. not sending Cogent enough packets to make it worth their while) to keep peering, and wants Cogent to either pay them for service or get transit from somebody else. They gave them about 50 days to make other arrangements, but Cogent decided to play chicken with them.