Blackout Shows Net's Fragility 287
It doesn't come easy wrote to mention a ZDNet article discussing a recent outage between Level 3 Communications and Cogent Communication. A business feud inadvertently highlighted the fragility of the Internet's skeleton. From the article: "In theory, this kind of blackout is precisely the kind of problem the Internet was designed to withstand. The complicated, interlocking nature of networks means that data traffic is supposed to be able to find an alternate route to its destination, even if a critical link is broken. In practice, obscure contract disputes between the big network companies can make all these redundancies moot. At issue is a type of network connection called 'peering.' Most of the biggest network companies, such as AT&T, Sprint and MCI, as well as companies including Cogent and Level 3, strike "peering agreements" in which they agree to establish direct connections between their networks. "
The small should pay for the big? (Score:5, Interesting)
What I don't get is why one of them would suddenly want the other to pay up. What's changed now, and why does the smaller company have to pay the big one's bills?
Am I missing something here?
Re:The small should pay for the big? (Score:3, Funny)
Yes.
Re:The small should pay for the big? (Score:3, Funny)
Re:The small should pay for the big? (Score:5, Informative)
http://www.merit.edu/mail.archives/nanog/msg12302
http://www.merit.edu/mail.archives/nanog/msg12350
Re:The small should pay for the big? (Score:2)
Re:The small should pay for the big? (Score:4, Funny)
WHat?! No! I haven't said a word about it!
Re:The small should pay for the big? (Score:2)
Am I missing something here?
Yes. "Greed".
Re:The small should pay for the big? (Score:3, Interesting)
Re:The small should pay for the big? (Score:5, Informative)
Re:The small should pay for the big? (mod this up) (Score:5, Informative)
Re:The small should pay for the big? (mod this up) (Score:3, Insightful)
Re:The small should pay for the big? (Score:3, Informative)
Am I missing something here?
I only read about this very briefly, but my understanding is it went beyond that. Just cutting the peering connection is fine and proper and packets then are rerouted through other peers, possibly costing more money, possibly not. Then the internet goes on as before and everyone is happy and the peers involved can negotiate a new link if they want and figure it will save them money by avoiding other routes where they have to pay for traffic.
My understanding is that in this ca
Re:The small should pay for the big? (Score:3, Informative)
what we are seeing here is a pissing contest between two "tier1". so there literally is no other route the packets can take to reach each other network (contractually speaking, not technically). each of these networks have peering contracts with other companies, not transit. a peer is only used to reach other's network, a transit lets you reach networks beyond the network you are transiting through.
Re:The small should pay for the big? (Score:3, Informative)
If Cogent users can get to Qwest and L3 users can get to Qwest, but cogent
That's not how peering works - here's the diff (Score:4, Informative)
Peering arrangements are different. Two networks that have a lot of traffic for each other will set up direct connections, split the direct costs of the connections, and not charge for accepting packets from the other carrier. But they'll only advertise the routes for their *own* customers. If two small ISPs peer with each other, typically they're each also buying transit service from big ISPs, but it's cheaper for them to dedicate a connection or put bits on a public peering point like MAE-West than to both pay their upstream ISPs.
The biggest ISPs in the US are called "Tier 1" ISPs, and they all peer with each other rather than buying transit, though they might buy transit for international connections, if they can't get the other side to buy transit from them. It seems flaky, but it makes business sense, or at least it did for a while. In some sense, being big enough that all the other Tier 1s will peer with you is what defines Tier 1, and aside from technical issues, it's a marketing thing - "See, we're one of the big players!" Peering and Transit don't mix very well - you either connect to a given carrier by peering, or by transit, or else you spend a long time hammering out custom arrangements about exactly which routes you'll accept and tweaking routing tables.
Cogent is a Wannabe-Tier-1. Their main business model is to put fiber into big multi-tenant office buildings and sell everybody 100-meg Ethernet for about the price other carriers charge for one or two T1s. If I were a customer, I wouldn't expect there to be enough upstream to really get that much bandwidth all the time, but I'd expect to get more than a T1 all the time, and a lot more than a T1 almost all the time. Level 3 has apparently decided they're not getting enough value out of the relationship (i.e. not sending Cogent enough packets to make it worth their while) to keep peering, and wants Cogent to either pay them for service or get transit from somebody else. They gave them about 50 days to make other arrangements, but Cogent decided to play chicken with them.
You have to define 'size' (Score:2)
So, when a mom & pop ISP connects to one of the big guys, there's very little of interest on their network as an endpoint, and they probably don't advertise better routes than the tier1 already has, so they have to pay for the priviledge to be connected.
Now
Re:The small should pay for the big? (Score:3, Interesting)
A solution can be... (Score:2)
Re:A solution can be... (Score:3, Funny)
Re:A solution can be... (Score:3, Funny)
Oh - hang on, if someone else runs a bit of Ethernet in, do I have to connect to them? Damn.
Re:A solution can be... (Score:3, Informative)
So you claim there are no Internet Exchange Points [wikipedia.org] ?
pray tell, what is this thing [mae.net] ? Or that one, not to mention the middle one [mae.net].
Oh, and what do you think those Guys do [switchanddata.com] for a living ?
Nobody expects you to be a fucking genius or know everything. But why are some folks constantly touting stupid nonsense instead of keeping their mouths shut and learning something ?
No worries (Score:5, Funny)
Re:No worries (Score:4, Funny)
Background info (Score:3, Funny)
Re:Background info (Score:3, Funny)
Daniel
Efficiency can be the enemy of robustness (Score:5, Interesting)
On a similar note, that's why there are 13 root DNS servers, and why most of us aren't supposed to use them. The DNS example though, is one where efficiency and robustness agree. It's more efficient, at least in terms of net bandwidth, to use a DNS server closer than the root servers.
Re:Efficiency can be the enemy of robustness (Score:2)
Can someone explain why this isn't as easy as I think it is?
Re:Efficiency can be the enemy of robustness (Score:2)
A standard refers to 2 types of connections:
Customer-Provider and Peering.
An ISP X publishes, at most, the routes as follows (they can choose to publish less):
X publishes to its peers and providers the routes to its customers.
X publishes to its customers all the routes it recieved.
This creates a layered model, where in the top you have the top level providers, the internet backbone.
Due to this, it is impossible to route:
* down to a customer and then b
Re:Efficiency can be the enemy of robustness (Score:2)
If they redirect the traffic that would have gone via each other out through their other connections, then their customers (smaller ISPs) would start complaining that Cogent/L3 are abusing the connections that the smaller ISPs are paying them for.
At least that's how I understand it, my understanding being based on what I have read over the last 24 hours on this issue.
Phil Hibbs.
Re:Efficiency can be the enemy of robustness (Score:2)
I understand it's 'corporate chicken' but they could accomplish the same thing without inconveniencing their customers, I would think.
Re:Efficiency can be the enemy of robustness (Score:3, Insightful)
It's a web, and when one strand breaks, it increases the strain on the other strands.
Call the helpdesk...wait, THEY don't even know! (Score:5, Interesting)
Good article on this situation here
This situation has adversely affected various users of both companies' services. The inability of Level 3 to handle this situation in a fair and equitable manner to the consumers has alienated many customers and will continue to do so until the current situation is remedied. At what point is it good customer service to discontinue services due to no fault of said consumer base? Market history shows us that the single worse thing a company can do is to arbitrarily allow influences beyond the control of consumers to negatively impact services, determined by consumers to be status quo, without any warning or notification. If left unresolved and unaddressed, the current situation could set dangerous precedents for internet users across the country by allowing service providers to instantly discontinue provided services at the moment they feel that the services they provide are not being adequately compensated for from outside companies.
On a side note, I was listening to Howard Stern (oh no!) this morning and he said that his Time Warner internet connection at home didn't work. Howard then called a tech guy to come and fix the problem, only for him to call a help desk to figure out what happened. The help desk didn't even know what was wrong. It sounds like Level 3 just pulled the plug and didn't notify ANYONE. Or maybe it was Cogent, the point is nobody outside of that dispute KNEW what was going on.
This sounds like a good way to alienate your customers and/or ruin your business model. But that is just my opinion.
Re:Call the helpdesk...wait, THEY don't even know! (Score:5, Funny)
I got a semi canned response but it did have some techincal details. It also stated that if you wish to discuss the techincal nature of the problem go to www.ask.slashdot.org With a full link to the other article.
Yep Roadrunner sent me to slashdot to get more information.
Re:Call the helpdesk...wait, THEY don't even know! (Score:2)
Monitor it yourself (Score:5, Interesting)
http://www.internetpulse.net/ [internetpulse.net]
I'm not affiliated with them in any way, and I'm sure there are other similar sites, but I thought it was worth mentioning.
Fixed now? (Score:4, Informative)
A New Approach (Score:3, Interesting)
Re:A New Approach (Score:2)
If you use Cogent or Level3, dump them. Find another provider.
Re:A New Approach (Score:2)
Some how I'd expect voluntary contributions would fall short.
An internet tax would work.... as evil as it might sound.
Re:A New Approach (Score:2)
I live in Burlington, VT and the city is doing this through an organization called Burlington Telecom [slashdot.org]. I'm one of the beta testers and we're going to be hooked up with a 5 Mbps symmetrical fiber connection in a couple of weeks. The best part? It's cheaper then Adelphia and I, as a citizen, own the network. Unfortunately, this still doesn't solve the "long haul" problem.
Re:A New Approach (Score:3, Insightful)
This would work in populated areas in theory. In practice, though, 95% of the bandwidth in any given system gets eaten up by 5% of the users unless there is heavy regulation. Actually, we pretty much need the big internet companies in order to get a particular level of QoS.
Like I said, all it takes is one in fifty who won't play nice to ruin it for everybody else. I'd be
Re:A New Approach (Score:2)
Very good point. Could there be some sort of system to moderate "good" and "bad" nodes? Hmm, kind of like on slashdot?
Re:A New Approach (Score:5, Informative)
If by "work in populated areas" you mean "slow the network to a crawl" then yes, it would work. Mesh networking is cool stuff, but you aren't going to build a backbone out of it. Wireless is really fast compared to your DSL line or cable modem. But it isn't even in the same ballpark as what you can do on fiber. Backbone links are running at 10Gbps or even 40Gbps. Full duplex, so that is 20Gbps or 80Gbps of "marketing bandwidth". Compared to what, 22Mbps or 54Mbps half-duplex for your wireless? You aren't going to build a comparable backbone out of wireless links running at roughly 1/1000th of the speed. Physics pretty much guarantees that fiber links will always be faster than wireless.
Re:A New Approach (Score:2)
A high speed mesh network is an engineering problem, not a theoretical on
Re:A New Approach (Score:2)
Peering (Score:5, Funny)
In other news, the RIAA announced they've stopped an extremely large P2P network.
Internet can route against natural calamities (Score:5, Informative)
Nothing can protect you in this case.
If on the other hand there was a natural calamity and every one was trying to get you access
then you would get it. Like it happened during Katarina.
This is not a natural calamity.
The best option is to ditch your provider if they are not a monopoly and if they are lobby to your government to create multiple providers.
Re:Internet can route against natural calamities (Score:3, Interesting)
It's dupealicious! (Score:3, Funny)
Not that I would ever do such a thing...
It always will be fragile (Score:5, Insightful)
The Internet will IMVHO always be quite fragile. While the design lends itself to robustness the reality is that there is only money for a few very big connections and therefore a disaster that affects one of these connections is going to cause wide spread outages.
Take, for instance, the connections running between Europe and America. I bet most of them run in almost exactly the same place on the sea bed because it's the cheapest / shortest path to take. A fairly localized geological disaster (at least in geological terms) could cut all the cables at once; or at least enough to make to difference.
If we wanted the network to be robust we would need to run cables up over the north pole and round the equator and probably stick in some satelite links as well. There just isn't money for that. People are willing to accept the risk that it might fail in extreme situations.
FWIW I think the problem is worse on the global scale than the country scale. I imagine most developed countries probably have enough redundancy in their own country. It's the interconnects between countries that are probably the biggest problem.
Re:It always will be fragile (Score:4, Informative)
This isn't a good example, because in this case most traffic would automatically be re-routed to go through Asia and the trans-Pacific cables. And if those went down it would go over South America Oceana.
It would get much slower, sure, but would not cause an outage.
There is no *technical* reason this peering relationship breaking down should be causing an outage either. If the both also peered with some third party that could service them both, like MCI or something, then the traffic would still get through. The companies are just being bull-headed.
Re:It always will be fragile (Score:2)
I have no hard data to back this up but IIRC when 9/11 happened it took out a major Internet switch center as well. Everyone talked about how traffic would get routed from Europe to America via Asia and the Pacific links but the reality was that most of America was unreachable from Europe for quite a few days. I suspect the places that were reachable had European mirrors. While there was a viable route it wasn't able to cope with the increase in traffic so basically every packet timed out.
Re:It always will be fragile (Score:3, Interesting)
North Pole Run coming soon (Score:2, Funny)
As a bonus, Santa's new underwater toy factory can tap into it.
Woo-hoo, faster email to Santa! Hope the jolly old elf doesn't discover online pr0n or he'll never get those presents made on time.
Re:It always will be fragile (Score:2)
Engineers do actually consider redundancy when designing these things. Take, for example, TAT-14 [tat-14.com]. They have a northern route that goes up to Denmark, and a southern route that lands in England. And that's just one of many fiber optic connections across the Atlantic.
Distance is important, but so is redundancy.
Keep it private (Score:3, Insightful)
Neither company involved in this dispute wants to do t is. They need to work it out, or other companies will find a solution and take the customers.
If you're desperate to provide data to multiple top tiers, pay for a host that is connected to multiple backbones.
There is zero need to mandate anything. Let the free market provide and we'll be safer in the long run. Let government provide and we'll see a slowly creeping tyranny online.
Re:Keep it private (Score:2)
All of which can, and have been, imposed anyway.
Let the free market provide and we'll be safer in the long run.
There is no such thing. It is an impossibility. Communism and Capitalism both have these utopian ideas at their hearts; that's why neither of them work in practice the way they're supposed to on paper.
TWW
Not true (Score:2)
Of course that need is generated by cmpanies that want the internet 'under control'.
So if you sit back and 'let the market decide' those who controll the mark will. It won't be for you.
Not a redundancy issue... (Score:4, Interesting)
bidding-based protocols (Score:2)
Bandwidth isn't (and never will be) truly free as long as the hardware and admin labor has a cost. But if we seek way to deliver the most packets at the least cost, then market forces will drive
Re:bidding-based protocols (Score:2)
This is a good idea. However, it demands an untamperable log system - that is, you need to be able to prove that router X sent you n packets, and the owner of router X needs to be able to prove t
The fragility of the net (Score:5, Informative)
ah peering (Score:4, Interesting)
But if most of the traffic from other networks is going to customers that are connected and already paying for your network's service then it makes no sense and is simply wrong for a network to start charging other network providers. It breaks the end to end communication model and is providing your customers with less than the service they are paying for. People pay for internet connectivity so they can transfer data between other users on the internet, not just the ones on your company's network.
If money exchanging hands is at all appropriate in this case it might be for the actual installation of routing equipment which establishes the physical connection between networks.
not a blackout (Score:2, Interesting)
So the internet is breaking down (Score:2, Insightful)
Re:So the internet is breaking down (Score:3, Insightful)
Are you arguing that government control moves power from the few to the many? That is backwards to my way of thinking. The quickest way I can think of to concentrate power is to put the government in charge of it.
The Internet doesn't match its sales brochure. (Score:2)
But when you choose to have a single critical link you don't have an interlocking web of connections.
even if a critical link is broken.
If it was a web there would be no critical links to break. The problem is that for various (technical/economic) reasons there is a backbone (or series of backbones) it isn't really an interlocking web.
The peering problem is bad.... (Score:2)
back in the day.... (Score:2)
re the fragility of the internet (Score:2)
http://politics.slashdot.org/article.pl?sid=05/10
Cogent Sucks (Score:2, Interesting)
I contacted Cogent's "premium" help desk last night when I found that I was suddenly no longer able to get to our networks in Australia. The tech had no idea that his own company was in the middle of a huge peering battle with L3. I had to tell them!
This was predictable (Score:5, Interesting)
Unfortunately, that is not the Internet that we have today. In the original Internet, every router knew about every network connected to the Internet. Most networks had connectivity to many other networks. Discovery protocols allowed alternative routes to be discovered if one failed.
Today, we don't have a (mostly) fully connected net, we have ISPs who don't know anything about networks which they don't "own", only that certain IP prefixes need to be passed to ISP x, y or z.
This makes the infrastructure much more fragile than it was originally intended to be. We ended up with this for a few reasons. First, the wimpy routers in use at the time had limited memory available to hold the network maps. The answer chosen was to no longer attempt to hold a full world view, but to divide the world into regions, certain IP prefixes would "belong" to those regions, and all any router would need to know about was networks in its region, plus how to route traffic to other regions, who would take care of routing within the region. This led to "backbone" connections - high capacity links needed because all traffic between regions now didn't "diffuse" through the network, but was channeled into specific connections. It also set the scene to allow the net to be commercialised, those regional centers were obvious "choke points" that an enterprising company could own and pretty much dictate the pricing to lower level enterprises who would do the dirty work of dealing with end-users.
Slowly but sureley the Internet evolved into a system dependent upon a few companies with high-speed links between them - prime candidates BTW, as locations for government control to be imposed. The self-healing nature of the original Internet was lost because all traffic HAS to pass via the top level companies infrastructure and over their interconnect backbone connections.
The "self healing" Internet is long gone.
Re:This was predictable (Score:3, Interesting)
This is what happens when you have an industry based upon a high cost of entry (physical infrastructure, here) and a low marginal c
Crazy Idea (Score:2)
Damage: Level3 won't accept Cogent traffic.
Horrible hack: tunnel BGP traffic to Level3 customer who masquerades requests as local traffic.
Yeah, the real solution is tier 2 folks having more peerings, but as a nasty workaround is that hack feasible?
Can you tunnel BGP traffic in TCP or ssh or something?
-l
Re:Crazy Idea (Score:3, Insightful)
Damage: Level3 won't accept Cogent traffic. Horrible hack: tunnel BGP traffic to Level3 customer who masquerades requests as local traffic.
You don't need to masquerade anything, if you're connected to Level3 and Cogent, just configure your router to advertise your route to the Level3 network on the Cogent side and vice-versa.
Then watch your router melt under the hundreds of gigabits of traffic -- that you'll have to pay for both ways. Congratulation, you're the new peering agreement between Level3 and
Point taken. Thanks. [nt] (Score:2)
-l
Pending Assingment (Score:2)
Re:Pending Assingment (Score:2)
The Internet Health Report shows the interruption
Your link doesn't work, use http://www.internethealthreport.com/ [internethealthreport.com]
The problem here is conflicting business models (Score:2, Informative)
At the fringes there are really two types of internet service offered: upstream and downstream. Most consumers (individuals) need a lot of downstream and very little upstream. They typically are sold assymetric service that is heavily biased in this direction. My cable connection, for example, gives me ~5Mbps down and 768kbps up. On the flip side are the content providers who typically need a lot of upstream bandwidth and less upstream bandwidth. ISPs have found that these customer are willing/able to
Baloney. Its just bad companies. (Score:3, Informative)
The reason these two jokers are having this problem is that they made a business decision to only move traffic with reciprocal peering and then failed to keep that peering alive. That's because they're both cheap-ass bastards; peering costs a heck of a lot less than buying transit.
Go buy from someone else who who isn't a cheap-ass. Someone who buys transit for anything they can't peer. You won't have a problem.
The only lesson here is that most time honored of lessons: you get what you pay for.
Peering is optional. Routing is mandatory. (Score:2)
It has probably been said... (Score:3, Informative)
The problem isn't soley with the business arrangements between the "big providers" - oh, certain, that does have impact, but the internet would be as robust as ever, if every participant on it could be a peer.
This is how the network was meant to be, a mesh comprised of stupid interconnects and smart nodes. Every node on the internet, from the largest colo to the smallest wireless handheld, should have the ability to be a true peer on the internet. In practice, this isn't really possible, but imagine a mesh network with a distributed p2p DNS system which many people could run if they wanted to - if only a fraction were running it, and were distributed enough, such outages might not occur (the traffic could continue to be routed, albeit at a slower pace).
Everyone should be able to be a peer on the network, everyone should be able to get at least one static IP, everyone should be able to run their own server(s) if they want to. Right now, the only way you can do it is by paying huge amounts of $$$ so you can get a garden hose instead of a straw. I am not saying access to the internet should be or could be free, but peering should be a natural right of being a part of the internet, not something you have to pay extra (a LOT extra) for.
Roadrunner affected (Score:2, Informative)
Re:Roadrunner affected (Score:3, Informative)
The Unix traceroute command can be used to do this:
$ traceroute slashdot.org
Re: Fucking Kids stuff (Score:2, Insightful)
Re:When did this blackout happen (Score:3, Informative)
Re:Didn't notice at all. (Score:2)
Re:Didn't notice at all. (Score:3, Informative)
Re:Didn't notice at all. (Score:3, Insightful)
If Level3 didn't want to peer anymore with Cogent. That's understandable, it wasn't an even exchange of traffic anymore. They could have done the right thing and simply stopped the peering. Insted, they have decided to be vendictive and filter any traffic to/fr
Re:Didn't notice at all. (Score:3, Informative)
FYI, smaller ISPs pay larger ISPs for bandwidth all the time. The larger ISPs have huge costs. Switches costing hundreds of thousands of dollars, filled with a bunch of cards in it that each cost hundreds of thousands. Lots of them. Lots of fiber and other costs. It gets real easy to h
Re:Didn't notice at all. (Score:3, Interesting)
Re:Ask Slashdot (Score:4, Interesting)
Physicians trying to use the internet to take care of critically ill patients are already experiencing this. Radiologists sitting home reading films are seeing this as well.
Is 100% on neccessary? Hell, VoIP is making money like crazy over this unstable network of ours.
My suggestion is to test with people that will understand the limitations of your service. Then get a little VC money to spread your servers out.
Re:Typical (Score:2)
Line of logic still holds. The museum has plenty of bigwigs and money makers in the upper levels. Maybe they could chip in sometime.
Re:designed to withstand? Says who? (Score:2, Insightful)
Re:designed to withstand? Says who? (Score:2)
Re:Net blackouts (Score:3, Insightful)
Re:Net blackouts (Score:2)
Re:Hmmm.... (Score:2)
This is a classic example of why all businesses who are internet dependent (most busineses nowadays) should either have high-speed DSL as a backup at the least, or some other provider ready to g
Re:This is so strange... (Score:3, Funny)
Re:It's Nobody's Fault (Score:4, Informative)
If the peering point had been taken out by a bomb, the re-routing would have been performed in fairly short order. However, this is not the case here.
Level3 think that Cogent is taking the piss and is not a real peer. Level3 want Cogent to buy transit to reach Level3, either directly from them (or from someone else) because at the moment the peering is very lopsided, and costing Level3 a bucketload of money and giving Cogent a boatload of free bandwidth.
Cogent on the other hand doesn't want to pay for transit to Level3.
Right now, Cogent could route all their traffic for Level3 over transit they pay for. They don't want to do that because it won't force Level3 back into the peering agreement. So what they do is leave the link severed and do not re-route so that Level3 customers cannot get to sites hosted by Cogent. This means Level3 customers will grumble at Level3. Additionally, they offer a year's free transit to single homed Level3 customers just to raise the brinkmanship with Level3 a notch higher. Basically it's war between L3 and Cogent.
If Cogent re-routes their traffic, they are defeated and L3 will never re-peer. What Cogent are hoping is that enough angry customers on the L3 end will whine at L3 so L3 will be forced to re-peer.
For the rest of us in the peanut gallery (i.e. those of us who aren't single homed customers of Cogent or Level3) we can just watch the fun and games and throw peanut shells at the squabbling combatants because we don't see any black hole at all.