Verizon's Accidental Mea Culpa 390
Barryke writes: Verizon has blamed Netflix for the streaming slowdowns their customers have been seeing. It seems the Verizon blog post defending this accusation has backfired in a spectacular way: The chief has clearly admitted that Verizon has capacity to spare, and is deliberately constraining throughput from network providers. Level3, a major ISP that interconnects with Verizon's networks, responded by showing a diagram that visualizes the underpowered interconnect problem and explaining why Verizon's own post indicates how it restricts data flow. Level3 also offered to pay for the necessary upgrades to Verizon hardware: "... these cards are very cheap, a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that's the case, we'll buy one for them. Maybe they can't afford the small piece of cable between our two ports. If that's the case, we'll provide it. Heck, we'll even install it." I'm curious to see Verizon's response to this straightforward accusation of throttling paying users (which tech-savvy readers were quick to confirm).
But scarcity! (Score:5, Insightful)
If people don't think bandwidth is a scarce commodity, how will we get them to pay through the nose for it?!?
Level 3 - start pulling cards (Score:3, Interesting)
Level 3 - start pulling cards (Score:3)
I wish it was that simple, but I'm on board with the general idea. I wouldn't publicly cut the cables. That's too extreme. I would, however, like to see Level 3 turn the tables and publicly (as noisily as possible) accuse Verizon of using up all their bandwidth and that if Verizon
Verizon FiOS customers downgrade! (Score:5, Interesting)
Re:But scarcity! (Score:5, Informative)
Verizonâ(TM)s Accidental Mea Culpa
Mark Taylor / 18 hours ago
David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflixâ(TM)s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizonâ(TM)s own paying broadband consumers.
His explanation for Netflixâ(TM)s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.
The diagram then shows this one little bar, suggestively color-coded in red so you know itâ(TM)s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.
But hereâ(TM)s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizonâ(TM)s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because thatâ(TM)s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network â" IN BOTH DIRECTIONS.
So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?
Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3â(TM)s network interconnects with Verizonâ(TM)s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.
So letâ(TM)s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the con
Re:But scarcity! (Score:5, Informative)
Google's cached copy...
http://webcache.googleusercontent.com/search?q=cache:DBHDyx7n4D0J:blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/+&cd=1&hl=en&ct=clnk&gl=us
Re:But scarcity! (Score:5, Funny)
It looks like the Level 3 post has been pulled. It goes to their 404 page which has a link to recent posts which lists the very post linked in the article.....and the recent post link ALSO takes you to a 404.
Strange, the link works fine for me. Your ISP isn't Verizon by chance is it?
Re:But scarcity! (Score:4, Interesting)
No, if he's getting a 404, there's a connection to the server. Would be hilarious if Verizon had something to do with it, but we can't pin that one on them.
It was a joke, but if we are going to take it seriously it is certainly not out of the realm of possibility for an ISP to redirect a specific URL to a different URL. Just get the 404 page from the site and redirect it there as it passes through the provider's network gear. Similar process to the one used by internet providers in countries that have mandatory blacklists for "pirate" sites.
Re:But scarcity! (Score:5, Insightful)
In what world do you live that Level3 is a "much smaller ISP"? Level3 is a global tier 1 ISP, FFS.
Re:But scarcity! (Score:5, Insightful)
In what world do you live that Level3 is a "much smaller ISP"? Level3 is a global tier 1 ISP, FFS.
In a world where schnell and shill sound a lot alike.
Re:But scarcity! (Score:5, Insightful)
Apples to oranges. Level3 and Cogent aren't last-mile providers; they're Tier 1 backbone providers. Tier 1 providers have things like peering agreements -- last mile providers do not. Last mile providers are (and sell) unbalanced connections, so it's impossible for them to ever have "peers."
A better way of thinking of it is that Verizon should be representing the interests of its customers, because Verizon is the gateway between the customers, and the rest of the internet. It's not doing that job -- it's trying to play both sides against each other. This is what middlemen do, of course, and they're entitled to do it, but as long as they have a monopoly (which they do), then there should be limits, oversight, and accountability.
Re: (Score:2, Insightful)
A better way of thinking of it is that Verizon should be representing the interests of its customers, ...
Nice sentiment, but, unfortunately, a public corporation's responsibility is to its shareholders and their interests - which is simply $$$. (and probably executives and cushy bonuses, etc...)
Re:But scarcity! (Score:4, Insightful)
It's not a sentiment; it's a responsibility as monopoly holders, as I mentioned .
And government has a responsibility too. (Score:5, Insightful)
Nice sentiment, but, unfortunately, a public corporation's responsibility is to its shareholders and their interests - which is simply $$$. (and probably executives and cushy bonuses, etc...)
And a government's responsibility is to take action against a company which is committing wholesale fraud against its customers by selling them Internet Service which promises bandwidth speeds which they are then purposefully not providing in order to shake down their customers and companies trying to provide services to those customers more money.
A government's responsibility is to ensure that companies that are given government licenses and franchise agreements which restrict competition in certain geographic areas are providing the service that the people of that area want and need at a fair price.
A government's responsibility is to ensure that companies which get too big, hold too much market share and are too horizontally or vertically integrated are broken up so that there can be real competition and a real free market.
Re:But scarcity! (Score:4, Informative)
a public corporation's responsibility is to its shareholders and their interests
No. A corporations responsibility is to fulfill its charter, that may mean 'make the most money possible', it may not.
Shareholders only really get to argue about fulfilling that goal, if the goal is to feed the world, the shareholders can't bitch about not making money, only about feeding the world.
Re: (Score:3)
Nice sentiment, but, unfortunately, a public corporation's responsibility is to its shareholders and their interests - which is simply $$$. (and probably executives and cushy bonuses, etc...)
Apple is a public corporation, but they seem to be eking out a living from making stuff that people voluntarily want to buy.
As it turns out, a greedy algorithm is a poor choice for those actually wanting to be greedy.
Re:But scarcity! (Score:5, Interesting)
[...] Level3 and Cogent aren't last-mile providers; they're Tier 1 backbone providers. Tier 1 providers have things like peering agreements -- last mile providers do not. [...]
Except that Verizon Enterprise (formerly Verizon Business) is also a Tier 1 backbone provider. Different part of the company, but the behavior does appear to be a conflict of interest, of exploiting the Verizon's ISP (last-mile) business actions (failing to resolve congestion to L3) to make a competitor (L3) to Verizon Enterprise (formerly UUnet, AS 701 / 702 / 703) less desirable to Level 3 customers, namely Netflix.
Arguably, Verizon is abusing its ISP customers as pawns in making a competitor to one its Enterprise IP business less desirable, in a very anti-competitive fashion. [ftc.gov]
Re:But scarcity! (Score:5, Insightful)
The problem is still the lack of competition in the market. If everyone had the choice between 4-5 ISPs, considering the popularity of Netflix, consumer ISPs would be paying Level 3 truckloads of money to ensure Netflix works flawlessly...and the roles may even be reversed (where Level 3 tries to gouge Verizon, since they'd know Verizon would have no choice or lose a ton of customers).
But since there isn't any competition, Verizon takes their own customers hostages...
Re:But scarcity! (Score:4, Interesting)
I've lived in Europe and I got to see first hand what very strong competition means.
Every ISP peers to the max with every other ISP it can, and with the backbone providers. Nobody charges for peering either way, everybody wants to open the pipes as much as possible.
At one place I lived at I had a choice of the biggest 3 providers in the country and 2 small ones. All of them offered bandwidth in the range of 100 Mbps, both up and down, to/from anywhere inside their network (which for the big ones meant pretty much the entire country) and varying levels of outside bandwidth (but 10-30 Mbps was usual). This was pretty much the standard on cable or copper connections in the cities. Outside it went down but you'd still typically get 30-50 Mbps. Fiber was only available in the cities – but it meant 1000 Mbps down (yes, 1 Gbps).
Lowest basic monthly subscription started from around 10$. It was 25$ for the fancy fiber stuff. I wish I was making this up.
Was there throttling, blocking, or shafting customers with lower-than-advertised bandwidth? You betcha, and plenty of it. Did anybody call for government regulation? Nope. They bitched about it to the ISP, and if the ISP didn't fix it (or couldn't) they switched to another one. Or they decided they don't care that much and stayed on. Whatever. Even with the most crap of the crappiest ISP's you still got something like 10 Mbps so, yeah, some people didn't care.
Re:But scarcity! (Score:5, Insightful)
Why don't I want city-owned fibre? I'm a big fan of city-owned roads and city-owned sewer pipes.
Re: (Score:3, Interesting)
In case you're honestly asking: city-owned roads are much the same as they were 50 years ago, same for sewer pipes. We can do things like that in a publicly owned manner due to the long cycles, and even then it's often not optimal. Note, too, that most roads that you travel on were not built by a government but were built by a developer and simply maintained by the government.
Trying to get government to run networks would work well up front, but in 5 years it would be outdated and there would be no money
Re:But scarcity! (Score:5, Insightful)
Prove it. At this point with stagnation or even reduction of service from the Internet providers it isn't at all clear that private companies are doing anything other than gauging customers with the exclusive franchises or licenses they are getting from communities in order to be the only one running wires.
All evidence is pointing to it being better for communities to treat wired communications along public ways as a public utility.
Much is made about the private capital that is used to invest in installing all these wires, but it is the capital of customers which is paying back those original investments. I would say the customers who are actually paying for this should be the ones that decide how they want their communications network managed.
Re: (Score:3)
Trying to get government to run networks would work well up front, but in 5 years it would be outdated and there would be no money to upgrade it.
What is the difference between that and what we have now? It isn't like Comcast is working around the clock to beef up their last-mile connections.
I feel the same way about internet service that I feel about Healthcare: I am all for private for-profit alternatives, but there needs to be a solid free or low-cost (non-profit) public option. Everybody should have a minimum standard with the option to go with a private entity if they want something better. Something needs to shake up the Internet monopolies and
Re:But scarcity! (Score:4, Insightful)
That long cycle worked fine for the telephone lines that serve your house. They served us well for a long time. The biggest flaw was that instead of managing the infrastructure themselves, they gave it to private companies to manage. Then, when they started abusing the monopoly, the government had to turn around and start requiring them to allow CLECs to use the lines. The phone companies, predictably, hated this, and did the absolute least that they could do to comply with the regulations, often refusing to fix problems with lines while blaming it on the CLEC (and vice versa).
All those problems would have been avoided if the government had simply maintained exclusive control over the lines and leased them out to third parties. That's how next-generation fiber networks in cities should be set up. The entire premise of letting a few companies maintain exclusive control over critical infrastructure is fundamentally flawed and can only lead to more of the same bulls**t we've had for the last two decades.
The only scheme that works is the public utility scheme, where the government owns the wires and private companies provide the service. We know this model works because it has worked with our interstate highway system and private shipping companies for decades. Is it perfect? No. The government historically hasn't charged those shippers enough money in gas taxes to cover infrastructure maintenance costs, resulting in some roads falling into disrepair. But that's mainly a problem caused by lack of a single management body that manages all of the roads in a region. Dozens of city governments working together isn't a great way to get things done except on an "It burns! It burns!" emergency basis. The solution to that problem, of course, is for all the cities in a metropolitan area to get together to form a non-profit corporation, and make that corporation responsible for the management and leasing of lines a la TVA.
Re:But scarcity! (Score:5, Insightful)
city-owned roads are much the same as they were 50 years ago, same for sewer pipes
What shithole do you live in where they haven't upgraded any roads in 50 years? Is your city shrinking in size because theres no way any city, with normal growth, has the same traffic it did 50 years ago. No roads have ever been made wider? No new roads have been added?
Your argument is bunk, its just your ignorance of how your city has dealt with the need for additional capacity either due stupidity or willfully ignoring the obvious upgrades that have been made.
Re: (Score:3)
I'm just assuming you're saying that and not judging from personal experience, because last I checked, the State of Michigan monitors road quality state-wide [state.mi.us] and funds repairs accordingly.
Re: (Score:2, Flamebait)
...a much smaller ISP like Level3...
You're fucking idiot. Shut up and get back under your rock.
No excuses left (Score:5, Insightful)
Too big to fail, too arrogant to concede, too greedy to care. This news is all the more reason to regulate.
Re:No excuses left (Score:5, Insightful)
Too big to fail, too arrogant to concede, too greedy to care. This news is all the more reason to regulate.
But, but, but... regulation is the antithesis of the Capitaist way that our republican Democracy has weaned its children on since it was formed!!
I do tend to agree though - regulation of ISPs is probably the only way to deal with this.
Capitalist theory says that if an incumbent merchant/provider is too inefficient to provide a good service or if another potential merchant/provider thinks they can do a better job for a lower price, then that new provider will step in and provide said service. The threat of that is what keeps the incumbent lean and competitive, and the result is a competitive environment that is generally good for the consumer and rival providers seek to offer better deals to entice custom away from their competitors.
However, that theory assumes that there is a very low or non-existent barrier to entry into that competitive marketplace. Given the initial infrastructure setup costs and, in many cases, exclusivity contracts between providers and the municipal areas which would present the profits to drive services out into more marginal areas, the barriers to entry into the Tier 1 ISP market are prohibitive, to the point where you need to be a corporate entity the size of Google to be able to reasonably make the capital investment required.
As such, the local markets for each ISP more closely resemble non-competitive monopolies with the illusion of choice being provided by third party suppliers who typically have to by access to the resources from the incumbent monopoly - they get wholesale prices, and the consumer sees some small price reductions if the third parties can make enough money to operate by charging the consumer slightly less than the discount they got from the incumbent. But fundamentally, everything is still controlled by that original monopolistic provider, so services suck, progress is stifled because there is no incentive for change, innovation is discouraged, and the level of capacity/reliability is never going to be any more than "just barely enough so that we can maximise our profit margins".
Re: (Score:2)
But, but, but... regulation is the antithesis of the Capitaist way that our republican Democracy has weaned its children on since it was formed!!
Regulating what to make is the antithesis of Capitalism. Regulating what to charge is the antithesis of Capitalism. Regulation of abuse of monopoly powers is not.
Re:No excuses left (Score:5, Insightful)
Also not terribly productive until you put reigns on it and channel that strength towards useful goals.
Regulations are the reigns by which the power of the free market is harnessed and made productive.
And like reigns...to much is bad, but none is worse. But nuanced conversations like this with 'but free market' morons in the current GOP are next to impossible.
Re:No excuses left (Score:5, Insightful)
Think about what you're saying, that anyone should be able to anything they want? Anything? or should there be some rules governing behavior? Unless you're in favor of wild west anarchy, you're in favor of *some* type of regulation on society and the blessed free market capitalism.
Individual Rights are Granted by Societies (Score:4, Interesting)
Individuals, and individual rights, are like single atoms. They only exist in the abstract sense. The real world is entirely dominated by groups and collective actions.
You're a confused anarchist. The problem with non-coercive government is that all government is coercive. Government is primarily a set of restrictions on the use of force, or alternately the monopoly on that use of force. Getting rid of a government, or disarming it, merely allows anyone with a larger arsenal to set up their own government -- anarchy is an unstable system. We all have a right to violence, because it cannot be taken from us except in extreme situations. Remember, the Code of Hammurabi was instituted, "...so that the strong might not harm the weak." Coercive government is a necessary evil, and it will remain necessary so long as men are capable of harming their fellows, for that is its justification and primary purpose.
Rights are not inherent, except in some abstract sense. In the real world, your rights are what the men with guns say they are. You may feel fortunate that the world has had a long, bloody time to work out semi-cooperative frameworks to restrain our darker impulses. Individual rights are an important conceptual counterbalance to the overwhelming powers of the collective, but they are no justification for anarchy, economic or otherwise. The "free market" is an ideal, even a good one, but in most cases removing government interference makes markets less free, more subject to collusion and fraud. In some cases, where the service is required to be universal, or when the barriers to entry would be insurmountable, it makes sense for the government to assume these functions directly. Govenment can also be thought of as the natural monopoly of natural monopolies, in that sense.
Slavery is a word that has a specific meaning; your definition is specious. You just fundamentally don't like being told what to do. To some degree this idealism is admirable. For the true individualist, I can recommend (from long personal experience) the Alaskan wilderness; you can get land for free still up there, provided you build upon it. Whatever romantic images your mind conjures upon thinking of Alaska are all true; I can't stand the weather, personally, but it's as close to a pure state of nature as you will ever find. If you'd like to enjoy the benefits of society, however, you have to play by the rules. "Slavery" isn't an option -- it's mandatory.
Re:No excuses left (Score:5, Insightful)
Regulation for the public benefit = good. Examples: Public Utilities, Healthcare, Agriculture, Air Quality/Environmental Protection.
Regulation for the sake of Regulation = bad. Examples: 70,000 + pages of IRS Regulations and 30,000 pages of tax code written by special interests and bureaucrats.
PR needs to talk to tech (Score:5, Insightful)
Re: (Score:3)
It is painful how true that is.
Most of the time they get away with it, I'm just ecstatic that they didn't this time.....and soooo badly......
Connect with a VPN (Score:5, Interesting)
Re:Connect with a VPN (Score:5, Interesting)
Re:Connect with a VPN (Score:5, Funny)
Re:Connect with a VPN (Score:5, Funny)
Re:Connect with a VPN (Score:5, Insightful)
According to tfa, they actually aren't throttling. Throttling implies that they are deliberately shaping traffic inside their network to limit your bandwith.
What they are really doing is deliberately creating a bottleneck at key peering locations through negligent inaction when it comes to upgrading infrastructure.
Small difference, I know, but very important when you actually talk about throttling, and likely the argument they would make if the FCC took them to court over it.
Re:Connect with a VPN (Score:4, Insightful)
If you read the Verizon page they are proposing a solution... Netflix should connect directly to Verizon and pay them.
This should be an argument between level 3 and verizon we wouldn't be hearing about this at all if that bandwidth was evenly split up between 100+ services.
Re:Connect with a VPN (Score:5, Insightful)
It's not artificial because of the details of the technical implementation, it's artificial because it's a scarcity that would not be expensive or difficult to resolve. Drought is geographic scarcity that cannot be readily resolved; an undersized water treatment plant is systematic scarcity that can be resolved but would be expensive and slow; a faucet that's rusted half-closed is artificial scarcity.
Re:Connect with a VPN (Score:4, Insightful)
What you're doing is using a VPN connections which has an different inbound interconnect than the one which the majority of Netflix traffic comes in on. Verizon is 100% correct. There's no throttling going on because the connection that is being "throttled" as you put it is actually at 100% utilization and congested causing dropped packets and bad performance.
Re: (Score:3)
In Verizon's defense (Score:5, Funny)
Netflix has *yet* to pull up a dump-truck full of money to Verizon HQ.
Re:In Verizon's defense (Score:5, Informative)
Actually, they did [bloomberg.com]. Verizon has just yet to deliver. Apparently they don't expect to deliver until the end of the year [arstechnica.com] in any case.
Re:In Verizon's defense (Score:5, Funny)
Actually, they did [bloomberg.com]. Verizon has just yet to deliver. Apparently they don't expect to deliver until the end of the year [arstechnica.com] in any case.
Which this article seems to implies it takes Verizon a year to send a technician to 7 cities to connect up a few cables between routers. (And / or maybe install a couple of cards). Maybe Verizon should stop having their techs travel by horseback, they might get it done faster.
Re: (Score:3)
Horseback?
From Verizon, I'd be surprised if they were even given shoes.
End Of the Year is a schedule determined carefully by verizon so that Verizon can figure out how to offer their own competing service "Without the problems of netflix!" ....or possibly because Verizon has to force netflix into a complex solution to hide the fact that it's a simple problem.
Re: (Score:2, Funny)
Maybe Verizon should stop having their techs travel by horseback, they might get it done faster.
Gotta support those buggy whip manufacturers.... ;-)
Re:In Verizon's defense (Score:4, Funny)
Re:In Verizon's defense (Score:5, Funny)
Re:In Verizon's defense (Score:5, Informative)
http://www.wolframalpha.com/in... [wolframalpha.com]
$260,000. I'm sure Verizon loses that in the couch cushions every other month.
Re: (Score:2)
After some Google searches, it looks like a dump truck would be about 27 cubic yards [yahoo.com]. A penny is about 0.44 cubic cm [yahoo.com]. This gives us about 46,900,000 pennies in a dump truck (rounding down to the nearest hundred thousand) or $469,000 worth.
Re: (Score:2)
Never underestimate the bandwidth of a trebuchet loaded with a dumptruck full of pennies.
We are talking about bandwidth, right?
I disagree (Score:3, Interesting)
We all know most top tier network providers are running over multiple bands of fiber just sitting there idle. What Verizon is saying is Level 3 has not worked out an agreement with Verizon to upgrade capacity. The physical part is the easy part; it's just about upgrading port usage. Now, if Level3 is paying for X bandwidth and they're not getting X bandwidth because Verizon hasn't upgraded their equipment, I'm sure Level3's lawyers would be all over that.
Re:I disagree (Score:4, Insightful)
Level 3 doesn't pay Comcast for bandwidth. Why should they? Comcast customers have already paid Comcast for the links to their house and they're the ones pulling data from Level 3. Level 3's customers pay Level 3 to deliver to the edge of their network. As the Level 3 post points out, the cost for Verizon to add more bandwidth between the Level 3 network and the Verizon network is minimal.
Re: (Score:3)
This is a peering agreement, not a service agreement. Even if it were a service agreement, then it should be Verizon paying Level 3 so that verizon's customers can access the content they want.
Peering agreements don't usually pay each other because both networks gain advantage from the peer-connection.
Re:I disagree (Score:5, Insightful)
Part of the issue is that Verizon is a last-mile network, and does not sell symmetric bandwidth to its subscribers. So, the typical agreement between providers - where they each send about the same amount of traffic to each other and upgrade the interconnects to handle that traffic - will not work between Verizon and Level 3. Verizon (and the vast majority of other last-mile providers, including Comcast) will NEVER have a balanced interconnection with Level 3, because the home subscribers can all download far faster than they can upload.
Really, it's Verizon's customers who are causing all this bandwidth usage, so it should be Verizon ensuring that their interconnects can handle the requested bandwidth. If anything, Verizon (and Comcast) should be paying Level 3 for additional download capacity... but we all know that is never going to happen.
Re: (Score:3)
No. Absolutely not. Costs should be driven by the sender of data and not the recipient.
Re:I disagree (Score:5, Insightful)
Costs should be driven by the party responsible for the traffic being on the network. In the case of neflix traffic, that's _me_, the end recipient. And I've already ponied up to the cable company to cover their cost to transfer the bits to me. The cable co just wants to double dip.
Re:I disagree (Score:5, Insightful)
Re:I disagree (Score:4, Interesting)
I don't think Netflix *wants* to fight dirty, or they would have done this already (and with Comcast, as well).
Re: Verizon's customers (Score:3, Insightful)
"it's Verizon's customers who are causing all this bandwidth usage"
Let's pause to re-read that: Verizon's customers. Ah yes, those people who pay Verizon $x each month for y mbp/s of bandwidth. Those foolish people who actually expect Verizon to deliver on what is being paid for.
And then Verizon passive aggressively acts like the problem is not of their own making.
Re: (Score:2)
Yeah, that would make sense. Verison's blog just had a small hidden reference to that. Its not immideatly obvious, probably because thats a difficult thing to explain to people not versed in peering agreements.
Re: (Score:3)
Re:I disagree (Score:5, Informative)
Verizon to level3: "Our traffic from netflix moved over to Level3 last night... very strange, anyways we need to increase our capacity..."
Level3 to Verizon: "Ok, that will be $X"
Verizon to level3: "um... That's 300% higher than any other provider out there..."
Level3 to Verizon: "suck it... your monies are belong to us"
Except that this fictional exchange you've created, in which Level 3 is extorting Verizon for more, is easily refuted by using either blog post. For instance, from Verizon:
Netflix did not make arrangements to deliver this massive amount of traffic through connections that can handle it.
[...] Netflix is responsible for either using connections that can carry the volume of traffic it is sending, or working out arrangements with its suppliers so they can handle the volumes. As we’ve made clear before, we regularly negotiate reasonable commercial arrangements with transit providers or content providers to ensure a level of capacity that accommodates their volume of traffic.
Which is a nice way of saying, "Level 3 is refusing to negotiate rates for more capacity with us, so we've refused to give them more." Level 3's blog post also affirms that the issue is Verizon's refusal to act:
Verizon has confirmed that everything between that router in their network and their subscribers is uncongested – in fact has plenty of capacity sitting there waiting to be used. Above, I confirmed exactly the same thing for the Level 3 network. So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused.
Even without the blog posts, it should be obvious your notion makes little business sense. Level 3 is in no business position to play hardball like you've suggested. If they sacrificed on performance as a ploy to double-dip (i.e. get both Netflix and a lower-tier ISP* like Verizon to pay), Netflix would simply take its traffic to a different Tier 1 ISP that doesn't play those sorts of games, since the double-dipping would be hurting their bottom line. Or, at the very least, they'd be calling out their own ISP, rather than calling out the customer's ISP.
On the other hand, as a lower-tier ISP, Verizon has a monopoly on its own end users: if you want to reach them, you MUST go through them. If Verizon tries to double-dip by getting money out of both the higher-tier ISP and its end users, the end users won't understand what's going on, and in many cases they lack any viable alternatives anyway. Meanwhile, the higher-tier ISP can't switch out for a different peer, since Verizon is the only way to get to those end users.
Besides which, it's not like Netflix's switch from Akamai to Level 3 took Verizon by surprise, as you suggest, since it happened way back in 2010 and has been working fine for most of that time. If there was a problem resulting from the switch, it would have come up before now. Which is to say, this isn't a "Wow! Level 3's traffic is suddenly skyrocketing and we can't keep up!" situation. Rather, it's almost certainly a, "Hey, that Comcast company had a good idea to try getting money out of both sides...let's see if we can do it too!" situation, given the timing of it all.
* A quick aside: I'm well aware that Verizon also maintains a Tier 1 network, but Tier 1 networks rarely connect directly to end users. That's what lower-tier networks do. Moreover, the defining characteristic of a Tier 1 network is that it enjoys free peering with other Tier 1 networks. As such, the Verizon network being discussed here is clearly not their Tier 1 network, but rather a lower-tier one they control (e.g. a Tier 2 or 3 network) that has direct access to their end customers.
Re:I disagree (Score:4, Informative)
Sorry that scenario doesn't fly:
http://www.cidr-report.org/cgi... [cidr-report.org]
Netflix has peerings with:
AS2828 XO-AS15 - XO Communications,US (Tier 1)
AS55095 AS-NFLXCORP - Netflix Inc,US
AS3257 TINET-BACKBONE Tinet SpA,DE (Tier 1)
AS4436 AS-GTT-4436 - nLayer Communications, Inc.,US
AS3356 LEVEL3 - Level 3 Communications, Inc.,US (Tier 1)
AS16397 ALOG SOLUCOES DE TECNOLOGIA EM INFORMATICA S.A.,BR
AS26592 ALOG SOLUCOES DE TECNOLOGIA EM INFORMATICA S.A.,BR (Tier 2 - Has large footprint in latin america).
AS1299 TELIANET TeliaSonera International Carrier,SE (Tier 2 - Apparently the largest fiber providers in Europe).
AS174 COGENT-174 - Cogent Communications,US (Tier 1)
So no this isn't a case of exclusive peering, Level 3 being such a large provider just happens to be the best connection between Verizon and Netflix.
Secondly, that whole thing of 'Level3 to Verizon: "Ok, that will be $X"' has no bearing on a peering agreement, the statement would have been more like "The link between us is congested, want to upgrade the link?" each side upgrades their switch (if neccesary) and they connect the cable / fiber (given that they are in the same location we are talking about a multimode fiber patch at the high end).
Blog post gone? (Score:2)
L3's blog still has a summary blurb, but the link to the actual post gets a 404 - did they take it down or did they just link it wrong? Anybody have a cached copy?
Re: (Score:2)
Its L3's blog, not Verizon's.
Verizon's Response (Score:5, Funny)
Verizon's response was "Ok, but these cards tend to wear out pretty quickly so we'll need you to pay that amount each month. 5,000 streams may sound like a lot, but they don't last very long. A person watches a few movies a week, maybe a couple of youtube videos per day, that's like 20 streams in one week, and that's only one customer. Before you know it, you've used up all 5,000 of those streams and the card needs to be replaced."
"Oh yeah, and if it's coming from Netflix then we're using twice as many streams. We use one stream from Netflix to us, then another stream from us to our customers. Maybe you should really pay us that amount every week."
Re:Verizon's Response (Score:5, Funny)
If this scenario made sense, you'd see Cisco routers with magazine-fed 10gb cards. Automatically eject a spent card and load the next.
That may be a rare example of an expendable with a higher per-unit and per-use price than HP inkjet cartridges.
Re: (Score:3)
Woosh? Continuing from the part I quoted:
Does that sound serious to you? I'm sure the part about them being willing to provide the cable is serious. The part about maybe Verizon not being able to afford the cable ... probably not. Verizon are trying to get other people to pay for the service their customers are already paying them to provide. They have to justify that somehow. L3 seems to be pointing
It's not just Netflix that is suffering though (Score:3)
I have FIOS... Yes my Netflix performance is piss poor, but so are the connections to other services that just happen to use the same transit providers as Netflix.
Particularly the VPS providers that I was using (I just switched due to the latency). I have 2 VPS providers, 1 in Reston, 1 in the UK. The one in Reston is just down the street from Verizons datacenter (used to be UUNET), but the provider to the VPS company I use was Cogent, heavy latency right at the peering point.
Of course, Verizon likes to blame Netflix for picking crappy transit providers, but had it been Company XYZ instead of L3 and Cogent, Verizon would have done the exact same thing to XYZ and let the peers saturate.
I did manage to switch to a different VPS that does not use Cogent or L3, and I have consistent low transit times, which I use as a VPN endpoint. Seems to do the trick (I have been doing this long before any people started publicizing using VPN's to get around Verizon and Comcasts shenanigans, mostly to keep Verizons prying eyes from monetizing my internet behavior, not to keep gov spying eyes out. If VZ wants to pay me [no, not give me a discount on already overpriced service, but give me cold hard cash] for my browsing and internet habits, then I will more than be happy to let them snoop)
Salesmen vs Engineers (Score:5, Funny)
They obviously didn't understand the presentation so they assumed no one else would either.
L3 blog post that has now disappeared (Score:2, Informative)
David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix’s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon’s
Re: (Score:3)
Original article is 404 (Score:3, Informative)
Here is a copy of the text, just in case:
Verizon's Accidental Mea Culpa
Mark Taylor / 23 hours ago
David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix's customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon's own paying broadband consumers.
His explanation for Netflix's on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.
The diagram then shows this one little bar, suggestively color-coded in red so you know it's bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post , I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.
But here's the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon's depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that's what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network â" IN BOTH DIRECTIONS.
So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?
Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3's network interconnects with Verizon's in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.
So let's look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion exists.
lvltvzw
Verizon has
Re: (Score:2)
Robber Barons guarding the bridge (Score:2)
Thou shall not pass
Level3 (Score:5, Interesting)
Level 3 Communications is (or at least was) a really great company to work with. When the company I worked for was a huge customer of theirs, they did anything and everything to satisfy us. The claim of them volunteering to install 10GE cards really does sound like something they'd just do to make a large customer happy.
I really miss working with them.
Re:Answer needed (Score:5, Insightful)
Customers.
You're paying for a service, and nowhere does it say that they will discriminate against a particular service, such as Netflix.
It's obstructive business, against your customer's best interests, for no particular reason. It will also violate any given "net neutrality" laws that are / may come into effect.
Those laws are the answers. The reason for their existence is this sort of unnecessary posturing. And governments make companies do a lot of things against a company's best interests - all the time. It would be in the company's best interest to not pay tax, screw over its customers, not ship goods that have been paid for, be monopolistic, collude with others to enforce market prices, etc. The laws are brought in to stop that shit in the PEOPLE'S best interest, not the company's.
Not saying it's anywhere near perfect, but your post seems to want to back a corporation screwing over its customers and then (falsely) blaming its competitors and random third-party companies for that.
Re:Answer needed (Score:5, Insightful)
"I want it and my government friends have guns..." Is this the best we can do?
The reason Verizon can stay in business despite having "very limited interest in what their customers want" is because of municipal and state granted monopolies, federal grants and subsidies, and the reason they even exist at all is because of a government approved corporate charter. Why is "government friends with guns" an acceptable argument for them getting their way, but not an acceptable argument against it?
Re: (Score:3)
Laws are one of the tools we use to align the interests of groups that are potentially in conflict. Why aren't they a valid option here?
Re:Answer needed (Score:5, Insightful)
Re: (Score:2, Informative)
...because maybe Verizon's paying customers want it and expect to get what they pay for?
I know. Serving your customers. Totally alien concept.
Re:Answer needed (Score:4, Insightful)
The issue with this is that its good for consumers but bad for investors, and we all know who our esteemed congress men actually represent.
Re: (Score:2)
Perhaps a rule where cable or satellite TV providers are prohibited from operating centralized peering points. If Verizon had to buy their bandwidth from upstream providers, they wouldn't be able to choke L3. And L3 would have to bid against Verizon's upstream providers to get Netflix's business.
Essentially, less economic centralization in the network infrastructure would provide for more opportunities for competitive bidding all along the chain. Everyone would end up with more customer-focused incentive
Re: (Score:2)
because Verizon is being paid by their customers to access this data.
Why should verizon have any peering agreements with anyone?
Why not just operate their own network that doesn't peer out to the internet as a whole?
As a customer of an ISP, I'm not paying for the ISP to deliver me to their own internal network. I'm paying them to deliver my data to the peered connections. ALWAYS.
If my ISP has one peered connection that's beyond capacity, I expect them to upgrade it, as it's the cost of doing business.
And es
Re: (Score:2)
The traffic isn't uniform in both directions. When it is, its an easy agreement. When its not, someone usually pays. Level3 offered to pay for the hardware to make additional connections.
But thats kind of like me telling verision that I'll pay for the router and won't charge them for using their network and expecting them to provide me with free service. There isn't enough of an incentive to allow them to agree to my free internet for Bill proposal. Same thing for most networks with unequal traffic pattern
Re: (Score:2)
Re: (Score:2)
to make their customers happy. that is it. The customers are purchasing something, then finding out that they cant get what they thought they paid for. When customers find out that they can get some things at the speeds that they paid for , but not others, then it should be their ISPs job to provide that service. If there was true competition then this would happen quickly. Verizon doesnt really compete, so they can play these bullshit games.
Lets reverse your question. Why should people not be entitled to b
Re: (Score:2)
The government should threaten them because they are not providing the service their customers have paid for and they advertised. The Internet works because tier 1's all connect to each other and upgrade ports as they get saturated (good ISP's are proactive, ok ones last minute and crappy ones only after the link is saturated). While this is related to net neutrality it's primarily a false advertising issue, if they change their wording to correct that it's a DPUC (or similar) issue as they are no longer
Re:Answer needed (Score:5, Insightful)
Got anything better?
Remove the laws and regulations holding back community fiber projects.
Re:Answer needed (Score:5, Insightful)
+1. Politicians at the state level have been paid off by cable companies and ISPs to squash competition from local municipalities. The cities who got in before the legislation are loving their services. It is absolutely insane what money can do in the political process.
Re: (Score:2)
How about to make their fucking customers happy. I pay Verizon (because my only other choice is Comcast, and I hate them more). I request a service, I expect my provider to give me access to this service. Netflix pays L3, L3 is their service provider. Service providers peer, that is the way the internet has always worked.
Re: (Score:2)
Because of the way they are using them, these links (which are set up to provide balanced access) are saturated in one direction while only 30-60% utilized in the other direction.
That's true, but you have the responsibility backwards. The peers that download and don't upload are the problem. That's Verizon, not Netflix.
Re: (Score:2)
Netflix pays level 3 to get their bits from their servers all the way to the edge. Customers pay to get bits from the edge all the way to their house. Level 3 and Verizon make an agreement for the parts where those 2 networks touch each other.
Verizon is saying: "The direction of bits matter. Because our customers are paying to receive bits and not to send bits, YOU owe US more money. If our customers paid to SEND that much bit instead of receiving them, then we'd owe you money....like if it changes anything
Re: (Score:3)
Re:ugh (Score:5, Interesting)
So when Netflix decided to pay Comcast, they were able to upgrade all of those remote trunks in ~24 hours, even though they cost of fortune?