Verizon's Accidental Mea Culpa 390
Barryke writes: Verizon has blamed Netflix for the streaming slowdowns their customers have been seeing. It seems the Verizon blog post defending this accusation has backfired in a spectacular way: The chief has clearly admitted that Verizon has capacity to spare, and is deliberately constraining throughput from network providers. Level3, a major ISP that interconnects with Verizon's networks, responded by showing a diagram that visualizes the underpowered interconnect problem and explaining why Verizon's own post indicates how it restricts data flow. Level3 also offered to pay for the necessary upgrades to Verizon hardware: "... these cards are very cheap, a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that's the case, we'll buy one for them. Maybe they can't afford the small piece of cable between our two ports. If that's the case, we'll provide it. Heck, we'll even install it." I'm curious to see Verizon's response to this straightforward accusation of throttling paying users (which tech-savvy readers were quick to confirm).
Re:In Verizon's defense (Score:5, Informative)
Actually, they did [bloomberg.com]. Verizon has just yet to deliver. Apparently they don't expect to deliver until the end of the year [arstechnica.com] in any case.
L3 blog post that has now disappeared (Score:2, Informative)
David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix’s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon’s own paying broadband consumers.
His explanation for Netflix’s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.
The diagram then shows this one little bar, suggestively color-coded in red so you know it’s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.
But here’s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon’s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that’s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network – IN BOTH DIRECTIONS.
So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?
Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3’s network interconnects with Verizon’s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.
So let’s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion
Re:But scarcity! (Score:5, Informative)
Verizonâ(TM)s Accidental Mea Culpa
Mark Taylor / 18 hours ago
David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflixâ(TM)s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizonâ(TM)s own paying broadband consumers.
His explanation for Netflixâ(TM)s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.
The diagram then shows this one little bar, suggestively color-coded in red so you know itâ(TM)s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.
But hereâ(TM)s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizonâ(TM)s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because thatâ(TM)s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network â" IN BOTH DIRECTIONS.
So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?
Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3â(TM)s network interconnects with Verizonâ(TM)s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.
So letâ(TM)s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the con
Re:Answer needed (Score:2, Informative)
...because maybe Verizon's paying customers want it and expect to get what they pay for?
I know. Serving your customers. Totally alien concept.
Re:But scarcity! (Score:5, Informative)
Google's cached copy...
http://webcache.googleusercontent.com/search?q=cache:DBHDyx7n4D0J:blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/+&cd=1&hl=en&ct=clnk&gl=us
Original article is 404 (Score:3, Informative)
Here is a copy of the text, just in case:
Verizon's Accidental Mea Culpa
Mark Taylor / 23 hours ago
David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix's customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon's own paying broadband consumers.
His explanation for Netflix's on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.
The diagram then shows this one little bar, suggestively color-coded in red so you know it's bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post , I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.
But here's the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon's depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that's what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network â" IN BOTH DIRECTIONS.
So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?
Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3's network interconnects with Verizon's in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.
So let's look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion exists.
lvltvzw
Verizon has
Re:In Verizon's defense (Score:5, Informative)
http://www.wolframalpha.com/in... [wolframalpha.com]
$260,000. I'm sure Verizon loses that in the couch cushions every other month.
Re:But scarcity! (Score:0, Informative)
I instantly could tell this was a verizon smear post when you said that level 3 is a "small isp". I'm just going to post literally the very first lines from the wikipedia article, to show how horrible you shills are.
It operates a Tier 1 network.[1] The company provides core transport, IP, voice, video, and content delivery for most of the medium to large Internet carriers in North America, Latin America, Europe, and selected cities in Asia.[2] Level 3 is also the largest competitive local exchange carrier (CLEC) and the 2nd largest provider of fiber optic internet access (based on coverage area) in the United States.[3]
You idiots can't even take the time to have BASIC knowledge about what they are making you smear. If shills had honor, you'd probably kill yourself, but if you had any honor, you wouldn't be a shill.
Ironically the captcha is "informed". Something a shill really should be.
Re:I disagree (Score:5, Informative)
Verizon to level3: "Our traffic from netflix moved over to Level3 last night... very strange, anyways we need to increase our capacity..."
Level3 to Verizon: "Ok, that will be $X"
Verizon to level3: "um... That's 300% higher than any other provider out there..."
Level3 to Verizon: "suck it... your monies are belong to us"
Except that this fictional exchange you've created, in which Level 3 is extorting Verizon for more, is easily refuted by using either blog post. For instance, from Verizon:
Netflix did not make arrangements to deliver this massive amount of traffic through connections that can handle it.
[...] Netflix is responsible for either using connections that can carry the volume of traffic it is sending, or working out arrangements with its suppliers so they can handle the volumes. As we’ve made clear before, we regularly negotiate reasonable commercial arrangements with transit providers or content providers to ensure a level of capacity that accommodates their volume of traffic.
Which is a nice way of saying, "Level 3 is refusing to negotiate rates for more capacity with us, so we've refused to give them more." Level 3's blog post also affirms that the issue is Verizon's refusal to act:
Verizon has confirmed that everything between that router in their network and their subscribers is uncongested – in fact has plenty of capacity sitting there waiting to be used. Above, I confirmed exactly the same thing for the Level 3 network. So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused.
Even without the blog posts, it should be obvious your notion makes little business sense. Level 3 is in no business position to play hardball like you've suggested. If they sacrificed on performance as a ploy to double-dip (i.e. get both Netflix and a lower-tier ISP* like Verizon to pay), Netflix would simply take its traffic to a different Tier 1 ISP that doesn't play those sorts of games, since the double-dipping would be hurting their bottom line. Or, at the very least, they'd be calling out their own ISP, rather than calling out the customer's ISP.
On the other hand, as a lower-tier ISP, Verizon has a monopoly on its own end users: if you want to reach them, you MUST go through them. If Verizon tries to double-dip by getting money out of both the higher-tier ISP and its end users, the end users won't understand what's going on, and in many cases they lack any viable alternatives anyway. Meanwhile, the higher-tier ISP can't switch out for a different peer, since Verizon is the only way to get to those end users.
Besides which, it's not like Netflix's switch from Akamai to Level 3 took Verizon by surprise, as you suggest, since it happened way back in 2010 and has been working fine for most of that time. If there was a problem resulting from the switch, it would have come up before now. Which is to say, this isn't a "Wow! Level 3's traffic is suddenly skyrocketing and we can't keep up!" situation. Rather, it's almost certainly a, "Hey, that Comcast company had a good idea to try getting money out of both sides...let's see if we can do it too!" situation, given the timing of it all.
* A quick aside: I'm well aware that Verizon also maintains a Tier 1 network, but Tier 1 networks rarely connect directly to end users. That's what lower-tier networks do. Moreover, the defining characteristic of a Tier 1 network is that it enjoys free peering with other Tier 1 networks. As such, the Verizon network being discussed here is clearly not their Tier 1 network, but rather a lower-tier one they control (e.g. a Tier 2 or 3 network) that has direct access to their end customers.
Re:I disagree (Score:4, Informative)
Sorry that scenario doesn't fly:
http://www.cidr-report.org/cgi... [cidr-report.org]
Netflix has peerings with:
AS2828 XO-AS15 - XO Communications,US (Tier 1)
AS55095 AS-NFLXCORP - Netflix Inc,US
AS3257 TINET-BACKBONE Tinet SpA,DE (Tier 1)
AS4436 AS-GTT-4436 - nLayer Communications, Inc.,US
AS3356 LEVEL3 - Level 3 Communications, Inc.,US (Tier 1)
AS16397 ALOG SOLUCOES DE TECNOLOGIA EM INFORMATICA S.A.,BR
AS26592 ALOG SOLUCOES DE TECNOLOGIA EM INFORMATICA S.A.,BR (Tier 2 - Has large footprint in latin america).
AS1299 TELIANET TeliaSonera International Carrier,SE (Tier 2 - Apparently the largest fiber providers in Europe).
AS174 COGENT-174 - Cogent Communications,US (Tier 1)
So no this isn't a case of exclusive peering, Level 3 being such a large provider just happens to be the best connection between Verizon and Netflix.
Secondly, that whole thing of 'Level3 to Verizon: "Ok, that will be $X"' has no bearing on a peering agreement, the statement would have been more like "The link between us is congested, want to upgrade the link?" each side upgrades their switch (if neccesary) and they connect the cable / fiber (given that they are in the same location we are talking about a multimode fiber patch at the high end).
Re:But scarcity! (Score:4, Informative)
a public corporation's responsibility is to its shareholders and their interests
No. A corporations responsibility is to fulfill its charter, that may mean 'make the most money possible', it may not.
Shareholders only really get to argue about fulfilling that goal, if the goal is to feed the world, the shareholders can't bitch about not making money, only about feeding the world.