Forgot your password?
typodupeerror
Verizon Businesses Networking The Internet

Verizon's Accidental Mea Culpa 390

Posted by Soulskill
from the information-wants-to-be-hamstrung dept.
Barryke writes: Verizon has blamed Netflix for the streaming slowdowns their customers have been seeing. It seems the Verizon blog post defending this accusation has backfired in a spectacular way: The chief has clearly admitted that Verizon has capacity to spare, and is deliberately constraining throughput from network providers. Level3, a major ISP that interconnects with Verizon's networks, responded by showing a diagram that visualizes the underpowered interconnect problem and explaining why Verizon's own post indicates how it restricts data flow. Level3 also offered to pay for the necessary upgrades to Verizon hardware: "... these cards are very cheap, a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that's the case, we'll buy one for them. Maybe they can't afford the small piece of cable between our two ports. If that's the case, we'll provide it. Heck, we'll even install it." I'm curious to see Verizon's response to this straightforward accusation of throttling paying users (which tech-savvy readers were quick to confirm).
This discussion has been archived. No new comments can be posted.

Verizon's Accidental Mea Culpa

Comments Filter:
  • But scarcity! (Score:5, Insightful)

    by StikyPad (445176) on Friday July 18, 2014 @09:39AM (#47481997) Homepage

    If people don't think bandwidth is a scarce commodity, how will we get them to pay through the nose for it?!?

    • by emil (695)
      Find locations where you will hurt Verizon customers, and cut the cables. Do so publicly. Precondition repair on upgrades of Verizon's network as you direct. If Verizon doesn't want network neutrality, then punish their customers.
      • Find locations where you will hurt Verizon customers, and cut the cables. Do so publicly. Precondition repair on upgrades of Verizon's network as you direct. If Verizon doesn't want network neutrality, then punish their customers.

        I wish it was that simple, but I'm on board with the general idea. I wouldn't publicly cut the cables. That's too extreme. I would, however, like to see Level 3 turn the tables and publicly (as noisily as possible) accuse Verizon of using up all their bandwidth and that if Verizon

    • by bigpat (158134) on Friday July 18, 2014 @12:01PM (#47483359)
      If you are paying for 75/25 or 50/25 and they are throttling it at the borders of their network, then you aren't getting the bandwidth you are paying for... downgrade your service. That $10, $20 or more per month they aren't getting from you because of their throttling practices should get their attention.
  • No excuses left (Score:5, Insightful)

    by Anonymous Coward on Friday July 18, 2014 @09:41AM (#47482005)

    Too big to fail, too arrogant to concede, too greedy to care. This news is all the more reason to regulate.

    • Re:No excuses left (Score:5, Insightful)

      by Stolpskott (2422670) on Friday July 18, 2014 @09:58AM (#47482139)

      Too big to fail, too arrogant to concede, too greedy to care. This news is all the more reason to regulate.

      But, but, but... regulation is the antithesis of the Capitaist way that our republican Democracy has weaned its children on since it was formed!!
      I do tend to agree though - regulation of ISPs is probably the only way to deal with this.
      Capitalist theory says that if an incumbent merchant/provider is too inefficient to provide a good service or if another potential merchant/provider thinks they can do a better job for a lower price, then that new provider will step in and provide said service. The threat of that is what keeps the incumbent lean and competitive, and the result is a competitive environment that is generally good for the consumer and rival providers seek to offer better deals to entice custom away from their competitors.
      However, that theory assumes that there is a very low or non-existent barrier to entry into that competitive marketplace. Given the initial infrastructure setup costs and, in many cases, exclusivity contracts between providers and the municipal areas which would present the profits to drive services out into more marginal areas, the barriers to entry into the Tier 1 ISP market are prohibitive, to the point where you need to be a corporate entity the size of Google to be able to reasonably make the capital investment required.
      As such, the local markets for each ISP more closely resemble non-competitive monopolies with the illusion of choice being provided by third party suppliers who typically have to by access to the resources from the incumbent monopoly - they get wholesale prices, and the consumer sees some small price reductions if the third parties can make enough money to operate by charging the consumer slightly less than the discount they got from the incumbent. But fundamentally, everything is still controlled by that original monopolistic provider, so services suck, progress is stifled because there is no incentive for change, innovation is discouraged, and the level of capacity/reliability is never going to be any more than "just barely enough so that we can maximise our profit margins".

      • by wrf3 (314267)

        But, but, but... regulation is the antithesis of the Capitaist way that our republican Democracy has weaned its children on since it was formed!!

         

        Regulating what to make is the antithesis of Capitalism. Regulating what to charge is the antithesis of Capitalism. Regulation of abuse of monopoly powers is not.

      • Re:No excuses left (Score:5, Insightful)

        by pixelpusher220 (529617) on Friday July 18, 2014 @11:15AM (#47482937)
        Free market capitalism is like a wild horse. Powerful, fast and strong.

        Also not terribly productive until you put reigns on it and channel that strength towards useful goals.

        Regulations are the reigns by which the power of the free market is harnessed and made productive.

        And like reigns...to much is bad, but none is worse. But nuanced conversations like this with 'but free market' morons in the current GOP are next to impossible.
    • Re:No excuses left (Score:5, Insightful)

      by Virtucon (127420) on Friday July 18, 2014 @10:45AM (#47482667)

      Regulation for the public benefit = good. Examples: Public Utilities, Healthcare, Agriculture, Air Quality/Environmental Protection.
      Regulation for the sake of Regulation = bad. Examples: 70,000 + pages of IRS Regulations and 30,000 pages of tax code written by special interests and bureaucrats.

  • by MrDoh! (71235) on Friday July 18, 2014 @09:41AM (#47482009) Homepage Journal
    Was obvious people were going to figure out everything Verizon was saying is BS, and that they'd continue to get bad press about this. You'd think the PR droids spouting this stuff would talk to their tech people and listen. But they probably said "look, just give us a pretty graphic right?" "But, techs will see through your spin" "Leave that to us" "But it'll make us look even worse" "You don't get paid to deal with this" All too predictable, and the same techs are probably still being yelled at.
    • It is painful how true that is.

      Most of the time they get away with it, I'm just ecstatic that they didn't this time.....and soooo badly......

  • Connect with a VPN (Score:5, Interesting)

    by LinuxFreakus (613194) on Friday July 18, 2014 @09:46AM (#47482047)
    Just connect to a VPN first and then use Netflix. You'll be able to clearly see how much Verizon is throttling. I've been using this as a workaround for a while now. I'm not sure why more people don't think of pointing this out when Verizon's tech support people claim there is no throttling.
    • by i.am.delf (1665555) on Friday July 18, 2014 @09:51AM (#47482077)
      You can also escape this bottleneck using an IPv6 tunnel to he.net.
    • by jythie (914043) on Friday July 18, 2014 @09:54AM (#47482103)
      I wonder if Netflix themselves could provide such a service, maybe even run it through Comcast? Now there would be a fun bit of PR... "yes, your netflix connection runs better if we route from your Verizon DSL through Comcast, imagine how much better it would be if you just switched to Comcast?"
    • by Anonymous Coward on Friday July 18, 2014 @09:55AM (#47482115)

      According to tfa, they actually aren't throttling. Throttling implies that they are deliberately shaping traffic inside their network to limit your bandwith.

      What they are really doing is deliberately creating a bottleneck at key peering locations through negligent inaction when it comes to upgrading infrastructure.

      Small difference, I know, but very important when you actually talk about throttling, and likely the argument they would make if the FCC took them to court over it.

      • by pr0fessor (1940368) on Friday July 18, 2014 @12:13PM (#47483465)

        If you read the Verizon page they are proposing a solution... Netflix should connect directly to Verizon and pay them.

        This should be an argument between level 3 and verizon we wouldn't be hearing about this at all if that bandwidth was evenly split up between 100+ services.

    • by Talderas (1212466) on Friday July 18, 2014 @10:40AM (#47482629)

      What you're doing is using a VPN connections which has an different inbound interconnect than the one which the majority of Netflix traffic comes in on. Verizon is 100% correct. There's no throttling going on because the connection that is being "throttled" as you put it is actually at 100% utilization and congested causing dropped packets and bad performance.

    • by harl (84412)
      Verizon is most likely telling the truth in a highly deceptive way. Technically there is no throttling. There are not fucking with any packet that comes through that connection. They are not actively doing anything to inhibit the flow from Netflix. They've simply chosen to architect a bottle neck which only impacts one provider. The end result is the same but they can truthfully say they're not throttling.
  • by NotDrWho (3543773) on Friday July 18, 2014 @09:47AM (#47482049)

    Netflix has *yet* to pull up a dump-truck full of money to Verizon HQ.

  • I disagree (Score:3, Interesting)

    by Dishwasha (125561) on Friday July 18, 2014 @09:53AM (#47482093)

    We all know most top tier network providers are running over multiple bands of fiber just sitting there idle. What Verizon is saying is Level 3 has not worked out an agreement with Verizon to upgrade capacity. The physical part is the easy part; it's just about upgrading port usage. Now, if Level3 is paying for X bandwidth and they're not getting X bandwidth because Verizon hasn't upgraded their equipment, I'm sure Level3's lawyers would be all over that.

    • Re:I disagree (Score:4, Insightful)

      by putaro (235078) on Friday July 18, 2014 @10:04AM (#47482215) Journal

      Level 3 doesn't pay Comcast for bandwidth. Why should they? Comcast customers have already paid Comcast for the links to their house and they're the ones pulling data from Level 3. Level 3's customers pay Level 3 to deliver to the edge of their network. As the Level 3 post points out, the cost for Verizon to add more bandwidth between the Level 3 network and the Verizon network is minimal.

    • This is a peering agreement, not a service agreement. Even if it were a service agreement, then it should be Verizon paying Level 3 so that verizon's customers can access the content they want.

      Peering agreements don't usually pay each other because both networks gain advantage from the peer-connection.

    • Re:I disagree (Score:5, Insightful)

      by SighKoPath (956085) on Friday July 18, 2014 @10:14AM (#47482345)

      Part of the issue is that Verizon is a last-mile network, and does not sell symmetric bandwidth to its subscribers. So, the typical agreement between providers - where they each send about the same amount of traffic to each other and upgrade the interconnects to handle that traffic - will not work between Verizon and Level 3. Verizon (and the vast majority of other last-mile providers, including Comcast) will NEVER have a balanced interconnection with Level 3, because the home subscribers can all download far faster than they can upload.

      Really, it's Verizon's customers who are causing all this bandwidth usage, so it should be Verizon ensuring that their interconnects can handle the requested bandwidth. If anything, Verizon (and Comcast) should be paying Level 3 for additional download capacity... but we all know that is never going to happen.

      • by Talderas (1212466)

        No. Absolutely not. Costs should be driven by the sender of data and not the recipient.

        • Re:I disagree (Score:5, Insightful)

          by suutar (1860506) on Friday July 18, 2014 @11:21AM (#47483005)

          Costs should be driven by the party responsible for the traffic being on the network. In the case of neflix traffic, that's _me_, the end recipient. And I've already ponied up to the cable company to cover their cost to transfer the bits to me. The cable co just wants to double dip.

          • Re:I disagree (Score:5, Insightful)

            by BronsCon (927697) <social@bronstrup.com> on Friday July 18, 2014 @12:37PM (#47483699) Journal
            Bingo. Well, almost; it's a little more nuanced than that. Costs should be driven by the party responsible for the traffic being on *your* network. For Verizon, that's Verizon's customer; for Level3, that's Netflix. And they both already pay their providers. Where Verizon and Level3 peer, it's a matter of recognizing that the imbalance of traffic across that link is caused by Verizon's customers requesting more traffic than they (can) return. Thus, Verizon caused the imbalance and should therefore pay for it. If Verizon primarily sold symmetrical access and allowed their users to run servers, there would likely be a balance, and if there was not, they'd have a leg to stand on here, but they don't sell symmetrical access to the end user and they don't have a leg to stand on in this debate; what they do have is a monopoly on Verizon customers, which they're attempting to abuse right now, which should warrant an anti-trust suit, if anything. No additional regulation needed.
            • Re:I disagree (Score:4, Interesting)

              by BronsCon (927697) <social@bronstrup.com> on Friday July 18, 2014 @12:46PM (#47483789) Journal
              Ahh... I thought of this as soon as I clicked the button... If Netflix *really* wants to drive the point home, they can simply start peering or buying transit (more likely) from the providers Verizon has non-congested links with and stop routing to Verizon through L3 and Cogent. When Verizon refuses to upgrade *those* links, Netflix will be able to say "Either Verizon is refusing to upgrade their links, as we've been saying, or they only use providers which, as they claimed of Level3 and Cogent, can't handle the throughput their customers are requesting. In either case, this should be a wake-up call for Verizon customers to stop giving them money for a service they aren't, for whatever reason, delivering." And they shouldn't stop there; after that statement, they should re-enable routing to Verizon over all available links and watch the congestion continue; regardless of Verizon's response (which will likely be something along the lines of "Netflix performance continues to be slow because they have disabled routing to our network over multiple providers"), Netflix can stand up and say "We are currently routing to the Verizon network through every provider Verizon also uses, and make our routing decisions based on performance metrics, including packet loss and ping time to each user, to ensure that our users get the best possible experience we can provide. Unfortunately, as every link Verizon maintains appears to be congested, packet loss and ping times are high in all cases; the only solutions that exist are for Verizon to upgrade their links or peer with us, or for Verizon customers to find an alternate provider."

              I don't think Netflix *wants* to fight dirty, or they would have done this already (and with Comcast, as well).
      • by Kevoco (64263)

        "it's Verizon's customers who are causing all this bandwidth usage"
        Let's pause to re-read that: Verizon's customers. Ah yes, those people who pay Verizon $x each month for y mbp/s of bandwidth. Those foolish people who actually expect Verizon to deliver on what is being paid for.

        And then Verizon passive aggressively acts like the problem is not of their own making.

    • Yeah, that would make sense. Verison's blog just had a small hidden reference to that. Its not immideatly obvious, probably because thats a difficult thing to explain to people not versed in peering agreements.

  • L3's blog still has a summary blurb, but the link to the actual post gets a 404 - did they take it down or did they just link it wrong? Anybody have a cached copy?

  • by CanadianRealist (1258974) on Friday July 18, 2014 @09:59AM (#47482157)

    Level3 also offered to pay for the necessary upgrades to Verizon hardware: "... these cards are very cheap, a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more

    Verizon's response was "Ok, but these cards tend to wear out pretty quickly so we'll need you to pay that amount each month. 5,000 streams may sound like a lot, but they don't last very long. A person watches a few movies a week, maybe a couple of youtube videos per day, that's like 20 streams in one week, and that's only one customer. Before you know it, you've used up all 5,000 of those streams and the card needs to be replaced."

    "Oh yeah, and if it's coming from Netflix then we're using twice as many streams. We use one stream from Netflix to us, then another stream from us to our customers. Maybe you should really pay us that amount every week."

    • by idontgno (624372) on Friday July 18, 2014 @11:03AM (#47482811) Journal

      If this scenario made sense, you'd see Cisco routers with magazine-fed 10gb cards. Automatically eject a spent card and load the next.

      That may be a rare example of an expendable with a higher per-unit and per-use price than HP inkjet cartridges.

  • I have FIOS... Yes my Netflix performance is piss poor, but so are the connections to other services that just happen to use the same transit providers as Netflix.

    Particularly the VPS providers that I was using (I just switched due to the latency). I have 2 VPS providers, 1 in Reston, 1 in the UK. The one in Reston is just down the street from Verizons datacenter (used to be UUNET), but the provider to the VPS company I use was Cogent, heavy latency right at the peering point.

    Of course, Verizon likes to blame Netflix for picking crappy transit providers, but had it been Company XYZ instead of L3 and Cogent, Verizon would have done the exact same thing to XYZ and let the peers saturate.

    I did manage to switch to a different VPS that does not use Cogent or L3, and I have consistent low transit times, which I use as a VPN endpoint. Seems to do the trick (I have been doing this long before any people started publicizing using VPN's to get around Verizon and Comcasts shenanigans, mostly to keep Verizons prying eyes from monetizing my internet behavior, not to keep gov spying eyes out. If VZ wants to pay me [no, not give me a discount on already overpriced service, but give me cold hard cash] for my browsing and internet habits, then I will more than be happy to let them snoop)

  • by minstrelmike (1602771) on Friday July 18, 2014 @10:00AM (#47482163)
    What happened was a bunch of salesmen and marketers at Verizon asked how they could explain the network throttling.
    They obviously didn't understand the presentation so they assumed no one else would either.
  • by Anonymous Coward
    Verizon’s Accidental Mea Culpa Mark Taylor / 17 hours ago

    David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix’s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon’s
  • by fok (449027) on Friday July 18, 2014 @10:10AM (#47482307) Homepage

    Here is a copy of the text, just in case:

    Verizon's Accidental Mea Culpa
    Mark Taylor / 23 hours ago

    David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix's customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon's own paying broadband consumers.

    His explanation for Netflix's on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.

    The diagram then shows this one little bar, suggestively color-coded in red so you know it's bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post , I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.

    But here's the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon's depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that's what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network â" IN BOTH DIRECTIONS.

    So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?

    Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3's network interconnects with Verizon's in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.

    So let's look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion exists.

    lvltvzw

    Verizon has

  • Thou shall not pass

  • Level3 (Score:5, Interesting)

    by JWSmythe (446288) <jwsmytheNO@SPAMjwsmythe.com> on Friday July 18, 2014 @11:04AM (#47482833) Homepage Journal

    Level 3 Communications is (or at least was) a really great company to work with. When the company I worked for was a huge customer of theirs, they did anything and everything to satisfy us. The claim of them volunteering to install 10GE cards really does sound like something they'd just do to make a large customer happy.

    I really miss working with them.

Make headway at work. Continue to let things deteriorate at home.

Working...