Georgia Tech's ShaperProbe Detects ISP Traffic Manipulation 113
An anonymous reader writes with this excerpt from Ars Technica: "Two researchers at Georgia Tech can tell you exactly how American ISPs shape Internet traffic, and which ones do so. Bottom line: of the five largest Internet providers in the country, the three cable companies (Comcast, Time Warner, Cox) employ shaping while the telephone companies (AT&T, Verizon) do not — though that fact is less significant for the user experience than it might first sound."
Question (Score:1)
Re:Question (Score:4, Informative)
no, shaping is done per-protocol, and throttling is done per pipe
Re: (Score:3, Funny)
Re: (Score:3)
According to TFA, the shaping being done isn't protocol specific (although it can be). They equate it to a bucket of tokens, where you start your transfer with a full bucket, and at specific intervals, you must use a token to transfer X amount of data. What that means is that when you start your transfer, you would have full bandwidth until you empty your bucket, after which you have to wait for the bucket to be filled again to continue your transfer. This has the effect of giving you a full pipe during you
Re:Question (Score:5, Insightful)
Is shaping the same as throttling?
Sort of.
Online, shaping and throttling are something network companies do to customers. In meatspace, throttling is what customers want to to to network company executives.
HTH.
Re: (Score:2)
Surely we could think of some creative shaping of said executives as well - preferably applying shaping before throttling.
Re: Is shaping the same as throttling? (Score:2)
Re: Is shaping the same as throttling? (Score:4, Informative)
Shaping: controlling bandwidth among various protocols (whether DPI or QoS, port number, etc.). This can be enforced by throttling some traffic or by prioritization.
Throttling: capping or reducing the bandwidth available to some identifiable clump of traffic (I use clump because all the other appropriate terms I can think of have some technical definition more strict than what I want to say). It can be done solely in response to congestion, or in the absence of congestion. It can be done on some subset of a subscriber's traffic, or to the entirety of it. Throttling is a slowing or capping of traffic. Most shaping is a subset of throttling. Oversubscription could be considered a form of throttling. Throttling is much more general of a term than shaping.
Re: (Score:2)
I find that Wikipedia is good at giving a few people's opinions of terms, but not actually backing up the vernacular definition.
As opposed to a single poster on slashdot? At least Wikipedia insist on citations.
So in what way are your definitions superior to those on the linked Wikipedia page? What did the wiki get wrong?
Re: (Score:2)
What was "wrong"? Nothing worth dealing with. They give a mostly useless definition that doesn't help differentiate it from anything else, then talk about the technical implications and implementations without regard to the vernaclar, which for the average person is much more important than the overly-cited long and dry article about a te
Re:Question (Score:5, Informative)
Is shaping the same as throttling?
Shaping is when they give you a rate and enforce it. (The faster burst at startup is because you had accumulated some credit by not using your bandwidth in the immediately preceeding time.) There may be separate shaping mechanisms for different protocol families and there may also be shaping on aggregates - like total bandwidth across multiple users of a common DSLAM.
Throttling is when, after they notice that you've used a lot of bandwidth lately, they turn down the rate on the shaper ("traffic manager").
Shaping is mainly about things like keeping protocols from interfering with each other (by giving different classes of them separate allowances) and avoiding congestion and queue-too-full latency (by limiting the traffic sent to a following box to the amount it can handle.)
Throttling is about keeping a user's resource consumption down by slowing him down after he's run fast for a while.
Re: (Score:3)
Not quite right.
Throttling is where they simply slow down your entire internet connection. Typically when you download more than a certain amount of data the ISP punishes you with throttling. Kind of like going to McDonald's and ordering a Supersize Big Mac meal, then a member of staff comes over and punches you in the gut as punishment for overeating.
Shaping is where the ISP tries to slow down certain traffic to give priority to others. The typical use is to slow down P2P and large downloads so that web br
Re: (Score:2)
Throttling is where they simply slow down your entire internet connection. Typically when you download more than a certain amount of data the ISP punishes you with throttling.
I note that this can be accomplished in a hierarchical traffic manager / policer: The first level of the hierarchy might be each of several classes of traffic to (or from) your machine, while the second level might be the aggregate composed of all of this traffic. Throttling would consist of turning down the second level's programmed
Re:Question (Score:4, Insightful)
Shaping is like putting a bus lane / car pool lane on the motorway / freeway; Buses and car pool drivers can move through quicker, at the expense of car traffic having one less lane on the motorway, much like VOIP would be given priority over BitTorrent, or somesuch, but the cars and buses are all capable of going at maximum speed (should traffic allow).
Throttling is like variable speed limits. In the interest of keeping traffic moving freely across the whole motorway, the speed of heavily trafficked areas is slowed down so it doesn't cause congestion. 70MPH becomes 50MPH in the same way that 10Mbit becomes 2Mbit.
Data caps are like a bastard child of toll roads; You've driven a certain distance on this road which is covered by vehicle excise or fuel tax, now you have pay a toll. To travel further on this road, you pay more tolls. You can drive only so far each month on the toll roads for free.
HTH.
Re: (Score:1)
not all shaping / policing is bad. (Score:5, Informative)
I work for a college, and we shape / police traffic to / from the Internet.
This was a necessity on our 3Mb link of many years ago, but has still been useful on our 1Gb link of today.
This policy has greatly improved the user experience. Interactive protocols have low latency, bulk transfer protocols get sent to the end of the line. Where we do slow down things, it isn't really noticed by most folks. After first implementing this many years ago, we immediately got positive feedback. Now it is just "how things are."
Hell, I shape / police traffic at home to my cable modem. VOIP and interactive ssh are still usable even with huge downloads going on now, and users hammering the public wifi I provide to my neighborhood.
Re: (Score:2)
Exactly This. As a sysadmin for a small ISP we do the same. It's not about limiting traffic. It is about prioritizing to provide the best experience and utility.
And that is fine -to a point. As long as shaping does not become an excuse to limit traffic. In the case of the larger ISPs at least, it has become a choice -limit traffic in order to maintain higher profit margins vs invest in increasing capacity.
Re: (Score:2)
It's great that it improves the quality of service...but here's the problem: My neighbor and I both pay for the same service. So why should he get better service because he's using a more interactive protocol? Why should my file download slow down because my neighbors are all on VoIP calls? Depending on how the traffic shaping is set up, isn't it possible that someone who's paying the exact same amount I am for internet service could be getting several times more real data throughput simply because of what
Re: (Score:2)
why should he get better service because he's using a more interactive protocol?
"Better service" has multiple definitions here. I would say that interactive sessions get "worse service" when the latency increases, perhaps because you have some huge download that is eating away at bandwidth that an interactive session could be using. Do you want your VoIP calls to get choppy, or do you want your downloads to be a few minutes faster?
Re: (Score:1)
Re: (Score:2)
I have never had less than 8Mbits/sec on my consumer grade plan with Internode, makes at 15.
Re: (Score:2)
I don't think he's saying he actually has a guaranteed 3 Mbit DSL connection (i.e. a plan with an SLA). What he's saying is that ADSL is generally subject to a lot less contention than cable.
Which is true - with ADSL you have the bandwidth between you and the exchange/DSLAM, at whatever sync rate you have, all to yourself, whereas with cable you are competing with a dozen neighbours on the local loop. Once you hit the DSLAM and enter the ISP's network you are subject to the same contention as everyone else,
Re: (Score:2)
Um, having a low latency doesn't necessarily affect download speeds for large files.
Internet Protocol has this thing with sliding windows, etc., to deal with that.
Re: (Score:2)
If I'm in WoW
If there's anything that is more of a waste of bandwidth, it's this.
Streaming Sophie's Choice on Netflix?
Re: (Score:1)
I think it equates to the freeway, you have small cars that are tiny and capable of getting to their destinations quickly, then you have huge semi trucks that trundle along. Should it be fair to only have 1 lane and have the small cars get stuck behind the huge slow trucks, no. So you create lanes and ask that slow traffic keep right so the rest of the internet can move along.
Re: (Score:2)
Re: (Score:1)
It doesn't hurt your download if there's half a second nothing, then half a second double speed. The average data rate is still the same. However it massively hurts the VoIP connection.
Re: (Score:2)
See, that's the problem. You're a selfish, self entitled idiot. That's the only problem here in this discussion.
You're argument boils down to, everyone else's experience must suffer and frequently become unusable so you can download something a second or two faster. That's stupid. That's selfish. That's idiotic. And the simple fact is, its non-discriminatory. When you require interactive services, you get the same quality of service.
Re: (Score:1)
For now anyway, interactive services are low bandwidth, for example, a typical skype call is 400kbps / 400kbps (high-quality), or 1.2mbps/1.2mbps (HD).
Torrenting a popular 24hour+ old file uses significantly more (I get at least 2.4mbps on comcast, peaking at 3.6, I throttle the upstream to between 400 and 800 mbps, I peak at 1.2mbps otherwise). I download news articles at 16mbps, putting me about 6 times the maximum a skype call will use (highest bandwidth interactive connection I can think of).
An FPS is 1
Re:not all shaping / policing is bad. (Score:4, Interesting)
I understand you sometimes need to shape the traffic to prioritize services you want to perform better. But these companies are getting their money offering services they don't completely provide: First they charge you for access they can shape. And second, they put a cap just in case you get away with it.
Again, I understand your work case, because if you don't pay for it, normally you tend to abuse it. But if you pay for it, why would they need to mess with your traffic?
Re: (Score:3)
I hesitate to use words like "every," but I can think of no ISPs who advertise their plans as anything other than "up to X Mbps." Why you believe that not getting that is somehow a service they don't provide eludes me, particularly when the reason you are not receiving it is legitimate shaping activity*.
I tend to agree with you on this one, for a lot of r
Re: (Score:2)
because if you don't pay for it, normally you tend to abuse it. But if you pay for it, why would they need to mess with your traffic?
Usually when I'm on public wifi (even if paid for), I tend to put downloads on a slow trickle, overnight if possible. Ye olde wget is awesome for that:
wget --limit-rate=20k http://www.example.com/bigfile [example.com]
will limit the download speed to 20KB/s. All platforms have wget available.
(By the way, why does Slashdot make text URLs into hyperlinks even in code sections?)
Re: (Score:2)
ADSL ISPs in the UK are pretty much forced to shape and set ridiculously low caps because of the amount the owner of the telephone lines and other equipment, BT, charges them for access.
BT initially charged on a per-user basis, but then they switched to charging by the amount of bandwidth they supply to the ISP. So where as before an ISP paid £X per user regardless of how much data they consumed, under the new system they would pay for say a 144Mb "pipe" and choose how many users to put on it. O
Re: (Score:1)
ISP are able to install their own equipment/connections in BT exchanges and use the unbundled local loop for the last stage of the connection.
Users connected to smaller exchanges don't benefit from this as there is not enough customers to make it economic for the ISP. These users are forced to use the BT backhaul network at excessive cost.
Re: (Score:2)
Re: (Score:1)
Re: (Score:2)
You're providing for a bunch of students that don't have a contractual agreement with you.
You mean besides the enrollment contract, but otherwise you're spot on. Just like I have no contractual agreement with my ISP other than the one I signed when they installed my service.
So were you public about HOW? (Score:2)
I work for a college, and we shape / police traffic to / from the Internet.
This was a necessity on our 3Mb link of many years ago, but has still been useful on our 1Gb link of today.
This policy has greatly improved the user experience. Interactive protocols have low latency, bulk transfer protocols get sent to the end of the line. Where we do slow down things, it isn't really noticed by most folks. After first implementing this many years ago, we immediately got positive feedback. Now it is just "how things are."
Hell, I shape / police traffic at home to my cable modem. VOIP and interactive ssh are still usable even with huge downloads going on now, and users hammering the public wifi I provide to my neighborhood.
You make a good case, and I agree. I'd like to know whether or not you told your customers how you were shaping their traffic.
I have no issue with enforcing (your idea of) quality of service on a network. What bothers me about Comcast is the general lack of transparency behind it all. Their policies should be public and open to scrutiny, minimally so I know what's going on with the service I'm paying for and ideally so they can be held directly accountable if they implement an absurd form of shaping.
Yup (Score:2)
Unfortunately, I simply cannot trust that an ISP like Comcast will stick to shaping rules that only use the protocol. This is one of those cases where regulation is needed (particular given how many hand-out
Re: (Score:1)
You understand very little of shaping. Prioritization and fair distribution are two mutually exclusive domains. Under proper prioritization (QOS), you can fill your pipe (FTP BT mail download et.al.), but still be at a lower priority than a truly interactive critical data stream(gaming VOIP SSH). Both can max their pipe but the high priority packets need to go when they need to go or problems will surface.
Get a clue first and you will be qualified to find a good ISP.
Re: (Score:1)
Does your college run the last mile through public land that was taken or "borrowed" from the home owners? Does your college charge a monthly fee to the same owners of that land that was taken? Does your college negotiate a monopoly with the local government to ensure they are the only ones that can use that last mile? Effectively making the residents pay for that last mile over and over again and still only have one choice of internet provider over those lines?
I'd rather pay the local government for the
Re: (Score:2)
There's nothing wrong with QoS and putting some traffic at the front of the router queue. It just means light, latency important packets don't get drowned in a torrent of bulk traffic that doesn't really care what order the packets arrive in. But you still have the same amount of capacity for traffic at the start as the end, you're just determining which app gets to use it first, and which app you're prepared to throw away packets for first if necessary at heavy load.
The article is not about that, but bucke
Re: (Score:1)
Re: (Score:2)
Exactly. You can always tell when you've discovered a moron, and idiot, and a downright knuckle-dragger, because they immediately equate all traffic shaping as bad. In fact, traffic shaping is in wide use and is generally a good thing.
Traffic shaping frequently means your SSH session stays fast and responsive despite massive FTP downloads elsewhere. It means your games remain responsive and fun despite your neighbor stealing movies over the next several weeks. It means your HTML is delivered quickly, allowi
Traffic shaping in of itself is not a problem (Score:2)
As long as its done in a neutral manner based on whats being sent as opposed to whos sending it.
ADVERTISED FEATURE of Time Warner and Comcast (Score:5, Informative)
This is said (although almost in passing) in the article. But I will repeat it because i know how few of us RTFA. Time Warner advertises its PowerBoost feature (and Comcast has something similar) where you get like double your usual bandwidth limit for "burst" downloads and then you get throttled back to your limit after the burst is complete. This is a FEATURE they advertise, not something bad. It allows you to (for example) get 15mbit when download a web page or small file on your 7mbit plan. Notice its a 7 mbit plan, they are not throttling you below your plan's rated speed. They are giving you faster downloads for a quick burst. There is plenty wrong with Time Warner, but this isn't one of the the problems.
Re: (Score:2, Informative)
My neighbors add a couple more tiers to PowerBoost. Weekends the cablemodem borders on useless for any low latency or sustained streaming service (for example hulu, netflix). Weeknights interactive games are tolerable with hit & miss reliability for sustained streaming. Weekday mornings flawless service.
Re: (Score:2)
Weeknights interactive games are tolerable with hit & miss reliability for sustained streaming. Weekday mornings flawless service.
Ok, so I've designed this game network protocol which gives users with good bandwidth smooth gameplay and high precision, but gives lower bandwidth users smooth gameplay and lower precision -- client side prediction and synch rate are determined by a test -- a test at the beginning of your connection. This test says you have a monster awesome responsive network, and I'll give you the best experience you expect -- Until your BS ISP plan decides that you only get that for "bursts", and now you're in a fire-f
Re: (Score:1)
Trying to give messing with your bandwidth a positive spin, sounds like newspeak to me.
Re: (Score:1)
Not true, when I had Time Warner they throttled back my Netflix. I could tell because at first it came through at DVD quality and then dropped down to circa 1990 DIVX blocky crap quite quickly. But the speed was claimed to be 7 Mbit, which should be plenty fast enough for streaming DVD quality, let alone Netflix H.264 high compressed DVD stream. Dropped them for Grande, who was also 7 Mbit and the Netflix was crystal clear every time, all the time.
They may be fixed now, but I initially dropped them as they
Re: (Score:1)
It wasn't anywhere near "mentioned in passing" in the article. It was stated quite plainly and directly.
Re: (Score:2)
This is true, and furthermore "shaping" is one of the nicer/friendlier methods of managing traffic contention.
The "my plan says 10 Meg I demand 10 Meg!" argument is simply not technically valid. If the ISP has 10G of upstream to a carrier and more than 1,000 customers, it's not physically possible for everyone to run unlimited 10 Mbps all the time. So now that there's a possibility of contention, the good/bad lies in how the provider limits you, not if.
Possibly the worst is usage caps. After transferri
Telcos may appear innocent (Score:2)
Re:Cue the cable company bashing in 3...2...1.... (Score:5, Insightful)
Re: (Score:2)
Re: (Score:1)
You forgot, larger. Then it would be,... Wait for it,... Big Brother.
Re:Cue the cable company bashing in 3...2...1.... (Score:4, Insightful)
Since when is exclusivity a free market solution?
Re: (Score:2, Informative)
Are you some kind of retard? Deregulation doesn't lead to any sort of fair balance, it leads to excessive and unnecessary price increases across the board as monopoly situations are reached. Check out the New Zealand power market if you don't believe this happens.
Re: Unless you want millions of signals... (Score:2)
...and wires interfering with your private life yes, regulation is the answer.
Re: (Score:2)
There already is regulation (granted exclusivity) favouring the provider. Either that regulation is to be removed, or there has to be another regulation favouring the customer to counter it.
You need regulation whenever you have monopolies, or the danger of monopolies. That regulation has to either prevent forming of monolies, or if that is not possible, regulate those monopolies so that they cannot do too much harm. Remember, as soon as there's a monopoly (either granted or enforced through market power), t
Re: (Score:3)
Want free market? Stop taking subsidies. Stop filing suits against all the small local governments that try to install any system in areas that don't have service. Free market? Just stop being giant fucking douches, determined to milk every dime possible out of the system. Stop abusing the legal system, stop abusing the legislative system, stop ripping the people off. We don't HAVE free trade, you big dummy - not as long as a telco of cable company is permitted to file suit against a community that is
Re:Cue the cable company bashing in 3...2...1.... (Score:4, Insightful)
Re: (Score:2)
And how exactly do you think that is going to happen? Magic?
Re: (Score:1)
No they don't. The difference between AT&T/Verizon and just about any other ISP is that AT&T/Verizon are "tier 0" isps.
They have a worldwide backbone, with enormous capacity, and peer with major isp's everywhere in the world (even China). This not only has huge bandwidth, but these companies are capable of upgrading said backbone without needing anyone's permission. Comcast, Time Warner and Cox, basically hook up to AT&T/Verizon/Cogent/Level3/... Which is not the same thing. Every network engine
Re:Cue the cable company bashing in 3...2...1.... (Score:4, Interesting)
Personally, I agree that they own the network and should be allowed to do what they want. HOWEVER, I also think that we should pass laws FORBIDDING a monopoly into the home. At the least, we should change the monopoly to be from the home to the greenbox and any company can then sign up for a deal with providing service to the greenboxes, AT THE SAME RATES. IOW, if comcast wants to own the greenbox-home monopoly, not a problem. However, they charge other providers the same price that they charge the rest of comcast.
Basically, it is time to limit the monopoly's size.
Re: (Score:2)
Re: (Score:3, Informative)
HOWEVER, I also think that we should pass laws FORBIDDING a monopoly into the home. At the least, we should change the monopoly to be from the home to the greenbox and any company can then sign up for a deal with providing service to the greenboxes, AT THE SAME RATES. IOW, if comcast wants to own the greenbox-home monopoly, not a problem. However, they charge other providers the same price that they charge the rest of comcast.
That is kind of how it works in the UK (See how British Telecom has been split up [wikipedia.org]).
BT Openreach was created to "Ensure that all rival operators have equality of access to BT's own local network" and it works pretty well, I have a BT line and BT Wholesale broadband, but provided by a different company with their own service levels, prices etc. And there are a lot of ISP's like this.
If an ISP doesn't want to use BT's infrastructure in the exchange, they can even install their own [wikipedia.org] whilst still taking advantag
Re: (Score:2)
Re: (Score:2)
The thing is, the American government and the American people "own" a significant part of all the networks. Being granted a monopoly, and accepting subsidies, means that you don't get to make all the rules.
Re: (Score:1)
Which portion of the network does the government own, exactly? Do you think those tax breaks conferred some sort of property rights? That's an... interesting interpretation.
Re: (Score:2)
Not tax breaks, exactly. Subsidies.
http://www.reuters.com/article/2011/02/08/us-fcc-internet-rural-idUSTRE71759V20110208 [reuters.com]
http://www.nber.org/papers/w9090.pdf [nber.org]
The telcos and other ISP's have taken money from the government, many times, mostly to aid in that "last mile" crap. And, the "last mile" just never materializes.
Although - the idea of tax breaks conferring partial ownership upon the government isn't a bad idea either. Tax breaks are, after all, just another form of subsidy.