Slashdot Log In
Beating Comcast's Sandvine On Linux With Iptables
Posted by
timothy
on Sun Jun 29, 2008 11:20 PM
from the and-then-I'd-be-all-like-pow-and-reconfigure-iptables dept.
from the and-then-I'd-be-all-like-pow-and-reconfigure-iptables dept.
HiroDeckard writes "Multiple sites reported a while ago that Comcast was using Sandvine to do TCP packet resets to throttle BitTorrent connections of their users. This practice may be a thing of the past as it's been found a simple rule in the Linux firewall, iptables, can simply just block their reset packets, returning your BitTorrent back to normal speeds and allowing you to once again connect to all your seeds and peer. If blocking the TCP packet resets becomes a common practice, on and off of Linux, it'll be interesting to see the next move in the cat-and-mouse game between customers and service providers, and who controls that bandwidth."
Related Stories
[+]
Your Rights Online: AT&T Denies Resetting P2P Connections 112 comments
betaville points out comments AT&T filed with the FCC in which they denied throttling traffic by resetting P2P file-sharing connections. Earlier this week, a study published by the Vuze team found AT&T to have the 25th highest (13th highest if extra Comcast networks are excluded) median reset rate among the sampled networks. In the past, AT&T has defended Comcast's throttling practices, and said it wants to monitor its network traffic for IP violations.
"AT&T vice president of Internet and network systems research Charles Kalmanek, in a letter addressed to Vuze CEO Gilles BianRosa, said that peer-to-peer resets can arise from numerous local network events, including outages, attacks, reconfigurations or overall trends in Internet usage. 'AT&T does not use "false reset messages" to manage its network,' Kalmanek said in the letter. Kalmanek noted that Vuze's analysis said the test 'cannot conclude definitively that any particular network operator is engaging in artificial or false [reset] packet behavior.'"
[+]
Your Rights Online: Sandvine CEO Says Internet Monitoring a Necessity 171 comments
Khalid Baheyeldin writes in with a CBC interview with the CEO of Sandvine, Dave Caputo (bio here). Sandvine is the Waterloo, Ontario-based company that provides the technology that Comcast and other ISPs use to overrule Net neutrality by, for example, injecting RST packets to disrupt Bittorrent traffic. Caputo says, among other things, that Internet monitoring is a necessity. Some of the comments to the interview are more tech-savvy than the interviewee comes across.
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
It's a trace buster buster buster (Score:5, Funny)
It'll bust their trace buster buster.
Re:It's a trace buster buster buster (Score:5, Insightful)
Straight-up blocking it is probably more clearly illegal than throttling.
Parent
They are doing it because they are crooks...... (Score:5, Interesting)
They can't block the packets, they sold their users "unlimited" internet. If certain packets are just blocked that's not really unlimited, is it?
They sure didn't tell anyone they were secretly installing Sandvine boxes that nobody had heard of specifically to screw up certain kinds of traffic. They did it in secret. It was subterfuge. A dirty trick. Mischief.
Now that they are found out their story is they are just "managing bandwidth".
But what they are really doing is trying to stop 2% of their customers from using 98% of the bandwidth, bandwidth they have to pay for. Remember, though they are selling "unlimited" internet access at some level *all* bandwidth is measured. Theirs is certainly measured by their upstream provider. There is really no "unlimited" bandwidth.
Parent
Re:They are doing it because they are crooks...... (Score:5, Insightful)
Their own damn fault for selling something they don't have!
Parent
Re:They are doing it because they are crooks...... (Score:5, Insightful)
They always have. ISP's, especially those on the last mile, have historically sold 10 to 20 times the actual bandwidth to their customers. Except a while back the last mile was not a hot zone. There weren't so many things you can use huge amounts of bandwidth on.
Today there are lots. Desktop apps move to the web, there's streaming, online gaming, all kinds of legal, semi-legal and illegal things to download, malware and the list goes on and on.
The ISP's are caught in the middle of all this. They've entered this time period with pricing policies that belong in gentler times, and their infrastructure is also outdated and getting more so every day.
On top of everything, everybody seems to think it's their job to carry the Internet on its back and figure it out somehow. The end customer likes to have huge amounts of bandwidth for pennies. The websites and online apps have bulk deals for bandwidth with providers that have efficient distribution infrastructures all over the world. And the last mile ISP is left to fight a dog eat dog fight with other similar local ISP or with a bigger area ISP, both of which will drive it out of business eventually.
Not to mention the crazy politics involved, where they are required to act as copyright cops and other idiocies.
So they're desperate. They're trying anything to "fix things". There are a couple of sane solutions but not without problems. The obvious move would be to rethink their pricing and start selling capped amounts of bandwidth. Filtering will always be passed somehow but a hard cap upstream is a hard cap. And nobody will be able to protest they're not getting what they're paying for.
But this isn't easy either, because of the fierce competition. You do hard caps, you piss of customers. If they have a choice, they'll run to that new ISP that popped up in the neighborhood a week ago. Sure, that ISP will experience the same problems a while from now, but in the meantime you're short some income.
Another solution is a world-wide effort to update infrastructure (better throughput, either hardware or software). But who's gonna pay for that? The last mile ISP's can't and won't and granted, it's not fair they should pay all of it. But the other interested parties like the status quo and won't pay either, but will bitch just as readily about filtering and caps and whatnot. In the end, the ISP's will probably turn to insightful investors like Google's dark fiber and become their prisoners and people won't like that either, but will conveniently forget they're the ones that pushed the ISP's into that corner.
It's not just the ISP's fault, it's everybody's. The Internet has become an ecosystem, you gotta work together on all parts of it to see proper overall change.
Parent
Re:They are doing it because they are crooks...... (Score:5, Insightful)
Parent
Re:They are doing it because they are crooks...... (Score:5, Insightful)
This seems to me to be *exactly* what an ISP's job is. If they don't like doing this, they should get on out of the business.
Parent
Re:They are doing it because they are crooks...... (Score:5, Informative)
Um, in the US, we're already paying for it. We have since the late 90's when congress passed huge tax breaks on to telcos to develop our 40Mbps connections - you have one of those don't you? The telco's promised us one years ago, I'm sure mine is just around the corner.
Parent
Damn those customers (Score:5, Funny)
Damn, those lousy cellular customers are making a lot of calls on our unlimited rates plan. Let's just cut off their calls or make the service so distorted that they hang up themselves.
Damn, those idiotic customers are all watching hi-dev TV on their cable. Maybe we should switch the output signal to low-def.
Stupid drivers, since the population of the city has grown this roadway has been plugged. Let's give them a lesson by dropping speed limits and closing lanes.
Darnit, people are actually using our long-distance plan to call relatives in the other side of the country more... let's just block their calls randomly with a busy signal.
Too many nerds are visiting slashdot these days, it's getting bogged down. We're tired of upgrading servers, so let's just leave them with these Pentium III's and delete the account of anyone who posts too often.
We don't put up with this shit in other marketplaces, why should we put up with it in regards to the internet? Part of a company's planning procedures should be to map out weak areas in infrastructure, predict where/when capacity increases need to be made, and make improvements where necessary.
Parent
Re:They are doing it because they are crooks...... (Score:5, Insightful)
Parent
A Fitness center analogy.. (Score:5, Insightful)
Now, imagine you buy a year membership card.
Then you start showing up each morning, and again in the evening.
Then the fitness center comes to you and says: "You can come here, but we are going to lock all the doors when you show up, because you are using up to much resources and thus denying them to our other members.
Do you think there would be any outrage ?
Parent
Re:A Fitness center analogy.. (Score:5, Insightful)
Say that you found out a way to earn or safe a lot of money by staying on the fitness machines 16 hours a day.
Suddenly, the 28 fitness machines they expected to service 5,000 people are being used from opening until closing by the same 28 people.
Do you think the fitness companies and their customers would say "ah well... they've got us because of our advertising unlimited service."
No- the next time your contract came up, it would have a clause that allowed them to force people to share the machines or something to protect them.
You are being unreasonable. The cable companies are trying a weaselly scummy way to get out of the situation instead of just doing what they should do up front.
1) Determine the real usage of their desired customer (say 20gb a month).
2) Advertise 24gb a month for one "low rate" with a "reasonable $1 per gb"
And eventually they will. Even if you have you current company in an iron clad contract, if it is losing money the situation *will* fix it self.
---
The current isp situation in america is a complete joke and anti-capitalistic. We basically have duopolies in 99% of cities between AT&T and a cable company. That needs to stop and be broken up. The internet wires, like the roads, should put be put by the government.
Parent
Re:A Fitness center analogy.. (Score:5, Insightful)
Parent
Re:They are doing it because they are crooks...... (Score:5, Insightful)
But what they are really doing is trying to stop 2% of their customers from using 98% of the bandwidth, bandwidth they have to pay for. Remember, though they are selling "unlimited" internet access at some level *all* bandwidth is measured. Theirs is certainly measured by their upstream provider. There is really no "unlimited" bandwidth.
Pisshaw. Large regional and national ISPs don't have "upstream" providers. They have a presence in a NAP(s) and peering agreements with other networks. The only costs they have is for the infrastructure; physical cables, equipment, power and people. They don't pay for bandwidth on a "meter". Their bandwidth is limited by equipment; available technology and costs.
They are "managing bandwidth" to control last mile congestion. It is cheaper to mangle traffic than to upgrade the last mile. Plain and simple.
Parent
Re:It's a trace buster buster buster (Score:5, Interesting)
Last time this came up for discussion, some people suggested that RST-injection was computationally easier than packet blocking, because it works on the connection level rather than the packet level.
It still seems to me like you'd have to do quite a bit of DPI to determine which connections are being used for Bittorrent, but maybe you can identify a connection, send a forged RST packet, and then ignore the packets in that connection for a while (saving you load on the DPI box) for a while, maybe just until it closes.
I'm not entirely clear how these Sandvine boxes work, but it seems like it would be easier to identify "okay, this connection is being used for x," "this connection is being used for Y," and then not have to pay more attention to them, than it would be to examine every single packet. That's where you get your cost reduction, I suspect.
Sandvine has a few patents out there that probably describe in greater detail how their QoS tool works (and which I haven't read yet); apparently the QoS RST-forging are part of their "Stateful Policy Management" product.
Parent
When comments become articles (Score:4, Informative)
Re:When comments become articles (Score:5, Informative)
I posted the kludge last time this got mentioned, I'm rather amused that this actually got posted again
http://tech.slashdot.org/comments.pl?sid=591167&cid=23888479 [slashdot.org]
Parent
Re:Here;s an idea: Stop fucking stealing shit !! (Score:5, Informative)
"Here;s an idea: Stop fucking stealing shit !! If you don't steal you won't care if your stealing facilitation enablers get a fucking RST or not. "
rst hurts anyone trying to keep long lived tcp connections, regardless of how much or what traffic they are sending.
Parent
You know what I hate? (Score:5, Insightful)
It's when I see a comment on Slashdot, that seems to have no relation to the comment above it. Then I discover that the real parent post has been hidden by Slashdot's new comment system, and the child post linked to the grandparent.
It's damn annoying! Slashdot, please, at least link the child to the "hidden comments" link. That way, I won't get head spins when someone appears to viscously lash out at an interesting post.
Parent
sigh ... I'm my own spelling nazi (Score:5, Funny)
viciously, not viscously. I'll have to learn to read my previews more closely.
Parent
Re:sigh ... I'm my own spelling nazi (Score:5, Funny)
Parent
Tag: !news (Score:5, Insightful)
This trick has been around for a while, hasn't it?
The problem is, you can only filter out the RST packets on your end of the connection. But Sandvine also sends RSTs to the other end of the connection. That means it isn't enough for you to be running this iptables rule - all the peers you connect to have to be running it too.
Re:Tag: !news (Score:5, Informative)
Parent
Exactly. (Score:5, Informative)
I noticed my WoW connection suddenly became unstable at the beginning of the month.
I implemented similar firewall rules on my mac and the instability was cut in half.
Guess the other half is being forged to the blizzard servers.
Parent
Re:Exactly. (Score:5, Informative)
I did. I did some digging, found which ports the WoW client uses, and set ignore rules on only those ports.
Parent
This is why you select a specific port.... (Score:5, Informative)
iptables -I FORWARD 3 -p tcp --dport 36745 --tcp-flags RST RST -j DROP;
The above does what it says, drop TCP RST packets on port 36745. That is all you need to do to keep it from affecting your other network applications which may be getting legit reset packets.
Parent
Re:This is why you select a specific port.... (Score:5, Informative)
Parent
Re:This is why you select a specific port.... (Score:5, Interesting)
It's a response to a violation of the TCP protocol to begin with, so it's not surprising that it has some negative side effects.
Probably the best thing to do would be to build a filter that registers the presence of the RST packet and waits to see if you get more data from the site that supposedly sent it.
* If the site that the RST packet supposedly came from continues to act like it's got an open session, then you can ignore the RST as a forgery.
* If you have no more non-closure packets after the RST, then you can apply an aggressive timeout and then deliver the RST after 2-3 seconds of silence.
Parent
Re:This is why you select a specific port.... (Score:5, Interesting)
Although, I've never had this issue and am not a Comcast customer...
I'd assume that the RST coming from Comcast would probably have a different TTL than a legitimate RST.. As a matter of fact, all the RST coming from Comcast would probably have the same TTL.
Anyone looked into this?
Parent
Re:This is why you select a specific port.... (Score:5, Insightful)
This was an initial way researchers detected forged resets. And it still works for some appliances (think snort), but most appliances ISPs use forge TTL now.
The appliance is seeing everything including TTL, so it is rather trivial for these devices to forge it on top of everything else it forges.
One idea being played around with is looking at the arrival time of the reset. A much harder analysis, but a much harder thing for the appliance to control.
Parent
Re:Tag: !news (Score:5, Interesting)
'Sandvine also sends RSTs to the other end of the connection. That means it isn't enough for you to be running this iptables rule - all the peers you connect to have to be running it too.'
Isn't that your ISP committing fraud? Altering a private communication with the intent of disrupting it, or the very least it's the 'ISP' impersonating you and also the other party.
Parent
Re:Tag: !news (Score:5, Insightful)
The problem is, you can only filter out the RST packets on your end of the connection.
That's only a temporary problem. The real problem -- for the ISPs -- is that the same software is running on each end of a p2p, so all of their efforts are guaranteed to fail eventually.
For instance, p2p programs can start using UDP spread spectrum... pass packets on random ports. The receiver then basically implements a quick and dirty tcp-like connection over this (ie much worse for an ISP than actual TCP). Add encryption and random length so it's harder to filter out. Or there can be a shared random number seed for the shared ports. Just for example...
There's probably some computer science or information theory law stating this, but they can't ultimately reduce the targeted traffic by more than the loss from encoding it as 'normal' traffic. For instance, if they limit torrents to 100k/s and the loss is 33% from 'base64' encoding the data as some kind of an html-ish doc then if normal web pages get more than 133k/s then torrents would be faster encoding them as 'normal' traffic.
Parent
Sandvine? (Score:5, Funny)
Re:Sandvine? (Score:5, Funny)
Why my packets never made it through,
With some other peer I was sharin' files,
Between the two of us Comcast was runnin' wild,
Reset me by surprise (reset by surprise), I'm afraid,
From the R-I-Double-A,
Don'tcha know,
Not much bandwidth's gonna be mine.
Oh, I heard it through the sandvine...
Oh, I'm just about to lose my mind,
Honey, honey, yeah...
I know a geek ain't supposed to cry,
But these fears I can't hold inside,
Losin' the 'net and it's neutrality,
Yeah, it means that much to me,
You coulda told me (you coulda told) yourself,
That you're forgin' packets for someone else,
Instead I heard it through the sandvine...
Not much bandwidth's gonna be mine.
Oh, I heard it through the sandvine...
Oh, I'm just about to lose my mind,
Honey, honey, yeah...
People say "Believe half of what you see,
Son, and none of what you hear",
But my router's mighty confused,
So if it's true, please tell me dear,
Do you want (do you want) to make me go,
Back to the ISP (and USENET feed) I used before,
Or should I drop packets from your sandvine...
Plenty bandwidth's gonna be mine.
Oh, I don't listen to your sandvine...
MPAA's 'bout to lose its mind,
Honey, honey, yeah...
- Original work, composed under the influence of Slashdot and beer. Lyrics in public domain. Someone with vocal talent, feel free to improve, record, and youtube it as a parody under the fair use exemptions.
Parent
Re:Sandvine? (Score:5, Funny)
Parent
Good, but shouldn't be necessary (Score:5, Interesting)
While it is good that it is easy to ignore reset packets that were created by the ISP, the question still remains:
Why should we have to block forged packets made by the ISP? If the MAFIAA suits are banking on IP == identity, and the ISP is forging packets with an IP that doesn't belong to any computer they own, isn't that a fairly serious form of forgery?
And, wow that site went down fast.
Re:Good, but shouldn't be necessary (Score:4, Informative)
If the MAFIAA suits are banking on IP == identity, and the ISP is forging packets with an IP that doesn't belong to any computer they own, isn't that a fairly serious form of forgery?
Yet another reason why anyone who knows anything about computers and networks have been saying the **AA's methods of identification are a complete joke and don't amount to anything that could be considered evidence.
Parent
This Account Has Exceeded Its CPU Quota (Score:4, Funny)
Now he needs to add a rule to iptables to save the webserver from the Slashdot effect.
-
I wonder if they will simply start disconnecting. (Score:5, Insightful)
I wonder if they will just say that blocking their RST Packets is a violation of TOS and disconnect you.
First it was email and spam, then it was content.. (Score:5, Insightful)
There is no more good reasons and not any easier for the ISP's to block or rate limit our web-use than it is to centrally control spam. People are different, and have different needs plain and square.
Who should have priority, and how to determine it? I can guarantee that if it is a packet flag, then spammers, virus writers, and even bit torrent users will find a way to use it. And regardless, consider the following:
- Which priority should online Live football have from site X? Should it have over the one from site Y, and Z, and the 1000+ others with different commentators and different languages?
- What if you rather wanted live games? Or Live online music concerts? What should have higher priority?
- What about your live online video rentals - stream from Netflix over one from Blockbuster or should maybe your own ISP be allowed to rate limit all the competition to sell their own?
- What about my VoIP from Skype over Vonage, Gizmo, Provider X,Y,Z?
- What about Online games from Xbox 360 above Playstation 3?
Who are to set the priorities? How on earth should the ISP know what my priorities are? How on earth should the football channel know they should not send with highest priority flags?
And there is also a much easier way that leaves the internet neutral:
As with e-mail spam filtering - let the settings be neutral from the ISP side, then let us set up our own profile or custom rules for the downstream traffic.
Re:First it was email and spam, then it was conten (Score:4, Interesting)
By the way - While onto it - if they are to ratelimit live sports events and do on, they MUST prioritize the version for hearing impaired which have a square with a commentator speaking in sign language in the corner ABOVE the one for the rest. This simply because it is illegal to discriminate against hearing impaired and everyone is able to see the screen even though a part of it might not be of such interest to most of us. Of course - if the hearing impaired could set these option themselves, then we don't need to degrade the performance for those not hearing impaired neither.
Parent
encryption (Score:5, Interesting)
Comcast is evil and I want them to DIAF, but my torrents, which are legal, haven't been that impacted.
When I want fast, I use the Comcast sponsored newsgroups through Giganews.
Comcast has moved on; now they're delaying packets (Score:5, Interesting)
They recently bumped up service to a full megabit upload speed, mostly because of Verizon FiOS service (which still isn't available anywhere in MA except the rich white suburbs- Boston's completely "dark", yet surrounded by towns and cities which have it.) However, if you use it past the old limit (384kbit), after a few minutes, latency skyrockets.
It takes anywhere from a minute to several minutes to kick in, but when it does, ping times to google jumped from 20-30ms to over 300ms. Sometimes I found ping times would be *seconds* long, and ssh became almost completely unresponsive. Curiously, none of the packets would actually be dropped- they'd just very, very badly delayed.
Seems very clearly designed to a)look the same as Verizon "on paper", 2)Satisfy people who want to email photos of the kids to grandma and grandpa (I will admit, it's insanely nice to be able to upload at four times the speed, when it works).
Time to stop trusting TCP (Score:5, Interesting)
I expect we'll see development of protocols more robust than TCP to a MITM attack (this is ultimately a MITM denial of service).
First They Came (Score:5, Insightful)
First they came for the game crackers,
and I did not speak up because I did not play games
Then they came for the pornographers,
and I did not speak up because I did not view porn
Then they came first for the spammers,
and I did not speak up because I was not a spammer
First they came for the music pirates
and I did not speak up because I was not a pirate
Then they came for me,
and by that time there was no fair-use left.
Re:Already slashdotted... (Score:5, Informative)
Parent
Re:Piracy is wrong - plain and simple (Score:5, Funny)
The internet will be mine, mine! Mwa ha ha ha ha ha ha!
Parent
Re:Which rule? (Score:4, Informative)
Parent
Re:Usenet (Score:5, Funny)
The first rule of Usenet is, you do not talk about usenet.
The second rule of Usenet it, YOU DO NOT TALK ABOUT USENET.
Fscking n00bs.
Parent
Re:Port 25 (Score:5, Informative)
Not sure what you mean by sending work email from home.
If you mean your ability to establish a connection with a corporate mail server not located on your ISP's network, then port 25 is unnecessary. You should use port 465 with SSL instead. Problem solved since no ISP ever blocks port 465 in any direction. At least not that I am aware of.
If you mean your ability to run a mail server at your house, then your shit out of luck period. There are a large number of mail servers now that use policy block lists. Every ISP publishes their policy block lists which includes your IP address range. The moment your mail server tries to establish a connection to another mail server using this block list your packets could be dropped right at the router, or your connection terminated by the mail server itself.
Now as upsetting as that might be, it really is for the greater good. The vast majority of all the SPAM being sent every day comes from compromised windows machines on dynamic IP address ranges. Using the policy block list is very effective at immediately stopping those communications from ever reaching the mail server.
If you are absolutely determined to run your own mail server from home I would suggest getting a static IP address. Not only will port 25 not be blocked, but you will have a MUCH BETTER chance of your packets not being dropped by routers servicing the mail servers you will be sending email to.
Another option, depending on the amount of money you want to spend, is to retain the services of an email services provider. There are more than a few out there. You can use your own domain and they will host it for you. They can also provide a fair amount of security and usually are more reliable in getting the email to the destination.
Additionally, you could always get a virtual server someplace and run your own mail server software on it. They have linux and microsoft systems available pretty cheaply. Then you would be operating on IP address ranges used by big ISPs and data centers.
Parent