Google Caught in Comcast Traffic Filtering? 385
marcan writes "Comcast users are reporting 'connection reset' errors while loading Google. The problem seems to have been coming and going over the past few days, and often disappears only to return a few minutes later. Apparently the problem only affects some of Google's IPs and services. Analysis of the PCAP packet dumps reveals several injected fake RSTs, which are very similar to the ones seen coming from the Great Firewall of China [PDF]. Did Google somehow get caught up in one of Comcast's blacklists, or are the heuristics flagging Google as a file-sharer due to the heavy traffic?"
Not me... (Score:3, Informative)
Re:Not me... (Score:5, Interesting)
Re: (Score:2)
That's interesting. I have had resets when SSHing one specific Linux box that I use for work, whereas all others have been fine. I don't know if that box is on a Comcast connection or not. But I haven't had any troubles SSHing into my own box from elsewhere.
Re: (Score:3, Interesting)
Re:Not me... (Score:5, Informative)
Furthermore, the problem is very likely far more simple and less sophisticated than this issue of packet spoofing.
Set up a continuous ping to something "nearby" (your gateway, your DNS ser ver, your neighbor, whatever) in your Comcast network and tee it to a file. Leave it up for days and you'll likely see periods of time where you have no service for patches of time... often long enough to kill sessions.
I very often have problems with any sort of sessions (SSH, VPN, etc.) staying up for long periods of time because the underlying line level reliability is so poor. I can watch my cable modem logs and see many resets, timeouts, etc.
I laugh whenever asked about phone service via Comcast. Sadly, however, this pathetic reliability also precludes Vonage and the like. And I find this a bit sad since while I do not consider Comcast capable of running a world class network, I loathe the phone company. Those guys are more competent but much more directly evil.
Re:Not me... (Score:4, Informative)
If we look at what is promised, what is purchased, what is possible, and compare that to what is experienced, it is clear that some ISPs suck, and there is a reason that they suck. Suckiness is not 'normal' or 'average' or acceptable. With the FCC ruling to allow multiple ISP connectivity to many homes, the quality of service should improve to prevent customer churn. My advice is to switch if complaints are not resolved if you can. If not, register a complaint with the authority who gave your ISP broadband monopoly in your area. Document the complaint process and responses. The BBB, I believe, can be consulted in cases where they clearly are not giving you what you paid for.
Re: (Score:2)
It seems like a new article pops up every week that blasts Comcast for these pratices. I'm losing count. I just keep hoping it doesn't
Re: (Score:3, Insightful)
You know, since providers and governments are breaking TCP/IP with these strategies, I think it warrants some sort of firewall extension to run heuristics on RST packets and try to determine wh
Re: (Score:2)
One thing that doesn't bother me is that ISPS should do some traffic shaping if the line is saturated. That is OK by me. Hell, if there was really that big of a problem I would support having cache-technology on the ISP side that websites could enable. Why should I have to pull 'panda sneezing' from California when my neighbor just looked at it? Why am I not pulling it from my ISP's servers in downtown Chicago? Of course this would need to be approved by the site that has
Comment removed (Score:5, Informative)
Re: (Score:3, Funny)
Re:Not me... (Score:5, Insightful)
Re:Not me... (Score:5, Insightful)
There are two kinds of big mistakes you can make: those that are big for a company your size, and those that are just plain big. In a big company with lots of customers, small mistakes are multiplied by volume into just plain big mistakes. If you've got gross revenues of a million dollars, a mistake with a potential $100,000 impact is big for your business, but not that big. You can survive it, you can reestablish credibility with your customers (whom you know face to face) by personally eating a helping of crow in front of each and every one. If you're in a company a 100x as big, you're talking maybe a $10M impact that if laid to the account of any individual employee is a disaster beyond that individual's ability to make right.
That's why large companies can develop a special kind of stupidity, preferring a status quo that is certainly wrong to any alternative that is only probably right. Individuals protect themselves using exactly the same strategy that schooling fish employ. Any decision has to have so many fingerprints on it that firing the people who can be tied to a mistake is like cutting off your right arm. That's why big defense contractors are probably the most bureaucratic organizations on the planet. Ordinary mortals have to make decisions that can have impacts measured in hundreds of millions of dollars. In any such situation, you obviously need a form of collective responsibility, the question is what form it takes. It's all to easy to develop an organization that protects individuals by being unable to detect and respond to most problems. We didn't know about it, if we had we probably couldn't do anything about it, and if we could have, it wasn't my job.
The problem is not that a typical PHB is necessarily stupid. The problem is that organizations are built in a way that rewards people for acting in a stupid way. But stupidity is all too common. Even stupid people can manage to be cunning in bad organizations, because they are problems in an organization built around willful blindness to problems. It's more of a challenge for intelligent people I suppose, because it's hard for people with imagination to find much satisfaction in what it takes to get ahead in these places. It has even been suggested that sociopaths make good managers, which I doubt. But I can well believe that feigned stupidity is better in some cases than the real thing.
Re: (Score:3, Interesting)
That's the right way to handle traffic in the net - drop the priority for packages that aren't sensitive and promote packages that are sensitive to delays. If the lines are up to their throughput limit this is the way to go, and doing it right will not have any really bad effect on the users.
Intentionally dropping data packages is much more evil since that interferes with the functionality and ultimately drives up the network traffic - not down - since many more packages has to be sent and re-sent to pro
Re: (Score:3, Insightful)
Google home page, but not services (Score:2, Interesting)
I was working from home last week, so I was using my Comcast connection extensively every day. The problems with Google connection happened several times a day. Intermittently, my attempts to connect to www.google.com failed for 5-10 min at a time. Oddly enough, going directly to Google services (Gmail, Notebook, Bookmarks, etc.) worked just fine.
Re:Not me... (Score:4, Interesting)
Thanks for adding anecdotal noise to the discussion that adds absolutely nothing to the discussion.
Gee, I think that anecdotal evidence is interesting, especially if you're interested in understanding what rules Comcast uses to decide which packets to block. Questions like: "Is it the whole network or just portions (I suspect just portions)?" or "Is it all the time or during peak demand?" Please try to be civil. If a comment isn't valuable, it won't be modded up. If it is valuable it will.
Re: (Score:3, Funny)
I also posted my Comcast anecdote on Slashdot, and haven't been flamed for it yet.
Re: (Score:3, Informative)
Get the facts (Score:5, Funny)
Comment removed (Score:4, Funny)
Re: (Score:3, Funny)
("Who watches the watchers?")
Re:Get the facts (Score:4, Insightful)
I have noticed this stuff happening for over a year or more. Of course I speak my mind on a lot of issues that goes against the grain. For instance, stuff like the domestic spying- I usually point out that it is far from domestic which get troll, flame bait, and overrated modifiers all the time. It has been a situation for a while now and I have a working theory on it.
The theory goes something like this. When we started seeing the politics sections appear (that was supposed to be temporary but stayed forever) I started seeing political motivated posts that were basically rehashes of some party line talking point getting moderated insightful while common sense posts about the topic in hand was being modded off topic, under rated or some other negetive moderation. I began watching and it appear that either an organized group or groups of people have signed up in order to press a particular view or the sites own administration is doing it to some extent. Judging by the constant links to political sites like media matters and moveon.org by posters themselves, I'm starting to think it is a group of ideolgs doing it.
Of course I can prove anything other then by saying it is my personal observations. But if you start looking at it in this light, you will likely see the trend happening too. Of course to what degree will probably depend on your political bias. But you should definitely see a pattern rising that will worsen coming to a major election time.
Re: (Score:3, Informative)
You wouldn't happen to be one of the people I talked about attempting to dispel knowledge of this are you? There we go, the tinfoil hat is back in place and everythign feels right again.
Either look around or keep your eyes shut. It doesn't matter much to me. But I call them as I see them. I haven't been wrong often.
Re: (Score:2, Informative)
Wow, -1 Troll? Do people even think before moderating? For those who aren't subtle enough to get it on their own, the parent post is being sarcastic.
Edit : ha, nevermind, someone had the common sense to mod it Funny.
Edit #2 : Oh yeah, didn't you know? Now you can edit your posts on Slashdot.
Re: (Score:2)
Google *is* the file-sharer (Score:4, Insightful)
Gmail Notifier (Score:5, Informative)
Comcast is really pissing me off. But what's my other option: Qwest DSL.
Re:Gmail Notifier (Score:4, Insightful)
Call your city. Ask them to re-evaluate Comcast as the local Cable provider or do what my town did: offer RCN as a competing provider.
How can you tell? (Score:2)
Problem is, I never thought to dig into it as my connection is regularly 'comcastic' (pejorative) during peak hours.
I'm not sure if you should consider yourself lucky or unlucky that you can actually tell the difference between their incompetence and malice.
Re: (Score:2)
I hope they get slapped (Score:3, Interesting)
Re: (Score:3, Insightful)
unfair competition (Score:5, Insightful)
Comment removed (Score:5, Insightful)
Re:unfair competition (Score:5, Insightful)
It seems to me the whole rage against P2P traffic (which is how lots of games are played, BTW, and how almost all VPNs are set up) is not so much about capacity as about a conflict of interests on the part of Comcast. They're the content delivery network for TV programming and music (they have music channels like DirecTV does, don't they?). They are wanting to make sure you use your cable TV for getting video and audio, because that's where they get a bigger cut.
Comment removed (Score:5, Interesting)
Re: (Score:2)
Surely they'd rather have all that bandwidth going to paying HD content subscribers, rather than those filthy file-sharers!
Oh and they'd like to continue to oversell capacity too, thanks!
Re: (Score:2)
Re: (Score:2)
That's illegal. Ask Microsoft.
Fair? Who is saying anything about fair? (Score:3, Insightful)
I have to go with the dutch situation because that is the one I know.
In holland you used to have PTT (Post, Telecom, Telegram) which was owned by the state and also had banking services. Basically they where huge, slow, old but worked and kept things under control. For instance Postbank does NOT charge end users for tranferring money and has a free debit card. Essentially for normal people banking in holland was FREE and paid an interest if you had a postive balance.
But no that was not good enough, we nee
Re: (Score:3, Insightful)
Would be kind of awesome... (Score:3, Interesting)
Theory... (Score:2, Interesting)
Push it one step further... (Score:5, Interesting)
What if Google, a (justifiably) huge advocate of network neutrality, is deliberately sending the type of RST packets that imitate Comcast's faked packets, specifically to Comcast IP addresses, knowing the inevitable fallout that would result? It would make an already bad situation for Comcast far, far worse, and it's likely that the requested Senate investigation would turn into nails in the coffin for those who want preferential treatment of packets on the Internet.
For a company that does no evil, if they could pull it off, it would be absolutely diabolical. But then, it could easily be one of those "ends justify the means" kinds of situations. At any rate, all I can say is "MWAH HAH HAH HAH HAH!!!! Suckers!"
(No, I don't actually believe that's what's happening, but man, what an AWESOME plan to make network neutrality happen once and for all.)
Re: (Score:2, Insightful)
The ends should justify the means. The problem is when you start thinking the ends justify ANY means.
Far more likely (Score:3, Insightful)
It could be technical incompetence (Score:2)
Re: (Score:3, Informative)
Re: (Score:2)
How about running your own DNS server? Or get a list of DNS servers from various ISPs round the world that work and rotate through the IPs.
Re: (Score:2)
iptables fake RST detector (Score:5, Interesting)
iptables -I INPUT -j LOG -p tcp -m tcp --tcp-flags RST RST -m conntrack --ctstate NEW,INVALID
The fake RST will probably not have a valid sequence number for the established TCP connection, so the Linux stack will flag it as a NEW connection, and the fact that you're getting a RST for a NEW connection should be good enough alarm.
Or maybe it would also work with just the matching code
iptables -I INPUT -j LOG -p tcp -m tcp --tcp-flags RST RST -m state --state NEW,INVALID
What do y'all think?
Re: (Score:2)
Why wouldn't it have a valid sequence number? Don't they only need a single packet to get the proper sequence number? Wouldn't most TCP implementations throw away a sequence number that was so far off?
Re: (Score:2)
OK, I went back and RTFA. It appears they do send a correct SEQ RST, along with one in the 12xxx range. The problem is, they send them, spoofed, in both directions. So, even if you did come up with a way to ignore valid RSTs when there was an invalid RST very nearby, you'd also have to make the remote host not honor RSTs from you.
And, of course, since they're your ISP, they can just stop delivering your traffic. I'd suggest letting everyone you possibly can know about this, hopefully get it into the papers
Re: (Score:2)
Assuming that Comcast is injecting a RST with a valid sequence number (next in an open connection), it would be impossible* to distinguish between a generated RST and a real one. If they are indeed resetting your connection, and your kernel's tcp stack is not written by a 3-year-old, then they are most certainly using a valid sequence number.
Ignoring all RSTs would eventually fill your TCP stack with open connections and cause your kernel to barf. I *think* that RST is pretty uncommon on a decent network,
Go even further and ignore fake RST? (Score:5, Interesting)
Re: (Score:2)
Just because I can... (Score:2)
iptables -A log_and_drop -j LOG
iptables -A log_and_drop -j DROP
iptables -I INPUT -j log_and_drop -p tcp -m tcp --tcp-flags RST RST -m state --state NEW,INVALID
I'm not sure that INVALID is the same, though.
But I am saying that iptables rules, even though they're essentially a pile of GOTOs, should still at least strive for DRY -- don't repeat yourself. I don't know if it's actually more or less efficient, but it's sure a lot more maintainable. For example, if you wanted to try his fir
Sadly, NO (Score:2)
Re:iptables fake RST detector (Score:5, Insightful)
If Comcast truly is using Sandvine boxes, then this could be a network controller station with the preset examples still in place. The Sandvine sales presentation shows how to load up the system with all the prefixes from AS36561, and then interfere with a tiny percentage of TCP traffic after the first few hundred packets are transferred. What this does is provide a way of denying they are completely blocking those packets, but will blow away any connection hoping to do streaming video or cruise around on a web page heavy in graphic content like a mapping function.
The business model after installing Sandvine boxes is to then extort regular payments from large content providers to allow access to their network. Comcast, SBC/ATT and a few other monopolistic ISPs would like to see both sides of a connection pay for traffic in both directions, not the current economic model where each side pays for their own access or transit.
What Sandvine boxes do is break the end-to-end model of the internet. Even a tiny percentage of broken connections will put an end to all the cool applications everyone is currently enjoying. Streaming video and audio sessions, VoIP calls, file downloads, p2p exchanges, search engines, mapping and geolocation, and heavy web content sessions like social networking sites. The only traffic that can survive this kind of interference are from applications that make repeated attempts at connection in case of unexpected interruptions, like SMTP.
P2P protocol designers are pretty agile and clever. In the face of regular faked TCP RST bits on a connection, they'll evolve the protocol to make shorter connections, and to make repeated attempts to reconnect when an unexpected RST is received. Expect tuning "knobs" in clients very soon now, on how resilient to make the connections or how many bytes to transfer before tearing down and rebuilding the connection. There could also be a way to limit the numbers of attempted connections so as to fly under the radar of systems like this. I can open any bittorrent client with a single popular file, and see over 1000 completed TCP connections within 2 to 3 minutes. Limiting the number of new connections per minute could throw a spanner in Sandvine's current design.
the AC
Google could fix Comcast's ass tout suite (Score:5, Funny)
"Your ISP is interfering with the transmission of data requested from Google our users, and as a result we are unable to consistently provide advanced services to you. You will be redirected to a more basic version of Google's services so that we can provide as much as we can in the manner you have come to expect from us".
Wait 10 seconds, then redirect to Google's non-AJAX pages.
I predict hordes with torches and pitchforks (led by a little old lady with a claw hammer)
Re:Google could fix Comcast's ass tout suite (Score:5, Funny)
And links to your state's AG office...
And little adwords ads on the side for local law firms.
Google Web Accelerator Error (Score:2, Interesting)
Re: (Score:2)
Plug in another DNS server. May I suggest Verison, Open DNS, ScrubIT, or any of the other free DNS servers? I use ScrubIT as it is safe for work. As a bonus, most malware sites don't work. It keeps the AV software much quieter.
what the anti net neutrality crowd has to say (Score:2)
going on for months with google maps (Score:5, Interesting)
Getting Comcast to fix it seems unlikely.
Re: (Score:2)
I sometimes see this same problem both at work and at home. Neither use Comcast, so I suspect that the problem is on Google's side.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Comcast annoyed at Google for drop in PageRank? (Score:2)
Re: (Score:2)
Wait, they do still have customers, right?
time for IPSec? (Score:3, Interesting)
Now, whether MS would be cooperative in that, I dunno... I know XP supports it, but not too much about configuration specifics.
Re: (Score:2)
Re: (Score:2)
Establishing arbitrary IPSec connections on demand to each endpoint you want to contact would be extremely difficult.
You could, of course, tunnel to a host that was not connected to a braindead provider, but that would be extremely bandwidth inefficient - every packet you sent or received to host C when you were tunneled through host B would have to be both sent AND received by host B.
Re: (Score:2)
IPSec is primarily intended for creating a point-to-point tunnel.
No, IPSEC is primarily intended for securing internet connections, tunneling is just one way of using it.
Establishing arbitrary IPSec connections on demand to each endpoint you want to contact would be extremely difficult.
Unfornutely true for most cases. IIRC, the plan was to have DNS listings for host list the public key for the host which would then be used to initiate the secure connection. This would require secure connections to the DNS servers...
Got hit by this a few weeks ago (Score:2)
A few weeks ago I was at a house with Comcast, and none of us could reliably access Google. All other sites seemed to work. Several hours later (or perhaps the next morning) connections to Google were fine again. At the time I thought it might be a problem with Google, and that would be front page news on Slashdot, but nothing appeared, and I forgot about it.
That mystery is solved now...
Comcast shenaigans (Score:4, Interesting)
I've done bandwidth tests and my upstream STARTS at a nice 1.5MB/s and then 15 seconds later drops to 30K/s EVERY TIME.
What this does is give false results when people are doing speed tests. When you do your test you get great results (in my case 15Mb/s downstream and almost 2Mb/s upstream) for the first 15 or 20 seconds. Then after that it just BLOWS.
Re: (Score:2)
Wikipedia page (Score:5, Informative)
The way it's written now, everyone should use Sandvine - it sounds like wonderful software.
applications for testing ISPs? (Score:5, Insightful)
Don't care any more - no longer a customer (Score:2)
I got qwest up and running in 10 minutes, and i called Comcast when i got to work. I told him i was done dealing with their incompetance on cable TV (shows would start in HD, then go to SD for commercials, then never com
stab in the dark (Score:2)
Did (Score:2)
When Google calls Comcast (Score:5, Funny)
Comcast Secretary: Hello, thank you for calling Com-
Google Big Cheese: This is Google Inc. calling, I want to talk to whoever's in charge. Now.
Comcast Secretary: I don't know who you think you are but-
Google: Go visit google.com right now.
*secretary visits google.com, google recognizes the comcast head office IP range and serves up a pdf of a lawsuit document (Comcast as defendant) instead of the google homepage*
Secretary: Oh my, one moment please I'll transfer you.
Comcast Big Boss: What? I'm busy lining my socks with money and throwing darts at customer photos.
Google: This is Google Inc. You know why I'm calling.
Comcast: *stutters* y-yes, but we have the right to do whatever we need to, to ensure that our networks....
Google: Seriously?
Comcast: Seriously what?
Google: Seriously, you want to mess with us? Are you sure?
Comcast: *Long pause, and painful griding noises of "thinking"* Well... I think you overestimate how powerful you a-
Google: You have a lot to lose 'my friend'. You have 823 employees using Gmail. 138 office locations on Google Maps, 2,345 website pages indexed by the google search engine that recieve a collective 546 thousand search hits per day from Google Search. You currently rank first for the search term "cable internet" and nearly all your press releases are picked up by Google News. Do I need to go on?
Comcast: *speechless silence*
Google: That's right. And be quick about it. *snaps fingers*
--
(All numbers are made up)
Yeah, that's what I see coming...
From the guy in the second link (Score:3, Informative)
Here's the condensed version:
* Pings work fine, other websites work fine - only HTTP to google.com with a "google.com" host header is affected
* HTTP 1.0 without host header isn't affected
* Going to one of google's web servers by IP works fine (no "google.com" host header)
* I am typically seeding torrents and was at the time of each service interruption
* TCP RSTs follow a specific pattern. 2 RSTs in rapid succession in response to the initial GET statement (1 with a valid SEQ, one with a SEQ in the 12xxx range), followed by a second batch of the same. As the article here states (and as I posted in the linked thread), this matches perfectly with results from the China firewall
* The problem went away at almost exactly 12:00am EDT this morning (give or take a minute)
* This is from a Comcast subscriber in Grand Rapids, MI.
For more detail, visit the thread linked. I have links to the raw packet capture data in
Comcast & DNS (Score:3, Informative)
Using tcpdump showed that all the bad dns queries stopped after 4 frames, while the successful ones went 68 or 70 frames.
Switching from Comcast's regional DNS servers to their national DNS servers fixed the problem immediately.
Makes me wonder what they're doing on the regional ones.
Re: (Score:2)
My next questions would be: How bad is the disruption and how many users in what regions are affected?
Re: (Score:3, Interesting)
Re: (Score:2, Offtopic)
Re:Oh me oh my! (Score:4, Insightful)
Re:First hand experience here (Score:5, Informative)
Did you actually flush your DNS caches like, say, the one in your router, the one in your linksys box, the one on your PC? You can do it manually but the quickest way for a lot of equipment is to reboot. Hence the suggestion.
Additionally, it was quite likely google because something on your machine (maybe yourself "trying" the connection) had accessed google while the DNS redirection was in place (that was how they "redirected" you to their page). Once you'd done it once it'd linger until the TTL's had expired all the way back to your computer. Ping, nslookup, etc. would ALL show the Comcast IP until that happened, which could be minutes, hours, days, months, depending on your setup.
In your case, it looks like it was less than 24-hours, because it worked the next day without having to reboot. If you had rebooted immediately, it would have all worked when it came back up. That's WHY he was telling you that.
Before you start throwing accusations around, delve into such things just a little bit deeper.
Re: (Score:2)
Re: (Score:2)
Did you try using a non Comcast DNS server? Try using 4.2.2.1 (Verison) or another free server other than Comcast next time that happens. Delete the default settings in your router and plug them in. Reboot the computers to get new DNS info from the router and check it.
Re: (Score:2)
whois -h whois.cymru.com 4.2.2.1
AS | IP | AS Name
3356 | 4.2.2.1 | LEVEL3 Level 3 Communications
Verizon uses Level3's services.
Re: (Score:3, Informative)
I believe that 4.2.2.1 - 4.2.2.5 (or maybe 6) are all DNS servers for Level3, in case you want multiples available.
Not comcast (Score:3, Informative)
Your OWN COMPUTER was redirecting you to Comcast (maybe you should be indignant towards Microsoft? >_>). It's called DNS caching.
In Windows a simple ipconfig /flushdns can take care of that, although some applications, such as Firefox, keep their own DNS caches which must also be cleared (In Firefox there's a DNS cache timeout in about:config somewhere, you just set it to 0 and then back and that should flush the cache).
Also the tech was almost right... restarting your computer WOULD have fixed i
Re: (Score:3, Informative)
Re: (Score:2)
The post date is in the lower right corner (lower left for SA), and all of them linked in the story are from the past week or two.
Re: (Score:2)