Forgot your password?
typodupeerror
The Internet Networking

Crowdsourcing Confirms: Websites Inaccessible on Comcast 349

Posted by timothy
from the have-your-friends-drop-the-dime-on-your-non-friends dept.
Bennett Haselton writes with a bit of online detective work done with a little help from some (internet-distributed) friends: "A website that was temporarily inaccessible on my Comcast Internet connection (but accessible to my friends on other providers) led me to investigate further. Using a perl script, I found a sampling of websites that were inaccessible on Comcast (hostnames not resolving on DNS) but were working on other networks. Then I used Amazon Mechanical Turk to pay volunteers 25 cents apiece to check if they could access the website, and confirmed that (most) Comcast users were blocked from accessing it while users on other providers were not. The number of individual websites similarly inaccessible on Comcast could potentially be in the millions." Read on for the details.

My first clue came when a friend of mine set up the website http://www.helpmatt.org/ and asked her friends to donate. I said the website appeared to be down; they replied back that it was working fine for other people — and I narrowed it down to Comcast DNS servers not resolving the hostname www.helpmatt.org correctly. When I accessed the same website over my Frontier DSL connection, it worked. (I had recently signed up for Comcast cable Internet to save money over DSL, but I kept my DSL connection "just in case" something went wrong. At the time, I thought maybe I was being paranoid -- how hard could it be for a cable company to just run a straight Internet connection to my house and not screw anything up? Hollow laugh.)

I put out an informal survey to my Comcast-using friends, and a few of them said they couldn't access the website either. Still, I thought, this wasn't enough evidence that it was Comcast's fault; maybe the hostname was only resolving intermittently, and just by sheer coincidence it happened to be up when all of my non-Comcast-using friends tried it? I was about to do a more formal experiment, and recruit a larger sample of testers through Amazon Mechanical Turk to test whether the site was inaccessible to other Comcast users, when the problem spontaneously fixed itself and suddenly the website became accessible 100% of the time to everyone.

But, my curiosity had been piqued. Was there something wrong with Comcast's DNS servers -- whether deliberate or not -- that was causing other websites not to resolve correctly? I wrote a perl script to take a sample of websites -- part of the same list that I had used to find websites that were mis-blocked as 'pornography' by Smartfilter — and attempt to resolve them using both Comcast's main DNS server (75.75.75.75) and one of Google's public DNS servers (8.8.8.8). (You won't be able to do this experiment yourself unless you have a Comcast Internet connection, because while Google's DNS servers accept queries from anywhere, Comcast's DNS servers will refuse queries from any IP address not assigned to one of their customers.)

The script ran through a few hundred hostnames and flagged anything that failed to resolve on Comcast but resolved correctly on Google, although most of these were false positives caused by Comcast's DNS servers being temporarily unresponsive. But after running through the list of false-positives repeatedly, I found the first website that consistently failed to resolve on my Comcast Internet connection while resolving on Google: http://www.021yy.org/.

The website is for a second-hand furniture store in Shanghai; I have no idea what the domain "021yy.org" has to do with the business. (Perhaps the IP address that the domain name resolves to used to be occupied by a different website, and that IP address was inherited by the furniture store but the old hostname still points to it.) The hostname www.021yy.org resolves to the IP address 116.251.210.33 (for *ahem* non-Comcast users, that is), which according to the Asia Pacific Network Information Centre is part of a block of IP addresses assigned to a hosting company in Singapore. I'm not blocked from accessing the IP address of the website over Comcast; I can ping and send web requests to the IP address 116.251.210.33 with no problem. Only the hostname fails to resolve. (I can still access the site by using a VPN or a proxy server.)

So, I created a survey on Amazon Mechanical Turk, asking people three questions:

  1. Can you access the website http://www.021yy.org/?
  2. If you can't access the site, what error message does your browser give you?
  3. What provider are you using?

and offered 25 cents to every user who filled out the survey, up to a maximum of 50 people. Amazon Mechanical Turk, if you've never used it before, lets you create low-payment tasks and outsource them to a crowd of workers. Like any simple and powerful tool, it can be used for purposes that the original creators probably never imagined (presumably including this experiment), and someday I'd like to look into the most creative and bizarre things people have done with it. (Although, in this case, it seems like the site may not have done a great job of matching this task with available workers. Only 20 people filled out my survey in the 24 hours after I created it -- surely, out of all the available Mechanical Turk workers, there were more than 20 people who would have been interested in doing a simple website accessiblity check for 25 cents?)

20 unique users filled out the survey and reported:

  • Out of the 14 non-Comcast users, 100% of them were able to access the site.
  • Out of 6 Comcast users, 4 of them were blocked from accessing the site, and reported errors symptomatic of DNS failures ("Oops! Google Chrome could not find www.021yy.org" or "Server not found. Firefox can't find the server at www.021yy.org").

Even with such a small sample, that's enough to conclude that it's not a coincidence. (The real question is how two out of those six Comcast users were able to access the site at all. Maybe they're in a region of the country that's assigned different DNS servers. If I did the survey again, I'd ask people to include where they were living.)

So Comcast users -- at least some of them, probably most of them -- are blocked from accessing certain websites, which are perfectly accessible to users on other providers. I "only" had to test a few hundred domain names before finding one that would consistently fail to resolve on Comcast while resolving successfully on other companies' nameservers. With hundreds of millions of distinct websites "out there," if the same proportion holds, that would suggest that there about a million or more websites similarly affected. And that's not even counting all the other sites — like helpmatt.org, and also including some of the sites in my sample — which apparently resolve 100% of the time on other providers while sometimes failing to resolve on Comcast, but where the failure was not consistent enough to use them as a test case for the Mechanical Turk survey.

Unlike, say, the kerfuffle over Comcast threatening to de-prioritize content delivery from websites that don't pay them a fee, it's unlikely that Comcast is meddling with traffic intentionally here (especially since the sites' IP addresses are not blocked). It's more of a demonstration that if a company is sufficiently big and if it's sufficiently hard to prove that a problem is being caused on their end, the problem can exist for a long time without being solved. I called Comcast tech support after I discovered that sites were blocked on their network but not on other providers, and said that the problem really needed to be brought to the attention of the higher-ups, but tech support was adamant that it was impossible for a member of the public to reach anybody higher up than the call center.

Even if the number of affected sites is huge, at least it's only a small percentage of websites — I did have to run my script on a few hundred sites before I found one that appeared to be resolving on other DNS servers but not on Comcast. But that likely would have provided scant comfort to my friends who set up the helpmatt.org site, when they were urging people to visit the site and donate, and 25% of potential visitors were unable to reach the page. When it's your website, it's kind of a big deal.

This discussion has been archived. No new comments can be posted.

Crowdsourcing Confirms: Websites Inaccessible on Comcast

Comments Filter:
  • by krkhan (1071096) on Tuesday March 11, 2014 @02:44PM (#46456519) Homepage

    With hundreds of millions of distinct websites "out there," if the same proportion holds, that would suggest that there about a million or more websites similarly affected.

    Why are you assuming that this scales linearly? Are you suggesting that this is a technical glitch? If the websites are blocked due to the nature of their content it most certainly won't scale in a linear fashion.

  • by EvanED (569694) <evaned.gmail@com> on Tuesday March 11, 2014 @02:58PM (#46456657)

    OpenDNS hijackes NXDOMAIN failures, which is one of the big reasons to drop many ISP's DNS in the first place. I don't want to get into evaluation of motivation and such, but the effect is the same.

  • So... (Score:5, Interesting)

    by squiggleslash (241428) on Tuesday March 11, 2014 @02:59PM (#46456677) Homepage Journal

    Let me understand this correctly. You found Comcast's DNS isn't perfect and doesn't resolve some names. It does not appear to be malicious in any way, as the two domains you find affected are a foreign furniture store, and your friend's brand new website. It's fairly obviously a bug.

    So: you call Comcast Tech support, demand to talk to the Boss of Comcast, and then write a 10,000 word article (I didn't count) about it on Slashdot where you know 90% of the readers will take "Websites inaccessible on Comcast" as meaning "OUT OF CONTROL MEGACORP MONOPOLIST COMCAST IS CENSORING WEBSITES!!!"

    This makes sense to you? This is what you do? Really? Really?

    Just curious, but that time you got a duff cable modem and had to send it back, did you write a 60,000 article on how Comcast has banned you from the Internet, and did you demand to speak to the PRESIDENT OF THE INTERNET? When it rained that one time and you attempted to tune in the cable TV, only to find many of your channels were inaccessible, did you write a 75,000 word article on how COMCAST IS DROPPING CHANNELS and did you call tech support demanding to talk to THE LORD HIGH RULER OF TV?

    I think I've found an article where the discussion would be likely improved for once if the Betoddlers spammed it with anti-Beta comments.

  • Re:Stop (Score:4, Interesting)

    by jythie (914043) on Tuesday March 11, 2014 @03:16PM (#46456835)
    Comcast bought up hundreds if not thousands of smaller local ISPs and cobbled their networks together. so hardware policies are highly dependent on where you are and what the history of the local connection is. Even if it is over broadband that Comcast laid down, the back end could be any number of fragments of previous companies.
  • by FuegoFuerte (247200) on Tuesday March 11, 2014 @03:22PM (#46456907)

    Actually, there are a few major GTM (Global Traffic Management) schemes that do use the IP address of your DNS server, rather than your actual IP. They basically abuse the DNS system with super-short TTLs and give a different response to the DNS query based on the IP of the downstream DNS server. So, if you use a DNS server located on the east coast of the US when you're on the west coast, you'll get an east coast server even if that service has a west coast datacenter available.

    This is done primarily to free companies from the burden of having to design proper geolocation into their app/service, turning it into a more plug-n-play solution while breaking several of the finer points of DNS (like proper caching). This type of traffic management could easily be contributing to Comcast's DNS troubles, as it drastically increases load on the entire DNS infrastructure. Paul Vixie did a good detailed write-up about this type of traffic management a few years back. Unfortunately it's probably here to stay, and is used by some very major corporations and online services.

    If you want the most reliable DNS service, and want to be directed to the closest servers for the services you use, your only real option is to run your own recursive name server. A simple caching name server isn't enough, and will curse you with many of the same problems you see from your upstream. Fortunately, recursive name servers are pretty simple to set up, in both the *nix and Windows worlds.

  • by Geoffrey.landis (926948) on Tuesday March 11, 2014 @03:34PM (#46457039) Homepage

    You can use downforeveryoneorjustme.com, though it will use its own DNS and routing so it will still require you to figure out which of those is the problem.

    Say, that's a nice site. Wish I had mod points, I'd moderate you "informative".

  • There is one potential issue. I only found it when I was using a smaller regional ISP while I dealt with a billing dispute with Charter. If your ISP uses extreme levels of NAT and is used primarily by tech-savvy people (those who would be likely to use Google DNS in the first place). It may look to Google like a single IP address is hammering their DNS servers with queries and they may block that particular public IP address. I got that one explained to me by the president of that small ISP about a year ago when I asked why my DNS queries weren't going through and ended up being escalated to the top.
  • Re:Stop (Score:4, Interesting)

    by capedgirardeau (531367) on Tuesday March 11, 2014 @04:01PM (#46457317)

    OpenDNS has the terrible policy of turning back the error:

    "This website is not responding"

    When in fact it was a DNS lookup failure.

    I have written them repeatedly and filed a bug report, but they seem to think it is an acceptable response.

As far as we know, our computer has never had an undetected error. -- Weisert

Working...