Crowdsourcing Confirms: Websites Inaccessible on Comcast 349
My first clue came when a friend of mine set up the website http://www.helpmatt.org/ and asked her friends to donate. I said the website appeared to be down; they replied back that it was working fine for other people — and I narrowed it down to Comcast DNS servers not resolving the hostname www.helpmatt.org correctly. When I accessed the same website over my Frontier DSL connection, it worked. (I had recently signed up for Comcast cable Internet to save money over DSL, but I kept my DSL connection "just in case" something went wrong. At the time, I thought maybe I was being paranoid -- how hard could it be for a cable company to just run a straight Internet connection to my house and not screw anything up? Hollow laugh.)
I put out an informal survey to my Comcast-using friends, and a few of them said they couldn't access the website either. Still, I thought, this wasn't enough evidence that it was Comcast's fault; maybe the hostname was only resolving intermittently, and just by sheer coincidence it happened to be up when all of my non-Comcast-using friends tried it? I was about to do a more formal experiment, and recruit a larger sample of testers through Amazon Mechanical Turk to test whether the site was inaccessible to other Comcast users, when the problem spontaneously fixed itself and suddenly the website became accessible 100% of the time to everyone.
But, my curiosity had been piqued. Was there something wrong with Comcast's DNS servers -- whether deliberate or not -- that was causing other websites not to resolve correctly? I wrote a perl script to take a sample of websites -- part of the same list that I had used to find websites that were mis-blocked as 'pornography' by Smartfilter — and attempt to resolve them using both Comcast's main DNS server (75.75.75.75) and one of Google's public DNS servers (8.8.8.8). (You won't be able to do this experiment yourself unless you have a Comcast Internet connection, because while Google's DNS servers accept queries from anywhere, Comcast's DNS servers will refuse queries from any IP address not assigned to one of their customers.)
The script ran through a few hundred hostnames and flagged anything that failed to resolve on Comcast but resolved correctly on Google, although most of these were false positives caused by Comcast's DNS servers being temporarily unresponsive. But after running through the list of false-positives repeatedly, I found the first website that consistently failed to resolve on my Comcast Internet connection while resolving on Google: http://www.021yy.org/.
The website is for a second-hand furniture store in Shanghai; I have no idea what the domain "021yy.org" has to do with the business. (Perhaps the IP address that the domain name resolves to used to be occupied by a different website, and that IP address was inherited by the furniture store but the old hostname still points to it.) The hostname www.021yy.org resolves to the IP address 116.251.210.33 (for *ahem* non-Comcast users, that is), which according to the Asia Pacific Network Information Centre is part of a block of IP addresses assigned to a hosting company in Singapore. I'm not blocked from accessing the IP address of the website over Comcast; I can ping and send web requests to the IP address 116.251.210.33 with no problem. Only the hostname fails to resolve. (I can still access the site by using a VPN or a proxy server.)
So, I created a survey on Amazon Mechanical Turk, asking people three questions:
- Can you access the website http://www.021yy.org/?
- If you can't access the site, what error message does your browser give you?
- What provider are you using?
and offered 25 cents to every user who filled out the survey, up to a maximum of 50 people. Amazon Mechanical Turk, if you've never used it before, lets you create low-payment tasks and outsource them to a crowd of workers. Like any simple and powerful tool, it can be used for purposes that the original creators probably never imagined (presumably including this experiment), and someday I'd like to look into the most creative and bizarre things people have done with it. (Although, in this case, it seems like the site may not have done a great job of matching this task with available workers. Only 20 people filled out my survey in the 24 hours after I created it -- surely, out of all the available Mechanical Turk workers, there were more than 20 people who would have been interested in doing a simple website accessiblity check for 25 cents?)
20 unique users filled out the survey and reported:
- Out of the 14 non-Comcast users, 100% of them were able to access the site.
- Out of 6 Comcast users, 4 of them were blocked from accessing the site, and reported errors symptomatic of DNS failures ("Oops! Google Chrome could not find www.021yy.org" or "Server not found. Firefox can't find the server at www.021yy.org").
Even with such a small sample, that's enough to conclude that it's not a coincidence. (The real question is how two out of those six Comcast users were able to access the site at all. Maybe they're in a region of the country that's assigned different DNS servers. If I did the survey again, I'd ask people to include where they were living.)
So Comcast users -- at least some of them, probably most of them -- are blocked from accessing certain websites, which are perfectly accessible to users on other providers. I "only" had to test a few hundred domain names before finding one that would consistently fail to resolve on Comcast while resolving successfully on other companies' nameservers. With hundreds of millions of distinct websites "out there," if the same proportion holds, that would suggest that there about a million or more websites similarly affected. And that's not even counting all the other sites — like helpmatt.org, and also including some of the sites in my sample — which apparently resolve 100% of the time on other providers while sometimes failing to resolve on Comcast, but where the failure was not consistent enough to use them as a test case for the Mechanical Turk survey.
Unlike, say, the kerfuffle over Comcast threatening to de-prioritize content delivery from websites that don't pay them a fee, it's unlikely that Comcast is meddling with traffic intentionally here (especially since the sites' IP addresses are not blocked). It's more of a demonstration that if a company is sufficiently big and if it's sufficiently hard to prove that a problem is being caused on their end, the problem can exist for a long time without being solved. I called Comcast tech support after I discovered that sites were blocked on their network but not on other providers, and said that the problem really needed to be brought to the attention of the higher-ups, but tech support was adamant that it was impossible for a member of the public to reach anybody higher up than the call center.
Even if the number of affected sites is huge, at least it's only a small percentage of websites — I did have to run my script on a few hundred sites before I found one that appeared to be resolving on other DNS servers but not on Comcast. But that likely would have provided scant comfort to my friends who set up the helpmatt.org site, when they were urging people to visit the site and donate, and 25% of potential visitors were unable to reach the page. When it's your website, it's kind of a big deal.
Which is why I use OpenDNS, or Google, or (Score:5, Informative)
I stopped using comcast DNS servers years ago, and have avoided many an "outage".
I remember several large DNS outages on comcast that I was completely unaware of for hours or days, until some mention came up.
I have been using OpenDNS mostly, but I fall back to the google DNS servers if something there flubs up
208.67.222.222
208.67.220.220
Remember these numbers
Re:Fairly simple solution (Score:4, Informative)
You can set any DNS you want on your computer. You don't have to use the one handed out by the ISP's modem or router.
Re:Stop (Score:4, Informative)
Eh? I have Comcast and use my own cable modem and router. Whatchu talking 'bout, Willis?
Re:Stop (Score:5, Informative)
Re:Stop (Score:3, Informative)
Re:Fairly simple solution (Score:4, Informative)
I don't know if this is an issues with Comcast, but there are ISPs who force all DNS traffic to use their servers. It was a constant frustration when I was stuck with Excede (a US satellite internet provider).
Re:Quick change needed [Re:Stop] (Score:5, Informative)
You can use downforeveryoneorjustme.com, though it will use its own DNS and routing so it will still require you to figure out which of those is the problem.
Re:Stop (Score:5, Informative)
I wish kids with no experience would stop running their mouths. That is BS, and even you would understand it if you would think about it. On many of their routers, Comcast redirects port 53 to 75.75.75.75. It doesn't matter what DNS server you set the clients to because Comcast will transparently proxy to their server. As an example with our new IP block from Comcast that isn't yet setup on their DNS server to allow access:
$ nslookup aol.com 75.75.75.75
Server: 75.75.75.75
Address: 75.75.75.75#53
** server can't find aol.com: REFUSED
$ nslookup aol.com 8.8.8.8
Server: 8.8.8.8
Address: 8.8.8.8#53
** server can't find aol.com: REFUSED
$ nslookup aol.com 208.67.222.222
Server: 208.67.222.222
Address: 208.67.222.222#53
** server can't find aol.com: REFUSED
That shows they're intercepting traffic to both OpenDNS and Google's DNS. We're currently using a modem owned by Comcast, but last week when I swapped in an older modem for testing, I could use DNS on both OpenDNS and Google.
Possible botnet C&C related (Score:4, Informative)
The DNS for 021yy.org is rather fishy looking. The .org servers have NS records pointing to ns1.booen.com and ns2.booen.com, which have a 20 second time to live (vs. a normal 1 day TTL), which is common in botnet command & control networks. Also, the ns1/2.booen.com servers give answers to 021yy.org A lookups, but return NXDOMAIN for NS lookups (which is completely bogus; NXDOMAIN means that 021yy.org does not exist, not that it doesn't have NS records, which would still be bogus).
The NXDOMAIN for NS records would cause many caching servers to cache NXDOMAIN for all records (not just NS), which would cause the domain to not resolve (depending on the order things were looked up). Basically, I don't see this as a Comcast problem, but rather a problem with the DNS servers for 021yy.org. This may be accidental (although AFAIK no normal DNS server would reply with A records but return NXDOMAIN for NS records), but looks possibly like it is intentional and possibly part of a botnet C&C. There's a lot of that going on lately.
Re:Doctor that hurts (Score:4, Informative)
don't use the fast ISP? like you have a CHOICE??
I can pick dsl (dog slow link; that's what DSL means) or I can pick comcast.
what makes you think people in the US can actually choose an isp? they are all based on where you live. you'd have to MOVE to be able to choose an alternate.
not sure why you posted this BS but its not helpful in the least...
Re:Stop (Score:4, Informative)
No it will try them in the order listed until it gets a 'response'; I think if it gets a response like SRVFAIL it will also continue trying the remaining servers, but if gets a incorrect NXDOMAIN it will trust that value and not try the remaining servers.
Re:Old DNS cache? (Score:5, Informative)
DNS deals with this issue using TTL (time to live) for the records it hands out. The Authoritative DNS server for the domain gives out the TTL it wants for every query it receives. Other non-authoritative DNS servers are supposed to throw away any record they cache once it reaches it TTL Now if you have TTL's measured in days, you lower the load on your DNS server, but any IP changes can take a long time to propagate. The trade off is that lowering the TTL increases the load on the authoritative server. So, there are going to be differences in resolved domains that will resolve themselves over time.
However, that's not what the author is complaining about. He's getting no resolution for his request, meaning that the DNS server he queried was unable to retrieve the record from cache, nor find a DNS record for the domain when making a query upstream. My guess is that Comcast's DNS infrastructure is just overloaded so when trying to obtain information about more obscure domains like this it fails now and then. Such failures get cached for awhile so they hand out no matches to others as well. If enough folks start requesting the domain, it eventually will get cached properly and start to resolve. Of course, another possible option is that the domain got black holed by Comcast's DNS for being involved in a phishing expedition or other bad thing too, but it's hard to know.
Observations from Comcast's DNS Team (Score:5, Informative)
Re:Stop (Score:4, Informative)
It's OpenDNS's fault. They return a bogus A record instead of NXDOMAIN:
$ dig +noall +comments +answer test.example.com @8.8.8.8
-- Got answer:
-- -HEADER- opcode: QUERY, status: NXDOMAIN, id: 48729
-- flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
$ dig +noall +comments +answer test.example.com @208.67.222.222
-- Got answer:
-- -HEADER- opcode: QUERY, status: NOERROR, id: 31301
-- flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
-- ANSWER SECTION:
test.example.com. 0 IN A 67.215.65.132