FCC Relying On Faulty ISP Performance Data 89
alphadogg writes "The FCC recently used speed test results from comScore as an absolute indicator of specific ISPs' performance. Consulting firm NetForecast analyzed comScore's testing methodology and data to assess whether it accurately reflects broadband ISP performance, and to assess the appropriateness of using the data to reach general conclusions about the actual performance ISPs deliver to their subscribers. NetForecast uncovered problems on both counts. They found that the effective service speeds comScore reports are low by a large margin (PDF) because its data calculations under-report performance and place many subscribers in a higher performance tier than they purchased."
How Much (Score:2, Interesting)
I wonder which ISP owns comScore. Who got the worst rating?
Learn about statistics - both of you (Score:3, Interesting)
Both sides need to learn more about statistics.
The report fails to mention that across a large enough population, the results will be more-or-less correct within a certain percentage point because as he mentions, some people will test with a lot of bandwidth available at a certain point but others will test with their available bandwidth constricted. Overall, out of a large enough population the outliers are washed away.
comScore needs to realize that correlation != causation. It's not because your bandwidth correlates with other users' high-bandwidth plans, that it is caused by you actually buying the plan. But even then, even in the report the statistics show that it evens out pretty good with only a small percentage error.
Off course this brief report reeks more like paid research. Off course comScore measures the users' experience connecting to large-bandwidth centers like Akamai which has a lot of large sites on it and it doesn't accurately measures what the provider offers in the last mile. I don't care that I actually get my 10Mbps connecting to my neighborhood (unless a bunch of my neighbors actually host the Linux-ISO torrent I want) I care about getting on average getting maybe 50% of what I pay for which I usually don't get (I get closer to 1-10% depending on what I'm doing). comScore accurately reflects the poor status of broadband in this metropolitan area - dual-ISDN speeds (early 90's) on the best high-tier packages money can buy in this area. The only alternative is DSL which is horribly outdated.
Re:Wait for ACK? (Score:4, Interesting)
TCP/IP doesn't wait for the ACK. It keeps sending until the Window is full, or the ACK is received. If the Window fills it will wait until the ACK is received (or timeout and retry, etc).
If the test is trying to automatically place the users in specific Tiers then there could be a problem, however the rest of the issues are mostly a red herring. I use Speedtest.net and can readily attest to it's general accuracy, and I seriously doubt any other services are all that different.
by the way, I'm not in the U.S., I actually get what I pay for.
Re:Wait for ACK? (Score:3, Interesting)
I pay for 15 Mbit down and .5 Mbit up. .5 Mbit up.
I get 15-20 Mbit down and
I even tested it with the FCC test since I posted the first message. It rated my connection slightly faster than speedtest.net did. (Not significantly, and I'm sure it would vary).
Until recently I paid for 20 Mbit / 1.0 Mbit; but I wanted to save some money.
The general illusion in the U.S. is many markets are the numbers for a zip-code are good, or even fantastic, but only a tiny fraction of the zip-code may actually get any service at all. Other zip-codes where they actually have significant penetration often have poor, or even dismal results compared to what the consumer is promised. Due to the prior reporting requirements the FCC had the ISPs were using this difference to fudge their numbers and service levels. The truth took a back seat.
Re:FCC is faulty? (Score:3, Interesting)
Bad data == good data for a politician.
Especially when it's in favor of whatever they desire to happen. Politicians wanted healthcare so they generated a faulty "42.5 million americans uninsured" statistic. How? Using a couple mail-in postcards from voluntary recipients. Hardly scientific. (Real numbers from scientists estimate the number as 5-15 million uninsured U.S. citizens. +9 million if you include illegal non-citizens/intruders.)
And of course if the FCC stats show that ~40 million Americans don't have great than dialup speeds, that too works in politicians favor, and they'll justify it as a way to pass their favorite bill. (And also make their election funders happy.) Even on my DSL line which *never* falls below the advertised 750k, the FCC test showed only ~256k on the FCC test. Bogus.
Okay. Maybe I'm a little cynical.
Nah. I work for the government. More like - simple observation.
a few somewhat valid points, but mostly garbage. (Score:2, Interesting)
So here is the outline of their claims, with responses.
Data gathering errors
Only one TCP connection is used
Basically valid, it's a pretty rare net activity nowadays that actually maxes out the connection by itself, no idea if the promises the ISPs make contractually include any wording about per-connection performance.
Client-server delay is variable
Tough, this is a reality of how the network operates, if an ISP promises speed X, they need to invest in the infrastructure necessary to deliver speed X.
Participants’ computers may be resource constrained
Outside of listing minimum requirements for client computers, this is also a reality of how the customer will perceive network performance, and this is the important measure.
Test traffic may conflict with home traffic
semi valid-ish point, but I'm skeptical that it has a noticeable impact.
Decimal math is incorrect
This one seems like utter crap, they seem to be assuming that the testing company is saying MB and meaning MiB in one case, but that they say MB and really mean MB in another case. It's far more likely that they are saying MB and they mean MiB in both cases, in which case this point is moot.
Protocol overhead is unaccounted for
Another semi-valid point, but they claim the testers have the responsibility to make the ISPs numbers look better, why isn't it instead the ISPs responsibility to make their numbers more meaningful? IIRC, speeds are often advertised on the basis of file downloads, which means the protocol overhead should NOT be accounted for.
Data interpretation errors
Purchased speed tiers are incorrectly identified
This is probably the most significant claim, if true. However it's also the most wishy-washy of all the claims, going so far as to specifically state that it's the opinion of the company that it is even happening, rather than a factual claim:
"NetForecast estimates that it is highly likely that comScore incorrectly places many panelists' PCs into higher tiers than the subscribers purchased."
Overall, the report looks like a tiny bit of valid criticism of the testing methodology wrapped in a whole lot of weaseling about what the ISP should be expected to provide, and always siding with the ISP. The end result for me is that the validity of the entire report is fatally undermined by the obvious grasping at straws being done, and the impression that I get that if there were any errors in the opposite direction, they will not be reported.