How Microsoft Degrades Their Users (In a Good Cause) 174
blackbearnh writes "We all know that slow Web pages drive users crazy, but where is the boundary between too slow and too simple? As Microsoft's Eric Schurman points out, the fastest-loading page of all is a blank one, but it's also the most useless. In an interview with O'Reilly Radar leading up to his appearance at the Velocity Conference, Schurman talks about his experiences working on some of Microsoft's highest-volume sites, including the home page and Live Search. In particular, he discusses how Microsoft will selectively degrade the performance of pages to small sets of users so that they can see how various amounts of delay at different times and places affect user behavior. 'In cases where we were giving what was a significantly degraded experience, the data moved to significance extremely quickly. We were able to tell when we delayed people's pages by more than half a second, and it was very obvious that this had a significant impact on users very quickly. We were able to turn off that experiment. The reasoning... was it helps us make a strong argument for how we can prioritize work on performance against work on other aspects of the site.' He also talks about what it's like to be one of the most often-targeted DDoS sites on the planet."
Agile and all that (Score:3, Insightful)
What is cool about the Web is that it is the most Agile of all release environments. Unlike shrinkwrap software, web software can be changed very easily and universally for all users. It brings a raw edge to the development of the software.
In this, there is also the possibility of becoming complacent and ill-tuned to the needs of your users. Taking Google as an example, they keep their services in a perpetual state of beta, always in testing, never reaching a final v1. This type of reliance on constant feedback from customers may work for a short while, but unless the product reaches a state of relative stability (in terms of both not crashing and also not changing) the users will typically find some other software to use.
So when Microsoft decides to impact a few customers with degraded QoS, they may be setting themselves up for a bigger fall later. By introducing the possibility that MS may actively sabotage your user experience in the name of experimentation and testing, they degrade their own reputation (as much as it can be degraded from its current levels) and needlessly increase FUD regarding their proffered services.
It may be for a good cause, but customers should not be the ones testing Microsoft's software. As a professional software house, they should provide good quality control before software hits the servers. It doesn't matter if this is the age of Agile or not.
Re:Agile and all that (Score:5, Insightful)
In this, there is also the possibility of becoming complacent and ill-tuned to the needs of your users. Taking Google as an example, they keep their services in a perpetual state of beta, always in testing, never reaching a final v1. This type of reliance on constant feedback from customers may work for a short while, but unless the product reaches a state of relative stability (in terms of both not crashing and also not changing) the users will typically find some other software to use.
You just disproved your own point.
Re:Agile and all that (Score:5, Insightful)
Yep. Half a second delay is the end of the world. Age of Linux as SaaS...or whatever buzzwords are in play here.
I'm up for a good Microsoft rant any time, any place, but if a small batch of users have to take a performance hit to improve the experience in the end for all users, isn't that a positive thing? You can't really beta test this stuff. You can try running simulations, but nothing beats real world numbers.
Would we view this any different if Apple tried it? Google?
(Disclaimer: This is a logic exercise. In reality, I doubt there's actually much MS could and would do to their site to improve my experience using it.)
Re:As opposed to ... (Score:5, Insightful)
Re:As opposed to ... (Score:5, Insightful)
You're discussing the wrong topic. "graceful degradation" != "selective worsening".
In Gmail, the degradation is helping you cope with a poor connection. In Microsoft's experiment, the degradation is hurting you and is imposed by a coin flip.
Re:Agile and all that (Score:5, Insightful)
Yeah! I mean, take IE6 for example. That didn't change in a REALLY long time, and lots of people use it! That makes it good, right? [/sarcasm]
Your statement neglects quality. Yes, people want sites that're stable and don't crash, and yes, changing the design every week is bad and confusing, but improving on the design and function of a site is always a good thing, so long as you do so at intervals large enough for users to adjust to. The design of Gmail has only changed drastically two or three times in it's history design-wise, but they still consider it a Beta (depending on what you consider a drastic change, of course).
The issue is that Google, once simply a search engine, is now in the Web Services industry. The fact is, no matter what the label says, Gmail and many of their other apps are not in Beta, and haven't been in a long time. They're just hesitant to call it "v1" or something because that has a sense of finality, like customers shouldn't expect it to change very often. With the web, and Web Apps in particular, that's no longer really the case. They are often redesigned and redone to improve their performance, effectiveness, ease-of-use, and even aesthetics. You even point out yourself how agile the web is as an environment for releasing software. You neglect, though, that this keeps it interesting for the users as well, because they like the feeling that their product is continually being improved at no extra cost to them (unlike many shrink-wrapped software) (Note: When I say "extra cost," I mean in addition to any subscriptions they already have to the service, if any).
The "Beta" in Google's case is very much a marketing issue as much as it is a technical issue.
Move along, nothing to see here (Score:3, Insightful)
Passing on the low-hanging fruit, it seems to me that this is pretty much exactly the kind of thing that happens all the time at the packet layer. WRED, for example, selectively drops packets even when buffers aren't full and the network is still theoretically operating under capacity so that the next TCP connection doesn't bring down the uplink. How is the Microsoft strategy qualitatively different?
Re:Agile and all that (Score:5, Insightful)
I always thought that was Spock [youtube.com].
Re:Punch your customers in the face, selectively (Score:5, Insightful)
It sounds absurd because what you're saying is absurd.
If you ran an experiment where some customers had their orders delayed by a few minutes more than was necessary and had some kind of metric to determine their enjoyment of their dining experience, it wouldn't be so absurd. Perhaps you provide free internet access in your store, and the extra delay results in a greater chance of people making use of it. And once they've started using it, there's a greater chance they'll decide to order a coffee after their meal and stick around for a bit longer.
Or maybe you find they're less likely to return to the store. That might be hard to track, but the point stands. There are some things which are interesting and which may or may not give unexpected results when tried in real life. If an experiment like this shows that a few minutes delay significantly upsets customers, then it becomes clear that spending extra money to have more staff on is probably actually worth the expense. On the other hand if you can show that most people don't notice, then it makes sense to risk having a shortage of staff at peak periods if you can save a bit of money.
You might even find unexpected results, for example maybe a lot of people after waiting a few minutes with nothing to look at but the menu end up ordering more than they initially would, so it's actually profitable to make people wait longer. Who knows? The only way to find out is to experiment.
Re:Select groups of users (Score:2, Insightful)
They didn't do the experiment to find that out. They're only talking about what it's like (you even quoted it).
Re:As opposed to ... (Score:5, Insightful)
That would probably completely invalidate the results though, for two reasons. First, the sorts of people who would opt into that wouldn't at all be representative. (It would take an unusual person to even find the opt-in, let alone volunteer for a degraded experience knowingly.) Second, knowing about it would be way too likely to affect how the people behaved.
You could get halfway by saying "would you like to help us do research" or something like that, without saying in what way, which would reduce these problems, but not completely.
Blank page? (Score:1, Insightful)
In IE (I need IE for some corporate intranet and extranet apps), I have set my home page to 'about:blank' and even then it takes some time to load. I have no idea why IE insists on showing me the message 'Connecting...' while loading my blank homepage. Shouldn't it come up instantly?
Re:Punch your customers in the face, selectively (Score:3, Insightful)
"You may be modded funny, but to me, that's only because there isn't a "douchebag" mod"
I think the funny mod can actually be used defensively... when somebody says something, and they might be completely serious, but it's blatently obvious that it shouldn't be taken seriously, a +1 funny mod can help to set the context of which it is read :-)
Re:As opposed to ... (Score:5, Insightful)
I think this would only actually be a problem if anyone used LIVE search.
Re:As opposed to ... (Score:5, Insightful)
Re:As opposed to ... (Score:5, Insightful)
Modded as Funny (100%), but you've just laid out the environmentalist rationale for controlling everything everyone does, everywhere.
Re:Punch your customers in the face, selectively (Score:3, Insightful)