How Healthcare.gov Changed the Software Testing Conversation 118
An anonymous reader notes an article about how the tribulations of Healthcare.gov brought the idea of software testing into the public consciousness in a more detailed way than ever before. Quoting:
"Suddenly, Americans are sitting at their kitchen tables – in suburbs, in cities, on farms – and talking about quality issues with a website. The average American was given nightly tutorials on load testing and performance bottlenecks when the site first launched, then crumbled moments later. We talked about whether the requirements were well-defined and the project schedule reasonably laid out. We talked about who owns the decision to launch and whether they were keeping appropriate track of milestones and iterations. ... When the media went from talking about the issues in the website to the process used to build the website was when things really got interesting. This is when software testers stepped out of the cube farm behind the coffee station and into the public limelight. Who were these people – and were they incompetent or mistreated? Did the project leaders not allocate enough time for testing? Did they allocate time for testing but not time to react to the testing outcome? Did the testers run inadequate tests? Were there not enough testers? Did they not speak up about the issues? If they did, were they not forceful enough?"
Re:Were known management tools used? (Score:5, Informative)
It does not sound to me as though known management tools were used. Did they sit down with the government personnel in charge, and present their approach, and what the site would look like (menus, flow, etc) when finished? Were there testable milestones, and a final presentation of working software? It sure doesn't sound like it.
They might well have done all these things and still failed to catch the problems before the site's launch.
Performance, like security (ack! scary!) is a non-functional requirement -- that is to say it's not the kind of requirement where you can sit down with a checklist and say, 'yep, this it works,' or 'no, it doesn't.' You have to develop a more sophisticated test.
Load testing is a step in the right direction, but you also have to look at system architecture. Remember the days before people figured out that you had to load web ads asynchronously, after the page content was loaded? Sometimes the page load would be slow, not because the page's server was loaded, or because of the user's browser or internet connection were slow. Often it would be the ad server that was overwhelmed, which if you think about it is bound to be more common than the content server being overwhelmed. You could functional test and even load test the heck out of a page with synchronous ad loads, but until it went into production chances are you wouldn't catch the fatal performance flaw. That kind of problem is architectural; some of the data being delivered is coming from servers outside your control.
Ordinary tests are about ensuring reproducible results, but when the architecture leaves you vulnerable to things happening on servers and networks outside your control your problems *aren't reliably reproducible*. You have to design around *possibilities*.
Some of the problems with Healthcare.gov were of this nature, although with not so simple a solution as "use window.onload()." The site is supposed to orchestrate a complex, *distributed* process *synchronously*. You have to go out the Homeland Security's database to confirm citizenship, then to the IRS databases to confirm claims about income, then get quotes from the private insurers that meet the customer's needs. There is, in my opinion, no way to be 100% sure, or even 80% sure that a system like that will work under real work load, unless you present it with real work load.
Were I architecting such a site, I'd plan to do a lot of that work batch; that is I'd build the healthcare application offline on the user's browser, with internal consistency checks of course. Then I'd send the user's application through a batch verification system, emailing him when a result was ready. This is a clunky and old-fashioned approach, but it wouldn't force the user to chain himself to his browser . And it would have more predictable performance. Predictability is a vastly under-rated quality in a system. A system which is fast most of the time may not be as desirable as one which provides the answer consistently.
Re:Someone important *should* take the blame (Score:4, Informative)
The President doesn't write code.
No, but he presumably appoints the senior staff whose teams ultimately will write code, and capture requirements, and run tests, and all the rest. For something on this level he probably had a direct say in things like budget and timescales and the scope and high-level organisation of the project as well. If he asked for the impossible, he bears responsibility for that. If he didn't know it was impossible, he should have hired better advisors before committing to the project. And if it wasn't impossible but failed anyway, he should have appointed a senior management team that was up to the job.
(I'm not in the US and have no allegiance to any US political party, so please don't try to turn this into some sort of red vs. blue flamewar. I'm just translating the same general principles that apply to senior management for any infrastructure project into the specific context of healthcare.gov.)
Re: Someone important *should* take the blame (Score:4, Informative)
What a staggeringly ignorant post.
The house controls the purse strings and has oversight powers over the operation of the government.
Healthcare.gov has allocated some $634M and to date had expended just over half of it by the time it went live, lord knows what they spent since Oct. 1... The house withheld not one nickel from the project - it's fully funded.
As for House interference, you are simply inventing a conspiracy out of whole cloth... You really think the House 'secretly' met with those overseeing the development of the website and told them to mess it up? It was obvious to anyone looking at the situation that this was a train wreck waiting to happen - remember Republicans saying the best way to defeat Obamacare was to let it play out and fail on it's own? If there was a conspiracy as you imagine, how is it not one person has stepped forward to expose it?
The contractor selected had a history of failed projects, why is it so hard to believe they failed again?