How Healthcare.gov Changed the Software Testing Conversation 118
An anonymous reader notes an article about how the tribulations of Healthcare.gov brought the idea of software testing into the public consciousness in a more detailed way than ever before. Quoting:
"Suddenly, Americans are sitting at their kitchen tables – in suburbs, in cities, on farms – and talking about quality issues with a website. The average American was given nightly tutorials on load testing and performance bottlenecks when the site first launched, then crumbled moments later. We talked about whether the requirements were well-defined and the project schedule reasonably laid out. We talked about who owns the decision to launch and whether they were keeping appropriate track of milestones and iterations. ... When the media went from talking about the issues in the website to the process used to build the website was when things really got interesting. This is when software testers stepped out of the cube farm behind the coffee station and into the public limelight. Who were these people – and were they incompetent or mistreated? Did the project leaders not allocate enough time for testing? Did they allocate time for testing but not time to react to the testing outcome? Did the testers run inadequate tests? Were there not enough testers? Did they not speak up about the issues? If they did, were they not forceful enough?"
WTF Not forceful enough? (Score:3, Insightful)
Ya those damn testers, they just can't communicate the issues to management. Like that NASA engineer and the O-rings. Stop blaming the testers.
Re:WTF Not forceful enough? (Score:5, Insightful)
Well you either blame the help, or else someone important is going to have to take the blame.
Someone important *should* take the blame (Score:4, Insightful)
One of the most insightful truths ever told to me:
It is always management's fault.
This goes right to the root of the tree, because by definition if the people further out couldn't get the job done or didn't have the right resources to do it, it was management's responsibility to fix those problems. The buck stops with the most senior managers on a project, whose only two choices are to explain what is needed to succeed and then do so if given those things, or to fail.
Re:Someone important *should* take the blame (Score:4, Informative)
The President doesn't write code.
No, but he presumably appoints the senior staff whose teams ultimately will write code, and capture requirements, and run tests, and all the rest. For something on this level he probably had a direct say in things like budget and timescales and the scope and high-level organisation of the project as well. If he asked for the impossible, he bears responsibility for that. If he didn't know it was impossible, he should have hired better advisors before committing to the project. And if it wasn't impossible but failed anyway, he should have appointed a senior management team that was up to the job.
(I'm not in the US and have no allegiance to any US political party, so please don't try to turn this into some sort of red vs. blue flamewar. I'm just translating the same general principles that apply to senior management for any infrastructure project into the specific context of healthcare.gov.)
Re: Someone important *should* take the blame (Score:4, Informative)
What a staggeringly ignorant post.
The house controls the purse strings and has oversight powers over the operation of the government.
Healthcare.gov has allocated some $634M and to date had expended just over half of it by the time it went live, lord knows what they spent since Oct. 1... The house withheld not one nickel from the project - it's fully funded.
As for House interference, you are simply inventing a conspiracy out of whole cloth... You really think the House 'secretly' met with those overseeing the development of the website and told them to mess it up? It was obvious to anyone looking at the situation that this was a train wreck waiting to happen - remember Republicans saying the best way to defeat Obamacare was to let it play out and fail on it's own? If there was a conspiracy as you imagine, how is it not one person has stepped forward to expose it?
The contractor selected had a history of failed projects, why is it so hard to believe they failed again?
Re: (Score:3)
Bullshit. The President doesn't write code.
His staff was saying, in the week before the website went live, that everything would be great. That's on him, for not knowing (or lying?) about the state of the website.
Re: (Score:2)
Bullshit. The President doesn't write code.
His staff was saying, in the week before the website went live, that everything would be great. That's on him, for not knowing (or lying?) about the state of the website.
Perhaps that was in part due to CGI Federal SVP in charge of the Healthcare.gov project, Cheryl Campbell, testifying before the House of Representatives September 10, 2013 that the project was still on track [video [house.gov] - Testimony starting around 36m20s.] & [pdf [house.gov]]. While some of the other organizations present during this testimony were flagging issues, the primary contracting organization was communicating that the project was still on track and would be ready for the October 1, 2013 launch.
If the key play
Re: (Score:3)
Re: (Score:3)
It is always management's fault.
That is exactly the job of management: to get the job done. When the lead programmer quits? Management is the one who needs to find another one.
Ender's game gives a perfect example: Colonel Graff is management. He sorts through all possible candidates, develops a program with one goal in mind: beating the formics.
Ender is the guy who actually does the work, he is the engineer. He does the battle plan, and implements it. But the Colonel is watching, making sure things are going according to plan, and wi
Re: (Score:1)
That is why good CEOs like Steve Jobs takes responsibility to 'test' the products and involve himself in the design process. Because in the end it is his responsibility.
Trust but verify.
Re: (Score:1)
You are forgetting the first rule of bureaucracy:
It is NEVER upper management's fault.
inspecting an unfinished POS (Score:4, Interesting)
development not complete (Score:4, Insightful)
It is not really a question of testing. Parts of the software were missing or incomplete. You can't test what isn't there.
Re: (Score:1)
Re: (Score:3)
Are mixing it up with unit testing?
TDD isn't the only time you have tests before the code is written. It is a common practice to create test cases for functionality that has not been completed or even started on. The status on the test case may be "Implementation Not Complete", which is just one type of test failure.
This way you can still have reports on development progress and show that large sections of functionality have not been tested yet (because they aren't finished).
Re:development not complete (Score:5, Interesting)
This was very common in the
Ideally QA gets to help validate that the functional requirements are adequately addressed in the design. In many cases, lacking a spec of any kind I would have to create one of my own based on what the product was able to do or close to being able to do at that time in order to make a test plan.
When you are brought on board a sinking ship there is no point in blaming the crew for the state of the ship -all you can do is damage control to validate whatever is working and then lower the bar as to what constitutes 'working' or 'functional' -particularly if some major components or functionality are missing.
You're going to be seen by management as the people who are going to point out what idiots and incompetents the developers were and be seen as the enemy by the developers who were probably led down the rabbit hole by changing or nebulous requirements and unrealistic schedules...
So it is important to try to walk the middle line -making observations about the current situation without casting blame or making guesses about how the project got to that state (although it may be obvious when you look at the principals and the agenda). Gap analysis of both testing and in the product functionality and features is another thing that needs to be done more often in order to present a realistic picture of the current state of the product or project.
As a consultant it is nice to be able to come into these things knowing that you didn't help cause the trainwreck -you are just there doing triage and trying to save the patient....and sometimes management will listen to you about project and requirements that they ignored when brought up by their own people. Even if it is 20/20 hindsight perhaps they will heed the techies the next time they embark upon this path -Nah!
-I'm just sayin'
Re: (Score:2)
Re:development not complete (Score:5, Insightful)
The real world does not work like this and I find the best work comes when the guy who will be writing the software actually collects the requirements and does a good job of that including a plan of how to test the code.
The worst code I see is written when the programmer is given some narrow requirements on some ticket and they code directly to those requirements with little to no knowledge of the overall system they are working on. In some magical world where all of your requirements are perfect this could work very well. But part of being a solid developer is knowing how to spot the "smell" of bad requirements. While sometimes you can do this without any knowledge of the overall system, you are much more likely to have the right insights if you have some relevant domain knowledge.
I have a current project that I am maintaining where I wrote most of the code in a vacuum without understanding the customer's real needs. It is a horrible mess and I would have done things completely differently if I was involved in meetings with the clients early on. I'm not saying my code would have been perfect, but there were some massive disconnects between the assumptions I drew from the written requirements and the explanations I got from the clients once I was given more access to them. Now there is no time or money for massive rewrites or refactorings, so it continues to be a thorn in my side.
Re: (Score:2)
So it is important to try to walk the middle line -making observations about the current situation without casting blame or making guesses about how the project got to that state (although it may be obvious when you look at the principals and the agenda).
In my experience, it is *always* obvious when you look at the principals and the agenda how the project got screwed up. You're right though--diplomacy often requires walking the middle path.
Battleifeld 4 (Score:2, Funny)
These same questions plague Battlefield 4
Re:Battleifeld 4 (Score:4, Funny)
And not to mention, Battlestar Galactica too. A lot of the needless drama on that show could have been avoided if they had just licensed their testing framework from the Cylons.
Were known management tools used? (Score:3)
Re: (Score:1)
They didn't even use a RDBMS even near mainstream, much less one that can be used with standardized process.
Can MarkLogic pass the ACID test? Their website states that they can, but it would be nice to see real world results, especially well before this was thrown as a pivotal site the American public has to deal with.
Maybe the ACA website would have had a far better fate had they used DB/2, Oracle, or even MS SQL. No, it doesn't have the "pop" of NoSQL, but RDBMS servers are made to just do the job quiet
Re: (Score:3)
No, it doesn't have the "pop" of NoSQL,
More to the point, it doesn't have the scalability across distributed systems. Show me one application approaching this scale, just *one*, that relies RDBMS clusters and two-phase commit exclusively to support this kind of transaction volume. Don't get me wrong; I'm an old-school RDBMS guy myself; I know a lot about relational database systems, including their limitations. I'd look to the way outfits like Amazon, ,Google or LinkedIn design their infrastructure rather than Oracle on big iron. This is well o
Re: (Score:2)
Well, we're talking far too abstractly here to be very meaningful. I'm not saying an RDMS couldn't be *part* of the picture. I'm saying that a system architecture that punted all the persistence and data consistency problems to a distributed RDBMS is a non-starter for something on this scale. People don't build systems like this one that way. If they don't, even though that technology is a mature one, that's a good reason to be skeptical of the idea that that approach would be a panacea.
Re: (Score:2, Insightful)
How do you make a presentation of working software when the working software is designed to pull data from half a dozen databases outside of your control? You might as well throw a football in the air and then claim you're a world class quarterback, you're just waiting for the rest of the world class team to show up and assist you.
Re:Were known management tools used? (Score:5, Informative)
It does not sound to me as though known management tools were used. Did they sit down with the government personnel in charge, and present their approach, and what the site would look like (menus, flow, etc) when finished? Were there testable milestones, and a final presentation of working software? It sure doesn't sound like it.
They might well have done all these things and still failed to catch the problems before the site's launch.
Performance, like security (ack! scary!) is a non-functional requirement -- that is to say it's not the kind of requirement where you can sit down with a checklist and say, 'yep, this it works,' or 'no, it doesn't.' You have to develop a more sophisticated test.
Load testing is a step in the right direction, but you also have to look at system architecture. Remember the days before people figured out that you had to load web ads asynchronously, after the page content was loaded? Sometimes the page load would be slow, not because the page's server was loaded, or because of the user's browser or internet connection were slow. Often it would be the ad server that was overwhelmed, which if you think about it is bound to be more common than the content server being overwhelmed. You could functional test and even load test the heck out of a page with synchronous ad loads, but until it went into production chances are you wouldn't catch the fatal performance flaw. That kind of problem is architectural; some of the data being delivered is coming from servers outside your control.
Ordinary tests are about ensuring reproducible results, but when the architecture leaves you vulnerable to things happening on servers and networks outside your control your problems *aren't reliably reproducible*. You have to design around *possibilities*.
Some of the problems with Healthcare.gov were of this nature, although with not so simple a solution as "use window.onload()." The site is supposed to orchestrate a complex, *distributed* process *synchronously*. You have to go out the Homeland Security's database to confirm citizenship, then to the IRS databases to confirm claims about income, then get quotes from the private insurers that meet the customer's needs. There is, in my opinion, no way to be 100% sure, or even 80% sure that a system like that will work under real work load, unless you present it with real work load.
Were I architecting such a site, I'd plan to do a lot of that work batch; that is I'd build the healthcare application offline on the user's browser, with internal consistency checks of course. Then I'd send the user's application through a batch verification system, emailing him when a result was ready. This is a clunky and old-fashioned approach, but it wouldn't force the user to chain himself to his browser . And it would have more predictable performance. Predictability is a vastly under-rated quality in a system. A system which is fast most of the time may not be as desirable as one which provides the answer consistently.
Re: (Score:2)
I think you hit the nail on the head, although I'd probably code in a messaging similar similar to what banks have so a user logs onto the healthcare site to receive notices... this will protect against phishing.
The batch system is by far the best way to do this, as it disconnects one from having to depend on other databases in real time.
I'd also consider using content delivery networks for static content. That way, the data coming from the dynamic website is as small as possible. Google does a good job a
Re: (Score:2)
Google does a good job at this with YouTube.
I am not saying you are wrong, I have seen it being suggested, but it has never flown on governmental projects.
Re: (Score:3)
Let's talk about security of this website, because it illustrates what's going on here well.
The guy responsible for signing off on the security of Healthcare.gov - signing off on the waiver, mind you, not some actual audit - refused to sign. His boss signed for him and he resigned or was fired.
Last week this repeated - the new guy responsible for signing off on security - presumably the most compliant guy they could possibly find - refused to sign off. Again his boss signed for him, and he's gone.
Does any
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Management decided to go live with the site anyway. The were hoping that at least some people would be able to register.
Re:worst job ever... (Score:4, Funny)
I cannot imagine a worse job than to have worked on that project.. The ratio of "status update" meetings and management pud-pulls to useful work accomplished must have been damn close to infinity..
You haven't worked on my project.
Re: (Score:1)
There are over 2.8 million words of Obamacare regulations! [netrightdaily.com]
I challenge anyone to create a website that conforms to such a huge number of rules -- some of them probably contradictory!
Conservatives constantly point out how excessive regulation makes doing business difficult. Well it makes things difficult on the government, too. Let's be fair.
10 PRINT "This is screwed up, it won't work"
20 GOTO 10
Not relevant to the web site (Score:4, Interesting)
Most of the Affordable Care Act has nothing to do with the web site. The site didn't have to implement those "2.8 million words of Obamacare regulations" as code: it only had to match patients up with insurance plans, which means interacting with dozens (hundreds?) of government and industry databases.
Some states, like California, managed to implement their sites without any of the problems of the federal exchange. The federal exchange mainly suffered from (1) being rushed, and (2) having to deal with a larger number of external systems than any single state exchange.
Re: (Score:2)
It also had to determine whether you were eligible for subsidies, and if so, how much. Which was a non-trivial exercise, though nowhere near the whole ACA-worth of regulations.
Re: (Score:3)
> I challenge anyone to create a website that conforms to such a huge number of rules -- some of them probably contradictory!
Sounds like any other regulatory burden. Are the things at Amazon FCC approved? Are they UL listed? Do they pass muster by the USDA?
All of that stuff is outside the scope of the website and it should be the case for Obamacare too.
Your kind of thinking is why it was such a disaster and why 3 guys managed to throw together a window shopping frontend with little effort.
Re: (Score:3)
Conservatives constantly point out how excessive regulation makes doing business difficult. Well it makes things difficult on the government, too. Let's be fair.
Isn't that rather like saying "Ladies and gentlemen of the jury, have compassion on my client. By killing his parents he became an orphan."?
Nothing to see here. (Score:2, Redundant)
Re: (Score:1)
I don't know whether in each case the project goals were just too ambitious, or whether the projects were incompetently planned or managed (probably all of the above), but there seems to be
Re: (Score:2)
Nothing to C here (Score:2)
The little awareness they glean of the craft from this incident will just as li
"Average American was given nightly tutorials on (Score:5, Funny)
That's great but how about we teach the average American how to spot Europe on a map first.
Re: (Score:1)
Why? Is Europe's location somehow significant to average Americans?
Re:"Average American was given nightly tutorials o (Score:4, Insightful)
Why? Is Europe's location somehow significant to average Americans?
And there is the proof of the OP's implied statement
Re: (Score:2)
Yes. The average American is of recent European descent, speaks a European language, and lives under a system derived from European thought history.
Re: (Score:2)
Why? Is Europe's location somehow significant to average Americans?
Well you know, you might like to bomb us sometime. Then it comes in handy to know where it is!
Re: (Score:2)
That's great but how about we teach the average American how to spot Europe on a map first.
I think finding Europe is the least of their problems: The Chaser: War on Everything - Americans [youtube.com]
(Yes I know it was probably all in the editing .. still .. you've got to be elfin joking)
Re: (Score:3, Insightful)
They're no better with our geo than we are with theirs.
Re: (Score:2)
It happens occasionally here in the Southwest U.S. that European tourists forget they can drive for eleven hours and not see anyone, or anything. Including a petrol station.
Re: (Score:2)
What's "So Mo"? Southern Missouri? Technically that would be "So MO" to disambiguate from Montana...
Re: (Score:2)
That's great but how about we teach the average American how to spot Europe on a map first.
That's great but how about we teach the average American how to spot the USA on a map first.
Bad design, not just bad implementation (Score:3)
Re: (Score:2)
What a shit article (Score:2, Interesting)
No the project leads gave plenty of time for testing, development, and even kept the WH up to date on what was happening. What happened though was the Obama administration pushed through something that wasn't ready, and wouldn't be ready for long past it's actual inception date. And this of course is because the administration sat on it's backside for an extended period of time, then waved their hands and said a couple of years should be more than enough.
The committee meetings are chock full of very usefu
The underlying problem... (Score:1)
The problem underlying the entire fiasco — and the less-impacting others like it (Amtrak [usatoday.com], anyone?) — is that whatever the government does, is done poorly .
"Not bad for a government job," — is part of vernacular, yet, a curious mix of well-meaning idealists and self-serving demagogues manage to convince the public to try again every once in a while...
Re:The underlying problem... (Score:4, Interesting)
Yes, yes, because every bridge the government builds falls down three or four times a day in the first couple of weeks after it's opened.
Re: (Score:2)
I did not say, it does not get done at all — I said, it is done poorly. The government-managed highways and bridges suck — next time you are stuck in traffic, you'll be forced to agree with me....
Re: (Score:3, Interesting)
The problem underlying the entire fiasco — and the less-impacting others like it (Amtrak [usatoday.com], anyone?) — is that whatever the government does, is done poorly .
I realize that that's a right-wing meme, and it's rare for conservatives to change their minds based on the facts, but it's not true.
The military and Veterans Affairs medical centers give some of the best care in the world. I've read the studies that compare them to other centers around the world. They've got the data.
Ronald Reagan got his colon and prostate surgery at Walter Reed. Watch what they do, not what they say.
If you got a head injury in Iraq, you'd have the best chance in the world of surviving wi
Re: (Score:2)
Is not it a little early in the conversation for ad hominems?
Citation needed? [usatoday.com]
Re: (Score:3)
You — and others — seem to have misconstrued my argument to mean, the government simply can not do anything. That's not, what I said. They can do it — just poorly.
Is not it a little early in the conversation for ad hominems?
I base this on several years of the Wall Street Journal comments page, until I gave up on them. And I read lots of conservative articles on health care policy. There are conservatives who change their minds based on the facts, but in my experience they are rare. William Buckley is dead. The WSJ editorial page has turned into a Pravda for the conservative wing of the Republican Party. Sic transit gloria mundi.
The military and Veterans Affairs medical centers give some of the best care in the world
Citation needed? [usatoday.com]
http://www.ncbi.nlm.nih.gov/pubmed/?term=%22Veterans'+Affairs%22 [nih.gov]
http://www.nejm.or [nejm.org]
Re: (Score:2)
Given the amounts of money the government spends on this sort of research, it is not at all surprising, that few others do. Competing with government — like fighting the city hall — is usually fruitless and often dangerous. And not because the government does such a good work, but because their pockets are much deeper and, when that does not help, they c
Re: (Score:1)
You don't know much about Amtrak, do you? Amtrak is the result of rail companies no longer wanting to deal with the passenger business they had left after the rise of the automobile. So they got together and convinced the government to take over that side of things, while leaving them free to engage in the business they wanted, freight.
Which they do very nicely.
But passengers? They don't want the bother, they don't want the trouble, they'd rather leave it to highways (run by the government, in case you d
Re: (Score:2)
Cool story, bro. I could've asked for some references, but it does not change a thing. The point was — and remains — that Amtrak sucks. The link I gave earlier [usatoday.com] discusses — as an example — the cost of a can of soda... Despite selling it to passengers for $2, Amtrak
Been a problem for decades (Score:3)
SQA as a red-headed stepchild has been an issue for many, many years. It's just that most troubles/failed software systems don't have the widespread public exposure that Healthcare.gov has; even the most brain-dead corporation would not have launched such an incomplete and bug-ridden system to a vast end-user bases.
Some years ago, I led a review of a late (4 yrs vs 2 yrs estimated) and very over-budget ($500M vs. $180M estimated) corporate software project. The core problems had everything to do with SQA, starting with the fact that there was no SQA organization; all testing was done on an ad hoc basis by individual teams and organizations. Adding to that problem was the fact that there was no coherent architecture. After 4 years and $500M, there were no systems that were ready to go into production. Far too common in industry and especially in government. ..bruce..
Funny thing: testing is not that important... (Score:4, Insightful)
If you find out in testing that your architecture or design does not cut it, you are screwed. The only thing you can usually do is scrap the project and start again. Testing does only work for simple things like simple busiess logic and the like, where you know the characteristics very well beforehand. For anything that is a new design, the only thing that helps is very capable and experienced architects and designers that have a good change to get it right by intuition. This will be people that can do architecture, design and implementation and can do all three well. Not many of those exist, but there is no replacement for them. Those that think they can do things on the cheap without not only having this type of expert but also listening to them closely will fail. This can be observerd time and again and can alost be called a "well established industrial practice", because quite a few "managers" do not actually know that it can be done better. Funny thing, in other fields, you have chief enineers, architects and the like and the critical work is not given to people that are likely to fail. Only IT messess it up regularly, because talent and exerence is not respected.
Re: (Score:2)
Re: (Score:2)
I strongly suspect that healthcare.gov was not modularized or not modularized competently. Still, testing is really not that important. Sure, you have to do it, but it is not what prevents major software desasters. I know that there are a lot of "developers" out there that think testing is the gold standard, but there is at the same time a lot of software that sucks despite it having been tested. That should tell you something.
Re: (Score:2)
Why weren't Google or Amazon asked to do it? (Score:2)
You've got to ask yourself, who's building and deploying reliable, performant, extremely scaled apps these days? Who has been doing that successfully for over a decade? Why don't we ask them to build our big app? Or if they're busy, ask them who they would recommend.
It's usually the schedule, in my experience (Score:2)
Ok, so some projects, as has been pointed out, are doomed from the very first bad architectural decision (or lack of architectural decision.)
But regardless of that factor, the most common thing I've seen is management/corporate promising a particular release date, in a contract, say, and eventually getting around to telling development/engineering, who say, if they're brave, um, that's not possible. If they are less brave, they smile and get on with faking it, all the while knowing it's impossible in the ti
Maybe now Testers will have budget. (Score:1)
I'm a developer that has been working as a tester in the last 5 years.
Always coding to run performance / load / whatever not manual testing.
In my experience working at bank or big publishers, my main problem is always budget.
Enough money for everything except the right number of people testing, the right tools or infrastructure for testing, and so on.
I don't expect that was different with healthcare.gov.
Re: (Score:1)
Except for one project that I destroyed with performance testing, leading to a huge re-factoring, the others always get delayed to correct all serious bugs.
But it was, as always, management decision. Postpone or release with bugs?
Re: (Score:1)
Suffering from that very issue as we speak with a major software update due for rollout by the end of Feb. Argh.
At this point what difference does it make (Score:2)
By all accounts, there was some testing done, however inadequate. Thing is, the system utterly failed that testing and THEY DEPLOYED IT ANYWAY. If you're going to ignore the testing, why even do it? Just throw open the doors and hope for the best. Which is what they did, apparently.
Re: (Score:2)
Then they are people who understand insurance as poorly as you appear to. The healthy pay the unhealthy's medical bills because a) the risk and cost is shared by all premium payers when they are healthy (including those who get sick later) b) the unhealthy don't need the added stress of getting huge bills to impact their attempts to get better and usually can't work while they're in traction or a coma.
You know, sort of like how in fire or flood insurance the people whose houses are standing help pay for re
Re: (Score:2)
Democrats wanted it, so it was doomed to fail since nothing they do work.
So all those bills the Republicans have been desperately voting down "BECAUSE DEMONCRATS!!" are just mercy killings?
TDD (Score:1)