Forgot your password?
typodupeerror
Software Bug United States

How Healthcare.gov Changed the Software Testing Conversation 118

Posted by Soulskill
from the polls-say-software-testing-is-patriotic dept.
An anonymous reader notes an article about how the tribulations of Healthcare.gov brought the idea of software testing into the public consciousness in a more detailed way than ever before. Quoting: "Suddenly, Americans are sitting at their kitchen tables – in suburbs, in cities, on farms – and talking about quality issues with a website. The average American was given nightly tutorials on load testing and performance bottlenecks when the site first launched, then crumbled moments later. We talked about whether the requirements were well-defined and the project schedule reasonably laid out. We talked about who owns the decision to launch and whether they were keeping appropriate track of milestones and iterations. ... When the media went from talking about the issues in the website to the process used to build the website was when things really got interesting. This is when software testers stepped out of the cube farm behind the coffee station and into the public limelight. Who were these people – and were they incompetent or mistreated? Did the project leaders not allocate enough time for testing? Did they allocate time for testing but not time to react to the testing outcome? Did the testers run inadequate tests? Were there not enough testers? Did they not speak up about the issues? If they did, were they not forceful enough?"
This discussion has been archived. No new comments can be posted.

How Healthcare.gov Changed the Software Testing Conversation

Comments Filter:
  • by Anonymous Coward on Tuesday December 24, 2013 @05:21PM (#45778185)

    Ya those damn testers, they just can't communicate the issues to management. Like that NASA engineer and the O-rings. Stop blaming the testers.

    • Well you either blame the help, or else someone important is going to have to take the blame.

      • by Anonymous Brave Guy (457657) on Tuesday December 24, 2013 @06:23PM (#45778655)

        One of the most insightful truths ever told to me:

        It is always management's fault.

        This goes right to the root of the tree, because by definition if the people further out couldn't get the job done or didn't have the right resources to do it, it was management's responsibility to fix those problems. The buck stops with the most senior managers on a project, whose only two choices are to explain what is needed to succeed and then do so if given those things, or to fail.

        • It is always management's fault.

          That is exactly the job of management: to get the job done. When the lead programmer quits? Management is the one who needs to find another one.

          Ender's game gives a perfect example: Colonel Graff is management. He sorts through all possible candidates, develops a program with one goal in mind: beating the formics.
          Ender is the guy who actually does the work, he is the engineer. He does the battle plan, and implements it. But the Colonel is watching, making sure things are going according to plan, and wi

          • by Anonymous Coward

            That is why good CEOs like Steve Jobs takes responsibility to 'test' the products and involve himself in the design process. Because in the end it is his responsibility.
            Trust but verify.

    • by pigiron (104729)

      You are forgetting the first rule of bureaucracy:

      It is NEVER upper management's fault.

    • by harvey the nerd (582806) on Tuesday December 24, 2013 @06:17PM (#45778629)
      Imagine being the QA inspector on a 1985 Jugo car. No matter what you say, the entire thing is a POS. The only question is whether you need your paycheck that badly. Politics and unrestrained corruption simply don't mix well with code.
  • by BradMajors (995624) on Tuesday December 24, 2013 @05:23PM (#45778193)

    It is not really a question of testing. Parts of the software were missing or incomplete. You can't test what isn't there.

    • by bsolar (1176767)
      Of course you can test incomplete software. The relevant tests should fail until the implementation is complete and correct.
    • by hguorbray (967940) on Tuesday December 24, 2013 @06:28PM (#45778683)
      Having been brought on late to QA a few death marches/trainwrecks in my time I have found that many projects don't get QA involved until way too late in the game.

      This was very common in the .com boom days when everything was developer-centric and testing was seen as an unneeded cost that could be covered by the coders themselves -I don't need to tell anyone here why that is a bad idea.

      Ideally QA gets to help validate that the functional requirements are adequately addressed in the design. In many cases, lacking a spec of any kind I would have to create one of my own based on what the product was able to do or close to being able to do at that time in order to make a test plan.

      When you are brought on board a sinking ship there is no point in blaming the crew for the state of the ship -all you can do is damage control to validate whatever is working and then lower the bar as to what constitutes 'working' or 'functional' -particularly if some major components or functionality are missing.

      You're going to be seen by management as the people who are going to point out what idiots and incompetents the developers were and be seen as the enemy by the developers who were probably led down the rabbit hole by changing or nebulous requirements and unrealistic schedules...

      So it is important to try to walk the middle line -making observations about the current situation without casting blame or making guesses about how the project got to that state (although it may be obvious when you look at the principals and the agenda). Gap analysis of both testing and in the product functionality and features is another thing that needs to be done more often in order to present a realistic picture of the current state of the product or project.

      As a consultant it is nice to be able to come into these things knowing that you didn't help cause the trainwreck -you are just there doing triage and trying to save the patient....and sometimes management will listen to you about project and requirements that they ignored when brought up by their own people. Even if it is 20/20 hindsight perhaps they will heed the techies the next time they embark upon this path -Nah!

      -I'm just sayin'
      • by Dan667 (564390)
        Most of the time I find that many Developers want to be handed a perfect spec and let to go into a basement and code it. The real world does not work like this and I find the best work comes when the guy who will be writing the software actually collects the requirements and does a good job of that including a plan of how to test the code. I think part of the problem is pretending requirements are some uncertain quantity, when they are usually very straight forward to collect if you are willing to do the
        • by ranton (36917) on Wednesday December 25, 2013 @12:08AM (#45780303)

          The real world does not work like this and I find the best work comes when the guy who will be writing the software actually collects the requirements and does a good job of that including a plan of how to test the code.

          The worst code I see is written when the programmer is given some narrow requirements on some ticket and they code directly to those requirements with little to no knowledge of the overall system they are working on. In some magical world where all of your requirements are perfect this could work very well. But part of being a solid developer is knowing how to spot the "smell" of bad requirements. While sometimes you can do this without any knowledge of the overall system, you are much more likely to have the right insights if you have some relevant domain knowledge.

          I have a current project that I am maintaining where I wrote most of the code in a vacuum without understanding the customer's real needs. It is a horrible mess and I would have done things completely differently if I was involved in meetings with the clients early on. I'm not saying my code would have been perfect, but there were some massive disconnects between the assumptions I drew from the written requirements and the explanations I got from the clients once I was given more access to them. Now there is no time or money for massive rewrites or refactorings, so it continues to be a thorn in my side.

      • So it is important to try to walk the middle line -making observations about the current situation without casting blame or making guesses about how the project got to that state (although it may be obvious when you look at the principals and the agenda).

        In my experience, it is *always* obvious when you look at the principals and the agenda how the project got screwed up. You're right though--diplomacy often requires walking the middle path.

  • by Anonymous Coward

    These same questions plague Battlefield 4

  • by whizbang77045 (1342005) on Tuesday December 24, 2013 @05:31PM (#45778265)
    It does not sound to me as though known management tools were used. Did they sit down with the government personnel in charge, and present their approach, and what the site would look like (menus, flow, etc) when finished? Were there testable milestones, and a final presentation of working software? It sure doesn't sound like it.
    • by Anonymous Coward

      They didn't even use a RDBMS even near mainstream, much less one that can be used with standardized process.

      Can MarkLogic pass the ACID test? Their website states that they can, but it would be nice to see real world results, especially well before this was thrown as a pivotal site the American public has to deal with.

      Maybe the ACA website would have had a far better fate had they used DB/2, Oracle, or even MS SQL. No, it doesn't have the "pop" of NoSQL, but RDBMS servers are made to just do the job quiet

      • by hey! (33014)

        No, it doesn't have the "pop" of NoSQL,

        More to the point, it doesn't have the scalability across distributed systems. Show me one application approaching this scale, just *one*, that relies RDBMS clusters and two-phase commit exclusively to support this kind of transaction volume. Don't get me wrong; I'm an old-school RDBMS guy myself; I know a lot about relational database systems, including their limitations. I'd look to the way outfits like Amazon, ,Google or LinkedIn design their infrastructure rather than Oracle on big iron. This is well o

    • Re: (Score:2, Insightful)

      by Anonymous Coward

      How do you make a presentation of working software when the working software is designed to pull data from half a dozen databases outside of your control? You might as well throw a football in the air and then claim you're a world class quarterback, you're just waiting for the rest of the world class team to show up and assist you.

    • by hey! (33014) on Tuesday December 24, 2013 @06:32PM (#45778713) Homepage Journal

      It does not sound to me as though known management tools were used. Did they sit down with the government personnel in charge, and present their approach, and what the site would look like (menus, flow, etc) when finished? Were there testable milestones, and a final presentation of working software? It sure doesn't sound like it.

      They might well have done all these things and still failed to catch the problems before the site's launch.

      Performance, like security (ack! scary!) is a non-functional requirement -- that is to say it's not the kind of requirement where you can sit down with a checklist and say, 'yep, this it works,' or 'no, it doesn't.' You have to develop a more sophisticated test.

      Load testing is a step in the right direction, but you also have to look at system architecture. Remember the days before people figured out that you had to load web ads asynchronously, after the page content was loaded? Sometimes the page load would be slow, not because the page's server was loaded, or because of the user's browser or internet connection were slow. Often it would be the ad server that was overwhelmed, which if you think about it is bound to be more common than the content server being overwhelmed. You could functional test and even load test the heck out of a page with synchronous ad loads, but until it went into production chances are you wouldn't catch the fatal performance flaw. That kind of problem is architectural; some of the data being delivered is coming from servers outside your control.

      Ordinary tests are about ensuring reproducible results, but when the architecture leaves you vulnerable to things happening on servers and networks outside your control your problems *aren't reliably reproducible*. You have to design around *possibilities*.

      Some of the problems with Healthcare.gov were of this nature, although with not so simple a solution as "use window.onload()." The site is supposed to orchestrate a complex, *distributed* process *synchronously*. You have to go out the Homeland Security's database to confirm citizenship, then to the IRS databases to confirm claims about income, then get quotes from the private insurers that meet the customer's needs. There is, in my opinion, no way to be 100% sure, or even 80% sure that a system like that will work under real work load, unless you present it with real work load.

      Were I architecting such a site, I'd plan to do a lot of that work batch; that is I'd build the healthcare application offline on the user's browser, with internal consistency checks of course. Then I'd send the user's application through a batch verification system, emailing him when a result was ready. This is a clunky and old-fashioned approach, but it wouldn't force the user to chain himself to his browser . And it would have more predictable performance. Predictability is a vastly under-rated quality in a system. A system which is fast most of the time may not be as desirable as one which provides the answer consistently.

      • by mlts (1038732)

        I think you hit the nail on the head, although I'd probably code in a messaging similar similar to what banks have so a user logs onto the healthcare site to receive notices... this will protect against phishing.

        The batch system is by far the best way to do this, as it disconnects one from having to depend on other databases in real time.

        I'd also consider using content delivery networks for static content. That way, the data coming from the dynamic website is as small as possible. Google does a good job a

        • by ls671 (1122017)

          Google does a good job at this with YouTube.

          I am not saying you are wrong, I have seen it being suggested, but it has never flown on governmental projects.

      • by lgw (121541)

        Let's talk about security of this website, because it illustrates what's going on here well.

        The guy responsible for signing off on the security of Healthcare.gov - signing off on the waiver, mind you, not some actual audit - refused to sign. His boss signed for him and he resigned or was fired.

        Last week this repeated - the new guy responsible for signing off on security - presumably the most compliant guy they could possibly find - refused to sign off. Again his boss signed for him, and he's gone.

        Does any

      • by Dan667 (564390)
        There is lots of integration, but lets be honest. A site you have to use over and over in order to get your insurance ordered is going to suck and I personally would not use it.
      • The problem here wasn't one of testing. QA did good tests, and found that the site wouldn't handle load.

        Management decided to go live with the site anyway. The were hoping that at least some people would be able to register.
  • This is just yet another big government project gone awry. We get these in the UK all the time. I seriously doubt anyone is talking about the testing of this particular project though. Those involved in testing will just keep doing what they do, good ones doing it properly, bad ones doing it, well badly. The other 99% of the population will just bitch about the site as being generally crap but they won't be saying 'They really should have done more integration and load testing'
    • The average citizen may or may not be proficient in a given area of expertise, and perhaps a few gifted ones in several, but Ernest's ability to turn on a light in a room probably begins at 'flip the on switch' and ends at 'reset the breaker'. Coding, system administration, the quality control thereof are no more relevant to your average facebook poster than car mechanics to your mean 16 year old with his head under the hood.

      The little awareness they glean of the craft from this incident will just as li

  • by JoeyRox (2711699) on Tuesday December 24, 2013 @05:55PM (#45778461)
    load testing and performance bottlenecks"

    That's great but how about we teach the average American how to spot Europe on a map first.
    • by Anonymous Coward

      Why? Is Europe's location somehow significant to average Americans?

    • by OzPeter (195038)

      That's great but how about we teach the average American how to spot Europe on a map first.

      I think finding Europe is the least of their problems: The Chaser: War on Everything - Americans [youtube.com]

      (Yes I know it was probably all in the editing .. still .. you've got to be elfin joking)

    • Re: (Score:3, Insightful)

      by Oligonicella (659917)
      A Swiss friend of mine visited DC. I live in So Mo. We chatted on the phone and he suggested we get together for lunch the next day. So, I agreed and told him the city in Virgina we could meet in after driving eleven hours.

      They're no better with our geo than we are with theirs.
      • It happens occasionally here in the Southwest U.S. that European tourists forget they can drive for eleven hours and not see anyone, or anything. Including a petrol station.

      • What's "So Mo"? Southern Missouri? Technically that would be "So MO" to disambiguate from Montana...

    • That's great but how about we teach the average American how to spot Europe on a map first.

      That's great but how about we teach the average American how to spot the USA on a map first.

  • by Bengie (1121981) on Tuesday December 24, 2013 @06:05PM (#45778539)
    Performance and scaling should have been addressed in the design phase
  • Software get's so much grief about bugs and testing. Do you know how many times I had to send my Intel processor back for a replacement because of bugs? Seems like we're getting notifications every six to twelve weeks for that damn processor exchange again. Makes me so mad!

    Luckily processors, chip-sets, and mother-boards must be a lot easier than software.... right? Perhaps there's something else going on in the hardware world?

  • What a shit article (Score:2, Interesting)

    by Mashiki (184564)

    No the project leads gave plenty of time for testing, development, and even kept the WH up to date on what was happening. What happened though was the Obama administration pushed through something that wasn't ready, and wouldn't be ready for long past it's actual inception date. And this of course is because the administration sat on it's backside for an extended period of time, then waved their hands and said a couple of years should be more than enough.

    The committee meetings are chock full of very usefu

  • The problem underlying the entire fiasco — and the less-impacting others like it (Amtrak [usatoday.com], anyone?) — is that whatever the government does, is done poorly .

    "Not bad for a government job," — is part of vernacular, yet, a curious mix of well-meaning idealists and self-serving demagogues manage to convince the public to try again every once in a while...

    • by DeathToBill (601486) on Tuesday December 24, 2013 @06:33PM (#45778715) Journal

      Yes, yes, because every bridge the government builds falls down three or four times a day in the first couple of weeks after it's opened.

      • by mi (197448)

        Yes, yes, because every bridge the government builds falls down three or four times a day in the first couple of weeks after it's opened.

        I did not say, it does not get done at all — I said, it is done poorly. The government-managed highways and bridges suck — next time you are stuck in traffic, you'll be forced to agree with me....

    • Re: (Score:3, Interesting)

      by nbauman (624611)

      The problem underlying the entire fiasco — and the less-impacting others like it (Amtrak [usatoday.com], anyone?) — is that whatever the government does, is done poorly .

      I realize that that's a right-wing meme, and it's rare for conservatives to change their minds based on the facts, but it's not true.

      The military and Veterans Affairs medical centers give some of the best care in the world. I've read the studies that compare them to other centers around the world. They've got the data.

      Ronald Reagan got his colon and prostate surgery at Walter Reed. Watch what they do, not what they say.

      If you got a head injury in Iraq, you'd have the best chance in the world of surviving wi

      • by mi (197448)
        You — and others — seem to have misconstrued my argument to mean, the government simply can not do anything. That's not, what I said. They can do it — just poorly.

        it's rare for conservatives to change their minds based on the facts

        Is not it a little early in the conversation for ad hominems?

        The military and Veterans Affairs medical centers give some of the best care in the world

        Citation needed? [usatoday.com]

        Ronald Reagan got his colon and prostate surgery at Walter Reed. Watch what they do, not what

        • by nbauman (624611)

          You — and others — seem to have misconstrued my argument to mean, the government simply can not do anything. That's not, what I said. They can do it — just poorly.

          it's rare for conservatives to change their minds based on the facts

          Is not it a little early in the conversation for ad hominems?

          I base this on several years of the Wall Street Journal comments page, until I gave up on them. And I read lots of conservative articles on health care policy. There are conservatives who change their minds based on the facts, but in my experience they are rare. William Buckley is dead. The WSJ editorial page has turned into a Pravda for the conservative wing of the Republican Party. Sic transit gloria mundi.

          The military and Veterans Affairs medical centers give some of the best care in the world

          Citation needed? [usatoday.com]

          http://www.ncbi.nlm.nih.gov/pubmed/?term=%22Veterans'+Affairs%22 [nih.gov]

          http://www.nejm.or [nejm.org]

          • by mi (197448)

            If you go to any major medical conference, and go to the sessions on important diseases, you'll usually hear them talking about the "VA study."

            Given the amounts of money the government spends on this sort of research, it is not at all surprising, that few others do. Competing with government — like fighting the city hall — is usually fruitless and often dangerous. And not because the government does such a good work, but because their pockets are much deeper and, when that does not help, they c

    • by Anonymous Coward

      You don't know much about Amtrak, do you? Amtrak is the result of rail companies no longer wanting to deal with the passenger business they had left after the rise of the automobile. So they got together and convinced the government to take over that side of things, while leaving them free to engage in the business they wanted, freight.

      Which they do very nicely.

      But passengers? They don't want the bother, they don't want the trouble, they'd rather leave it to highways (run by the government, in case you d

      • by mi (197448)

        Amtrak is the result of rail companies no longer wanting to deal with the passenger business they had left after the rise of the automobile. So they got together and convinced the government to take over that side of things

        Cool story, bro. I could've asked for some references, but it does not change a thing. The point was — and remains — that Amtrak sucks. The link I gave earlier [usatoday.com] discusses — as an example — the cost of a can of soda... Despite selling it to passengers for $2, Amtrak

  • by bfwebster (90513) on Tuesday December 24, 2013 @06:34PM (#45778727) Homepage

    SQA as a red-headed stepchild has been an issue for many, many years. It's just that most troubles/failed software systems don't have the widespread public exposure that Healthcare.gov has; even the most brain-dead corporation would not have launched such an incomplete and bug-ridden system to a vast end-user bases.

    Some years ago, I led a review of a late (4 yrs vs 2 yrs estimated) and very over-budget ($500M vs. $180M estimated) corporate software project. The core problems had everything to do with SQA, starting with the fact that there was no SQA organization; all testing was done on an ad hoc basis by individual teams and organizations. Adding to that problem was the fact that there was no coherent architecture. After 4 years and $500M, there were no systems that were ready to go into production. Far too common in industry and especially in government. ..bruce..

  • by gweihir (88907) on Tuesday December 24, 2013 @06:46PM (#45778791)

    If you find out in testing that your architecture or design does not cut it, you are screwed. The only thing you can usually do is scrap the project and start again. Testing does only work for simple things like simple busiess logic and the like, where you know the characteristics very well beforehand. For anything that is a new design, the only thing that helps is very capable and experienced architects and designers that have a good change to get it right by intuition. This will be people that can do architecture, design and implementation and can do all three well. Not many of those exist, but there is no replacement for them. Those that think they can do things on the cheap without not only having this type of expert but also listening to them closely will fail. This can be observerd time and again and can alost be called a "well established industrial practice", because quite a few "managers" do not actually know that it can be done better. Funny thing, in other fields, you have chief enineers, architects and the like and the critical work is not given to people that are likely to fail. Only IT messess it up regularly, because talent and exerence is not respected.

    • Since when? Every large software vendor I've ever worked with breaks their product up into modules and do standalone module testing and once those tests are passed add it to the main code line and do additional integration testing from there. Why on earth would you not test a software project of this size until it was completed in it's entirety? While there may be modules that you can't go live without, there absolutely shouldn't be a requirement to start from the ground up because one module failed test
      • by gweihir (88907)

        I strongly suspect that healthcare.gov was not modularized or not modularized competently. Still, testing is really not that important. Sure, you have to do it, but it is not what prevents major software desasters. I know that there are a lot of "developers" out there that think testing is the gold standard, but there is at the same time a lot of software that sucks despite it having been tested. That should tell you something.

    • I can't believe this was modded so high. The point that testing is the wrong place to detect architectural flaws is spot on. The idea that the "only thing that helps is very capable and experienced ... get it right by intuition" is flawed. While I'm not going to knock expertise, modeling and simulation should be the preferred way to analytically determine whether something will work in complex systems. "Serious" engineering realizes the limitations of humans to reason about problems. Would you drive across
  • You've got to ask yourself, who's building and deploying reliable, performant, extremely scaled apps these days? Who has been doing that successfully for over a decade? Why don't we ask them to build our big app? Or if they're busy, ask them who they would recommend.

  • Ok, so some projects, as has been pointed out, are doomed from the very first bad architectural decision (or lack of architectural decision.)

    But regardless of that factor, the most common thing I've seen is management/corporate promising a particular release date, in a contract, say, and eventually getting around to telling development/engineering, who say, if they're brave, um, that's not possible. If they are less brave, they smile and get on with faking it, all the while knowing it's impossible in the ti

  • I'm a developer that has been working as a tester in the last 5 years.
    Always coding to run performance / load / whatever not manual testing.

    In my experience working at bank or big publishers, my main problem is always budget.
    Enough money for everything except the right number of people testing, the right tools or infrastructure for testing, and so on.

    I don't expect that was different with healthcare.gov.

    • A-freakin-men to that.

      Suffering from that very issue as we speak with a major software update due for rollout by the end of Feb. Argh.
  • By all accounts, there was some testing done, however inadequate. Thing is, the system utterly failed that testing and THEY DEPLOYED IT ANYWAY. If you're going to ignore the testing, why even do it? Just throw open the doors and hope for the best. Which is what they did, apparently.

  • by NewYork (1602285)
    https://en.m.wikipedia.org/wiki/Test-driven_development [wikipedia.org] is better for Govt projects

You've been Berkeley'ed!

Working...