Software Quality In a Non-Software Company? 308
Nicros writes "I work for a publicly traded biotech company that happens to write software that is, in fact, kind of critical for the business — without it no data would ever be read from our instruments, and no analyses would be performed on that data. The problem is that as a 'biotech' company, we are not taking software quality seriously. We have no senior management with any history of commercial software development — our C level has really no clue whatsoever what software really is, much less what is going on in software development. All of our quality processes are related to manufacturing our system (not software), so we are constantly forced into ad-hoc development since there is no real process for our development. Repeated requests to hire someone with some real commercial software development experience have gone unanswered. I have been to the CEO directly one-on-one and although he agreed this was an issue, thanked me, and said he would look into it, that was the end of it. He has bigger things to worry about. So the question: Is this just a fact of life and I need to deal the best I can? What else can I do to get some attention on software quality in the company?"
A suggestion on how to handle this (Score:3, Interesting)
Very interesting question. I see two things that might help. First, don't go to the CEO. You and him clearly speaks different languages. Go to your nearest manager instead and explain to him the consequences of not having procedures. I am sure you can convince him of this, and after the discussion do not settle for a "I'll look into that and get back to you". Always end your meetings with a list of action items with _who_ does _what_ and _when_. This way, you will have clearly defined dates you can follow up on. Have him commit to a date when he will do X (for instance - talk to the Director) and set a date for a follow up meeting with you where he will explain the outcome of said meeting. Should the meeting be canceled, be sure to get a replacement date and set follow up meeting accordingly.
Reminds me of Ariane 5 (Score:5, Interesting)
We have no senior management with any history of commercial software development
That reminds me of Arianespace. It took the crash of a 150M$ rocket to make them change that.
Open source it (Score:5, Interesting)
However, you should try and convince you're management to open source the software. In this industry the probability is that if you don't open source it someone else will write an open source replacement (see Phred/Phrap, and the open source replacements of the primary data analysis software on next-gen sequencers which are starting to appear). That means your company losses control of the primary data analysis and possibly device control software, and that's bad for your company.
Open source has the added benefit that your development costs will fall (you can start using GPL code), it'll help you engage with the scientific community and you'll get people outside the company doing free work for you (seriously people want to get this stuff working, they'll help). You'll also get free peer review on your code which will drive standards up.
Scared of showing your crap code? Don't be, in this industry I've seen enough to know that most of it sucks (a lot of it's written by Biologists with no formal training). The clincher? "ABI are doing it, why can't we!" http://solidsoftwaretools.com/gf/ [solidsoftwaretools.com]
Been There - Don't Burn Yourself Out (Score:2, Interesting)
I worked for a small medical electronics manufacturer in the UK. They had no software development team apart from me. I was fresh out of college and eager. I replaced the previous "software guy" who had walked out. No documentation or code was actually in place. The software standards that we were supposed to abide by were considered by the management to only be important when a product was finished. I ended up stuck in the middle between the customer the management, the marketing team and the hardware boys. I became quite adept at software/hardware debugging (for that project at least) and all the while attempting to keep the documentation going ( which was considered a waste of time by management of all levels). The crunch came when they took an order for 30 finished units before the prototype was finished (including documentation, third party software/unit tests etc..) despite my constant protests.
I burned my self out and am now a green oak carpenter. I blame my own young naive mind. And the fact that I trusted the management to be dealing with this sort of stuff.
If you don't abide by standards and have a half decent software development work flow organized by the management you're going to be in a fire fight. Get vocal and demand it now before you become a gibbering wreck trying to keep everybody happy. And the management will keep their jobs not doing there jobs.
It depends on if you are a dev yourself. (Score:2, Interesting)
If you are a software developer yourself, try to set up proper division yourself by talking to appropriate people in positions.
If not, stick to biotech. Software developing is engineering too, and it is not a good idea to do amateur in-house engineering, especially if your software products need to be of mission-critical kind. Outsource all software related job(s) to someone else who specializes in software design and development, and you yourself will sleep better.
Re:Anarchy is an opportunity (Score:2, Interesting)
It sounds like in your company there is no one doing this job. You've talked to the CEO. Get him to make you VP of software and tell him you'll solve the problem if he gives you responsibility.
Management may be doing its job perfectly well! This could quite possibly be R&D code that was hacked together by scientists with no formal training in programming (most likely Biologists). The R&D code then found it's way in to production (because they needed to get something out the door).
The software is good enough to get the device working, and it's the performance of the device that makes the company money, the software just has to work ``well enough'' the device is where the money is made.
Management maybe planning to hire a new development team to develop production software. Or they may realise that the current device isn't going to be in market long enough to warrant developing production style software. As I've mentioned in my other comment, I think the best solution here is to open source the software, it's the way to biotech industry is moving anyway and it will help you with you're software engineering problems.
Re:Mixing of two mindsets (Score:3, Interesting)
I had a family member who used to work with these kind of measuring machines...
He would often complain about how the old but functional machines would be replaced with fancy computer controlled ones, where the software would be extremely limited and often horrendously buggy.
"good enough" is good enough (Score:3, Interesting)
The company's troubles get worse when in the process of discovery, the plantiff attorneys find that instead of due diligence with respect to software development processes, there was no diligence.
The situation is a disaster waiting to happen. If the author has presented his concerns to top management and they've been ignored, if he's proposed to solve them himself (they probably do need somebody C-level in software development) and that fails, the guy needs to update his resume and call the headhunters.
Or become the fall guy when good enough is demonstrated to be not good enough in a court of law. CYA records of meetings and e-mails demonstrating that the writer saw a problem and tried to get it fixed to run into management non-cooperation might get the author off the hook for personal liability, but probably won't save his career.
Certification (Score:5, Interesting)
If you are in the bio tech field then all of your processes need to conform to ISO13485. There is a section specifically about software. Your company won't be in an FDA/CE regulated environment long unless you comply with those quality standards. I suggest you research the guidelines and point them out to your quality manager.
Re:Open source it (Score:2, Interesting)
It's gotten somewhat better over the past 5 years, but as the hardware lives for 20 years, we end users get to enjoy the crappy software long after the company has moved on. It certainly does color my decision about which hardware I will buy in the future, though.
Yea I wouldn't argue that ABI software sucks, but it's a useful stick to beat management with. I'm not sure it's got much better... They've not open sourced the primary data analysis or device control software for the SOLiD sequencers (which suck hard). From what I could see a lot of the device stuff uses messy perl scripts.
I really wish they, and Illumina the current next-gen leader, would open source their software (both device control and primary data analysis). If that were the case I'd be refactoring it, adding unit tests etc. right now.
Re:Anarchy is an opportunity (Score:4, Interesting)
Excellent point, if you present yourself as the man with the solutions, and especially if they promote you as such, you're going to be taking the heat when things do not go perfectly.
The thing about rigid development and testing processes is that they delay releases. You can do featureful but buggy from-the-hip releases rapidly, or you can do rock solid heavily tested releases very slowly.
It's the old speed, quality, cost diumverate. Sounds like today they're choosing speed and cost, and quality is there only because your product isn't yet so mature that regression bugs aren't common.
Companies starting a development methodology need time to adjust to the reduced agility that these require. It's best to work your way into it.
Start with introducing source control and some formal testing, along with build releases. It's not very likely that they're going to reject you on these things as their benefits are fairly clear and they shouldn't have that much impact on your bottom line. Create a branch per release, and suggest that the software not be released to manufacturing until that release passes formal testing.
Later you can introduce things like regression testing and a proper software development lifecycle.
You have to be careful how you present it. Make sure they're aware that you're easing the company into a larger methodology, and until fully implemented there's still going to be gaps for bugs to fit through. Explain that during this process you'll be performing gap analysis on each bug that does appear, identifying how it got through testing, and adjusting the process as necessary to ensure that such a thing is less likely in the future.
You don't want to present it as a silver bullet out the gate, you want to be sure to explain that it's iterative and the objective isn't immediate perfection but continual quality increase. Even if you have 30 years of software development lifecycle managerial experience, you can't convert a company to a full-blown lifecycle overnight, and depending on how a company works and what its needs are, lifecycles will in fact differ a bit from company to company. Even if a perfect lifecycle existed, you still need to give people time to get accustomed to it, so you can't just throw the switch over night or everything comes to a halt.
proactive synergy and high positive visibility (Score:3, Interesting)
Some people here will tell you to start dropping managerisms (like those in this message's title) and talk costs. They are correct, if you want to move into management. If you want to stay a programmer, however, just fix the damn problem. Nothing you described is too terribly difficult to correct on your own. Install and use Hudson [java.net]. It has plug-ins for .net and java language support (and probably more). Make sure you really use its code quality plug-ins (things like fxCop, findBugs, and PMD). In short, do al little every project to improve the development environment. These are free tools and fairly trivial to set up. Getting your environment right is part of your job as a developer. Don't abdicate that responsibility to management, especially if management doesn't understand what your development environment needs.
It is a fact of life that most non-software companies have not yet woken up to the emerging criticality of their software divisions. What you describe isn't surprising or unusual. Be better than 70% of your peers by fixing the problems as you see them. You will learn to be a better developer and management will learn to appreciate your efficiency. If they don't, so what? Move on with all the knowledge you gained building up their development infrastructure.
So That's Why Instrument Software Sucks... (Score:4, Interesting)
I've been a research scientist for, well, long enough and a computer nerd for a lot longer. I've always wondered how it is that a company can make a state-of-the-art piece of research equipment and then bundle it with a PC running Windows 95 with a serial interface and the worst, least intuitive, and most expensive software I have ever seen.
Now that I'm in charge of picking what we buy I make it a point to find companies that support Linux because it usually means they have a real software development team and that they don't outsource their development. (Jovin Yvon who bought up all their competitors were the worst at this, charging $5,000 to re-install their fluorimeter software, the installation CD for which they refused to sell based on "licensing" issues.). And I'm usually right.
Most recently I purchased a $70,000 high-speed InGaAs camera that came with Windows-only software (that wouldn't run in virtual machine either because it required low-level NIC access). They charged another $2,500 for the "intermediate" software package (which I have no problem with in itself) that had a bug in it. The bug? It wouldn't export movies longer than 10,000 frames, which at 1,000 fps is 10 seconds of video. It took them three months to get me a beta version of a different piece of software that would allow me to export longer movies.
Another company, which I love doing business with so I'll mention them here, EPIX Inc., makes less expensive high speed cameras, but develops their software in Java and releases Linux versions. The software is buggy--as is all instrument software--but I can actually call a real software developer on the phone and tell him/her what the problem is--imagine that!
Sorry for the rant, but this story makes it clear to me why instrument software is so terrible. The first major company that figures out how valuable intuitive, functional, cross-platform, (and for the love of god, not hardware-keyed) software that doesn't store data in some unholy proprietary format that physically ties people to the machine that the instrument is attached to just to process the data is... Ok, well the few companies (I'm looking at you Bruker) that have figured that out have staying power and brand loyalty in university research where as the others are used as pejoratives.
My erstwhile employer had a similar problem (Score:4, Interesting)
Let me tell you a little anecdote which was relayed to me by the IT manager at a previous employer.
When he came to the company, software quality had never been taken particularly seriously. They'd insourced IT where previously everything was handled by an outside company, presumably in the hope of getting better quality services for their money, but were seeing little benefit - mainly because the IT department was so busy implementing new features the business wanted they never had time to debug existing issues.
Helpdesk call levels were very high, the IT department wasn't particularly highly respected in the rest of the business and while the business probably did want less buggy software, they were always too busy chasing after the Next Big Feature to allow the IT department to concentrate on bugfixing.
So he went to the business (ie. the directors/senior management) and said "OK, here's a suggestion. We'll spend the next three months working on nothing but bugfixes. No new features. What glitches with the system are impacting your staff?". The business wasn't hugely keen on the idea of no new features for three months, but he was able to persuade them that the benefits of having more stable software outweighed this.
Three months later, the business was so impressed with the improvements that they asked if the IT department could spend another three months doing nothing but bugfixes.
Sometimes, the business needs a little poke from IT to understand how to get the best benefit from the IT department. Being able to recognise this and make a suitably diplomatic poke is what IT management is there for. If there isn't clear IT management in place to make such a poke - well volunteered.
Re:Anarchy is an opportunity (Score:1, Interesting)
The bad news is that it's because you're about to get fired.
At least that's what happened to me after I posted a question to Slashdot asking for information from my peers to help me help the organization I worked for. The sorts of people who rise to the level of CEO or President view this sort of thing as "insubordination". Mine practically held me responsible for every negative comment posted in reply about my superiors, as if I'd made them myself. God help you if anyone reads between the lines and figures out what company you work for (as happened to me), because airy its dirty laundry in public like this is "corporate sabotage".
software must be ready when product ships (Score:4, Interesting)
In most companies where software is not the main focus, the deadlines are for the main product and the software had better be done by that deadline. The software guys have nothing to say about the schedule.
I saw this effect first-hand at Digital Equipment Corporation. Although we did a lot of software, the corporate culture focused on hardware, so the software had to be ready when the hardware shipped. Fortunately, the software guys were generally able to get a prototype far enough in advance of "first customer ship" that it was possible to get the drivers written. From a software point of view the hardware documentation was sometimes lacking: once I recruited the assistance of an in-house field service guy to help me read the engineering drawings so I could figure out how to program a new device.
Re:Practice What You Preach (Score:5, Interesting)
Rarely does the statement "You've got a problem and you need to solve it" ever get a good response.
If you say "We have a problem, this is how other people solve it, and this is what I will need to solve it. Give me the budget and I'll solve the problem." then you are vastly more likely to get what you want. Then you'll have to deliver, but if you are like me (not that I am the best way to be), you'll find the responsibility gratifying and the deadline invigorating.
Therac-25 (Score:2, Interesting)
http://courses.cs.vt.edu/~cs3604/lib/Therac_25/Therac_1.html [vt.edu]
In summary, you're fucked. Get as far away as you can before the lawsuits come.
(And yes, I know that he isn't making therapeutic equipment - like that matters in today's legal climate)
I work for a Very Large Non-Profit with a biomedical branch, and they've been working on a new software system for years. After hundreds of millions spent, they still can't figure out how to process a lot of database transactions quickly. When asked why banks can do this with ATM data and not lose a single penny, management response has been "We're the Very Large Non-Profit Organization - we're different". They finally asked the project leader to retire (God forbid they fire him for incompetence), and are valuating whether to scrap the whole thing and start fresh, a la the FAA and IRS.
And no, I haven't quit.
Yet.
Re:Different theory (Score:1, Interesting)
There was probably some 'poor performance' underlying this that was not his fault, but that could have been fixed had his reccomendations been followed. Because shit rolls downhill, he got the blame. ( It very well may be that his reccomendations implicated those who fired him in being at fault )
As for spending more time on presentations than work, he should keep it up, but only with a view to appearing in charge, and without saying anything that would actually help. Really suggesting changes that could make you enemies is not the best strategy. Just do the presentation that explains the status quo, and fill it with buzzwords. Then you will appear to be the go-to-guy. Soon you'll be manager of something.
The more time you spend on red tape the better. When that's 100% what your job is, then you'll be making the big bucks. If you spend time on doing productive labor, then that's what you will be - labor.
Spending time doing labor as opposed to composing bullshit means that while your peers are busy making powerpoint presentations, going to meetings and being visible to higher ups, solving their problems ( by sending you an email to fix something ), you are drudging away in obscurity. Nobody who matters even has a clear idea of what you do, except when something goes wrong. Then you get to explain the problem. While you're fixing it, someone else is spending the time more wisely composing a power point presentation on how to prevent this sort of problem in the future. You thank them, because if they didn't do it, you would have had to in addition to fixing the problem which already ate half your weekend.
Some of us retch at the bullshit, but we end up being the bitches of those who revel in it.
Re:Opinion from the outside? (Score:3, Interesting)
Could you propose to hire a software test consultant for a day or two and let him point out serious quality issues (data integrity, security, correctness..)?
A serious, alarming report by an external software test professional could help reinforcing your requests?
Won't work. Obviously, the CEO doesn't actually care about this issue. I've been there - except it wasn't for a biotech company, it was a small software development company. The execs there were content to allow the development team push code to production whenever they cared to. The dev team was also responsible for production support. As the lone test/QA person, I had zero pull. "that's the way we've always done it" was the mantra. We were pretty good on security, but code quality was awful. We had minimum weekly production pushes, which started at 10:30 PM on a set night... everyone would log onto our IRC channel, and the push would begin. It was like monkeys trying to F a football in the dark. Mistakes happened all the time. Many times we were up until 2, 3 AM.
Everyone complained, but nobody wanted to do anything about it. In the company I was in, the executives were a bunch of used car salesmen. Quality didn't matter, but the promise and appearance of quality did. If your situation is anything like mine was, I have two words for you: get out. You may think you can be the person to turn things around, and make things better. But if the CEO doesn't want to change the company it won't change. My situation was different because it was private and the question here is with a publicly traded company. Maybe there's a difference.
But don't try to teach a pig to sing. It wastes your time, and annoys the pig.
Re:Certification (Score:3, Interesting)
Yah, but the FDA takes Section 201(h) of the FD&C act to define software as "a device".
It doesn't even have to be controlling hardware, things like a blood bank inventory database gets covered since if it screws up, then actions happen that are covered by the FDA(bad/wrong blood gets used).
Look at some of the letters in the other AC's google query, the FDA will come down on you for things you'd think are barely related, like your bug tracking software not requiring you to reference and collate similar incidents.
what your company must do (Score:3, Interesting)
The comments in this list about federal law requiring a quality system *including software quality procedures* are correct. There is no way out of this and the company has a tiger asleep in engineering. The reason the omission has not surfaced is that FDA's budget has prevented them from auditing deeply enough - yet. They haven't been able to send auditors with enough software background to be able to detect the absence of the expected levels of software QA. They definitely have the qualified people, just not enough of them.
An additional reason could be that the product/service has not hurt anyone, or if so, the incident(s) have not been reported - which is another federal law incumbent on the manufacturer AND the hospital/clinic/doctor. FDA audits and warnings can come any time if enough of these reports stack up. Or if the docs send them in and the company does not.
Even if the code is really good and no medical problems have come up, that will not stop FDA from pulling your product off the market if they find you non-compliant with their regulations.
So the company has 'enjoyed' a prototyping phase. Once the management has read the FDA regulations on the personal liability of the company officers, they will probably want to get started with the formal software QA system right away. It doesn't have to be completed overnight. But when/if FDA look deeply enough into your company, you would want them to find records of your diligent work in building up the software QA procedures and practice in an ongoing and steady way. And doing the right things first.
Are you out of your mind? (Score:3, Interesting)
Are you out of your mind? By going to the CEO with a quality problem, YOU are the one who is going to be fired when the flaws in the jury-rigged software cause a major problem in the company profits. You tell me that I'm paranoid and crazy. Well, sir, this is the way that American management works. You understand technology; they don't. They understand corporate politics, backstabbing, and manipulation of employees for their own gain. You don't. The fact that you went for a one-on-one with CEO proves this.
The only time that a mid-level technical employee goes for a one-on-one with the CEO is when you are setting up a fellow employee to get fired over a problem that you caused. It's a classic 'cover-your-ass and make someone else take the blame' type of situation. Which, if the situation really is what you say in your company, is not far in the future. By going to the CEO and discussing the situation, the CEO is going to assume that YOU caused it and are doing a preemptive bid to pass the blame on someone else. When the major problem does occur as a result of the unstructured software, the CEO is going to agree with all the other people in your group who are going to put the blame on you to save their own jobs. And you are the one who is going to be fired!
You should get another job in another company as quickly as possible. And, my friend, never go to management with a problem that has its underlying cause in someone's lack of management ability. It's corporate politics rule #1.
Re:Practice What You Preach (Score:1, Interesting)
I work at a fairly large blood banking/research company.
All of the manufacturing software that we use is considered to be a medical device by the FDA and as a result, requires a complete change control process that is coupled with a waterfall development methodology. This came from a time 'back in the day' where we wrote our own blood banking software in a mainframe environment.
We don't write blood manufacturing software anymore (executive decision), but we do write an awful lot of business intelligence and financial software, the development process has continued to live with company and, aside from attempts to implement a more agile process, serves us fairly well. Of course our busniess people howl at the need to develop real requirements and it more than doubles the duration of alll of our development efforts, but it creates a stable environment that I would not want to live without.
FYI, compliance to federal regulations (see how much your company should be complying with 510(k) regulations) can be a great way to get visibility with the C-levels folks and bean counters because the only other option is to go out of business.
Create two presentation slides (Score:4, Interesting)
1) We create software.
2) Software is used in medical devices.
3) We forego QA and industry best practices for software development.
4) Something goes wrong.
5) We get sued AND we lose (to the five-nines sure).
6) Change #1 to "Update resume".
SLIDE B
1) We create software.
2) Software is used in medical devices.
3) We follow industry best practices for software development and have a solid QA program.
4) Something goes wrong (yes, it still happens).
5) We get sued.
6) Our controls and best practices are a reasonable defense.
relate it to legislation / federal regulations (Score:4, Interesting)
One route to making people serious about IT processes is to relate it to relevant federal regulations.
For example, we've been doing work that will eventually involve us as a partner in upcoming clinical trials. There's a bunch of federal regulations about IT processes connected to clinical trials, and it has been easy to get management to accept that while our current processes can be as ad-hoc as we like, at some point having compliant processes will be essential to continuing the work we do, so we may as well get it right the first time rather than have to reimplement years worth of ad-hoc development somewhere down the track.
In my case, one trivial example has been being able to implement gpg signing of documents as a consequence of setting up the infrastructure to be able to be quickly compliant with 21 CFR 11 [fda.gov], which we'd need to do if we're part of a clinical trial.
Documentation is not optional - Federal Regs (Score:2, Interesting)
I work for a large biotech's regulatory department, and my job is to ensure we remain compliant with federal regulations vis a vis software development.
I want to underscore something that electroniceric touched upon: You work in a regulated industry, and your firm's practices are not compliant with federal regulations.
Your biotech firm's lack of SOPs and BPs on software development is in direct violation of the Code of Federal Regulations. If the FDA audits your firm and finds this lack of compliance to 21CFR820, then the FDA can issue a Warning Letter, which will absolutely have an adverse effect on the company. Warning letters always hurt the company's stock price; investors take WLs as a sign that the company is being mismanaged.
If your CEO doesn't want to believe this, then he or she shouldn't be in that position.
FYI, here's one example of a medical device company that received a warning letter from FDA for failure to properly document software development:
http://www.fda.gov/foi/warning_letters/s6752c.htm [fda.gov] [fda.gov]
But Warning Letters aren't the worst thing that FDA can do. For failing to follow CFRs, FDA can determine that drug products (small and large molecule) or med devices are misbranded or adulterated and can demand the company recall said products. This will destroy a start-up or small company....
Good luck with this issue.