What Makes Software Development So Hard? 567
lizzyben writes to mention that CIO Insight is running a short piece that takes a look at why the rocky culture of software development continues to exist despite all of the missed deadlines, blown budgets, and broken promises. From the article: "I was not really looking or thinking about big software projects. I was just coming out of my experiences at Salon, where we built a content management system in 2000, which was painful. I was one of the people in charge of it, and when the dust cleared, I thought, I don't really know that much about software development. Other people must have figured it out better than I have; I must go and learn. So I started reading, and talking to people, and realized it's a big subject and an unsolved problem. And the bigger the project, the harder the problem."
What large projects DON'T have problems? (Score:5, Interesting)
One wag of my finger, one tip of my hat (Score:5, Interesting)
First the wag: The idea that the interveiwee states early on, that software development is not introspective, is horse-hockey. We think about it all the time. We invent new, clever ways to diagram software, capture requirements, interview users, validate functionality and come with all sorts of certifications. More than, say accounting, we are process focused people. Maybe our processes suck, but we spend a lot of time, energy, certification exams, etc, on those processes.
Tip of my hat: He does mention starting small and iterating. I think that's the best way to build software.
Re:It's design not development (Score:5, Interesting)
You can go to school to learn to be a bridge builder and come out of it knowing all the exact specifications to build a bridge and probably design a fairly good bridge, or with a bit of creativity and some extra architectural skills a really cool bridge. Software design isn't really taught in this manor, sure your taught how all the bridge building tools work, and even a lot of the engineering specifications. But I have yet to see the software design school that covered more than a class or two into truly how to design software. Then again, we've been building bridges for thousands of years, and designing software systems for a few decades. It does take time for these things to really get figured out.
Complexity & Culture (Score:5, Interesting)
The first reason is the complexity of the systems. There are essentially extremely many interacting parts that cause all sorts of emergent phenomena. This is a field that we don't really know how to handle very well - good quantitative models of complex systems simply don't exist. We need more study of systems theory and chaos theory to build some form of predictive models on which in turn can base planning.
The second problem is a sort of 'freedom of thought' culture that sees coding as the vaguely mathematical expression of ideas. As there are a huge number of degrees of freedom most attempts to regulate it in practice have been abandoned. So instead of just the problem of describing a mathematical problem, you throw individual human preferences, thinking and biases into the equation.
Of course it can't continue that way forever and we need to move to a meta level of coding. We do have well-constructed systems, such as the software on-board an airplane - but they are laughably primitive (because you have to account for all possible states the software can get in to). And we have advanced software, but it is laughably unreliable.
Still, there is reason to be hopeful. We're not inventing the wheel every time any more (just the horse carriage and so on). Modern programming systems have extensive class libraries that are on average a more stable foundation than making a custom implementation from scratch every time.
Ultimately however we need to know how to handle complex systems and how to enforce convergence/stability to systems where the huge number of possible combination of states can be analyzed. And sad as it is, if we want it to be reliable, the liberal individual approach to coding has to be abandoned in favour of a much more strict industrial way of thinking. Right now we have handcraft and we need industrial precision, standardization and quality.
Re:It's design not development (Score:2, Interesting)
Of all the Renaissance painters that ever painted, only a few became forever famous.
Same thing with coders.
Short Answer (Score:5, Interesting)
Software is still a Science and not an Engineering practice.
As long as the design can also be the implementor and estimates based on actual analysis are optional, Software will NEVER be an engineering study.
These aren't changing quickly for what I believe are a few reasons:
IEEE has not created a Professional Engineer - Software and noone really wants them to.
Companies don't like to be told they have to hire something that sounds expensive to build something they cannot see.
Companies will NEVER open their software to outside inspection the way construction companies must open their buildings because of the concept (flawed, I think) of Intellectual Property.
If a company had to have their software inspected by a Software P.E. before using it in production or selling it to end users - If Software P.E.'s had to adhere to standards which included concrete estimates and testing - If companies were not allowed to use anyone they could find that has seen a computer to write their software... commercial software development would be much further ahead.
Do I believe any of this should happen? No. Why? Because I want it to continue to fail. I do not believe software development should be put on a pedestal or only performed by "experts". I believe shoot from the hip software projects allow open source software projects to exist and to succeed in the market.
Open source works without accurate estimates because the contributors can flock to good projects and don't have to adhere to a labour budget. Company employees can't get wind of the cool software project and leave the crappy one's - corporate structure and budget's won't let them.
I don't believe companies with more than 120-150 people are stable - once they breach this range empire building occurs and massive uncontrollable monsters result. If a company truly needs more than 150 people it should split into two and partner on the project at hand. I believe this is a human condition - humans work best in tribes where they can personally know all of the members.
All of this might be completely and utterly wrong. But it's my hypothesis.
Gap between computer science and person problems (Score:4, Interesting)
Every tool has its own problems. One problem is accounting, or keeping track of money. With pen and paper, you can run out of ink or paper. Humans aren't good at adding numbers in their heads. With computers, you can completely erase all of your work with a few clicks of the mouse or keyboard. So there are ways that problems inherent in the tool can emerge, which take energy and attention away from solving the pre-existing problem.
Currently with computers we are trying to abstract problems such as banking and business into simple logic puzzles. I think that's too much of a simplification. I think we need to create a virtual world full of basic human-percieved concepts, such as time, weather, humans, animals, etc. and create programs by manipulating those basic ideas and objects.
An example is an ontological system like OpenCyc. An ontological system holds hundreds of thousands of logical assertions like "Animals eat Food" and "Paris is the Capital of France". Basically, an ontology system has some basic common sense. From all of these assertions, it can make logical conclusions. So, if well tell Cyc that Duke is a Dog, it can conclude that Duke has a tail and eats food. If we tell it that Duke lives in Paris, it knows that Duke lives in France.
Now imagine, instead of dealing with animals and where they live, it has a bunch of assertions about generally accepting accounting principles. One day, you might be able to just sit down and talk with an ontological system via email or IM, and say, "We got a check from client A for $575, another check for $440." and then the computer balances the books with all the other accounting principles it 'knows'.
Current programs seems to be exclusively a digital re-creation of paper-based forms and filing cabinets. It's a sheet of paper with a bunch of field:value pairs, and reports are the resulting logical operations you can do with all of that data. This is *basically* the relational database. I think we are hitting the limit of the field:value model of reality. I think there are other models, such as virtual realities like online worlds, knowledge systems like opencyc, etc.
Programmers are working exclusively at too low of a level. Yes, of course, we will always need to teach and understand basic boolean logic and computer science terms, but we need to start working at higher-level, human friendly concepts.
Lack of a specification language! (Score:4, Interesting)
Home builders have architectural plans. Machinist have blueprints. Electronic equipment builders have schematics.
Software engineers are stuck trying to figure out the incoherent ramblings of marketing/sales/business analysts/corporate executives/users and a host of others who have no means to specify what they are asking for.
Software specifications are uniformly deplorable.
No you don't, you silly manager troll. (Score:1, Interesting)
Too much Management and Systems *ARE* what makes software hard.
Bay Bridge (Score:3, Interesting)
Re:Where software developers sell themselves short (Score:3, Interesting)
When you're building a bridge, there's a base requirement: something can cross it. Now what can cross it, how fast, structural integrity, etc, all play a factor in the specifics, but at the end of the day, either something can cross the bridge (safely / quickly / profitably) or it cannot.
If you were told to write a program to add two numbers together, and that was all, you'd have a pretty easy time rejecting "hacky" options, too - because you know all of the requirements.
But most of the time you don't, or you misunderstand them, or (in the classic case) you offer up something and the project suddenly takes a dramatic turn because you did one little neat thing in it and they want the whole system programmed around that tiny widget (at the expense of all others).
Comparing software to a bridge is useless. A bridge is like one specific program - a calculator, or a no-frills text editor. Building software is more like urban planning. Now go ask engineers how many times they've screwed the pooch on that one.
It does not have to (Score:2, Interesting)
- Testable means that each and every module can be tested independently, which normally implies no side-effects (not an absolute demand, though). I always write a test section for each module and make the tests cover as many parts of the code as possible (hopefully all sections of the code, but for slightly complex modules this may be infeasible or even impossible).
By sticking to these rules and by documenting the code thoroughly, coding does not have to be too difficult. These rules/principles, I know, are part of some of the major design methodologies, but you do not have to make it more fancy than this to make it work. The above principles are very hard to retrofit into existing (legacy?) code that was not built this way, but extensions could still be made in a modular way that, if not to the letter, then in principle, adheres to the rules.Re:It's design not development (Score:3, Interesting)
I suspect a lot of programmers are fully aware of that (though I would agree with another respondant that "craft" might be somewhat more accurate), but that a large number of managers (mainly those who don't have a software development background themselves) don't fully understand the truth behind the statement.
Software development is still a subjective process in many respects, and I'm not completely convinced that it's a bad thing. Sometimes the effective expression of a radical idea requires a solo programmer who just happens to have a brilliant insight or even just a vision of something.
Writing software isn't like building a bridge -- there are often a fair number of incompatible but equally elegant solutions to a give problem, and sometimes the elegance is readability and supportability rather than cute tricks in the code (which might legitmately save clocks or bytes but make also make life miserable for some unsuspecting support guy five years down the road).
Re:Mod parent up! (Score:3, Interesting)
Though the comment was party in jest, though I do believe that it is a prevailing issue. I believe there are far too many developers in the industry that are not competent in their ability to write quality software. This is evident in many developers adhering to patterns for the sake of using pattern when the patterns do not fit their needed problem set. Web programming is saturated with this kind of mentality; this is evident in new frameworks coming out weekly that rehash the same old problems and poorly I might add. Your comment about development being an artisan type endeavor might be more true than anyone wants to admit. The business side of the industry is scared of the idea that software may not be able to be manufactured. Crafting conjures the realization that each peace is unique and unlike an assembly line, carries unpredictability and risk as it cannot be quantified until it is completed. This takes away businesses ability to look like they are in control and masters of the software domain and makes people who otherwise feel as thought they are all controlling realize that in fact they have no control nor can they even begin to comprehend what needs to be done to control software development. Yet everyone goes on quietly knowing this and not saying anything because no one wants to look like a jackass. So the business goes on acting as though they are in control while not treating the process as a creative one because no one wants to admit that software cannot be manufactured. The sooner companies realize this, the better off the industry as a whole will be. Developers will be looked at as skilled contributors and not an cog that are replicable by the next guy with the same skills on the resume, Timelines will be more realistic and business personnel will realize that in fact they have no freaking clue how to manage a development environment and that they had better get someone who does know.
Re:Where software developers sell themselves short (Score:5, Interesting)
Can the sum of the numbers exceed MAXINT? MAXINT on what platform? Should overflows set a flag, throw an exception, or be reported in-band?
Can the numbers be floating point? If one is much smaller than the other, how large a fraction of the smaller number's precision are you willing to lose?
Can the numbers be multiple-precision? Is the multiple-precision library you're using compatible with the rest of your system? Does it fit into your memory requirements?
Do the numbers come from some real-world source that need to be checked for sanity, for example the output of a sensor?
In what form does the output need to be? Encoded in ASN.1? BCD? IBM "unpacked decimal"? String? If output is variable length, is there a maximum size that must be enforced to prevent buffer overflows? If the maximum size is exceeded, should the result be truncated, or should the function throw an exception, or
The sickening thing being that I've probably left out some important issues. Like thread safety.
Look up the bug history of "IEFBR14", originally a single return instruction that required four or five revisions to meet shifting requirements.
Re:Two things make software "hard" (Score:3, Interesting)
In a prior job I was asked (in front of everyone in the boardroom) to estimate how long it would take to do a project which had just come up and only been discussed for 15 minutes or so. When I said that I would need some time to spec it out I was told that would not be possible.
After thinking about it for a minute or two I decided that 90 days would be reasonable based on what I knew of the specs at the time and my current work load but that I could manage 60 if I could put this one first on the list of priorities.
I was then told that 60 was too long, how about two weeks? I finally compromised at 30 days with overtime and absolutely no other work on anything else in the company.
I finished the programming on the evening of the 30th day, went home and wrote up some instructions I could use for training the people who needed to use it. First thing the next morning I was taken to a conference room by my boss and literally *yelled* at because nobody was using the program yet. (Thisw as the 31st day.)
I explained that the program was done but I had to train the users that morning so they could be operational in the afternoon. (Only about 10 people needed to use it right away.) This was dismissed with a *shout* of "Nobody is using it, therefore it is not finished!" He considered the program a failure.
Do you see anywhere that management could have been at fault for the project "failing" here? It's all too common to have management fall into the Capt. Kirk role. (Scotty: "I have the engines at 100%", Kirk: "Damnit Scotty, I need more power!") Maybe it's the techies who do screw around that make them think the rest of us just need to be pushed harder to get better performance out of us. I don't know but I've seen it more than you might think.
BTW, when I say "yelling" I mean it. Face red, blood pressure skyrocketing, the works. BTW #2, the program worked just fine starting late that afternoon after staff were shown where it was and how to use it. BTW #3, that was the day I decided to find a new job and stop blowing off offers from headhunters. (I was somewhere else with a 40% pay raise three months later.)
Re:It's complicated (Score:4, Interesting)
1. Clients rarely know what they want. Most software projects are designed and written for someone else's requirements. These requirements often come in the form of "I don't like that, but I have no idea what I actually want."
2. Coders are systematizers, who tend towards the asocial end of the spectrum, which is why silicon valley produces so many autistic and Auspergers's kids. Large projects require communication, and there are a lot of coders out there who may be good with machines but are lousy with people.
3. As a result, lead programmers are often tantrum throwing prima-donnas, intellectual bullies who use their position to undercut those under them and make themselves look superior. You may have to promote and fire half a dozen programmers before you fill the lead programmer position with someone who is actually good at the job. In the meanwhile, you've just lost 5 good programmers. And if you actually manage to find a good one, you'll have to pay him in blood, because he's probably worth more than your house.
3. Salesmen, who deal with the client, are professional bullshitters, who are generally wafer thin on the technical details. That's actually not a slam--this is literally their job--to sing whatever lullaby the client wants to hear. Programmers will generally learn their lesson after being burned a couple of times and will not promise the impossible. The sales people never get burned--it was the programmers who screwed up, right? Calls for heroics on the part of programmers invariably begin with someone in marketing.
So, a few tips:
1. Get someone to hammer out the details of the specifications before signing the contract.
2. Get someone with some technical knowledge to study those specs, and the deadlines, before signing the product.
3. Get a lead programmer who fights for his team, not for the himself, and who is more interested in getting things done the right way than in getting them done his way.
4. Get a project manager (preferrably female--tends to calm the more socially inept coders, less testosterone poisoning) who handles communication between the coding team and the client. Do not, repeat DO NOT allow the coder to meet the client. They will eat the client! You will have a hard time getting paid, and hiding the bones is a bitch.
Re:Mod parent up! (Score:3, Interesting)
For this to do any good, you'd need reasonable criteria upon which to base the certification. IMO, TFA was mostly correct in pointing out that right now, we don't know what those criteria should be. The Mythical Man-Month remains relevant today largely because relatively little progress has been made in solving the problems it explored. As Brooks pointed out in No Silver Bullet, we've cured a lot of the accidental problems of software engineering, but done relatively little about the essential problems.
In terms of the certification itself, we're just about as lost. The correlation between formal schooling and success as a software engineer is weak at best, and quite possibly negative. Experience indicates more, but still very little. Certification based on such criteria would do more harm than good.
There are other problems as well. With buildings, it's pretty easy for most people to realize that "skyscraper" and "dog house" have different requirements as to the level of expertise involved in their design. In software those differences often aren't nearly so obvious. The managerial people who'd need to enforce the requirement for certification don't know the difference between a skyscraper and a scale model of one for a train set (and sometimes the developers don't either). "Web developers" (in particular) seem prone to jumping (metaphorically) from doll houses to 18-lane suspension bridges in high-wind, earthquake-prone areas.
Re:good question (Score:3, Interesting)
Re:Where software developers sell themselves short (Score:3, Interesting)
As the old joke goes, Microsoft made IIS susceptible to worms like Melissa and Code Red, costing billions of dollars to companies around the world, and didn't have to spend a dime in restitution, while Intel made the Pentium with a floating point error that affected a handful of people doing extremely precise simulation work, and had to spend a small fortune recalling chips.
Even if we just had software liability, we'd see a much greater focus on quality in our industry. Although we tread a thin line trying to introduce that concept without having horrendous effect on open source.