What Makes Software Development So Hard?

lizzyben writes to mention that CIO Insight is running a short piece that takes a look at why the rocky culture of software development continues to exist despite all of the missed deadlines, blown budgets, and broken promises. From the article: "I was not really looking or thinking about big software projects. I was just coming out of my experiences at Salon, where we built a content management system in 2000, which was painful. I was one of the people in charge of it, and when the dust cleared, I thought, I don't really know that much about software development. Other people must have figured it out better than I have; I must go and learn. So I started reading, and talking to people, and realized it's a big subject and an unsolved problem. And the bigger the project, the harder the problem."

For Me

[KermodeBear]

What makes software development so difficult for me in my particular case:

1. Requirements are not clearly defined
2. Requirements that are defined change constantly
3. No existing documentation on the system I work with
4. Outdated technology (Not a big issue, but many things are just easier to do with newer software)
5. The sales department sells features that do not exist, promise a date, and do not consult IT at all about the feasibility or time estimates

4 out of the 5 things I have listed above have to do with bad information / lack of communication.

Re:It's design not development

[truthsearch]

The last project I was on which had requirements that didn't change at all during development was 7 years ago. And it was built right on schedule. For me it's been the ever-changing requirements which delay projects the most. The other half of the problem is usually bad estimates, but that's almost irrelevant after the requirements change.

Re:good question

[larkost]

You are close, but the real piece you are missing is that most of a conductors work goes on long before you see them up on stage conducting. The real work is in music selection (you have to know your orchestra and what its strengths and weaknesses are), properly managing rehearsal schedules (what groups should meet, and what they should work on), choosing the right players and the right section leaders (the latter is both a delegation and a political art), and the actual concert conducting is much more show than anything else (unless things start to go wrong).

I know a player in the Philadelphia Orchestra and he tells me that they largely ignore the conducting that happens during a performance (granted that conductor's stick work is impossible to precisely read). They know how he wants the piece played because of what they did in rehearsal.

Oh... and I do have the experience to comment on this, I was in orchestras for 12 years.

Re:Where software developers sell themselves short

[Grishnakh]

I think one of the biggest problems in our industry is accountability. The engineer would never put the unsafe option on the table, because the engineer knows he'll loose his license and go to prison if the bridge collapses. With software, on the other hand, we just expect our customers to deal with the fact that it fails, and we behave accordingly - and unprofessionally.

You seem to be missing something: software engineers are not "professionals" in the sense that civil engineers are, and they're not licensed. There's no liability to a "software engineer" if anything happens, only to his employer.

You can't say "no" to your boss and expect to keep your job long. Bridge-building engineers don't have that problem because they ARE the boss, just like doctors don't work for bosses as they themselves are the boss.

You want to make software engineers accountable for their work? You'll need to mandate government licensing, and then you'll need to pay those developers 2x-3x as much because they'll have to have malpractice insurance. Somehow I don't think any software companies want this.

Re:It's design not development

[smellsofbikes]

>we've been building bridges for thousands of years

And despite that, they still fall down. [bris.ac.uk]

Read Henry Petrowski's book, "To Engineer Is Human" some time. The point of the book is: an engineer, whether bridge or software, is building a compromise between deadlines, money, user requirements, and laws of nature. As time marches on, that compromise will *always* tilt towards deadlines and money, and eventually it will break, at which point we've learned something for next time.

Re:good question

[nigelo]

It is certainly true that if you don't know what it is supposed to do, it'll be probably be very hard to figure out how to do it.

But don't overlook that it is human nature to say 'I don't really know how this works, but how hard can it be?' (to be an orchestra conductor...) 'No matter, let's do the parts we know, and get them behind us' thus nicely taking up most of the time allowed, leaving little or no time for the little-understood-but-hope-it's-easy 'then-a-miracle-occurs' piece that never got sized or included in the estimating/planning, etc. This (rightly) leads to the claim that 'the hard parts didn't fall into view until after most of the work was done'.

Again, do the hard parts first - if you don't know what it does or how it does it, consider it hard. If you go over budget on the early stuff, at least it was on the hard parts, and maybe your team can subsequently pull off the implementation of the easy parts in less time than you originally thought, especially now that they've learned how to do the hard stuff.

Either way, it'll be easier to deliver the latter parts under the pressure at the end of the project, particularly if you've overrun budget/schedule.

Re:good question

[Matje]

The reason you attack the hard problems first is that you have less money invested if the hard problems turn out to be unsolvable (within your time/money/skill scope).

The big difference between a school test and a software project is that you only need to answer as many questions correctly as necessary to get a passing grade (you can thus skip the hardest questions), whereas in a software project you must solve the hard problems if you want to complete the project.