Donald Knuth Rips On Unit Tests and More

eldavojohn writes "You may be familiar with Donald Knuth from his famous Art of Computer Programming books but he's also the father of TeX and, arguably, one of the founders of open source. There's an interesting interview where he says a lot of stuff I wouldn't have predicted. One of the first surprises to me was that he didn't seem to be a huge proponent of unit tests. I use JUnit to test parts of my projects maybe 200 times a day but Knuth calls that kind of practice a 'waste of time' and claims 'nothing needs to be "mocked up."' He also states that methods to write software to take advantage of parallel programming hardware (like multi-core systems that we've discussed) are too difficult for him to tackle due to ever-changing hardware. He even goes so far as to vent about his unhappiness toward chipmakers for forcing us into the multicore realm. He pitches his idea of 'literate programming' which I must admit I've never heard of but find it intriguing. At the end, he even remarks on his adage that young people shouldn't do things just because they're trendy. Whether you love him or hate him, he sure has some interesting/flame-bait things to say."

Shocked

He pitches his idea of "literate programming" which I must admit I've never heard of

I'm shocked to discover that Knuth is taking an opportunity to push literate programming, given that he's been pushing literate programming at every opportunity for at least 25 years.

Now, I've no problem with literate programming, but given that even semi-literate practices like "write good comments" hasn't caught on in many places, I think Don is flogging a dead horse by suggesting that code should be entirely documentation driven.

Re:Shocked

Now, I've no problem with literate programming, but given that even semi-literate practices like "write good comments" hasn't caught on in many places, I think Don is flogging a dead horse by suggesting that code should be entirely documentation driven.

To be fair to Knuth, I don't think the failure to write good comments detracts from literate programming. What Knuth wants is an inversion of thr traditonal code/documentation relationship: you write the documentation and embed the code within that, as opposed to concentrating on code, and incidentally embedding documentation (as comments) within the code. Ultimately the failure of good comments and good documentation is because people are focussing on the code; as long as documentation and comments are an afterthought they will be forgotten or poorly written. If you switch things around and focus on the documentation and insert the code, comment-like, within that, then you're focussing on the documentation and it will be good.

The reason I think literate programming doesn't catch on has mostly to do with the fact that a great many programmers don't bother to think through what they want to do before they code it: they are doing precisely what Knuth mentions he does use unit testing for -- experimental feeling out of ideas. Because they don't start with a clear idea in their heads, of course they don't want to start by writing documentation: you can't document what you haven't thought through. This is the same reason why things like design by contract [wikipedia.org] don't catch on: to write contracts it helps to have a clear idea of what your functions and classes are doing (so you can write your pre-conditions, post-conditions and invariants) before you start hammering out code. The "think first" school of programming is very out of favour (probably mostly because it actually involves thinking).

What?

You've heard of TeX, written in Web, the language designed for Literate Programming, but you've not heard of Literate Programming?

I have a lot of respect for Knuth as an algorithms guy, but anything he says about programming needs to be taken with a grain of salt. When he created the TeX language, he lost all credibility - designing a language in 1978 which makes structured programming almost impossible is just insane. TeX gets a lot of praise as being 'bug free,' but that's really only half true. The core of TeX is a virtual machine and a set of typesetting algorithms, both of which are very simple pieces of code (they'd have to be to run on a PDP-10). Most of the bits people actually use are then metaprogrammed on top of the virtual machine, and frequently contain bugs which are a colossal pain to track down because of the inherent flaws in the language (no scoping, for example).

If you want to learn about algorithms, listen to Donald Knuth and you will learn a great deal. If you want to learn about programming, listen to Edsger Dijkstra or Alan Kay.

Re:What?

Amen about TeX (and even LaTeX). I consider myself pretty knowledgeable about many computing languages, but every time I've hit a non-trivial problem with making TeX do what I want, I've had to consult with a TeXpert (i.e. the utterly invaluable comp.text.tex). And, sadly, in almost every case the solution has been either insanely baroque, or there's been no real solution at all. LaTeX makes brilliant looking documents, but Jesus wept, it's hard to make your documents look like YOU want, as opposed to how it thinks they should look.

Re:What?

If you want to learn about algorithms, listen to Donald Knuth and you will learn a great deal. If you want to learn about programming, listen to Edsger Dijkstra or Alan Kay.

For those that didn't read the article, Knuth expressed criticism on several of the questions asked, but he didn't want to just duck the question. For instance, the 'trendy question' he said this.

With the caveat that thereâ(TM)s no reason anybody should care about the opinions of a computer scientist/mathematician like me regarding software development

Programming for the human VM

Literate programming is an old friend for developers of functional programming languages. I see it like "code for the human mind": it provides a source code that is well adjusted to the needs of the developer, not just the machine.

It interleaves code and documentation in the same files, and provides a specialized compilator to tell the two kinds of codes apart. Just like Doxygen and Javadoc can extract the comments from a source code project, the "tangle" process can extract all the code from a Literate program and pass it to a clasic compiler.

Now that C and C++ seem to have a declining popularity [slashdot.org], maybe we can look for better ways of getting away from the bare metal (which, don't forget it, is why those languages become popular at the beginning). Don't get me wrong, they served us well for 36 years, but I think it's time again to begin caring more for the developers' requirements and less for the hardware requirements.

Unit Tests are not wasteful

The reason for his dismissive attitude of unit tests - that he knows exactly how all of his code works, and what impact a change will have - is exactly the reason you need them. In the real world, most programmers do in fact have to share their code with others. You're not always going to know the ramifications of refactoring a particular block of code, even if you wrote it yourself. And if you can keep all of that in your head at once, either your program is trivial, or you are some sort of supergenius. Now while I think the TDD guys are a little bit overzealous sometimes with their "100% coverage or die" attitude, unit testing is still a good habit to get into, regardless of what Knuth thinks.

Re:Unit Tests are not wasteful

Unit tests, especially if written and organized in an intelligent fashion, can be of tremendous value in eliminating small coding errors that were not inteded but are bound to creep in if the project is large enough. If you are clever about your tests then you can usually inherit the same test multiple times and get multiple uses out a test or part of a test. If unit tests are not used then it is more likely that several small errors in seemingly unrelated classes or methods can combine in an unforseen way to produce nasty and unexpected emergent behavior that is difficult to debug when the whole system is put together. Unit tests will not make up for crappy design, but they will help detect minor flaws in a good design that might otherwise have gone undected until final system integration where they could be much more difficult to debug.

I actually have a great deal of respect for Knuth, but I think that he is wrong about unit tests. Perhaps it is the difference between the academic computer scientist and the career programmer who admires the ivory tower, but is willing to make concessions in the name of expedience and getting work done on time.

He's right

He's right about unit tests... sort of. Just as most coders shouldn't be designing interfaces, most coders don't know how to test. It can often be more work writing the unit tests than writing the code.

If you have a function that multiplies two integers, most coders will write a test that multiplies two numbers. That's not good enough. You need to consider boundary conditions. For example, can you multiply MAX_INT by MAX_INT? MAX_INT by -MAX_INT? Etc. With real world functions you are going to have boundaries up the whazoo. In addition, if you have a function that takes data coming from the user, check for invalid cases even if another function is validating. Check for null or indeterminate values. Write tests that you expect to fail.

Re:He's right

The thing about unit testing is that it's subject to the law of diminishing returns. A simple test of the basic functionality gets you a lot for minimal effort. Writing dozens of carefully chosen tests to examine boundary conditions etc. gives you a little bit more, but for a great deal more effort. Whether or not it's worth it depends very much on the situation and the nature of the code you're writing.

In other news, Chuck Norris rips on safety gear...

The headline is misleading. Donald Knuth represents the epitome of the solitary super-programmer stereotype, so it's only natural that he sees no need for unit tests to catch mistakes or extreme programming to improve team development practices. I don't think he's necessarily saying that those things are without value for ordinary programmers.

The Summary Exaggerates the Interview

Knuth said many of these supposedly outrageous things in passing, and does it while noting that he is an academic. Most of these claims in the summary vastly exaggerates the strength of the claims in the interview. Knuth specifically states;

there's no reason anybody should care about the opinions of a computer scientist/mathematician like me regarding software development.

Knuth doesn't claim that unit testing is a waste of time for everyone, just that it is a waste of time for him, in his circumstances. This makes sense, considering he follows his own (diametrically opposed) doctrine of "literate programming", which, if the summary author has never heard of, should cause him to be cautious about interpreting Knuth.

Worst Summary Ever

The summary sounds like it was written by the headline-producing monkeys at Fox, CNN -- or hell, at the Jerry Springer show. Donald Knuth is not "playing hardball." Nobody needs to call the interview "raw and uncut," or "unplugged."

The interview has almost nothing to do with unit testing and the little Knuth does have to say about the practice is hardly "ripping."

When will people stop sullying peoples' good names by sensationalizing everything they say?

Knuth is a well-respected figure who makes moderate, thoughtful statements. From the summary, you'd think he was a trash-talking pro-wrestler.

Re:Did anyone claim the bug prize on TeX?

the prize was not US$1000. it started out very small. Knuth did indeed pay out, and indeed doubled it, several times. From wikipedia: "The award per bug started at $2.56 (one "hexadecimal dollar"[24]) and doubled every year until it was frozen at its current value of $327.68. This has not made Knuth poor, however, as there have been very few bugs claimed. In addition, people have been known to frame a check proving they found a bug in TeX instead of cashing it."

Re:Did anyone claim the bug prize on TeX?

It is probably folklore. But the story during my grad school days was that, Knuth offered 1000$ prize to anyone fining a bug TeX and he doubled it a couple of times. And it was never claimed. If that was true, it is very unlikely he was just flame baiting.

He offers rewards for reporting errors is his books and for finding bugs in his code:

http://en.wikipedia.org/wiki/Knuth_reward_check [wikipedia.org]

Many people save these (usually small) checks as souvenirs. My father -- a frugal mathematician -- received a few $2.56 checks from Knuth, and he promptly cashed each one.

You misunderstand

Using "MULTIPLYBY" instead of "*" is asinine, because both are equally descriptive. Putting a comment above the line telling people why you're doing it isn't.

Re:Literate programming...

... looks like it falls into the same trap as COBOL. The idea that by making programming languages incredibly verbose, they will somehow become easier to use is a fallacy.

Using "MULTIPLYBY" instead of "*" isn't going to make your code easier to read.

From what I've seen (particularly of CWEB), literate programming doesn't change the programming language itself, it just adds a TeX style markup to the comments so that detailed (and nicely typeset) documentation can be generated from the source code. Take a look at some of Knuth's CWEB code, such as his implementation of Adventure:

http://sunburn.stanford.edu/~knuth/programs/advent.w.gz [stanford.edu]

It appears to be ordinary C once the CWEB documentation is stripped out.

Documentation is the source

So basically it's the same as the XML comments you can put in your .Net or Java code to create JavaDocs, or whatever they are called in .Net, based on the comments in the code?

Not quite. In Javadoc (or the C/C++ equivalent) the C/Java code is the source, and documentation is generated from that. In literate programming, the documentation is the source, and it has code snippets, like you would see in a Knuth textbook.

The snippets have markup to indicate when some snippet needs to come textually before another to keep a compiler happy, but mostly this is figured out automatically. But in general, the resulting C code is in a different order than it appears in the source documentation. For instance, the core algorithm might come first, with all the declarations and other housekeeping at the end. (With documentation about why you're using this supporting library and not that, of course.)

Re:Literate programming...

That's not literate programming at all. A tad more research on your part is required. I actually remember when "web" in a computing context a literate programming tool rather than that thing you're surfing right now.

Literate Programming interleaves the documentation (written in TeX, naturally) and code into a single document. You then run that (Web) document through one of two processors (Tangle or Weave) to produce code or documentation respectively. The code is then compiled, and the documentation built with your TeX distribution. The documentation includes the nicely formatted source code within.

You can use literate programming in any language you want. I even wrote rules for Microsoft C 7.0's Programmer's Workbench to use it within the MSC environment.

I've frequently thought about going back. Javadoc and/or Sandcastle are poor alternatives.

Re:Literate programming...

Most people using C++/Java/C# end up writing "} //end while" anyway,

Pray god I never have to work on code written by these fictitious "most people".

I'd kill any colleague of mine who wrote such a vacuous comment. With a golf club, in front of its cow-orkers to drive the lesson home,

I sometimes add comments like that if the brace is closing a deeply nested block, but then the comment indicates which particular loop or block is ending.

Spaghetti-O Code

I somewhat disagree with what you and... *sigh* Monkeybaister posted. Yes, there are many times when long stretches of code should be broken out into functions. But I tend to do that mostly when the same bit of code is used in several different cases. The reason being is that when you start modularizing off all your while loops that are more than a dozen lines long, you create a whole new type of spaghetti code. I'm going to coin a term and call it "spaghetti-O code." You try to track down a bug and what would have been a straightforward couple pages of code now has all kinds of functions at different places in the code. As such, it can often make debugging or forming a mental map of the code much harder.

Re:Literate programming...

I concur. Comments should tell WHY the while block is there and what it DOES. Not where it starts and where it ends, the code tells us that descriptively enough.

I've met code blocks several hundred lines long and it was never ambigious where they started and ended.

Re:Literate programming...

I myself solve the problem using this construct:

#define BeginWhile {
#define EndWhile }

#define BeginFor {
#define EndFor } ...

Re:On multicore

I've been involved with "parallel programming" for 20 years. There is no mystery involved. The C family of languages are particularly ill suited for parallel programming which may be why we are seeing reports recently claiming that it is "too difficult". Pointers in particular make life difficult for both the programmer and the compiler.

There are a few techniques to be mastered and using a language designed with parallelism in mind helps hugely with the picky details.

Maybe More Like 50-75 Times a Day

Well, I guess I might as well try to defend myself.

Well, if you are testing your code 200 times a day, you are almost certainly wasting time. Lets run some numbers:

Assuming you work an 8 hour day, that means you are testing your code every 2 minutes and 24 seconds. Given that most of your tests will take this long to run (you've got a suite of them right?), that leaves you with zero time to actually do the work you are testing.

Frankly, if you are using Unit Tests you should be using them after major chunks of work, not in a trial and error fashion. Now if you were using them in a trial and error fashion - "lets change this, run the tests and see if they pass, no that didn't work, lets try this", etc, I could understand how you hit the 200 times per day mark.

That is a completely respectable position to take. I used to work in this way during college. My several thousand unit tests do take 2-3 minutes to build. My estimate of 200 times was probably over shot and should be more like 50-75 times a day. I would also like to point out that I can continue developing while the tests run. I use Maven2 as a build tool and enjoy it immensely, it helps me do test driven development. Since you are obviously far superior to me, I will assume you know what this means but point it out to the rest of the idiots like myself. TDD is where you write your unit tests before you code. Then you satisfy your unit tests with code. When you need to change code, you change the tests and then you change the code to fix the tests. Crazy waste of time right?

If you are coding in a trial and error fashion and using unit tests that way, I'd advise getting some tuition or changing career.

Thanks, I love you too.

But the thing is that my employer loves my work and my code rarely breaks. Now why is that? Perhaps because I'm regression testing at all times? Perhaps it's because I take the time to think about things before I do them and, as a result, I really begin to understand what it is that I'm writing.

An added benefit is that I've found I can look at my or others unit tests and really understand what was going through their mind when the first wrote the method that I am expanding. It's quite interesting, but I'm sure you are a supreme being like Knuth and don't bother with such trivialities.

I'm a highly productive individual ...

Congratulations, I'm glad that you view yourself in such light while being able to cut me down with a few keystrokes. I'm also glad I don't work with you. Really glad. I would sacrifice all productivity in the name of being able to come into work and not feel like shit.

Making the unit tests part of the build process is like requiring a roadworthiness test for you car every one mile you drive it. Sure the car is safe, but its not very productive at getting you from A to B, you could walk faster.

I respect your decision to use a car analogy here although I find it flawed as I most often do.

Here's a question: how much time do you spend working out what happened when your code breaks? TDD is a trivial amount of time compared to that. I am concerned about my software in the present and future. I wish others were also.

What counts is that when you run the unit tests, they pass, and that they accurately test the conditions that need testing.

I disagree with you. I run unit tests that fail all the time--on purpose.

I know it will most likely result in a swift abrasive response but I implore your highness to really spend some time understanding how unit tests can help the really stupid coders (like me).