Are All Bugs Shallow? Questioning Linus's Law 596
root777 writes to point out a provocative blog piece by a Microsoft program manager, questioning one of the almost unquestioned tenets of open source development: that given enough eyeballs, all bugs are shallow. Are they? Shawn Hernan looks at DARPA's Sardonix experiment and the Coverity static-analysis bug discovery program in open source projects to conclude that perhaps not enough eyeballs are in evidence. Is he wrong? Why? "Most members of the periphery [those outside the core developer group] do not have the necessary debugging skills ... the vast numbers of 'eyeballs' apparently do not exist. ... [C]ode review is hardly all that makes software more secure. Getting software right is very, very difficult. ... Code review alone is not sufficient. Testing is not sufficient. Tools are not sufficient. Features are not sufficient. None of the things we do in isolation are sufficient. To get software truly correct, especially to get it secure, you have to address all phases of the software development lifecycle, and integrate security into the day-to-day activities."
Re:Bugs are an error in the... (Score:5, Interesting)
Except the point he is trying to make is that his code is better then the competing individual because he follows process doctrine.
Unfortunately, to make his claims stick he took a failed project as an example to support his theories. While being quite pointed in defining what projects failed he did not cite which projects of his has succeeded. This would have been at least a good starting point for a real argument.
Is good process doctrine wrong? No, it won't hurt of course, but it's not quite a kevlar vest against root shells.
Besides more examples from both sides of the camp he really does neglect several facts. Many open source projects are often led or particpated by professionals as well. In fact a recent article suggested a great more open source projects are corporate sponsored.
It's just an awful piece when you consider he is painting his enemy as both unprofessional and only arming that foe with one failed project example.
Personally, I wanted to read something useful that I could learn from and grow with, but this is pretty standard tripe.
Re:Bugs are an error in the... (Score:5, Interesting)
I also think a big difference is that you psychologically don't write shitty code when you think others are going to look at it.
Coders that write shitty code don't know that they write shitty code. From their perspective the code is just fine and even very good. When ever I told someone I don't like his code and challenged him to explain what he did and why, he only answered: "erm, well that is what the code should do as the requirements demand that", they had no idea what my point was and when I pointed i tout they shrugged and did not understand or value my concerns.
angel'o'sphere
Re:Bugs are an error in the... (Score:5, Interesting)
What the essay fails to capture is the nature of the functioning of the eyeballs in practice, between open source and closed source. In closed source, the eyeballs only look at what they are paid to look at, if the code is just barely good enough to sell, then out it goes and nobody looks at that code again until the complaints start rolling in and then and only the do they fix it, well, sort of fix it, they of course only fix it just barely enough to silence the noisiest of complaints and the only if there are real consequences for failing to do so. Don't think so then try this http://social.technet.microsoft.com/Search/en-GB?query=this%20is%20a%20know%20fault&ac=8 [microsoft.com] and a huge number of them have never been fixed.
Open source follows a completely different series of routes;
1) People looking for faults because they get a kick out of finding them and fixing them.
2) Tweaks to functions that indirectly remove bugs by simply replacing them with better code.
3) Discoveries in user interactions, less of a complaint because there is no force in pushing the fix.
5) Governments and government departments directly pursuing more secure code.
6) Corporations seeking to build a public reputation by demonstrating coding expertise.
So in the case of open source software there are many 'different' kinds of eyes, so those eyes all working from different perspectives do in reality make bugs very shallow. In the closed source proprietary world the bugs are buried in the depths of the code, hiding in the dark, basically because of profits versus workmanship issues, which means no light is shone on them because only one set of eyes looking from a single 'shallow' perspective looks at them.
There is of course one other set of eyes looking at code, the saboteurs both private and government, looking for faults to exploit. Hard with open source because it can rapidly turn around and bite you on the arse if you use it (if you protect against it everybody notices). Closed source (mostly but a lot of less than honourable eyes lend up looking at it), of course can be targeted as long as you, well, use open source code yourself whilst promoting closed source to everybody else (hmm, kind of reminds me of all those mainland China computer companies, odd that, isn't it).
Re:Bugs are an error in the... (Score:5, Interesting)
True, but that's not what he is questioning. Given two identical projects that are fairly complex (i.e. an OS kernel) he's saying that just being open source doesn't necessarily provide "more eyes". While I think there is a bit of merit to this, it certainly doesn't hurt to have more eyes possible - especially when you don't have to pay for them.
Agreed, of course. However, the converse is important, too:
Given two identical projects that are fairly complex (i.e. an OS kernel), being closed source virtually guarantees that there won't be 'more eyes'.
But the real question is: How many eyes are enough?
The answer is its own problem: Only one more pair. The tricky part is figuring out whose they are. (Yes, I'm in screaming agreement with what the OP is saying.)
It's a quality issue as much as it's a question of quantity. Ben Laurie [links.org], writing about the Debian OpenSSL Fiasco [imagicity.com], states:
So yes, it does matter whose eyes are turned to a particular problem. The difference between FOSS/Open Source and Closed Source is therefore whether the Closed Source company has hired the right people and whether the FOSS project has gained the attention and interest of the right people.
Neither of those situations is guaranteed, but they are not at all equivalent. (Especially when we consider that for many of the best FOSS products, gaining the attention and interest of the right people is done by employing them.) Realistically, FOSS faces better odds of having bugs found and fixed, all else being equal.
Re:Bugs are an error in the... (Score:1, Interesting)
Since you know so much, point out the design problems in the NT kernel. Lets go for a small number - 10.
It's a fair point, with regard to security (Score:2, Interesting)
Open source bugs get fixed because people notice and are bothered by the bugs. This is the biggest motivator of open source contributions - everybody has an itch to scratch. The bugs that get fixed fastest are the bugs that are encountered the most. And this is why the Microsoft guy is absolutely correct in his analysis.
Bad security is not a user-facing bug. Unlike functionality bugs, there is little incentive for community members to identify and fix security bugs. Sure, the Linux kernel and other key packages will attract expert eyes, but the average random piece of open-source software will not.
Security analysis is both complicated and un-glamorous. There are not a lot of people attracted to that kind of work. There are even fewer people who would do it for free. The position of the linked article is that it's better to pay people to think about security than it is to rely on the principles of OSS. I agree 100%.
Only early bugs are shallow (Score:3, Interesting)
Actually, most bugs that survive initial testing are not shallow. If they were, they'd have been caught early.
A key point of the article is that almost nobody in the open source world is really looking hard at old code. An experiment was run to encourage code review, but nobody really wants to do that. This is related to the phenomenon that many open source projects stall out at version 0.x. The basic functionality is in, the fun part has been done, and the boring grind of making the last bits work isn't getting done.
Some bugs are so deep the open source process can't fix them. Search Google for "prune_one_dentry oops". The Linux kernel is known to crash when all free memory has been taken over as file cache, a process needs memory, and due to some lock being set, file cache space can't be released. Bugs of this type have been reported steadily since 2004, and it's still not fixed. It will probably take a redesign of some fragile code to fix that, and nobody wants to take that on.
Re:Bugs are an error in the... (Score:3, Interesting)
Specifics?
Some of my favorite FOSS stuff -- things I'd pick over the commercial alternatives any day of the week, purely on their own merits: The Linux kernel and GNU command-line utilities. PostgreSQL. The Dojo toolkit. Firefox. Thunderbird. Eclipse. CUPS. Apache (web server, many of their other projects suck). Various Debian package managers. VirtualBox.
Some cheap and clunky and altogether second-rate things that attempt to duplicate functionality of commercial software that does the job much better, that I (hate to but nevertheless) use, for any of a number of reasons: GIMP. OpenOffice (especially the Word and Excel clones -- and good grief, it oughtn't be that hard to do better than *Word,* of all things!) GNOME/KDE/any other Linux desktop. Various RAW conversion utilities.
Some commercial software that does stuff better than the FOSS stuff they're riffing off or building on: Jira. Confluence. Mac OS X.
Some areas where the FOSS world has consistently failed to deliver, despite years and years of effort and constant promise, and the fact that the problems appear ideally suited to being solved the FOSS way:
Content management systems. There are a gazillion FOSS ones out there, and all of them suck in some significant way -- either they're a big ol' mess of vaguely connected utilities (Drupal), they make very big assumptions about how you want your site to work (Joomla), or they're half-finished while incorporating several internally competing ways of doing things (Lenya and its plethora of editors, none of which really work very well.)
Anything related to proper graphic design tasks. This requires a full chain of utilities from the RAW file in the camera to the finished file to be sent to the printer (or put up on the web). Most of the chain just isn't there: no system-wide color management, no RAW conversion software with accurate, consistent profiles for a wide range of cameras, no genuinely functional (and color managed) page layout software.
I could go on, but you get my drift. I don't care for ideological arguments. If FOSS is a genuinely and universally better way to make software, it would have incontrovertibly proved it by now. If it was genuinely and incontrovertibly unworkable, it would have failed by now. Instead, it's done neither -- it works brilliantly for some things, fails miserably in other things, and muddles along for lots of others. Just like any other way of making software.
Whew. I feel better now.
Re:Bugs are an error in the... (Score:3, Interesting)
This argument is logical, and reasonable.....but for all that is wrong
Linux is (surprisingly) not riddled with bugs, not full of security holes, and is maintained
Microsoft products (surprisingly) do have security holes, do have bugs, and these are left unpatched until people complain ...
The problem is that in general people who use Linux complain about the flaws, and if the people who manage the code agree it is a flaw, then it does get fixed, the only cost is time .... with Microsoft fixing/not fixing each flaw is evaluated as to the cost in money and time
Linux people use Linux and so have an incentive to make it work (at least how they want it to) Microsoft only need to keep selling, so it has to be "good enough", people have noticed that internet explorer development ground to a halt when there was little or no competition and picked up again when people started using other browsers ....
Who Wants to be A Millionaire has the Answer (Score:2, Interesting)
I learned something about "Group intelligence" from the quiz show Who Wants to be a Millionaire in which contestants are given three lifelines to help them answer difficult questions.
The weakest lifeline by far is to appeal to the wisdom of the crowd and ask the audience. This only works for the simplest question.
Phone a friend works better IF you know the right friend.
However the most powerful lifeline. The one smart players keep till last is 50:50 - randomly removing two wrong answers.
So if open source debugging is equivalent to "Ask the Audience" then closed source debugging by the specialised team of developers is "Phone a Friend". Now all we have to do is figure out what is the debugging equivalent of 50:50 and all our problems are solved.
Re:Yeah, right.... (Score:3, Interesting)
To play devil's advocate, there is the issue that Microsoft products have 10 times the "eyes" looking for security vulnerabilities than Linux-based products do. They also tend to have more features included.
And frankly, the proof *is* in the pudding. Windows 7 is an excellent product, and I've yet to run into a single bug in it. That's not to say there are no bugs, just that I haven't experienced any. So far, it's running far better than *any* Linux distro I've ever tried-- for one thing, it knows what to do with my Tablet PC hardware! (How to use a pen, and how to sleep/suspend.)
Re:Bugs are an error in the... (Score:3, Interesting)
Mr Graphic Designer Man: "Linux still doesn't do proper color management."
Me: "I don't know what that means. You may be right."
Funny that you mention that; I've throughly compared several CMS engines, from Adobe, Apple, Esko (formerly Barco), Efi (makers of the Fiery) and LittleCMS (the OS CMS used everywhere Linux need color management). Believe it or not, LCMS was right there with Efi on quality, a little better than Adobe, and waaay better than Esko (Apple was erratic, it seems to be a modification of Efi's engine). And it ran circles around everything else on terms of performance and resource utilization.
So, quality _can_ be better on OSS, but it won't dispel long help myths.
Re:Perfect code may not be perfect.. (Score:3, Interesting)
Re:Then why does IE suck as a browser? (Score:3, Interesting)
Well the catch is that linux developers are mostly being paid. That's IMO what's taken say pre ubuntu linux from being an amusing nerd gimmick to something a normal human being might actually use for something. Firefox developers are being paid too.
If you pay people to do the wrong things the product will suck. Linux benefits from several eyes on the problem of deciding where development money should go. But that's the same weakness. What happens to firefox when the product they sell: A google startpage, is usurped by google launching its own competing browser (or if their privacy ideals conflict with what google does to that search info)?
The quality of software is judged by it doing what you want it to do, whether you want it to do the right or sensible thing or not is another matter. IE was cheap (i.e. free), and preinstalled, netscape wasn't. Since then they worked hard to maintain backwards compatibility with a product who's main selling feature was that I didn't have to download it. Which is why they have more marketshare than firefox still. Vista was competing with XP, not Linux. And it offered no real compelling reason to upgrade, on the other hand most of the core software stuff in Vista carried over to 7, which is apparently wonderful, even though it's main difference from vista seems to be they changed the priorities on some processes, changed the look of the skin and put a big 7 on the box.
Where are the results of those paid developers you ask? http://www.computerworld.com/s/article/9142978/Windows_market_share_slide_resumes down to 92% of the OS market. Office went from what, 10, 15% of the market in 90/91 to probably 80% today, and I bet 90% of the revenue. The market of people who pay for the product they use seems to have spoken. MS fulfills their requirements. Just because their requirements may make no rational sense, they paid people to do what the customer wanted - and the customer got a product he/she will pay for.
Linux developers obviously haven't put enough effort into making the product people want. It may, in the formal proof sense of computer science be 'better', but it obviously doesn't meet as many user requirements on the desktop.
As for who produces the best quality, the passionate volunteer or the guy in it for the money? Well how long is the project? If my developers aren't getting paid, what are they eating, where are they living? If they're getting paid, but not by me, how much work can I expect them to actually do? The guy guaranteed a job no matter how terrible a job he does (Yugo), in the socialist/communist system doesn't really count.
You realize software isn't free right? Duplication costs being nothing isn't he same as the product costing nothing to make. A fighter jet costs 40 billion dollars to develop, and 100 million dollars to actually manufacture. If you only sell 100 of them (say the french Rafale) 80% of your total costs (50 billion) are in 'development' and 20% in actual production. Or maybe it will be 40/60 if they make a full 200 of them, whatever. The passionate volunteer who really really wants to do it still needs to eat, so if you want him to stay on time and a budget you kinda need to pay him. Or else he's going to work for the person who offers him food, and not work on your project.
Re:Bugs are an error in the... (Score:3, Interesting)
What do you mean by unmaintainable? How can it be unmaintainable, yet still be in use to this day?
You can use something that's unmaintainable. All "maintenance" is, in this context, is applying the necessary fixes to keep the OS reasonably current and secure. Every product, sooner or later, reaches a point where it's "not economically feasible" to continue making proper repairs-- you just wind-up spending too much money and time doing so.
Maintainability of --anything-- begins with good documentation and an engineering framework that allows for repairs of whatever is broken or functionality upgrades without breaking other things. Since well-documented APIs that say what they do and do what they say are still something I'd assume are lacking in Windows 7 based on complaints of software and drivers not working correctly on it, it would seem that this fundamental flaw in building a foundation for the OS is indeed still present.