Forgot your password?
typodupeerror
Software Bug Open Source Security

Are All Bugs Shallow? Questioning Linus's Law 596

Posted by kdawson
from the defending-a-shibboleth dept.
root777 writes to point out a provocative blog piece by a Microsoft program manager, questioning one of the almost unquestioned tenets of open source development: that given enough eyeballs, all bugs are shallow. Are they? Shawn Hernan looks at DARPA's Sardonix experiment and the Coverity static-analysis bug discovery program in open source projects to conclude that perhaps not enough eyeballs are in evidence. Is he wrong? Why? "Most members of the periphery [those outside the core developer group] do not have the necessary debugging skills ... the vast numbers of 'eyeballs' apparently do not exist. ... [C]ode review is hardly all that makes software more secure. Getting software right is very, very difficult. ... Code review alone is not sufficient. Testing is not sufficient. Tools are not sufficient. Features are not sufficient. None of the things we do in isolation are sufficient. To get software truly correct, especially to get it secure, you have to address all phases of the software development lifecycle, and integrate security into the day-to-day activities."
This discussion has been archived. No new comments can be posted.

Are All Bugs Shallow? Questioning Linus's Law

Comments Filter:
  • by ralphdaugherty (225648) <ralph@ee.net> on Tuesday February 16, 2010 @12:39AM (#31152168) Homepage

    They become a lot shallower when you can look at the source code.

    • by Interoperable (1651953) on Tuesday February 16, 2010 @03:35AM (#31153024)

      Presumably Microsoft employees do look at the source code but some days I have my doubts. (I'm kidding of course)

  • Yeah, right.... (Score:5, Insightful)

    by socceroos (1374367) on Tuesday February 16, 2010 @12:40AM (#31152176)
    As we can all see, this has gone famously for Microsoft.

    What do they say? ...the proof is in the pudding?
    • Re: (Score:3, Funny)

      by iserlohn (49556)

      "The proof of the pudding is in the eating", or as in the case of Microsoft, "the proof of the FUDding is in the beating"...

    • Re: (Score:3, Interesting)

      by Blakey Rat (99501)

      To play devil's advocate, there is the issue that Microsoft products have 10 times the "eyes" looking for security vulnerabilities than Linux-based products do. They also tend to have more features included.

      And frankly, the proof *is* in the pudding. Windows 7 is an excellent product, and I've yet to run into a single bug in it. That's not to say there are no bugs, just that I haven't experienced any. So far, it's running far better than *any* Linux distro I've ever tried-- for one thing, it knows what to d

  • by filesiteguy (695431) <kai@perfectreign.com> on Tuesday February 16, 2010 @12:41AM (#31152182) Homepage
    Unless you're writing some insanely complex application like a launcher for thermonuclear missiles, you pretty much will have user error as a major instigator of bugs.

    Until you get your code into the hands of users who - for example - will repeatedly hit the ENTER key wile waiting for a response, you don't have a clue what might happen.
    • Re: (Score:3, Insightful)

      by shadowbearer (554144)

        I don't know that one could always consider user error as a "bug" in the software.

        Given the potential variety of human experience and the ways in which software can be misused or abused, it's likely there is no way to make any piece of software "user proof", as you point out. ;)

      SB

       

    • by Interoperable (1651953) on Tuesday February 16, 2010 @01:18AM (#31152368)

      That's simply not true. Proper, bug-free code should fail gracefully in the event of odd user behavior. It may be that random mashing of the keyboard will give the user some unexpected results but it should never cause the program to go into a state that it was not designed to go into, such as trying to access 0x00000000.

    • by grcumb (781340)
      Posting this only to highlight the perfect irony of the title.
  • by Statecraftsman (718862) * on Tuesday February 16, 2010 @12:44AM (#31152206) Homepage
    This is precisely the kind of argument you become susceptible to if you think that an attribute of software (security) is more important than your freedom. Shawn makes some good points about the technical quality of software and it's true there may not be enough eyeballs to find bugs in free software let alone hands to fix them. What Shawn would have us take from this article is that free software may not be technically superior. It's an attempt to frame the argument and shape what's people think is important in software. Unfortunately, if you care about software freedom, Microsoft's FXCop and PreFast-clean mean nothing. Their software disrespects you as a user and keeps pushing the limits in dividing and taking power away from their user base. Don't buy this line. Choose freedom first and interested parties will take care of attributes like security, ease-of-use, and compatibility over time.
    • by amiga3D (567632) on Tuesday February 16, 2010 @01:06AM (#31152320)
      I'm not convinced that Free Software isn't superior. If he'd let us look at his code maybe we could tell. No?, well...without proof it's just opinion.
    • Re: (Score:3, Insightful)

      by shadowbearer (554144)

      Exactly.

        Microsoft is a business that exists to make money. (Obscene amounts of it, if you want my opinion.)

        People who code free software generally do so to make better software.

        I know which one I trust.

      SB

  • by jamienk (62492) on Tuesday February 16, 2010 @12:55AM (#31152260)

    Since when does MS have the right to say "To get software truly correct..."? They KNOW how to make software secure?

    • Re: (Score:3, Funny)

      by rasputin465 (1032646)
      I had the exact same thought. "Getting software right is very, very difficult" ... "trust us, we know; we still haven't figure out how to get it right".
  • by c0d3r (156687) on Tuesday February 16, 2010 @12:58AM (#31152268) Homepage Journal

    One of my most difficult bugs was fixed by simply rescheduling the time a datamining job was to run (which was integrated in to a massive ERP system with other major components of which i had no insight). It took at least 24 hours to test everytime i created a new build. Essentially it was a scheduling ordering issue, where pre-processing of other processes wasn't done in time.. It took me a month to figure this one out. Some times the bugs are outside of the scope of your own system, and the bug will probably re-arise as data grows. I've also had some difficult threading issues where a wait is never notified caused by bad error handling, which was fixed by simply renaming a file (after 1 month of multi threaded debugging with the final session taking 3 days for one execution).

  • Code fixes (Score:5, Insightful)

    by JWSmythe (446288) <jwsmythe@jws[ ]he.com ['myt' in gap]> on Tuesday February 16, 2010 @12:59AM (#31152272) Homepage Journal

        That's kinda funny.

        I spent part of today working around problems with a closed source application.

        The other part of the day has been working with an open source program, where I've already solved the problem, and am documenting my changes to pass back to the author for the next release.

        I'm not a "core" developer for any public projects. I've never submitted a bug fix to someone like Microsoft (but have sent bug complaints that went unanswered). I have sent quite a few bug fixes for open source applications, most of which were used in future release. I'm just another guy, or as indicated, another pair of eyes.

    • Re:Code fixes (Score:4, Insightful)

      by Kjella (173770) on Tuesday February 16, 2010 @01:56AM (#31152606) Homepage

      I'm not a "core" developer for any public projects. I've never submitted a bug fix to someone like Microsoft (but have sent bug complaints that went unanswered). I have sent quite a few bug fixes for open source applications, most of which were used in future release. I'm just another guy, or as indicated, another pair of eyes.

      Well, in my experience what's annoying about closed source software is that you can't solve your own problems. I've reported quite a few defects and gotten quite a few of them fixed, but when you're working with a large vendor just getting through the support organization, down to development and back out through the normal release process means the implementation project is normally over before you get it. There's also a hotfix process but that creates its own headaches both in getting it, running other support cases on the same module and getting rid of it when it's rolled into a normal release.

      Sometimes I really wish you could just patch it and roll your own build to solve your own problems. Right now, reporting bugs is more of a chore in the project and really more of a long term investment in not getting as many headaches in the future. I honestly admit there's been times where I've thought "man, am I glad I reported that six months ago" but other times I've cursed that I "wasted" time on support rather than just accept that it'll never work and get what works working and just do damage control on the rest. Ah well, nothing like a little undeserved flak for the consultant.

    • I did the same the other day for a proprietary security camera setup software (under windows) which is a fairly blatant ripoff of zoneminder.

      The company that produced this software (some of you may know who I'm talking about) was no help to the business owner I was working for, he'd already spent a few hundred dollars working with their "support techs" - who were unable to solve his problem (conflict with a anti-malware app) and there is almost no support available online even for basic issu

    • Re: (Score:3, Insightful)

      by houghi (78078)

      Same here. I once saw an error in a css file. This was a file that was created on the fly by the closed source program (don't ask) and it took them 8 months implementing the solution. The irony is that the part that they complain about is what brought up this solution. I was just an extra pair.

  • Many bugs are caused by the silent L in in the word USER.

  • If one qualified programmer can write a bug and it takes at least one qualified programmer to find that bug then how can it actually be damaging to have many look for the bug, once it is identified even by a "non qualified" programmer others can address the issue much quicker. He seems to try to relate literal depth in the code to the comment "bugs are shallow", while some bugs maybe subtle and complex like all software after QA, first release and further have been completed others maybe be found much late
  •   Any technological endeavor human beings work towards will always be subject to "more eyeballs means improvement". If there's not enough eyeballs, then there simply isn't enough people working on the problem.

      I haven't RTFA. but from the summary, most of what this program manager says is intuitively obvious.

    SB

    • Re: (Score:3, Funny)

      Any technological endeavor human beings work towards will always be subject to "more eyeballs means improvement".

      So that's why the more people there are on the committee that designed a language or protocol, the better the result. I'd always wondered about that.

  • He's partly right. (Score:4, Insightful)

    by slimjim8094 (941042) <slashdot3@justconne c t e d . n et> on Tuesday February 16, 2010 @01:15AM (#31152358)

    ...though perhaps not in the way he intends.

    Look, software is *hard*. Building an OS kernel is like assembling a thousand watch movements by hand. You're going to screw up. It's not a matter of "if". There Are Always Mistakes.

    Now, when he says "truly correct", I'm assuming he doesn't mean formal proving. That would be absurd, especially for an operating system as complex as Windows or Linux (or really anything with limited resources). Anything short of the formal proof and you just have empirical evidence that it works - but if there's a billion branches and trillions of code paths, nobody will hit all of them with all data.

    Fact is, stuff is going to break. You can't prevent it.

    So if we can't keep code from breaking - if all significant code is buggy - what's the answer? Well, with open-source code you can find a bug in your application and debug through the kernel itself, finding out why your syscall isn't returning the right information, and fix it yourself. Then everybody benefits from your work - keep in mind, you only did it (or needed to) because your application exposed a flaw. If you're using Linux 1.8 for some unholy reason, well you can fix it anyway (just nobody else will care).

    But if you're using Windows, and you get bad return data from a method, your best shot is probably going to be to just coerce the data how you want it. This happens *all the time* in closed-source software - handle a buggy OS method with a special case.

    So "many eyeballs" is correct, but not because there are thousands of expert code analysts poring over every git commit. It's correct because any piddly little application developer can debug the kernel itself, following his own method calls around to make sure they do the right thing. Even if he doesn't know how to fix it, he'll be able to say "doThis(*myData) isn't returning the right value" and lead the experts (writers/kernel hackers) straight to a fix.

    This is the strength of open source, at least from a code quality standpoint.

    • Even with a formal proof you can't be sure there are no bugs. You can prove the code matches a specification, but you can't prove the specification to be correct.
  • by bcrowell (177657) on Tuesday February 16, 2010 @01:16AM (#31152362) Homepage

    The funny thing about this article is that he essentially never mentions (a) design flaws or (b) perverse economic incentives to sell defective software. IMO these are probably the two biggest reason why MS has such a terrible reputation on security.

    As an example of a design flaw, there are lots and lots of things that MS designed for ease of use, while ignoring security. MS software is way too willing to execute code in an email or on a web page just because they wanted to do something flashy without putting any responsibility on the user to know what the heck was going on. This is a design flaw. No amount of debugging will ever fully succeed in working around it.

    The economic incentives to ship buggy, insecure software are also huge. Companies gather revenue by putting out a new version of the software with a long list of features. Users who buy the new version of the software generally have no way of knowing that it's full of bugs. MS is of course infamous for this.

    Of course the implication of the whole article is that MS pays people to fix bugs, while nothing like that is going on in the open source world. This is complete nonsense. Most well known open-source projects are written by paid coders. But let's not let facts get in the way of MS advertising.

    • From TFA

      Mathematically, the many-eyeballs argument, and the million-monkeys argument are equivalent.

      Yes, but this is only true if N(eyeballs) = 2 million - N(one-eyed monkeys) - 2*N(zero-eyed monkeys). Of course, once we have humans and their eyeballs involved, we will need modify this recently discovered Microsoft monkey-eye theorem. We should inquire if Microsoft considers human and monkey eyes equivalent in order to determine the effective conversion factor between human and monkey eyes.

      A Microsoft Creationist would set the conversion rate at infinite, since our eyes are in the image of Go

  • by snowwrestler (896305) on Tuesday February 16, 2010 @01:17AM (#31152366)

    From the article:

    One cannot deny the logic. In fact, it is a tautology. If you assume that all individuals have a non-zero probability of finding and fixing a bug, then all you need is "enough" individuals.

    Emphasis added by me to show where I think his argument goes off the rails. "Linus' law" does not assumed that each eyeball is a bug fixer--it simply states that bugs are made shallow. Often the hardest part of fixing a bug is knowing about it, and finding it. The open source process makes it easier to do both, even if there are only a small group of coders actually fixing things.

    This is not about how many software engineers you have reviewing your code. It's about how your end users can interact with the software engineers.

  • by Antique Geekmeister (740220) on Tuesday February 16, 2010 @01:21AM (#31152396)

    Ladies and gentleman, the article author is making a strawman argument. By transforming the "Linus' Law" into a badly written syllogism, and pointing out examples where _his invented syllogism_ fails, he's implying that closed source is _better_. Unfortunately, the vulnerabilities of closed source are often worse, by comparison and from experience.

  • by JoshuaZ (1134087) on Tuesday February 16, 2010 @01:22AM (#31152398) Homepage
    This is a classic absolutist fallacy. The author has taken what is essentially a rhetorical way of stating a more precise claim (that bugs become more shallow with more eyes and that as you increase this number the shallowness increases). The author has then found that that statement in its most general form might not be correct or might not be the whole story. And therefore decides to throw out moderate versions of the claim. I am not impressed.
  • Getting software right is very, very difficult. ... Code review alone is not sufficient. Testing is not sufficient. Tools are not sufficient. Features are not sufficient.

    One of these things is not like the other...

    Features are to software correctness as apples are to oranges.

    Really, do not subscribe me to your newsletter, mr 'program manager'

  • by haruchai (17472) on Tuesday February 16, 2010 @01:39AM (#31152514)

    Let me rephrase this for him -

    "For 25 years, we deliberately chose to ignore the bitter lessons that were learned by the big vendors, to take shortcuts
    to ship shit software first and fix it later and to build up massive layers of cruft in the name of backward compatibility. Now we are caught in a nice pickle
    as we've spent years trying fill the leaks in our crap - some of which is so insecure that, 8 years after the launch, we still have record numbers of bugs in
    Windows XP almost every fucking Patch Tuesday -and restructure it into something rock solid.
    However, until we can get this done, we need to play smoke and mirrors, convince you to toss Win XP - and your old PC, most likely, buy our latest
    and greatest and spit out evermore FUD about how nobody else can get stuff done except us.

    Ladies and gentlemen, I give you the M$ business plan and I'm pleased to say that it's working as well as ever and thank you all"

  • I'm all for open source software. I could give you a dozen reasons why it's a great thing.

    But does anyone REALLY believe it's bug-free because there are lots of eyeballs on it?

    From the first time I heard that argument I thought it was laughable and not backed by any solid evidence.

    He's attacking that argument for a simple reason: Because he can. It's a stupid argument.

    And he's getting people all worked up and distracted over it.

    Meanwhile, in the next room, Microsoft salespeople are convincing your boss they

  • bla, bla, bla ,bla, bla

  • NEWS! (Score:5, Insightful)

    by nudicle (652327) on Tuesday February 16, 2010 @02:00AM (#31152620)
    Ok, I've got some news for you. The quotation is not meant like an immutable law. There's a really good, important point there, but it's still just a meaningful aphorism. Let me help you with this -- when you see "given enough eyeballs, all bugs are shallow", read it as "given enough eyeballs, [almost all] bugs are shallow". Does that help? Can we move on now? This discussion is so stupid it's almost painful. Here are some other things to know: MS blog author wants attention; ESR is a self-important moron. Thank me later.
    • Re: (Score:3, Insightful)

      by Just Some Guy (3352)

      The quotation is not meant like an immutable law. There's a really good, important point there, but it's still just a meaningful aphorism. Let me help you with this -- when you see "given enough eyeballs, all bugs are shallow", read it as "given enough eyeballs, [almost all] bugs are shallow".

      But that's not true, and the original version is correct. Given enough viewers - where "enough" might possibly be more than the number of people alive - every error will be obvious to at least one person.

  • FUD (Score:4, Insightful)

    by mbone (558574) on Tuesday February 16, 2010 @02:18AM (#31152716)

    One big piece of FUD here is the notion that Microsoft programmers are paid, while open source programmers are not. The open source projects I know of advance mostly because of paid programmers, and I suspect that that is the case in general. That gives them the usual capitalist incentives for finding and removing bugs.

  • by Animats (122034) on Tuesday February 16, 2010 @02:59AM (#31152840) Homepage

    Actually, most bugs that survive initial testing are not shallow. If they were, they'd have been caught early.

    A key point of the article is that almost nobody in the open source world is really looking hard at old code. An experiment was run to encourage code review, but nobody really wants to do that. This is related to the phenomenon that many open source projects stall out at version 0.x. The basic functionality is in, the fun part has been done, and the boring grind of making the last bits work isn't getting done.

    Some bugs are so deep the open source process can't fix them. Search Google for "prune_one_dentry oops". The Linux kernel is known to crash when all free memory has been taken over as file cache, a process needs memory, and due to some lock being set, file cache space can't be released. Bugs of this type have been reported steadily since 2004, and it's still not fixed. It will probably take a redesign of some fragile code to fix that, and nobody wants to take that on.

  • Let me be ... (Score:5, Insightful)

    by http (589131) on Tuesday February 16, 2010 @03:55AM (#31153122) Homepage Journal

    I feel the need to explicitly call this guy a shill, rather than imply it. IF he honestly believes what he wrote, he's merely an idiot.

    Shawn Hernan has deliberately misconstrued what Raymond wrote. Raymond explicitly said that the phrase "Given enough eyeballs, all bugs are shallow" was an informal phrasing of the lesson, in the very first sentence of the lesson. The actual phrasing was given as "Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone." There's not even one sentence separating the two.

    Trying to rip apart an informal phrasing, and ascribing hidden syllogisms to it, tells me this man is either an ideologue or an idiot. Given his position, he's a dangerous ideologue or idiot.

  • by jthill (303417) on Tuesday February 16, 2010 @04:56AM (#31153388)

    He reports Coverity's results on open source software
    ... but doesn't report Coverity's results on Microsoft's software.

    He reports that Coverity scanned 280 open-source projects
    ...but doesn't report that only 180 of those have "active developer support".

    He can't be bothered to present any data at all on the distribution of the reported or corrected defects — how many are in nethack or aalib or that long-abandoned "flash-based photo album generator"?

    He doesn't, for instance, mention that Samba and several others have no defects Coverity can discover. None.

    Vim has none. X.org has ... three. All of KDE, nearly five million lines of code, has ... ninety. glibc has none.

    There have been MySQL and PostgreSQL and Berkeley DB versions with none. His bioblurb says he's "currently working to ensure that Microsoft SQL Server is secure". That's interesting. You mean it isn't, now? How many defects can Coverity find in SQL Server?

    TFA is a nauseating pile of sneers and aspersions ("Hope is not a security strategy"?) built on a very carefully selected and very few facts. "No one is doing auditing" he quotes. "No one is doing auditing" and reporting it to some self-styled central authority almost no one ever heard of is what's true, but telling the truth isn't what he's doing here. He's a "Program Manager", and he works for Microsoft.

Mommy, what happens to your files when you die?

Working...