Stories
Slash Boxes
Comments
Search

News for nerds, stuff that matters

Slashdot Log In

Log In

Create Account  |  Retrieve Password

Firefox Analyzed for Bugs by Software

Posted by CowboyNeal on Sat Aug 12, 2006 10:47 AM
from the killing-bugs-dead dept.
Mozilla The Internet Bug
eldavojohn writes "In a brief article on CNet, a company named Coverity announced that Firefox is using software to detect flaws in Firefox's source code. Even more interesting is the DHS initiative for Coverity to use this same bug detection software on 40 open source projects." An interesting tidbit from the article: "Most of the 40 programs tested averaged less than one defect per thousand lines of code. The cleanest program was XMMS, a Unix-based multimedia application. It had only six bugs in its 116,899 lines of code, or .51 bugs per thousands lines of code. The buggiest program is the Advanced Maryland Automatic Network Disk Archiver, or AMANDA, a Linux backup application first developed at the University of Maryland. Coverity found 108 bugs in its 88,950 lines of code, or about 1.214 bugs per thousand lines of code." We've covered this before, only now Firefox is actually licensing the Coverity software and using it directly.
+ -
story

Related Stories

[+] Developers: Programmers Learn to Check Code Earlier for Holes 212 comments
Carl Bialik from WSJ writes "Many companies are teaching programmers to write safer code and test their security as software is built, not afterward, the Wall Street Journal reports. This stands in contrast to an earlier ethos to rush to beat rivals with new software, and, of course, brings tradeoffs: 'Revamping the software-development process creates a Catch 22: being more careful can mean missing deadlines.' The WSJ focuses on RIM and Herb Little, its security director, who 'uses Coverity every night to scan the code turned in by engineers. The tool sends Mr. Little an email listing potential red flags. He figures out which problems are real and tracks down each offending programmer, who has to fix the flaw before moving on. Mr. Little has also ramped up security training and requires programmers to double-check each others' code more regularly.'"
This discussion has been archived. No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

Firefox Analyzed for Bugs by Software 50 Comments More /

 Full
 Abbreviated
 Hidden
More
Loading... please wait.

  • Math (Score:5, Informative)

    by Anonymous Coward on Saturday August 12 2006, @10:50AM (#15894412)
    That's .051 bugs per thousand lines of code for XMMS, an order of magnitude better.

    • Re:Math (Score:5, Informative)

      by It'sYerMam (762418) <thefishface@gmail . c om> on Saturday August 12 2006, @12:56PM (#15894981) Homepage
      Quite possibly because XMMS is practically stagnant. I don't even use it, but Amarok is by far the best audio app I've tried for Linux, quite possibly because the people developing it have some idea of which decade they're in.

      • Re:Math (Score:3, Insightful)

        by Bert64 (520050)
        Surely a program having very few bugs is reason for it to be stagnant, very little work needs doing in the way of bugfixing and the program already provides all the features it needs (i personally hate programs which suffer from feature bloat - pointless features added for the sake of it, that usually result in the core program not doing it's original task as well as it used to)
    • That's an interesting datum, since XMMS (well, XMMS2, actually) is in the middle of a re-write because the XMMS code is said to be quite inflexible. And it reportedly has a lot of bugs [debian.org].

  • If this is the same (Score:3, Interesting)

    by Anonymous Coward on Saturday August 12 2006, @10:53AM (#15894425)
    If this is the same as most automated testing software I've seen, it detects many things which aren't truly bugs as bugs. Accuracy on automated testing tools I've been exposed to is around 40%.

      • These tools can't neither find all bug in your code nor swear that those it finds are real bugs
        That's true. It's impossible to have both (all bugs and no false positives, soundness and completeness), and even one of them is usually extremely expensive (computationally).
        It helps to follow "good practices" and be more precautious
        That's not true of Coverity (disclaimer: I work for them), we find real bugs. You can see a couple examples here [slashdot.org]. The engineers are usually the ones excited about it, once they've see
  • I will definitely take another look at Coverity's products, if the Firefox team is finding value in it.

  • this slashdot news is already outdated (Score:5, Informative)

    by msh104 (620136) on Saturday August 12 2006, @10:56AM (#15894444)
    if you look at the coverity site ( http://scan.coverity.com/ [coverity.com] ) you will see that there are already multiple projects who have brought there bugs down to zero. samba being on of the earliest.

    • Re:this slashdot news is already outdated (Score:5, Insightful)

      by StrawberryFrog (67065) on Saturday August 12 2006, @11:03AM (#15894477) Homepage Journal
      there are already multiple projects who have brought there bugs down to zero.

      You mean "who have brought down the count of their bugs that this tool can detect down to zero." I'm sure they will have other bugs in code and design.

      How does this tool compare to tools that do analysis by introspection on bytecode from languages like C# and Java. I use FxCop [gotdotnet.com] on C# code, and while it is very cool, using it is not newsworthy at all. Does this tool do more? Is is the news that it's used in a high-profile C++ program?

      Integrating tools like this into your build process may be cutting-edge best-practice at present, but give it a while.
      • Yes. Automated testing/code inspection is handy, but a signifigant source of bugs is actually missing logic paths, or combanatorics errors which can't be detected by tools generally.

      • Re:this slashdot news is already outdated (Score:5, Interesting)

        by bcrowell (177657) on Saturday August 12 2006, @11:14AM (#15894540) Homepage

        You mean "who have brought down the count of their bugs that this tool can detect down to zero." I'm sure they will have other bugs in code and design.
        Yeah, if they could make a program that would detect all bugs in a program, it would violate Turing's proof that the halting probelm is undecidable. [wikipedia.org]

        From the articles, it sounds like they're basically looking for mistakes that could lead to security flaws, e.g., buffer overflows. If AMANDA is particularly buggy by their metric (detectable bugs per thousand lines of code), it's probably because AMANDA doesn't interface to the web, so the people coding it knew that certain classes of buffer overflow "bugs" wouldn't be a problem, because they wouldn't be exploited through an internet-facing interface. If you went back and ran this program on Unix apps written in C from the 1980's, you'd probably find zillions of bugs, but it wouldn't indicate low quality, it would just mean that the programs weren't written for an internet-facing environment in the year 2006, when the internet has become a battle zone for evil spammers, botnets, etc. If the only way such a bug can show up is for the user to supply carefully tailored input, and the result is simply that the program dumps core, then that's not a bug for a program that isn't facing the modern internet.

        • if they could make a program that would detect all bugs in a program, it would violate Turing's proof

          Unless the program's domain is restricted to context-sensitive languages. In fact, it is impossible for a computer to try to decide anything more general than a context-sensitive language because anything bigger requires the resources of a Turing machine, which has infinite memory. Computers implementable in a finite amount of matter are equivalent to linear bounded automata [wikipedia.org], not Turing machines.

    • No major piece of software is ever bug free.

      I follow the news:linux.samba [linux.samba] Newsgroup a bit. Various Samba features have been shipped broken in various recent releases.

      CIFSfs? (it is replacing smbfs and some Linux distributions have taken to disabling smbfs in the kernel to force people to switch) Cifsfs was broken in the newest major release. An intermediate release fixed that.
      'Valid Users' used with 'smbpasswd': that was broken in the intermediate release. The next intermediate release will cover that.

      N
    • Thank you for that link, it rather nicely confirms that at the time amanda was inspected last, it was clean.

      --
      Cheers, Gene

  • Interesting... (Score:3, Interesting)

    by porkThreeWays (895269) on Saturday August 12 2006, @10:58AM (#15894451)
    I find the AMANDA results interesting because AFAIK it hasn't recieved a code rewrite since the early 90's. I think an interesting study would be the to compare older projects with ones that have been rewritten from the ground up. Comparing the rate of new bugs introduced as opposed to those hidden in legacy code.

  • Bug/Lines of Code (Score:5, Funny)

    by X43B (577258) on Saturday August 12 2006, @11:00AM (#15894460) Homepage Journal
    "It had only six bugs in its 116,899 lines of code, or .51 bugs per thousands lines of code."

    Sounds like someone needs to run this debugger on their calculator.

  • Once detected can they also fix the flaws? (Score:3, Funny)

    by Browzer (17971) on Saturday August 12 2006, @11:03AM (#15894476)
    Or that job is left for the monkeys banging on the keyboards.

  • Which type of bugs? (Score:3, Informative)

    by Erixxxxx (920617) on Saturday August 12 2006, @11:09AM (#15894514)
    One has to wonder if these are coding/language bugs or logical bugs. Finding coding bugs is of course a valuable time saver, but the challenging and usually most costly bugs are of the logical sort, and invariably app specific.
  • Even more interesting is the DHS initiative for Coverity to use this same bug detection software on 40 open source projects.
    Before the F/OSS nay sayers toss out the obligatory (and to be expected) "Meh. So much for the 'many eyes' theory" let's point out that having the ability to run a code checker on source code is only possible to the holders of said source code. So, while absolutely true that a proprietary vendor can run the code checker on their code as well as an open source project, there is a huge difference when it comes to the customer/user of said software: with Open Source the user has the freedom to run such a tool over the source code themselves.


    In this age of SarbOx and risk management there is a real competitive advantage to F/OSS over proprietary code to large companies: audit-ability. In previous roles I've had to attest under HIPPA::Security that proprietary code was "secure" -- how? All I could do was obtain a vendor statement that was as non-commital and burden-shifting as possible. Yet, with a true ability to audit the code my pharmaceutical company depended on it would tilt the balance between similar-featured Closed vs Open source solutions. Especially today.

    Ok, maybe nobody really cares about the 'many eyes' theory anymore. Regardless, the "open the hood" theory still applies, perhaps more than ever.

  • Meanwhile... (Score:3, Funny)

    by Skiron (735617) on Saturday August 12 2006, @11:14AM (#15894537) Homepage
    Coverity segfaulted whilst auditing MS Vista.

  • For those who are interested in Firefox' results (Score:5, Interesting)

    by RebelWebmaster (628941) on Saturday August 12 2006, @11:14AM (#15894539)
    Here are some links to show the bugs in the Bugzilla database which were turned up by Coverity.
    Open Coverity Bugs [mozilla.org]
    All Coverity Bugs [mozilla.org]

  • Why AMANDA is buggy (Score:3, Insightful)

    by swordgeek (112599) on Saturday August 12 2006, @11:26AM (#15894589) Journal
    AMANDA could easily be the buggiest OSS program in existence, and it would still be OK. The reason? It just has to be less buggy than Netbackup, and more usable than Legato. Luckily for the AMANDA developers, this are very very difficult criteria to miss.
  • Coverity's own site [coverity.com] shows how many defects each product has fixed. the number of outstanding defects on AMANDA is now zero. zdnet reported the fixes back in April. [zdnet.com]

    Those that follow amanda-hackers will know that there was less than a week [yahoo.com] between when coverity released the report on March 6th and it was announced that all bugs were fixed in AMANDA on March 12th.

  • No rsync? (Score:3, Interesting)

    by ortholattice (175065) on Saturday August 12 2006, @12:06PM (#15894756)
    Funny selection of programs; I don't see rsync on the list. From the article: DHS wants to reinforce the quality of open-source programs supporting the U.S. infrastructure. So, XMMS (an MP3 player) is more important to the U.S. infrastructure than rsync?

  • In defense of amanda (Score:3, Informative)

    by Almost-Retired (637760) on Saturday August 12 2006, @12:06PM (#15894757)
    I'm somewhat surprised to see amanda being badmouthed here by this tool. It was mentioned on the amanda-users list a few months back that the amanda tree had been checked by coverity, and the 2 bugs coverity found were promptly fixed.

    Thats not to say that as new features are added, new bugs haven't been too, but to actually call amanda a truely buggy application does stretch this users belief a wee bit. I'm currently running a 20060424 dated snapshot of the 2.5.0 tree, with no hiccups at all.

    --
    Cheers, Gene

  • Types of bugs (Score:3, Interesting)

    by Dan East (318230) on Saturday August 12 2006, @01:23PM (#15895050) Homepage
    After looking at some of the results from the Firefox sources, I see that "bugs" include unreferenced variables and dead code that never gets executed.

    It looks like most of the real bugs consist of not checking return values, the worst being routines that act upon an object allocated by another routine without checking for null pointer.

    Dan East

  • Homeland Security Tested XMMS?! (Score:3, Insightful)

    by Pulsar (4287) <champ77.hotmail@com> on Saturday August 12 2006, @01:31PM (#15895067)
    XMMS, a multimedia/mp3 player was tested as part of what the article calls a "$1.2 million, three-year grant [the Department of Homeland Security] awarded to a team consisting of Coverity, Stanford University and Symantec Corp" that was setup to "reinforce the quality of open-source programs supporting the U.S. infrastructure".

    40 programs were tested. 40 open source programs. Not even all the programs installed by, or regularly used on, a default install of a particular distro or two; just 40 programs. I thought maybe these 40 were just the first 40 tested, but the original announcement of the award of the grant states that 40 programs would be tested.

    And yet they didn't test BIND? ssh? Also, PostgreSQL is on the results list, but MySQL isn't? Did Homeland Security put this list together?! Using a dartboard and a list of open source applications, or what?!

    This seems like a great software package, and I'm glad that Homeland Security acknowledges that "much of the critical infrastructure runs on open source", but I could think of a few other ways they could've spent $1.2 million, or at least a few other applications they should've tested before they got to XMMS.

  • Coverity on Windows? (Score:4, Insightful)

    by Money for Nothin' (754763) on Saturday August 12 2006, @05:41PM (#15895937)
    "Coverity was also run on the Windows source code. Unfortunately, the 32-bit integer iterator in Coverity was 1 count too small to store the count of the number of bugs found, and so Coverity's counter rolled-over, showing that Windows actually has -2,147,483,648 bugs. Microsoft employees were ecstatic at the results, and Steve Ballmer was said to be seen dancing in his office, yelling 'developers, developers, developers, developers!!'."
    • Err?... I always thought Bugzilla was just where you reported bugs in Mozilla suite products?...

      How does this bug detection software work anyway?

    • Re:not just any software? (Score:3, Informative)

      by Anonymous Coward
      Didn't they always use software of some sort, Bugzilla, etc?

      Bugzilla is a issue tracking software; it's useful only after you've already found a bug. The only other bug-related tool they use is the FullCircle crash reporter thingy, again, after-the-fact thing. This is different - this tool finds flaws from the source code automatically.

    • I assume they specifically mean "Firefox is using Coverity software to detect flaws in Firefox's source code."

      And I'm assuming that they mean "Mozilla is using Coverity..." or "Firefox developers are using Coverity...". After all you don't hear about what Internet Explorer is doing, but rather what MS are doing with it.

      Wouldn't it be great if the summary was clearer and neither of us had to make mental amendments? :(
    • > I hope these Coverity guys aren't pompous enough to think that their tool can find ALL bugs in a program with... magic...

      I am sure that they know their tools limitations, but I am pretty sure that others will interpret
      no outstanding bugs as if the application is secure or bugfree. Ethereal (now known as wireshark) has
      a very low bug count, but I will not use it due to numerous past remote exploits coupled with
      little interest in fixing bugs contra adding new features.

      > Hmm, they should run their tool

    • Re:Errr... (Score:5, Interesting)

      by twiddlingbits (707452) on Saturday August 12 2006, @11:07AM (#15894500)
      Finding all POSSIBLE bugs in a software program means traversing all possible paths in the code with all possible inputs. That's a HUGE problem. You can "model" the code using Logic Equations and that helps some but any errors in the conversion from code to logic equations invalidate results. The DoD and NASA have spent many millions on solving this problem over the last 10-12 yrs. When I was at NASA we used several different tools (CodeSurfer, Purify, Lint, Polyspace as I recall) as each tool was better at one thing (i.e memory leaks vs null pointer dereferences). A The complete process took a couple of days to weeks and then human eyes and expertise were still needed to remove false positives. A good site for all the tools out there, old & new is http://spinroot.com/static/ [spinroot.com]. Looks like Coverty might be a good one to look into, as the best I had seen was CodeSurfer. All the good tools I have seen are commercial (NOT open Source) and EXPENSIVE!! I'd love to see a decent open source tool to run as a first pass before applying the other tools. Another point is that these tools are STATIC analysis. Run-Time Analysis is a whole 'nother animal but that area is improving with tools like DTRACE in Solaris.
      • As an object oriented programmer, I always follow the general rule of having a function always give the same output for the same inputs. That is, you then don't have to worry about the 'state' of an object and you as a result have fewer paths to test and fix. This is why, IMO, global variables aren't such a good thing unless they are constant/rarely change.
        This should be common knowledge to a good object oriented programmer, but I wonder how often it's employed in the 'C' discipline.

        • Re:Errr... (Score:5, Insightful)

          by John Nowak (872479) on Saturday August 12 2006, @12:23PM (#15894841)
          A function that always returns the same value given its inputs is part of functional programming, not object-oriented programming. Most OO code is littered with side-effects and state-dependent behaviour. If you like to program in such a way, you may find yourself much more comfortable with a functional programming language. Languages like Haskell even enforce this.
        • Good programming practice says ANY function should give the same outputs ALL the time for the same inputs (i.e if you put in a 2 today you get out a 4 and the same thing tomorrow). What you seem to be talking about are "side effects" where a global variable or input parameter is modified within the context of a function. Some programming languages DO allow you to change the value of a parameter within the function and that result is passed back to the caller. In fact thats easy to do in C with pointers. Har

        • Re:Errr... (Score:5, Interesting)

          by twiddlingbits (707452) on Saturday August 12 2006, @12:13PM (#15894799)
          I had some extensive conversations with the team at CodeSurfer and they think they the problem is NOT impossible, maybe more like Polynomial time. The DOD was funding them (this was about 3 yrs) ago to try to develop a solution that worked for C/C++ and Ada. NASA wanted to tag along on the research but we were told it was "classified" and DOD only. It's rare when someone turns down research money so they must be on to something.

    • Re:Errr... (Score:5, Informative)

      by chgros (690878) <`charles-henri.g ... ' `at' `m4x.org'> on Saturday August 12 2006, @01:52PM (#15895146) Homepage
      I hope these Coverity guys aren't pompous enough to think that their tool can find ALL bugs in a program
      We aren't (I'm a Coverity employee). We find real bugs, and we find false positives (but not too many of those).
      Hmm, they should run their tool on its own source code, that would be fun.
      We do that regularly.

    • Re:I dislike the idea of Coverity (Score:5, Insightful)

      by Animats (122034) on Saturday August 12 2006, @12:02PM (#15894737) Homepage

      It is not possible for a program to analyze another program and find all the bugs; see halting problem .

      Wrong. It is quite possible to analyze a program and find all the bugs that violate the language constraints (null pointers, buffer overflows, etc.). That's what program verification is for. For some programs, you can't tell whether a bug condition will occur, so you treat that as a bug.

      Automated program verification is a good idea that went away because C and C++ have such ambiguous semantics. It's hopeless for those languages. The "pointer equals array" concept alone makes it very tough, because the language has no idea how big an array is. Worst idea in the language, and the root cause of buffer overflows.

      Good verifiers were written for Pascal (I headed one of those projects [animats.com]), a good one was written for Java [dec.com] (at DEC, just before DEC went under), and Microsoft is working on one for C#. [microsoft.com]

        • The halting problem is not an issue (Score:5, Informative)

          by Animats (122034) on Saturday August 12 2006, @01:21PM (#15895047) Homepage

          The halting problem is not an issue for program verification. This claim is raised repeatedly by the clueless, and it just isn't an issue.

          Yes, you can construct a program that's formally undecideable. It's a hard way to write a bad program. It takes some work, and the resulting program is unlikely to be useful.

          Most crash-type and security-hole problems in programs are entirely decidable. This is because almost all subscript calculations are composed from addition, multiplication by constants, and logic operations. Those are totally decideable, and there are good decision algorithms for that problem. Only when multiplication of two variables (both non-constant) is introduced can formal undecidability appear. See Presburger arithmetic [wikipedia.org].

          In fact, halting is decidable for all deterministic machines with finite memory. Either you repeat a previous state, or halt within a finite number of cycles. The decision process may be made arbitrarily hard, but that's not undecidability. True undecidability in the Turing sense requires infinite memory.

          Most of the practical problems with program verification come from dealing with interactions between various parts of the program. Containing those interactions well enough that you can localize problems is constraining on the programmer. "Design by contract" languages like Eiffel try to do that, but they're not popular. Retrofitting design by contract into C and C++ has been discussed, but the proposed schemes all have holes you could drive a truck through. A big truck.

          Although software work seldom uses proof of correctness techniques, there's a whole industry doing it for hardware. There was a machine-generated formal proof of correctness for the FPU in AMD's K7 processor. [onr.com] AMD thus avoided the "Pentium division bug".

    • Re:I dislike the idea of Coverity (Score:5, Informative)

      by anpe (217106) on Saturday August 12 2006, @12:25PM (#15894850)
      If you'd followed the lkml, you could have seen actual patches fixing real bugs, found by Coverity. Just run this search on google: "by coverity" patch site:lkml.org to convince yourself.
      The fact that it is impossible to solve the whole problem of program correctness and that false positives will come up doesn't mean that the problem Coverity is adressing isn't usefull.

      Regards,

All trademarks and copyrights on this page are owned by their respective owners. Comments are owned by the Poster. The Rest © 1997-2010 Geeknet, Inc.