How Long Does It Take to Fix Linux Kernel Bugs? (itsfoss.com) 36
An anonymous reader shared this report from It's FOSS:
Jenny Guanni Qu, a researcher at [VC fund] Pebblebed, analyzed 125,183 bugs from 20 years of Linux kernel development history (on Git). The findings show that the average bug takes 2.1 years to find. [Though the median is 0.7 years, with the average possibly skewed by "outliers" discovered after years of hiding.] The longest-lived bug, a buffer overflow in networking code, went unnoticed for 20.7 years! [But 86.5% of bugs are found within five years.]
The research was carried out by relying on the Fixes: tag that is used in kernel development. Basically, when a commit fixes a bug, it includes a tag pointing to the commit that introduced the bug. Jenny wrote a tool that extracted these tags from the kernel's git history going back to 2005. The tool finds all fixing commits, extracts the referenced commit hash, pulls dates from both commits, and calculates the time frame. As for the dataset, it includes over 125k records from Linux 6.19-rc3, covering bugs from April 2005 to January 2026. Out of these, 119,449 were unique fixing commits from 9,159 different authors, and only 158 bugs had CVE IDs assigned.
It took six hours to assemble the dataset, according to the blog post, which concludes that the percentage of bugs found within one year has improved dramatically, from 0% in 2010 to 69% by 2022. The blog post says this can likely be attributed to:
The research was carried out by relying on the Fixes: tag that is used in kernel development. Basically, when a commit fixes a bug, it includes a tag pointing to the commit that introduced the bug. Jenny wrote a tool that extracted these tags from the kernel's git history going back to 2005. The tool finds all fixing commits, extracts the referenced commit hash, pulls dates from both commits, and calculates the time frame. As for the dataset, it includes over 125k records from Linux 6.19-rc3, covering bugs from April 2005 to January 2026. Out of these, 119,449 were unique fixing commits from 9,159 different authors, and only 158 bugs had CVE IDs assigned.
It took six hours to assemble the dataset, according to the blog post, which concludes that the percentage of bugs found within one year has improved dramatically, from 0% in 2010 to 69% by 2022. The blog post says this can likely be attributed to:
- The Syzkaller fuzzer (released in 2015)
- Dynamic memory error detectors like KASAN, KMSAN, KCSAN sanitizers
- Better static analysis
- More contributors reviewing code
But "We're simultaneously catching new bugs faster AND slowly working through ~5,400 ancient bugs that have been hiding for over 5 years."
They've also developed an AI model called VulnBERT that predicts whether a commit introduces a vulnerability, claiming that of all actual bug-introducing commits, it catches 92.2%. "The goal isn't to replace human reviewers but to point them at the 10% of commits most likely to be problematic, so they can focus attention where it matters..."
While interesting... (Score:2)
Without context, I don't know what to do with this. .7 years median seems quite okay to me.
Re: (Score:3, Funny)
Without context, I don't know what to do with this. .7 years median seems quite okay to me.
You want context? OK.
How about that one time at bug camp when the OS became irrelevant after we discovered speculative execution bugs that “only” affected damn near every x86 CPU that rolled off an assembly line for the last twenty-five fucking years.
Re:While interesting... (Score:5, Funny)
How about that one time at bug camp...
You couldn't find a way to work the flute into that?
Re: (Score:1)
How about that one time at bug camp...
You couldn't find a way to work the flute into that?
Not until “Flute” makes it to the tippy top of the TIOBE index. I only fuck the best.
Re: (Score:2)
Re: (Score:2)
What does that have to do with what I said?
How long to Microsoft, Apple, or the BSD people take?
Spectre and Meltdown, according to the Google AI, took three to four weeks to fix.
Seems reasonable to me as somewhat of a layman.
Re: (Score:2)
Re: (Score:2)
It's because you have to look inside, not "on" it. /s
Re: (Score:1)
Re: (Score:2)
Not just that, now if there are, shouldn't AI be able to fix it and apply those changes everywhere?
BS (Score:5, Insightful)
Re: (Score:2)
Re: (Score:3)
It depends on what you are looking for. The total time from the creation of the bug to its fix is important. A bug might be having adverse effects without being explicitly logged and known to developers (something like random disk corruption that nobody can reproduce - maybe it's a hardware problem, maybe it's software). Adverse effects include black hats keeping the bug to themselves. The total reaction time of the developers (e.g. once a bug is known how long does it take to fix) is also important.
Re: (Score:2)
If you don't know it exists you cannot fix it. So all that time until you find it should be subtracted. How long it takes to fix it should start at the time it was found.
That depends on whether your goal is to measure the efficiency with which kernel devs fix known bugs, or to estimate how many latent vulnerabilities exist in the kernel for attackers to exploit.
Both are important metrics for evaluating kernel security, but they address different sorts of risks. The time from identification to fix is particularly crucial because the vast majority of attackers don't do their own vuln hunting, they rely on already-discovered vulns that have been published or leaked. So min
Re: BS (Score:2)
Time from bug creation to fix and time from bug discovery to fix are both important. But only one is easy to measure, and thatâ(TM)s what the authors measured. This blog post wasnâ(TM)t driven by âoewhat question is important to answer?â, it was driven by âoewhat can we say with these data?â.
Re: (Score:2)
"How long it takes to fix it should start at the time it was found."
If I spend no time looking for bugs in my code, my code has no unfixed bugs.
& how long does it take for patches to get mer (Score:2)
More naunced (Score:5, Interesting)
I think that some bug scenarios are more complex than described in this article.
I recall working on a Bluetooth kernel crash more than 10 years ago. The crash was a race condition between 2 Bluetooth threads when the RFCOMM link was being disconnected. To trigger the crash, the execution order of the 2 threads had to be reversed from "normal" so that 1 thread freed the memory containing the reference counter that the other thread relied on to not free the memory! This was a case of inappropriate use of reference counters when an abrupt disconnect had to be handled.
The crash was observed under very heavy processor loading that caused CPU starvation to the Bluetooth threads resulting in an "abnormal" execution order. Static analysis suggested that the reference counter would provide lock protection because reference counters are used through-out the kernel's networking protocol implementations. However, the flaw is that the reference counter can momentarily go to zero to trigger memory freeing but a second thread can come along and add 1 to the reference counter despite the memory now being in a freed state.
Looking through the commit history revealed about 4 attempts to fix this issue by 4 individuals over a period of about 5 years. I had to remove these ineffective and misguided workarounds to implement a proper fix that completely removed the use of the reference counter and immediately delete the state memory as soon as the disconnect was detected. NULL pointer checks prevented any use-after-free scenarios.
Therefore, kernel bugs may go through several workaround fixes before the root cause is fully understood and fixed. So saying that a bug was resident for 5 years may not mean that no attempts had been made to resolve it during that period.
Re: More naunced (Score:2)
Yup, and other areas of nuance, too.
The story says half of the issues are fixed quickly. The actual research post (not the short writeup /. links to) adds that 30% are what they called "self fixes" and incomplete features being implemented rather than stable features with defects. Most developers consider this just normal development processes, not the dangerous types of bugs.
It doesn't break down issues by severity and frequency. Bugs caught in experimental features not used by anyone VS bugs in features
Re: (Score:2)
There are systems that have been in place for years, even decades, and it's not showing errors in the billions of computers using it, instead the issue it "fixes" was a source code scan where a theoretical exploit is discovered; follow this specific pattern under that specific load with this variety of race condition in order to reveal a trivially small amount of program state information.
You're correct, of course, that fixes for ordinary bugs that don't actually cause any problems in production don't matter very much, but this is definitely not the case with respect to the subset of bugs that cause security vulnerabilities. The fact that a bug doesn't manifest under normal use says nothing about whether or not it can be triggered by an attacker.
Re: (Score:2)
Re: (Score:2)
The fun part of this Bluetooth crash in an embedded ARM system was that a particular test bench doing a software update procedure on the ARM system had a Linux kernel crash in the Bluetooth RFCOMM sub-system but Bluetooth was not used or required for the software update procedure. The failure was only observed on this test bench.
It transpired that a previous tester had put a paired Bluetooth enabled smartphone in the desk draw underneath the test rig. The guy running the software update procedure was unawar
Re: (Score:2)
Therefore, kernel bugs may go through several workaround fixes before the root cause is fully understood and fixed. So saying that a bug was resident for 5 years may not mean that no attempts had been made to resolve it during that period.
Indeed, there are always corner cases that will not be evaluated correctly by any given metric. That doesn't invalidate the research, though. The fact that the metrics are imperfect doesn't make them useless; they still provide a useful indicator of the state of kernel security and how it has changed over time.
Re: More naunced (Score:3)
My favorite bug was when they started using message-signaled interrupts. When enabling MSI, they didn't disable the traditional IRQ, and my machine would keep delivering it. In particular, the network card would do something to toggle the IRQ line whenever a packet came in, but would leave the line triggered when idle. If this persisted for five minutes, the kernel would decide that line was stuck and mask it, but it was shared with my hard drive, whose driver would then never find out that operations had c
Tootsie Roll Pop Owl says... (Score:2)
One.. Two... Three... decades. That's how long I've been trying to get working video drivers. Every time I retire a Windows or Mac system, I try to extend its life with Linux. I fail every time. All the way back to the ET4000.
Quality of pre-release (Score:2)
The AI model will not help (Score:2)
It may catch "92%", but these will be the simple-to-find ones. CVE-level? Probably almost none.
I'd like to see some comparisons. (Score:3)
Re: (Score:2)
IIUC, you can keep on wondering. This report depended on public information, which Linux discloses and neither Mac not MSWind make public.
Re: (Score:2)
But if it's something you'd like to see, regardless of whether or not you ever will, I think my point stands.
I've heard people say open-source gets fixed
This is about how long it takes to FIND bugs (Score:2)
Nothing here (in the summary -- I didn't click through) about how long it takes to FIX them, once found.