Google Security Researchers Accuse CentOS of Failing to Backport Kernel Fixes (neowin.net) 42
An anonymous reader quotes Neowin:
Google Project Zero is a security team responsible for discovering security flaws in Google's own products as well as software developed by other vendors. Following discovery, the issues are privately reported to vendors and they are given 90 days to fix the reported problems before they are disclosed publicly.... Now, the security team has reported several flaws in CentOS' kernel.
As detailed in the technical document here, Google Project Zero's security researcher Jann Horn learned that kernel fixes made to stable trees are not backported to many enterprise versions of Linux. To validate this hypothesis, Horn compared the CentOS Stream 9 kernel to the stable linux-5.15.y stable tree.... As expected, it turned out that several kernel fixes have not been made deployed in older, but supported versions of CentOS Stream/RHEL. Horn further noted that for this case, Project Zero is giving a 90-day deadline to release a fix, but in the future, it may allot even stricter deadlines for missing backports....
Red Hat accepted all three bugs reported by Horn and assigned them CVE numbers. However, the company failed to fix these issues in the allotted 90-day timeline, and as such, these vulnerabilities are being made public by Google Project Zero.
Horn is urging better patch scheduling so "an attacker who wants to quickly find a nice memory corruption bug in CentOS/RHEL can't just find such bugs in the delta between upstream stable and your kernel."
As detailed in the technical document here, Google Project Zero's security researcher Jann Horn learned that kernel fixes made to stable trees are not backported to many enterprise versions of Linux. To validate this hypothesis, Horn compared the CentOS Stream 9 kernel to the stable linux-5.15.y stable tree.... As expected, it turned out that several kernel fixes have not been made deployed in older, but supported versions of CentOS Stream/RHEL. Horn further noted that for this case, Project Zero is giving a 90-day deadline to release a fix, but in the future, it may allot even stricter deadlines for missing backports....
Red Hat accepted all three bugs reported by Horn and assigned them CVE numbers. However, the company failed to fix these issues in the allotted 90-day timeline, and as such, these vulnerabilities are being made public by Google Project Zero.
Horn is urging better patch scheduling so "an attacker who wants to quickly find a nice memory corruption bug in CentOS/RHEL can't just find such bugs in the delta between upstream stable and your kernel."
People still use CentOS? (Score:3)
Re: (Score:3)
No, not everyone.
A bunch moved to AlmaLinux instead. :-P
Re: (Score:2)
Some people went to Oracle Linux as well, while others jumped ship to Amazon Linux (which is also mostly RHEL compatible).
IBM really screwed the pooch by getting rid of their free RHEL clone.
Re:People still use CentOS? (Score:4, Informative)
>A bunch moved to AlmaLinux instead. :-P
Yep, Alma
But in this case, Alma and Rocky both use upstream RHEL and RHEL didn't have updates for this. So none of them did.
Re: (Score:1)
Re: (Score:2)
>"Well, no. RHEL has the updates, and so has Alma and Rocky. But Centos Stream lags behing, as was intended. Which sucks."
OK, I was basing what I said on feedback from others that RHEL did *not* have the update, I didn't know for sure. The point being that Stream is supposed to be AHEAD of RHEL now (defeating the whole point of CentOS), unlike Alma/Rocky and old [non-stream] CentOS, which will be slightly behind.
>"We had some c8stream machines, but the very slow updates forced us to abandon it."
Yeah,
Re: (Score:1)
Well, no. RHEL has the updates, and so has Alma and Rocky.
That's false. If you look at the Red Hat CVE pages, they all list RHEL as "affected". If RHEL had the fix, that page would say "fixed" with a link to the errata and the release date. So RHEL and all RHEL rebuilds are vulnerably to these CVEs.
In fact, of these distros, CentOS Stream 9 is the only one that has any of the fixes. It got the fix for CVE-2023-0590 even before the Project Zero announcement went out.
Re: (Score:1)
Re:People still use CentOS? (Score:4, Insightful)
Some use it, because migrating to newer versions or alternate distros is a pain. I find it frustrating that we have an important system still on CentOS 6, and it's mostly because the team for it is extremely tiny and overworked.
Also, sometimes it's needed - we have older releases that need compiling with older distribution releases. They won't work with newer compilers, without spending a ton of work backporting fixes, and you can't backport fixes to existing releases, you can only make a bug fix release for them. This is quite the complete opposite maybe of web software where you just upgrade and forget that the older code ever existed, but when customers have support contracts and you can't force them to upgrade immediately you have to support older stuff. Even if that means working on the Linux equivalent of XP.
Also a pain, some people are using CentOS for some things, older releases, and there is a binary compatibility problem. Tools or utilities build for newer GNU C libraries won't run if the libraries are too old. Also a pain, bigger than the pain of trying to stamp out the 32-bit utilities. That whole department I think painted themselves into a corner, the people who knew how to set it all up left and the remaining people aren't the Linux experts anymore.
Anyway, real life means you deal with problems that don't exist in the imaginary world where everything is done in an ideal way the first time.
build static... (Score:2, Interesting)
Re: People still use CentOS? (Score:1)
Huzzah for Rocky! But its good for everyone that there are others. The best place I've seen for metrics is EPEL and these graphs pull from that:
https://rocky-stats.tiuxo.com/ [tiuxo.com]
A decade behind ? (Score:2)
Re: (Score:3)
It's on purpose. CENTOS follows Red Hat's kernel tree because it was for people who wanted a distro as close as possible to RHEL without paying for it.
Red Hat have a published policy of backporting bugfixes without changing the release number in a major release. The 'E' in CENTOS and RHEL stands for Enterprise and the presumption is that enterprise customers want this to be stable for the whole 10 year lifecycle.
Re: (Score:3)
Yup, bleeding edge isn't so good in the enterprise. Not for Linux, not even for Windows or MacOS. Stability is important.
Re: A decade behind ? (Score:2)
Bleeding edge is pretty much what you want for security patches, though. The whole "don't touch it" mentality is likely the source of most security breaches behind mis-configuration.
Re: (Score:1)
Re: (Score:1)
Re: (Score:2)
Not only does it create a bunch of work but occasionally new software depends on new kernel features, too. Then they can sell you another license for a newer RHEL for the new box you'll need to run the new software, while you're still maintaining your old box with ancient kernels with backported fixes because it is still working and you don't want to modernize it.
Re: A decade behind ? (Score:1)
I know from experience that RHEL backports fixes, not just code. So they may recommend a specific setting or firewall configuration to mitigate the problem.
There are various problems, having looked into it a bit further: the latest 5.15 kernel has major issues with the Intel 25/40/100G Ethernet adapters, the fix is in Kernel 6.2 but it leaves a ton of âenterpriseâ(TM) customers in the lurch when you have your LTS kernel not supporting major hardware. The CVEs indicated arenâ(TM)t necessarily
Re: (Score:2)
Yes, true. Especially for linux and "well known" software. However we have an embedded system with customers who are reluctant to upgrade because upgrades involve time and expense, they do their own testing ahead of upgrades, and mistakes are a major cost.
Re: (Score:2)
RHEL 6 / CentOS 6 is based off of the 2.6.32 kernel and is considered EOL (unless you’re paying for the extended maintenance but that’s just about to run out, too).
RHEL 9 is the latest and is based off of the 5.14 kernel.
RHEL releases get standard support for 5 years plus another 5 if you want to pay for it. Companies like having stability, including ABI stability, for that length of time.
Re: (Score:1)
Re: (Score:2)
It's so that your kernel extensions and everything else keep working. CentOS is supposed to be stable in the sense that stuff that works keeps working for the lifetime of the major release (although that ceased to be true to some extent with CentOS 7 where there were significant breaking changes on point releases that broke compatibility for filesystem, IPv6, Infiniband and various other things).
The real problem (Score:1, Funny)
If there were just one Linux distro this (and many other problems) would not exist
Re: The real problem (Score:1)
nonsense, could have one lazy ass distro where things are unpatched. I sure as hell am not using the distro you like .
Re: (Score:2)
Or they would, but there wouldn't be a way to get out of it because there would be no more current alternative.
CentOS Is Dead. (Score:2)
Long live CentOS.
Re: (Score:2)
CentOS is supposedly a dev stream now (Score:2)
IBM states that CentOS Stream is supposed to lead RHEL proper, which makes this behavior even more inexplicable.
Unless, of course, Stream only exists to let IBM pretend it didn't nerf CentOS strictly to screw everyone over. In that case, everything fits nicely.
Re:CentOS is supposedly a dev stream now (Score:4, Informative)
IBM states that CentOS Stream is supposed to lead RHEL proper, which makes this behavior even more inexplicable.
I don't know why CentOS has been picked out in this article, when it's clear that the problem also exists in RHEL:
"As detailed in the technical document here, Google Project Zero's security researcher Jann Horn learned that kernel fixes made to stable trees are not backported to many enterprise versions of Linux. To validate this hypothesis, Horn compared the CentOS Stream 9 kernel to the stable linux-5.15.y stable tree.... As expected, it turned out that several kernel fixes have not been made deployed in older, but supported versions of CentOS Stream/RHEL."
Re: (Score:2)
I don’t think anyone seriously uses CentOS Stream in a production environment. I’m sure someone will chime in and say they do but there’s really no commitment to fix CVEs there with urgency like there is for RHEL.
I’m more concerned if once these changes make it into an actual RHEL point release they don’t bother fixing them.
Re: (Score:3)
Okay, the article sucks. I checked out the CVE page on Red Hat’s portal and the problem exists in the stable releases of RHEL.
Red Hat considers them to be “moderate” issues. I don’t have a comment on their interpretation of severity but that’s likely why they’re not in a hurry to fix.
Didn't need another reason to avoid RHEL (Score:3)
but this is an even more compelling one than "remember, this is IBM", or all the bullshit RH has pulled in the last few years with systemd (and gnome for the desktop releases).
Not patching known exploits completely destroys the whole "stable" argument that is pretty much the entire point of using RHEL in the first place. It means the only sane approach is to switch to LTS mainline kernels instead, and do all the regression testing yourself - at which point you have to start wondering what you're paying RH for, when you'd get better support from several of the generic off-the-shelf distros.
Re: Didn't need another reason to avoid RHEL (Score:1)
The question is whether the CVEs are at all relevant, the kernel brings out continuous updates for security that depend on established access whether that is physical or root. That being said, Iâ(TM)m waiting on Ubuntu (yes paid version) to fix an actual remote buffer overflow that results in full compromise bug in one of the packages in their repo thatâ(TM)s been open for going on 6 months.
Re: (Score:1)
Re: (Score:1)