CERN Transitioning From RHEL To Debian (phoronix.com) 47
Longtime Slashdot reader Microsplat: CERN plans to have all 2,200+ industrial computers and embedded systems in its accelerator-control infrastructure running Debian 13 by the end of 2026. These are systems used for accelerator control, laboratory equipment and other operational functions ... They also generate and interact with a hefty amount of data, although this Linux.com article refers more broadly to CERN's computing infrastructure. It's unclear whether the Debian migration covers all server infrastructure or what scientific-research-based systems are included. Our shops have done much the same in recent years, mostly due to CentOS and CFEngine getting the can. Phoronix adds some additional context in its article: CERN was a longtime RHEL/CentOS shop, previously co-maintained the Scientific Linux RHEL derivative, moved to CentOS in 2015, and later considered CentOS Stream. CERN says Red Hat's adoption of the "-march=x86-64-v2" compiler flag by default, which it viewed as "forced obsolescence" of older hardware, was the "straw that broke the camel's back."
Rocky Linux (Score:4, Interesting)
I'm a bit surprised they didn't go for Rocky Linux. Nothing wrong with Debian, mind, but Rocky is downstream from CentOS/RHEL and would be an easier migration.
Re: Rocky Linux (Score:3)
Do they not use the same arch?
Re: (Score:1)
Debian even shipped 32-bit x86 kernels until Debian 12, so it supports architectures longer than other large Linux distributions.
A major factor I'm guessing.
Re:Rocky Linux (Score:4, Funny)
RHEL to Debian. Clearly they want to use a major distribution. Not something that may not be around next year.
Re: (Score:2)
Roky would probably have the same problem that made them switch. CERN will have a lot of radiation hardened and the like hardware that would be hard and expensive to replace.
Re: (Score:3)
Based on the skim of their forum, they declined to support citing their desire for 1:1 binary compatibility with RHEL. AlmaLinux considered it, but the compilation costs (which also pull in EPEL) are very high. Then there is the issue of VM support, which now apparently requires some more advanced processor features. It's a heavy lift and not one that I expect Debian to maintain for much longer either.
Re: (Score:3)
They used to do their own white label distribution of RHEL called Scientific Linux with lots of cool add on packages. I'm not sure why they stopped doing so?
Re: (Score:1)
The normal servers run a mix of RHEL and Alma Linux, Alma chosen over Rocky because of better patching record for security issues and some other things.
For the control systems that are ancient CPUs in expensive embedded boxes, Alma and Rocky have the same issue as RHEL in not having working support for older CPUs.
Re: (Score:2, Flamebait)
Just say no to RPM.
Re: (Score:2)
Did something change that I am unaware of? I had thought that Redhat had "bought" CentOS" and permanently killed it. Is it alive again? A viable option?
"I never thought I'd see a Resonance Cascade, let (Score:4, Funny)
"I never thought I'd see a Resonance Cascade, let alone create one..."
Does this affect CERN and only CERN? (Score:3)
I do wonder what the Venn Diagram is of people who are running CPUs and systems which are older than 18 years but equally need to run the absolute latest OS version.
I would have thought there'd be a desire to not upgrade software to prevent the old "forced obsolete" systems to have to deal with modern day bloat, or the desire to not deal with breaking changes in major OS version upgrades.
But on a slightly adjacent thought, presumably CERN is highly optimising their chosen version of Linux for the purpose they are using it for, I'm actually genuinely surprised they haven't basically rolled their own version.
Re: (Score:2, Interesting)
Maintaining and running different distributions or major OS versions for different hardware obviously has additional overhead, so you want to avoid that as much as possible. The older hardware is likely still networked, for example, so you definitely have to judge security if not upgrading.
Groups like CERN do roll their own OS setups but it saves a lot of continued work if the main configurations start with a base you don't have to build and maintain yourselves. Standard OS builds are more reliable if other
Re: (Score:2)
You can run Kernel 7.2 on a 486 https://github.com/Sharktastic... [github.com]
Re: (Score:2)
Which does not help you of the rest of your distro does not run ...
Re: (Score:2)
You "can" do a lot of things. You "can" play Doom on TI82 calculator. The question was: does it make sense to do so, what are the benefits it brings vs the downsides?
Re:Does this affect CERN and only CERN? (Score:4, Insightful)
[...] but equally need to run the absolute latest OS version.
In the age defined by cybersecurity, this is a very very [*beep*] comment.
They have said "the last straw", not "the main reasons". I've seen recently a number of migration to Debian, because the current generation grew up, got tired of the "latest greatest" stuff that's half-broken all the time, and simply want stuff that "just works". Funnily enough, Debian is still "it". (Heck, even for the gamers.)
Doubt RHEL goes anywhere, but it's funny how many people grow up out of the denial that RHEL, well, sucks. Started gradually sucking even more after RH started going after market share of commercial *nix systems, and started adopting many traits of the monolithic commercial *nix OSes.
Re:Does this affect CERN and only CERN? (Score:4, Insightful)
RHEL/CentOS was fine until IBM bought them.
Re: (Score:2)
THE single most right-on-the-point comment here tonight. That's exactly it.
I moved to AlmaLinux almost when it was born, but am always worried that it may get killed, just as CentOS was when IBM bought it and made it the guinea pig between Fedora and RHEL.
An institution like the CERN cannot live with that kind of instability.
Re: (Score:2)
IMO: even before that. My cut-off date is when RHEL was certified to run Oracle RDBMS. A cash-cow has appeared, and the rest of the OS (along with community) became just a baggage.
Re: (Score:3)
RHEL isn't remotely considered the latest and greatest. I'm not sure specifically what CERN's problems were (no doubt they had many legitimate ones), I was just curious why the idea of the latest OS not running on 18+ year old hardware was a design consideration to them.
Cybersecurity is not achieved by running the latest and greatest OS. It's achieved by defence in depth. It's achieved by isolation of systems and control of information. You very much can have an OS out of date and insecure on a network if y
Re: (Score:2)
I wouldn't be a bit surprised to learn that systems like that are completely air gapped and that updates/updates are administered via removable media.
Re: (Score:3)
I would guess it effects anybody with Linux-based industrial control systems. These can be a lot more expensive than regular computers. Also, Debian is generally not about the "absolute last" version of anything, but rather conservative in its updates.
Re: (Score:2)
Well this is where I'm coming from. Industrial control systems are not normally kept up to date with the latest OS. For actual control systems they are usually certified combinations of hardware and software. You don't get to upgrade Windows or Linux, what you do is buy a new database server or a new engineering workstation with the new OS on it.
Re: (Score:2)
The thing is that CERN is probably doing very custom stuff here and needs to do their own thing on industrial hardware.
Re: (Score:2)
I do wonder what the Venn Diagram is of people who are running CPUs and systems which are older than 18 years but equally need to run the absolute latest OS version.
I hope normal distros don't start pulling this shit. Have an Atom server circa 2009 that won't even boot if this sort of crap started happening. It's not like you can elect not to update because they stop updating older versions.
Also have a laptop of the same vintage I use regularly... perfectly good hardware even by today's standards. Completely unnecessary, wasteful and pointless in the current environment with absurd hardware pricing. There is no reason they can't just use dispatching to exploit new
Re: Complains about compiler flags (Score:1)
Systemd doesn't do that, although it causes other problems, like breaking early boot logging when part of its mission was supposed to be fixing logging. It doesn't have significant requirements...
Re: (Score:3)
I just moved my last remaining Debian to Devuan. Essentially Debian without the bullshit.
Re: (Score:2)
Re: (Score:2)
I've been on Devuan for several releases. Root on [mirrored] ZFS based on the Debian instructions with some tweaks like writing init scripts, Nvidia driver and CUDA installed from the runfile, X11, KDE, wireplumber, etc etc. My Zen3 MiniPC and Zen+ laptop also, less Nvidia. The laptop is even still on the last version because it just runs a browser and CHIRP.
Re: (Score:2)
I was on non-systemd Debian before, but that seems to get harder. And yes, I have some custom init-scripts as well. Easy with sysV init. I regard Devuan as Debian with an added sanity-filter.
Re: (Score:1)
But the forced obsolescence caused by systemd is fine.
What hardware features forced by systemd have obsoleted old hardware?
If you just mean increased overhead, I'd like to know how much systemd adds on a minimal non-GUI systems. Any references to measurements would be appreciated!
Re: (Score:2)
First, Debian can still be run without systemd. Well, mostly and it is getting harder. Second, migration to Devuan is really easy.
That said, systemd is mostly a gross violation of all UNIX system design principles, but it should run on older hardware.
RHEL versus Debian philosophy? (Score:5, Interesting)
With RHEL 10 adding more complex stuff like Lightspeed and immutable kernels, it is starting to have a lot more moving parts. RHEL used to be the production choice, but as time went on, there is a lot of oddball stuff, and oddball decisions made. For example, even though Fedora has btrfs, RHEL doesn't have btrfs, even as a -supported- option. (Yes, you can slap ZFS on it, but supported is the big thing).
Debian, on the other hand, doesn't have snaps, flatpaks, or other subsystems present by default. It "just works" out of the box. Plus, other systems like Proxmox can be installed on vanilla Debian with ease.
I would assert that because that fewer moving parts are needed, it makes things easier. You can take filesystem snapshots without Stratis, for example.
It's a treadmill (Score:4, Interesting)
RHEL 8 / Rocky 8 just needs x86-64-v1, which is really old and is essentially all 64-bit x86 CPUs
RHEL 9 / Rocky 9 requires x86-64-v2 (if you're on x86), and I understand it's supported until 2032 - if that featured in their decision
RHEL 10 / Rocky 10 requires x86-64-v3 (see https://docs.rockylinux.org/10... [rockylinux.org] ) and can make 2013-era Xeons obsolete.
I've an older Dell R720 server that can't run RHEL 10/ Rocky10 on bare metal, but it's still quite functional otherwise. So I suppose it comes down to what hardware CERN have available and want to keep operational, whilst using the same OS across the whole campus.
If interested, https://en.wikipedia.org/wiki/... [wikipedia.org] has more details on the CPU levels.
And I'll note for completeness that like Debian, AlmaLinux still only requires x86-64-v1, so CERN did have the option of an "almost clone" of RHEL.
Re:It's a treadmill (Score:5, Informative)
New Intel Elkhart Lake CPUs without x86-64-v3 and AVX2 were still being introduced as recently as 2023!
https://en.wikipedia.org/wiki/... [wikipedia.org]
Re: (Score:2, Informative)
And I'll note for completeness that like Debian, AlmaLinux still only requires x86-64-v1, so CERN did have the option of an "almost clone" of RHEL.
The update to the article said all the experimental and compute nodes will remain on alma. Which is what the nodes I work on over there are. Not sure why that won't work on the accelerator control machines, would simplify things labwide. The teams at CERN and Fermilab that did Scientific Linux now contribute to Things That Work on Alma, for example.
Re: (Score:1)
That's pretty old. (Score:2)
Isn't it x86-64-v3 ? (Score:2)
AFAIK it's v3 since RHEL10.
It's a pretty sensible floor, it means 15+ years old hardware isn't supported.
If it was v2 it would give you 30 years maybe? Maybe time to upgrade?
AlmaLinux anyone? (Score:1)
It's strange they're moving to the Debian family when AlmaLinux exists and is by far the best RHEL-based distro for running on old server kit. They've kept old RAID drivers that RHEL scrapped and still support x86-64-v2 even in AlmaLinux 10 (there's a separate build specifically for that). AlmaLinux is also the creator of ELevate, which piggy-backed on the RHEL Leapp utility to provide an impressive set of upgrade paths going back all the way to CentOS 6 - see https://almalinux.org/elevate/ [almalinux.org]
how old? (Score:1)
CERN chooses Debian so its hardware can outlive RH (Score:2)
The problem is RHEL 9 moved to x86-64-v2 as its baseline CPU target. That excludes some of CERN's older machines. About 47% of the fleet u