Over 14 Million Servers May Be Vulnerable To OpenSSH's 'RegreSSHion' RCE Flaw (zdnet.com) 90
An anonymous reader quotes a report from ZDNet, written by Steven Vaughan-Nichols: Hold onto your SSH keys, folks! A critical vulnerability has just rocked OpenSSH, Linux's secure remote access foundation, causing seasoned sysadmins to break out in a cold sweat. Dubbed "regreSSHion" and tagged as CVE-2024-6387, this nasty bug allows unauthenticated remote code execution (RCE) on OpenSSH servers running on glibc-based Linux systems. We're not talking about some minor privilege escalation here -- this flaw hands over full root access on a silver platter. For those who've been around the Linux block a few times, this feels like deja vu. The vulnerability is a regression of CVE-2006-5051, a bug patched back in 2006. This old foe somehow snuck back into the code in October 2020 with OpenSSH 8.5p1. Thankfully, the Qualys Threat Research Unit uncovered this digital skeleton in OpenSSH's closet. Unfortunately, this vulnerability affects the default configuration and doesn't need any user interaction to exploit. In other words, it's a vulnerability that keeps security professionals up at night.
It's hard to overstate the potential impact of this flaw. OpenSSH is the de facto standard for secure remote access and file transfer in Unix-like systems, including Linux and macOS. It's the Swiss Army knife of secure communication for sysadmins and developers worldwide. The good news is that not all Linux distributions have the vulnerable code. Old OpenSSH versions earlier than 4.4p1 are vulnerable to this signal handler race condition unless they are patched for CVE-2006-5051 and CVE-2008-4109. Versions from 4.4p1 up to, but not including, 8.5p1 are not vulnerable. The bad news is that the vulnerability resurfaced in OpenSSH 8.5p1 up to, but not including, 9.8p1 due to the accidental removal of a critical component. Qualys has found over 14 million potentially vulnerable OpenSSH server internet instances. The company believes that approximately 700,000 of these external internet-facing instances are definitely vulnerable. A patch, OpenSSH 9.8/9.8p1 is now available. Many, but not all, Linux distributions have made it available. If you can get it, install it as soon as possible. If for whatever reason you're not able to install a patch, Vaughan-Nichols recommends you set LoginGraceTime to 0 in the sshd configuration file and use network-based controls to restrict SSH access, while also configuring firewalls and monitoring tools to detect and block exploit attempts.
It's hard to overstate the potential impact of this flaw. OpenSSH is the de facto standard for secure remote access and file transfer in Unix-like systems, including Linux and macOS. It's the Swiss Army knife of secure communication for sysadmins and developers worldwide. The good news is that not all Linux distributions have the vulnerable code. Old OpenSSH versions earlier than 4.4p1 are vulnerable to this signal handler race condition unless they are patched for CVE-2006-5051 and CVE-2008-4109. Versions from 4.4p1 up to, but not including, 8.5p1 are not vulnerable. The bad news is that the vulnerability resurfaced in OpenSSH 8.5p1 up to, but not including, 9.8p1 due to the accidental removal of a critical component. Qualys has found over 14 million potentially vulnerable OpenSSH server internet instances. The company believes that approximately 700,000 of these external internet-facing instances are definitely vulnerable. A patch, OpenSSH 9.8/9.8p1 is now available. Many, but not all, Linux distributions have made it available. If you can get it, install it as soon as possible. If for whatever reason you're not able to install a patch, Vaughan-Nichols recommends you set LoginGraceTime to 0 in the sshd configuration file and use network-based controls to restrict SSH access, while also configuring firewalls and monitoring tools to detect and block exploit attempts.
beyond belief (Score:5, Interesting)
Re: (Score:2)
Re:beyond belief (Score:5, Interesting)
Looks that way. I really expect better from the OpenSSH team. To be fair, this is the first time in a long while that they have screwed up. The last SSH security problems was clueless distro maintainer patching systemd crap into OpenSSH.
It is not quite as bad, however. Apparently, with 100 concurrent connection attempts the attack takes about 8h on average and apparently the defaults, at least on Debian, are more like a max of 10 concurrent connections. So anybody that patches fast is probably still pretty safe. On the other hand, sshd announces its version on connection (it needs to, apparently - just telnet to port 22 on the target), so attackers can identify targets very easily and cheaply. Hence anybody stupid or incapable enough to not patch fast will probably get attacked.
My servers, incidentally, are not vulnerable. They are either Debian/Devuan oldstable (sshd 8.4) or automatically patched from secureity.debian.org and hence already had the patch active yesterday. If this continues, I will add an sshd restart to the automatic update process though, had to do that manually on two machines yesterday, fortunately both not directly Internet-reachable.
Re:beyond belief (Score:4, Informative)
FYI Ubuntu patched the existing versions of OpenSSH, so you don't necessarily need 9.8p1 to be safe. For example Ubuntu 24.04 latest openSSH is OpenSSH_9.6p1 Ubuntu-3ubuntu13.3 and includes the fix.
More info: ubuntu.com [ubuntu.com]
Re: (Score:2)
Interesting. At least if the Ubuntu part pf the version is not advertised by sshd (on Debian it is per default, although I have turned that off), this will make it harder for attackers to find out whether a potential target is vulnerable.
Re: (Score:3)
FYI Ubuntu patched the existing versions of OpenSSH, so you don't necessarily need 9.8p1 to be safe. For example Ubuntu 24.04 latest openSSH is OpenSSH_9.6p1 Ubuntu-3ubuntu13.3 and includes the fix.
You'll see the same thing with AlmaLinux and the others - that's typically how LTS / Enterprise distributions of Linux work. The idea is to provide a stable, known platform that software makers can depend on for the entire life of a distribution's major release (e.g. Ubuntu 22, AlmaLinux 9). So for any packages that other software might depend on, the distro maintainers pick a particular release version at launch. Then, when any necessary fixes come up, they back-port those to that fixed release.
Re: (Score:1)
wrong, it's a bug distro makers made in the port of openssh using glibc. The bug doesn't exist in the BSD world.
Re: (Score:2)
I apply security updates as they're rolled out by the vendor of whichever distro a system has. Which, apparently, means I was "force fed" bad code sometime ago, but I was then force fed "good" code more recently.
I suppose you manually review every single patch for every single software package on all of your systems?
Re: (Score:2)
Re: (Score:2)
Looks that way. I really expect better from the OpenSSH team.
This only affects Linux glibc systems. OpenBSD does not have this issue. My guess is due to to how complex Linux has become, I would not be surprised if there are a few more of these. From what I have read, right now, you can get in on a slow 386 Linux System after 5 or so hours of trying. No one was able to, as of yet, get into a 64 bit system.
Re:beyond belief (Score:4, Interesting)
Re: (Score:2)
Debian, are more like a max of 10 concurrent connections. So anybody that patches fast is probably still pretty safe.
The other thing is This is on 32-bit systems in a lab.
Some embedded 32-bit devices exposed to the internet may be in grave danger, but 64-bit servers Not so much.
Re: (Score:3)
Re:beyond belief (Score:5, Insightful)
That is why everyone should have a suite of regression tests.
Regression testing [wikipedia.org]
Re: (Score:2)
To be fair, this requires a very elaborate test and the old test code (if there was any) would likely not even work. I think the problem here is more in the area of code documentation, change review and security architecture.
I hope this will lead to some introspection in the OpenSSH team. Although, again, to be fair, their native OpenBSD is not vulnerable. This is a glibc problem.
Re: (Score:3)
To be fair, this requires a very elaborate test and the old test code
The canonical TDD way to do it is:
1. Get a bug report.
2. Fiddle around until you can reproduce the bug.
3. Write code that can reliably reproduce the bug.
4. Fix the bug.
5. Run the code from #3 to verify the bug is fixed.
6. Add the code from #3 to the test suite so it is run automatically.
If you're doing it right, you shouldn't need to do much additional work because you had to write the test to fix the bug in the first place.
Re:beyond belief (Score:4, Informative)
Yes. And ordinarily this works just fine. But for a low-probability race condition that, on top, does not happen at all on the native OpenSSH platform _and_ that does likely require very specific conditions not quite the same as when it happened before, that approach does not really work.
That is one reason why you should always be very careful to completely avoid race conditions. They typically cannot reliably be tested for and they may be very sensitive to details that usually do not matter in regular functional testing. They could well happen on one test system and not at all on a slightly different test system, that may even have exactly the same software configuration. Even things like exact server room temperature may have an influence. These types of bugs are really the worst-case for any testing approach.
Re: (Score:2)
the correct way to ammend a poster you believe is wrong is:
1. carefully read the post you think contains an error
2. carefully read at least the abstract, making sure you understand the context
3. if in doubt, repeat #1 and #2
4. if you still believe you know better, post your correction
if you're doing it right, you shouldn't appear as a clueless doofus.
Re:beyond belief (Score:5, Informative)
The current test case can hit the bug on 32-bit systems by maxing out the number of sshd connections for 6 to 8 hours. Indications are it is harder to hit on 64-bit systems. And it is not directly the sshd code but a thread-unsafe issue in a library that they are calling (syslog call).
Slightly less bad than it could have been (Score:5, Informative)
OpenSSH’s release note for version 9.8/9.8p1 indicates that successful exploitation has been demonstrated on 32-bit Linux/glibc systems with ASLR enabled. They state that the attack typically requires 6-8 hours of continuous connections to the server.
While exploitation on 64-bit systems is considered possible, it has not yet been demonstrated. The advisory notes that these attacks will likely be improved upon.
https://www.openssh.com/releas... [openssh.com]
Re: (Score:3)
Also, the 8h average seems to be for 100 concurrent connections. The defaults, at least on Debian, seem to be only 10 concurrent connections. Hence this may take a while. As the 64 bit version seems to not yet be vulnerable, anybody that patches now is probably good. Anybody that does not patch will likely get attacked, however, as sshd announces its version on a simple telnet to port 22. Apparently ssh needs that to work. Incidentally, Qualys probably just scanned the whole IPv4 range for that number. Not
Re: (Score:3, Insightful)
Not really.
A version known not to be vulnerable won't be attacked.
A version that is known vulnerable if unpatched will be attacked. Versions allow attackers to narrow the focus of their attack resources to possible targets from all targets.
Re: (Score:3)
What if my system is 36 bits? https://en.wikipedia.org/wiki/... [wikipedia.org]
Re: (Score:2)
What if my system is 36 bits? https://en.wikipedia.org/wiki/... [wikipedia.org]
Then you're out of luck, dude. I upgraded to 32-bit [wikipedia.org] from 36-bit and Linux still doesn't support that!
You could have better success with other 36-bit machines, though. Like IBM 709/709x, GE-645 (a.k.a. Multics machine) or even the modern UNIVAC 1100/2200 .
Time for CVEs to include likelihood of exploit. (Score:4, Informative)
Yes, this has a high CVE rating because ***IFF*** it were to be exploited things would be bad.
However, 1/10000 chances of exploit to get the race condition failure mitigates that CVE to
be about a 1.0.
This SHOULD be re-fixed immediately, no doubt. But given the odds of exploitation, it's less
dangerous than running with scissors.
> It's hard to overstate the potential impact of this flaw.
Yeah, it is hard to overstate, so please stop overstating it.
Speaking for my ITSEC friends who are tired of the wailing.
Re: (Score:1)
Re:Time for CVEs to include likelihood of exploit. (Score:4, Interesting)
It has an expected time-to-hack of 8h with 100 connections. As one attempt takes 2 minutes, that is 1/24000 per try, but the attack can be parallelized. As a result, _everybody_ affected should patch now. That is "patch today", not "patch sometime in the next weeks". As Linux generally has reliable updates (unlike Windows), that should not cause problems, but it needs to be done. And do not forget to restart sshd.
There is some protection from the current attack not working on 64 bit and apparently at least Debian seems to have a (soft) limit of 10 concurrent connections. But, since sshd announces its version (just telnet to port 22, seems to be needed for ssh to work), attackers can find potential targets fast and easily.
Re:Time for CVEs to include likelihood of exploit. (Score:5, Informative)
You can only parallelize it up to the number of simultaneous connections allowed by the server, which defaults to 10.
By tying up all the allowed connections, you will also cause a DoS and prevent anyone else from logging in - which is likely to get noticed.
Re: (Score:3)
True and that is another good thing. To nit-pick, that is the maximum number of unauthenticated connections, so if an attacker uses just 9, regular users will still get in just fine.
Still a "patch or update now" situation. Updating ssh and restarting the service is enough though, this is Linux after all. No risky reboots or other crap packaged in with the security fix. In a situation like this it really pays to have competent security people that also understand system administration.
this vulnerability affects the default config... (Score:2)
err.. security professionals wouldn't be using the default config...
Thanks for telling us how to think and feel "media"
Re: (Score:2)
And why not? OpenSSH has a pretty good security history. Using the default config for it is typically fine. "Security professional" does not mean "insane paranoid".
Note that this is the first bad OpenSSH vulnerability in quite some time. The last ssh issues was morinic distro maintainers parching parts of systemd into OpenSSH and not actually an OpenSSH vulnerability. It did provide another good reason to not use systemd and that may be something a security professional should at least seriously consider.
Re: (Score:2)
Eww, those default cipher suites, MACs, and KeX?
Re: (Score:2)
So? Yes, I change them, but is it really needed? No.
Depending on who one asks... (Score:1)
Alma patched, Rocky not yet (Score:3)
Re: (Score:3)
I assume you're referring to AlmaLinux 9. AlmaLinux 8.10 is running openssh 8.0p1, which according th the description should not be vulnerable.
Re: Alma patched, Rocky not yet (Score:1)
Re: (Score:1)
If you hang around the EL community at all you'd know the two projects work together frequently. There was even collaboration in the EL IRC channel on this. No need to try to stir stuff up between the two. Alma is ABI compatible so they can patch early; Rocky waits on Red Hat to remain 1:1. Red Hat and Rocky both patched officially today. Some people want fast; some require 1:1 - use the distro that fits your needs better, but don't try to stir up something that isn't there.
Re: (Score:2)
Finally (Score:5, Funny)
Now I finally know what the word "Open" in OpenSSH does mean.
Sigh (Score:5, Informative)
Re: (Score:2)
Great. I was wondering what happened. I got the vulnerability info from some other sources two days ago.
Et Tu, macOS? (Score:2)
Are they sure macOS is vulnerable?
This would tend to say that some BSD-related Systems Dodged this Bullet:
"OpenBSD systems are unaffected by this bug, as OpenBSD developed a secure mechanism in 2001 that prevents this vulnerability. "
https://vulcan.io/blog/cve-202... [vulcan.io]
I can't find any evidence of macOS being vulnerable to regreSSHion, other than the one statement that SSH is used for "File Transfer in Linux and macOS."
Can some macOS/BSD Experts help clear this up? I don't think macOS is built on GNU Libraries
Re: (Score:2)
Does MacOS use glibc? AFAIK, this is a problem with ssh only in glibc. You could do an "ldd sshd" (does not require root) in a terminal and check.
Re:Et Tu, macOS? (Score:4, Informative)
glibc is not required, it's just that glibc is known to have the issue. The issue is that lots of long-time library routines are not re-entrant and cannot safely be called from a signal handler context. For glibc, this includes the syslog() call. OpenBSD has a re-entrant syslog_r() call instead (which OpenSSH uses there). So this issue is a security problem on any platform which OpenSSH uses syslog() (which I believe is all but OpenBSD) _and_ where syslog() is not re-entrant (which may be everywhere).
Re: (Score:2)
Yes. But what does MacOS use?
Re: (Score:2)
glibc is not required, it's just that glibc is known to have the issue. The issue is that lots of long-time library routines are not re-entrant and cannot safely be called from a signal handler context. For glibc, this includes the syslog() call. OpenBSD has a re-entrant syslog_r() call instead (which OpenSSH uses there). So this issue is a security problem on any platform which OpenSSH uses syslog() (which I believe is all but OpenBSD) _and_ where syslog() is not re-entrant (which may be everywhere).
Very Informative, I guess; but is macOS actually affected?
Re: (Score:2)
macOS does not have glibc which is required for this vulnerability.
Re: (Score:2)
See comment just above yours: https://it.slashdot.org/commen... [slashdot.org]
There's always telnet in the meantime! (Score:3)
Just saying ;)
This is why port knocking should be used (Score:5, Interesting)
Port knocking eliminates issues like these.
You can simply have the ssh server not available on any port, until a specific packet arrives at (a) specific port(s). After that happens, you set your firewall rules to only allow ssh traffic in from the IP address where the knock came from and launch the ssh server. Then, when the work is done, the ssh server is stopped and the port is closed. Good luck making use of this vulnerability.
Re: (Score:1)
what if unprivileged employee knows about the port knocking and then does exploit to get root?
Re: (Score:1)
If your own employees are hacking you from the inside you have bigger problems.
Re: (Score:1)
Insider threats are huge, absolutely I agree.
But port knocking isn't a fix. They're already inside and are often the same people responsible for the data.
Re: (Score:1)
Bold of you to assume rank and file data entry people in finance and insurance aren't using terminal based apps in 2024. Yes they are.
You think everyone on a character terminal is IT admin? Hahaha, not true even where I work and I don't even work in finance nor insurance now. Lots of ERP/MRP systems have good ol terminals
Re: (Score:1)
Uh yeah the finance people are using ssh to get into finance tools.
In some universe, maybe.
I've never seen finance use ssh. My sister is in insurance. Can't spell ssh. My wife is in hotel administration. They're using dos for half their back end and a browser for the rest.
Re: (Score:1)
Your sample size of two isn't the whole universe.
and who said anything about hotel administration, yeah I'm sure there are plenty of windows craplications for that.
I'll even do you one better, some finance people even still using and terminal program in windows that does Telnet its emulated terminal to get into MRP/ERP in this world, for the older apps written that way with curses. They can use arrow keys and enter for a kind of menuing and selection.
Re: (Score:2)
Re: (Score:1)
Agreed. I've posted several times here about the insanity and incompetence of companies that can't manage to lockout ex employees as they walk out the door instead of never or months later or only after a problem.
Re: (Score:1)
what if unprivileged employee knows about the port knocking and then does exploit to get root?
You'll see something in your logs, because one of the glories of port knocking is that you don't see the rash of doorknob rattling you get without it. So literally anything sshd logs is signfiicant.
Re: (Score:1)
if you see it in logs it's too late. but thanks for showing why most SOC people and companies are pretty worthless, screaming lost horses in empty barn
Re: (Score:2)
Port knocking eliminates issues like these.
Good thing tcpdump doesn't exist, right?
does this affect public key only systems? (Score:3)
I have interactive/password login disabled. does this affect rsa/dsa key login also?
Re:does this affect public key only systems? (Score:4, Informative)
Yes. The exploit happens before any authentication is even attempted.
Re: (Score:2)
So... uh... what's it doing with the key the user supplied? Maybe searching it for secret messages or perhaps just evaluating the numbers based on their artistic merits?
Re: (Score:3)
The exploit works by the client not doing any authentication at all, but just sitting and waiting until the server times out. Once it has figured out the timeout, then it connects again and tries to initiate authentication right around when the SIGALRM handler would fire, hoping to cause a non-async-safe function to be called unsafely. It's a very tricky timing attack that takes thousands of attempts (on average) to succeed.
Do your servers need wide-open SSH access? (Score:2)
It's time to ask this question: Do our servers need wide-open SSH access? In many cases, it's feasible to lock down access to a set of fixed IP addresses using firewall rules. Or if that's not possible, to use a VPN so that at least the server isn't listening directly on a publicly-accessible IP address. While this trades an OpenSSH vulnerability for a potential VPN vulnerability, there are probably a lot more SSH scanners prowling the Internet than various VPN scanners.
I have a couple of colocated ser
Re: (Score:2)
Of course most VPN server code is equally crusty C(++). If you want big corp support for something in a safe language it's slim pickings.
Re: (Score:2)
OpenVPN or Wireguard are probably reasonable choices.
Does libc == glibc ? (Score:2)
Bad workaround in summary. (Score:5, Informative)
It's worth noting that the workaround listed in the summary is probably wrong. Per the man page for sshd_config
"The server disconnects after this time if the user has not successfully logged in. If the value is 0, there is no time limit."
Re: (Score:2, Informative)
Re: (Score:3)
Hyperbole much? (Score:2)
"It's hard to overstate the potential impact of this flaw."
LOL.
If it takes multiple hours and many many thousands of tries against a completely known configuration on the easiest target possible in order to exploit it *once* in a lab environment, then I'd say that a great number of people have already found it very easy to overstate the potential impact of this flaw in a desperate attempt to increase their click-throughs.
Like, sure it's very bad that sshd could be used to get a remote shell but simple measu