Over 47,000 Supermicro Servers Are Exposing BMC Ports on the Internet (zdnet.com) 57
Catalin Cimpanu, writing for ZDNet: More than 47,000 workstations and servers, possibly more, running on Supermicro motherboards are currently open to attacks because administrators have left an internal component exposed on the internet. These systems are vulnerable to a new set of vulnerabilities named USBAnywhere that affect the baseboard management controller (BMC) firmware of Supermicro motherboards. Patches are available to fix the USBAnywhere vulnerabilities, but Supermicro and security experts recommend restricting access to BMC management interfaces from the internet, as a precaution and industry best practice.
IPMI is not for public networks (Score:3)
OTOH, IPMI shouldn't be insecure-by-default either (Score:2)
Re: (Score:2)
The one BMC BIOS I have had the displeasure to work with came with a psychotic number
of services preconfigured in addition to IPMI channels, and to boot, it had IPv6 link-local
addresses accepting traffic on the management port with no way to turn that off.
Port isolation is the only good solution for this crap. You can't just put it on a private VLAN
with the rest of your BMC/ILO crap, because a compromise could come out the BMC/ILO of
one machine to extend to your entire server farm.
Re: (Score:2)
Our setup has IPMI access between two servers (i.e. if one is still ok, IPMI on the other can be reached by it), but exposing it to the Internet was never an option. This does make things more complicated, but it is the only secure option. At the very least, you should put IPMI access behind a jump-host that is hardened and only accessible via SSH or VPN. Use two of these and a physically seperated IPMI network if you need redundancy.
Thise that place convenience over security will have neither convenience n
Re: (Score:2)
Note that fundamentally, the protocols used by BMCs can be properly secured if implemented correctly by vendor and owner sets really strong passwords. This should be done even if the network is ostensibly physically segregated.
The problem is that *any* appliance/firmware is a risk given the general inconsistency and update difficulties of the vendors. Those home routers are probably a much more enormous glob of security holes in practice.
What is maddeining to me is that despite the existence of software t
Re: (Score:2)
I do agree on all of that. In particular, use of very defensive and careful engineering on the software of the BMCs, a minimal interface with very strict input validation, good crypto right from the start, secure update mechanism, etc. and the thing could have been exceptionally hard to hack. But all of that needs people that really know what they are doing, and these are rare and expensive. I expect that, as usual, some "managers" though their bonus much more important than the welfare of their customers a
Re: (Score:2)
Except you might have your BMC's hooked up for authentication via your active directory for example, going to need a gateway setting. Other things are flinging their event log at a syslog host, contacting NTP servers so all those log entries are synchronized etc.
That said all inbound traffic should be cut off at the knees.
I take the view for example that running special NTP servers for the BMC network is introducing another potential security issue, where as letting them contact the campus NTP servers is ju
Re: (Score:2)
True, a complex environment should be able to competently mitigate the network, but then again I've found shops to be worse at it than they think they are.
I take the view for example that running special NTP servers for the BMC network is introducing another potential security issue
A well configured and automatically patched NTP service from a reputable OS vendor I would think a lower risk than putting appliances at greater risk of being globally available.
I also take the view that connecting my syslog server directly to the BMC network is a bad idea as it means any compromise on that and my BMC network is wide open
This is a key argument I can't agree with. If you can't reasonably secure an instance that is bridged to the same L2 segment, than you have much bigger problems corporate wide.
Really, this still needs to be said in 2019? (Score:2)
but Supermicro and security experts recommend restricting access to BMC management interfaces from the internet, as a precaution and industry best practice.
So, why in the heck would you put an unprotected management port on the internet. Management functionality should come through a fairly airgapped network, a single jump host or something like that... Why oh Why
Re: Really, this still needs to be said in 2019? (Score:4, Insightful)
Re: (Score:3)
Indeed. And to make matters worse, many of these incompetents do not understand they are incompetent (Dunning-Kruger at work) and do not even get help with doing things. Worst case I have seen so far was where "management" sent the future staff of a newly established SOC to evening school to learn the basics of TCP/IP networking. If they ever have an incident, these people will be completely lost.
Re: (Score:2)
Why? Because it is easier. With the "cheaper than possible" IT personnel modern "management" likes to hire, people struggle to make things work at all. Of course, anybody competent will never expose IPMI to the internet, but the people doing it are anything but competent.
Re: (Score:2)
You may not realize you're doing it.
I have a FreeNAS box built using a Supermicro board, which has two gigabit NICs. I am aware it has IPMI/BMC, and I'm aware that I need to shut it off. I am also aware that IPMI is enabled by default on the "other" NIC. So I wandered into the BIOS, found a setting for IPMI, and turned off the switch that said (in effect), "Enable IPMI on the second NIC." All set, right?
What I didn't re
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
that's generous.
In fact it was shown (article in slashdot) to be utter bullshit and the component in question normal
Re: (Score:2)
Well, the idea was not completely without merit, but it had numerous practical issues that Bloomberg missed.
Re: (Score:2)
Re: (Score:1)
Yes, though that did not appear under further scrutiny to be more than just an isolated occurrence. It appears to have been some sort of attack on the supply chain, rather than a manufacturer-installed feature.
Re: (Score:2)
Yes, though that did not appear under further scrutiny to be more than just an isolated occurrence.
Oh? They got that far? Got a link, because everything I read on the topic turned out to be 100% unsubstantiated.
Re: (Score:2)
So let's have some pix! (pr it didn't happen).
Re: (Score:2)
It's always great when an unsubstantiated claim from a news source that isn't backed up anywhere and refuses to name a source is only verified by a completely anonymous internet post without any evidence at all.
Don't worry AC, I believe you. We'll get our moment when they prove the aliens in Area 51 are real.
Its an accidental feature (Score:3)
By default, if the IPMI is unplugged, the IPMI fails over the the first onboard ethernet port. If that happens to be your WAN interface it's probably not the desired state. Failover behavior can be controlled in the BIOS, or plug the actual IPMI port back in.
Re: (Score:2)
And if you have the minimal competence to look at things and documentation and configure them right (and to block the IPMI port in the firewall you should have as well, just to be sure) then the port will not get exposed. Looks like at least 47'000 supermicro-based systems lack administrators with that minimal competence.
If you assign it an IP on your public network (Score:2)
That's an issue if you assign IPMI a reputable IP on your public network. You should never, ever do that.
The failover is a different MAC on the same physical plug.
* routable. Ducking autocorrect (Score:2)
That should say "a routable public IP".
You'd only do that if you're crazy.
Re: (Score:2)
That's an issue if you assign IPMI a reputable IP on your public network. You should never, ever do that.
Right, except that most by default will DHCP so if you have a dynamic address pool and you are oblivious, you get BMC on the network. Note a lot of these designs share physical access through one of the OS nics, in lieu of or in addition to a dedicated management port.
The failover is a different MAC on the same physical plug.
That would be pretty useless. Common configurations are either "both shared and dedicated are live by default", "shared is disabled if dedicated is plugged in" and "dedicated only". The last is the only behavior where you don't run the risk
Re: (Score:2)
>> The failover is a different MAC on the same physical plug.
> That would be pretty useles
It's called a vlan.
My wording was rude (Score:2)
Let me rephrase that better:
One use case is when you have IPMI on a separate vlan, which seems to be a common and recommended configuration. Not quite as foolproof as a separate physical switch, but just as secure until somebody goofs up and changes the IPMI configuration.
Re: (Score:2)
Ok, there was a misunderstanding. The failover behavior the poster referenced was configuring multiple NICs on the BMC in a failover fashion.
The various vendors have made a mess of the fairly simple concept that a BMC frequently has two ethernet interfaces, one with a dedicated phy/port and one that uses NCSI to subscribe to packets from an RMII wired NIC (either onboard, through OCP, or propreitary add-on). They call it selectable or failover or whatever.
Having the BMC traffic segregated onto a distinct
Re: (Score:2)
You should never use DHCP to hand out a pool of routable IP addresses in the first place.
Re: (Score:2)
This is a valid point, though we also have to recognize that a lot of places do. It's poor practice security wise, but firmware developers should design with the flawed status quo in mind and not use the fact that it is poor as a free pass.
Re: (Score:2)
They can't account for everything. Nothing is fool proof because fools are so ingenious.
They do it that way so that people who know what they are doing can easily get in to the thing remotely for initial configuration and OS install. It's supposed to be behind a jump box when you do that. Having done remote setup before, sometimes you're lucky if you can get the person on the remote end to plug the cat-5 into any of the ethernet interfaces (as opposed to trying to cram it in the USB, power, or serial port),
Re: (Score:2)
They can't account for everything. Nothing is fool proof because fools are so ingenious.
While true, there are ways to do a decent job mitigating.
They do it that way so that people who know what they are doing can easily get in to the thing remotely for initial configuration and OS install. It's supposed to be behind a jump box when you do that.
So for example, a default network configuration skipping any offered gateway until the password set (unorthodox, but still), or locking down default password while an OS is booted (system firmware can alert the BMC to the fact that boot services exited). It's a bit odd to tell the on-site guys 'no no leave the servers *off*, but it's not a difficult instruction to follow/implement. Once you have set your actual password, then open it on up to whatev
Re: (Score:2)
The BMC can't rely on knowing when the OS has booted successfully. All the firmware can know is that it handed off to the bootloader which might or might not have actually found an OS which might or might not have actually booted to a state where it is reachable.
As I said, if someone has little enough knowledge to hand out public IPs in a DHCP pool, not change default IPMI passwords, and not set it up to use a management port behind a jump box, they probably use administrator/administrator as a server user/
Re: (Score:2)
Note I was only suggesting those sorts of lockdowns for users that never bother to change from admin/admin or equivalent. Such a device would automatically lift restrictions once a password was configured that wasn't what the device shipped with. This is only to enable inital setup of the bmcs not to diminish the value of the bmcs after setup.
Re: (Score:2)
Understood. Unfortunately that's the time when the admin has the least resources to deal with limitations in a constructive way. For example, what if the box being set up is to BE the jump box? Or it would be if it would honor the gateway settings on the LAN so the temporary forwarding rule in the firewall could be useful and let the remote admin change the IPMI passwords, set it's LAN up, and install the OS.
That scenario can happen a lot on corporate intranets with tunnels between sites.
In other cases, Cis
most should be on an hardware MGT network (Score:2)
most should be on an hardware MGT network that does not have public IP's and may need an MGT VPN / jump sever to get to. EG not on the main traffic network.
Yes, the hardware issue is failover (Score:2)
Agreed. The problem I was addressing is that it if the cable on the management network comes loose, it can failover to the other NIC - THE NIC that is attached to the main, customer-facing network.
BUT only the hardware NIC, it's a different MAC and different IP. So the hardware failover shouldn't hurt you since it shouldn't use an IP that's on the main network.
Re: (Score:2)
But you would ALSO have to be serving routable IPs via DHCP to public facing interfaces on your network, so that's strike 2.
The people who have done this aimed very carefully before they shot their feet.
Not seeing much cause for concern (Score:2)
Do you have port tcp 623 on your router allowed access from the internet?
username and password required, yes it unencrypted is allowed... you're saying someone will use unencrypted mosde and also be traffic snooped... again over the internet through the 623 tcp port they opened because.... they're a moron?
if encryption used is weak, rc4 with fixed key....mkay, someone taking the time to decrypt what the dumb-ass in previous paragraph is doing could hit paydirt
and the worst thing, authentication state persis
Re: (Score:2)
What's funny is that is a pretty silly scan.
623 is significant because that's the port number of IPMI. IPMI is UDP only. Generally speaking, TCP ports 443, 80, 22 are all adequate to let an attacker ruin your day if you don't manage your BMC security. Listening on TCP port 623 means the BMC vendor is being a bit dumb.
username and password required, yes it unencrypted is allowed... you're saying someone will use unencrypted mosde and also be traffic snooped... again over the internet through the 623 tcp port they opened because.... they're a moron?
While I agree that encrypted protocols are only a weakness when they are used, that would require informed consent. The issue is why not use standard TLS/SSH to protect non-IPMI traffic (IPM
Re: (Score:1)
Re: (Score:2)
Unfortunately, those oblivious users help make botnets that ruin everyone else's day.
Also, a lot of these servers for convenience make the on board OS NIC act like a switch and let the BMC out through there by default.
Amul Franchise, Haldiram, Franchise (Score:1)