Ask Slashdot: Patch Management For Offline Customer Systems? 78
New submitter Nillerz writes: What, in your experience, is generally the best way to distribute patches in a way so customers can download them, considering that the machines are offline? Are there any software packages (open source preferred) that pretty much allow engineers to upload a patch with a description to a web server, and allow customers with credentials that are registered in LDAP to browse and download them quickly? And if not, how do you distribute patches to air-gapped machines?
Re:Is there even a reason to patch airgapped machi (Score:5, Insightful)
To fix non-security related bugs.
Re:Is there even a reason to patch airgapped machi (Score:5, Informative)
Or maybe you might have an airgapped "kiosk", with a keyboard and/or mouse and a dedicated application running modal (so it can't be bypassed to access the OS, perhaps without some hardware hacking). If it's non-networked, or only networked locally to some other system on-site, but still accessible to "users" who aren't fully trusted to the same level as the CEO (e.g., line employees, general public customers, etc.), you might want to patch it *for* security vulnerabilities, such as "if the user presses Ctrl+Alt+Del, they can access the desktop" (or something equally based on the concept of user input -> system access). That would be an example of a software-based security exploit on airgapped equipment.
Re: (Score:2)
Maybe it was a typo. - "users" who _ARE_ fully trusted to the same level as the CEO (e.g., line employees, general public customers, etc.). Because around here we assume the CEO is no more computer aware than the guy guarding the loading dock.
Re: (Score:1)
No, the CEO will be even less computer aware than that guy. Most likely, the CEO will have his secretary print out his email. And read it to him.
Download while offline? (Score:1)
How exactly would they download patches from a web server if they are offline?
Re: (Score:2)
This whole question is a trainwreck.
"Are there any software packages (open source preferred) that pretty much allow engineers to upload a patch with a description to a web server, and allow customers with credentials that are registered in LDAP to browse and download them quickly?"
If you've already got LDAP set up, you probably already got a network-accessible directory. Just put the patches in there.
Re: (Score:3)
Re: (Score:2)
It does when you know the context of the situation. The reason to have an "offline" or "airgapped" machine is so that it is not able to be hacked into from other untrusted computers on the network (i.e. the internet).
A computer can be completely isolated from other computers, but if you believe this makes the computer secure, then 2 computers talking to each other over an isolated network is also secure.
The distinction between "computers" is artificial anyway. You can have multiple systems running in the
sneakernet (Score:4, Informative)
If you really want to be fancy, make the installer check for something that is supposed to be on a legitimate customer system before it even prompts for credentials to decrypt the files, to make sure that it is being used on the correct machines and that it actually is the customer calling.
Re: (Score:2)
Ship encrypted files on flash with instructions for them to call when the media arrives. Provide phone support to walk them through the install process,
Why bother with encryption? Why not have an automated process where you just put the USB stick in the USB hole and reboot, and the system finds the update and installs it? If you're worried about authenticating the update, sign it.
where you provide the password to the files at that time.
Free clue: if you don't want the user to get the update, don't send him the files. Why bother with a password?
Once the patch is installed, walk them through formatting the flash media and mailing it back to you.
The price for a USB stick is so low these days that it will cost more to manage the mailing and return than the stick is worth. And why do they have to format the media
Re: (Score:1)
Re: (Score:1)
Because I'm an asshole that is going to replace your patch/binary with malware. At least with a password you can quickly tell something went wrong when there is no password requirement anymore.
Re: (Score:1)
Which can be done much better with signed software, distribute the public key on your software, and distribute software as an archive, that archive contains 4 files, a file containing a version number, a file containing the software installer/all files, and signatures for both files. To install you check the signatures and then compare the version number to the current version number, if the checksum is valid and the version number is newer then you install the software, Passwords are hackable/sniffable, an
Re: (Score:2)
You don't have my private key to sign it with. When the signature doesn't match, you don't win. No password needed. No encryption needed.
Re: (Score:2)
NO, you cannot reuse the stick.
First off, the network is probably airgapped for a reason. There are many known attacks to airgaps, and using a USB drive is a great way to infiltrate and exfiltrate information.
Think something like Stuxnet - it infected an airgapped networ
Re: (Score:2)
NO, you cannot reuse the stick.
First off, the network is probably airgapped for a reason. There are many known attacks to airgaps, and using a USB drive is a great way to infiltrate and exfiltrate information.
Think something like Stuxnet - it infected an airgapped network, and for that to work, the creators probably did tricks to exfiltrate information to get a map of the network layout.
The only safe way is after the USB stick is used, is to destroy it.
That's part why I suggested formatting and then sending it back, if the method of couriering the media is secure then the originating party can inspect the flash (as formatting isn't terribly thorough) but formatting it might stop the malware from actually executing on a random third-party's computer if something happens and the media is lost. I suppose that is inadequate in some applications though. Maybe a way to preserve the USB media for forensic inspection would be to have a literal bin like hospital
Re: (Score:2)
That's part why I suggested formatting and then sending it back, if the method of couriering the media is secure then the originating party can inspect the flash (as formatting isn't terribly thorough)
And just what good to the customer is it if you inspect a formatted USB stick and find nothing on it? I suppose you did a full byte-level dump on it before sending it and are going to compare to see if any of the blocks outside the filesystem management ones are different. And if you find something, does that just mean that the system wrote a file to it, or is it a sign of infection? Do you call the customer in a panic, just to have him tell you "yeah, we copied the system log files onto it since we were t
Re: (Score:2)
The only safe way is after the USB stick is used, is to destroy it.
If you can't trust your vendor to not send you malware on a USB stick, you don't put it into the airgapped system in the first place, and then you find a different vendor. So I guess "USB stick" as a way of getting data into an airgapped system is not acceptable. That would seem to apply to other methods of entering an update except, perhaps, getting a printed sheet of paper with source code on it. You first have your programming experts look through the source code, then they type it in and recompile.
Also, always assume that your airgapped network is infected.
The
MIF "message" mode?? (Score:1)
when the drive starts up have a message come up "Good[Morning|Afternoon|Evening] Mr %name% Your mission if you decide to accept it is to patch this system with the included files. As always if you or anyone on your team ..." and then when the patching program completes
"This message will self destruct in 5 seconds"
this will prevent reuse of the media
Re: (Score:2)
Re: (Score:2)
At the end of the day, the only way to update an airgapped machine is via sneakernet. USB, DVD, 3.5 floppy, it doesn't really matter. If you can beam an update into a machine via say, IR, it wouldn't be air-gapped.
If you have an entire air-gapped network, then any normal package manager would work. Just have to update the server via sneakernet and push the patch out from there.
Re:sneakernet (Score:4, Insightful)
No. Not on flash. Flash can be intercepted and modified. Send it on a CD/DVD that's not rewritable, and send a hardcopy of the MD5 hash in a second package. Then, before running the update, calculate the hash and compare it by eye with the hardcopy. I won't say that it's impossible for anybody to slip an infection past this, but it's not going to be easy, especially if you send the two parts of the message by different companies.
Re: (Score:2)
If it has a specific reason to not be online, then I expect that it might not even have an optical drive, depending on what it's used for. If it's that important then it might not even have external USB ports either. A service technician would have to open up the computer to use an internal USB header to interfa
Re: (Score:2)
It does look like there's a shortage of modern read-only tech these days, optical disc appears to really be the only game in town and it comes with its own baggage.
It's a buck for a blank CD-ROM, or less. It has no more baggage than a USB stick. If you don't have a USB port on the system and are going to have to open the box to plug a USB header onto the MB, then plug a USB CD into it. It's YOUR verified CD reader so there's no issue with it having malware.
And if you say you can't trust the CD from the vendor to be malware free, then you need to explain why you would trust a USB PROM to be malware free from the vendor.
Re: (Score:2)
It gets a bit more complicated, trusting a USB PROM; a CD filesystem can be end-user inspected a
Re: (Score:2)
Re: (Score:1)
Easy (Score:2)
And if not, how do you distribute patches to air-gapped machines?
Sneakernet + authorized tech + calendar/ticket tracking
Am I really that old? (Score:2)
Am I really getting so old that people find this to be a legitimate question?
Have I really been doing this for so long that there are now people who don't remember a time when disconnected machines was the norm?
Am I the only one left here who remembers (for example) dialling for hours to get into the local IBM-run BBS to download the 21 diskette images needed to update OS/2 2.1 to the latest patch level, digging into the cabinet for several boxes of diskettes, de-imaging each and every one of those diskette
Re: (Score:2)
You're getting grumpy, old man. :-)
Re: (Score:2)
You're getting grumpy, old man. :-)
I'd reply, but I'm too busy shaking my fist and venting my anger at a passing small, fluffy white cloud.
Yaz
swap drives (Score:3)
I worked on an airplane-based system, and we had removable hard drives which we swapped any time we had to update the software. This way, each upgrade also restored the system to a pristine condition.
I've also done this with CD-ROMs. One nice thing about booting and running from a CD-ROM is that it's impossible for it to be "hacked" (short of creating a new version and sneaking it in to the physical machine).
WSUS Offline (Score:2)
Re: (Score:2)
Yep. WSUS Offline for the windows boxes (although most of my offline windows installations are XP VMs so they are already as fully patched as they are ever going to be). Then we have an Umbongo server that serves all the Umbongo patches to the various offline workstations that host the VMs. A download script and a bit of rsyncing and the update server stays fresh.
The only issues are the rare times someone needs a MS patch that isn't covered by WSUS Offline, in which case they deal with it manually using MBS
MBSA + WSUS CAB File (Score:2)
With some small scripting (VBS, Powershell, etc), you parse the XML and find the needed patches in a patch repository.
Then you can remotely push all of that out via PS-Remoting or PSExec, and your offline/air-gapped network can stay patched.
rpm/yum deb/apt ? (Score:2)
Re: (Score:2)
Re: (Score:2)
What part of offline and airgapped did you have a problem understanding?
Re: (Score:2)
First of all you can install .rpm and .deb packages without being "online". Secondly you can be "airgapped" (i.e. not connected to the internet), and still install software using yum and apt from a server on an intranet, and actually this is quite convenient as it allows you to install software updates on many different machines simultaneously.
Maybe you should be more knowledgeable before you decide to act like a prick.
Re: (Score:2)
And look at the subject.
It's implicit that GP is not talking about .rpms and .debs, but rather the yum and apt files. And yes, you can run an airgapped intranet (I've done it myself for classified data), but it pushes back the question one level.
How do you get said updates onto the airgapped network?
Re: (Score:2)
It's implicit that GP is not talking about .rpms and .debs, but rather the yum and apt files.
I have read this sentence 5 times, and I still have no idea what you are talking about.
And yes, you can run an airgapped intranet (I've done it myself for classified data),
congratulations
but it pushes back the question one level.
How do you get said updates onto the airgapped network?
How do you get anything on to an airgapped network? You don't have many options which makes this easy. Write once optical media (e.g. CDR, DVDR, BDR), provides the most security, as it prevents one avenue for data to escape the airgapped network (presumably the reason for the airgap in the first place).
To a person who has run an airgapped intranet for classified data, I would have assumed that part was o
Re: (Score:2)
Pardon me. Yum and apt applications, not files.
How do you get anything on to an airgapped network? You don't have many options which makes this easy. Write once optical media (e.g. CDR, DVDR, BDR), provides the most security, as it prevents one avenue for data to escape the airgapped network (presumably the reason for the airgap in the first place).
Which is what the story poster was asking. There was no need for yum or apt.
Re: (Score:2)
Which is what the story poster was asking. There was no need for yum or apt.
If you read what the story poster wrote, he/she was asking about software packages that allow patches to be uploaded to servers where customers could download them, *and* how to distribute them on airgapped machines.
the software packages I am recommending are versatile package management systems (dpkg, rpm), that also come with tools for distribution of packages (apt, yum).
Once the software update package files are passed the airgap, they still need to be installed. If one were to set up an apt/yum server
Re: (Score:2)
You know what, I may have misinterpreted the story poster's question. Let's let it be, because there's no point to arguing here.
Just use Stuxnet (Score:2)
Re: (Score:2)
Yeah, I was gonna suggest this. Just put your update on a USB thumb drive with a rootkit, leave a few out in the parking lot, and wait. Someone will plug it in, at which point your drive can take over the machine and update it!
At that point, you might also consider installing something else to help dodge the air gap to make future updates for your customer even easier; for example, try data transmission via ultrasonic frequencies to another compromised^w updated machine with a network connection!
Windows Patching? (Score:1)
Hate to plug a windows product.. but we use shavlik protect with cloud.
It allows you to control what is patched and handles laptop users who don't always connect via VPN often.
The product needs work to work in larger environments. GUI is sometimes a little confusing too. Things are not where you would expect them for easy use.
It also patches our ESXi environment.
Yabba dabba doo (Score:2)
You have to have someone upload the patches to the airgapped machines using the sneakernet.
I'm not quite getting what we're talking about here? Operating system patches? or program patches?
Program patches are frequently offered in an offline format and MS offers patches like that as well.
So I don't get what we're talking about here. What are we patching?
To paraphrase Samuel L... Be specific, motherfucker.
git? (Score:2)
If you are patching at the code level, I believe that git is a very good option. Because the git repository can be moved as a standard directory and essentially contains all the patches and history. So even if the customer as custom patches, they can probably easily rebuild around it.
And since a git repository is essentially a directory, you can simply put it on a disk or flash drive and send it by snail mail or courier.
Just use git bundle (Score:3)
Sneakernet (Score:1)
You're OFFLINE, bust out the thumb drive, image it, and start making your rounds to ensure they are properly applied like a good IT guy.
Re: (Score:2)
"Ask" section is getting dumber and dumber :( (Score:2)
> What [...] is generally the best way to distribute patches
Patches for what? For PC operating systems? PC software? Embeded computers?
> in a way so customers can download them, considering
> that the machines are offline?
Well you can't download anything when you are offline. You mean that customers download the patches, put it on removable media and install them on their machines...
> Are there any software packages (open source preferred) that pretty much
> allow engineers to upload a patch wi
I've only had to deal with this on one project (Score:2)
I've only had to deal with air-gapped machines on one project, but we used to send out engineers/developers to upgrade those machines. They were security-critical, so we couldn't have customer support staff getting root access to do upgrades, and they were too operationally-critical to trust to automated update disks (not USB sticks -- those can be modified in shipping.)
So we sent staff to do the installs and updates. Not that there was a huge client base, and it was a pricey project, so the cost was n
unidirectional gateways (Score:1)
(1) If safety is the goal (eg: power plant control networks, train control systems) people deploy a Waterfall FLIP. The FLIP unidirectionally replicates industrial servers to external networks so corporate users can see the data, and reverses direction on a schedule to pull updates.