Become a fan of Slashdot on Facebook

 



Forgot your password?
typodupeerror
Bug Software Upgrades

Ask Slashdot: Patch Management For Offline Customer Systems? 78

New submitter Nillerz writes: What, in your experience, is generally the best way to distribute patches in a way so customers can download them, considering that the machines are offline? Are there any software packages (open source preferred) that pretty much allow engineers to upload a patch with a description to a web server, and allow customers with credentials that are registered in LDAP to browse and download them quickly? And if not, how do you distribute patches to air-gapped machines?
This discussion has been archived. No new comments can be posted.

Ask Slashdot: Patch Management For Offline Customer Systems?

Comments Filter:
  • by Anonymous Coward

    How exactly would they download patches from a web server if they are offline?

    • by Fwipp ( 1473271 )

      This whole question is a trainwreck.

      "Are there any software packages (open source preferred) that pretty much allow engineers to upload a patch with a description to a web server, and allow customers with credentials that are registered in LDAP to browse and download them quickly?"
      If you've already got LDAP set up, you probably already got a network-accessible directory. Just put the patches in there.

    • The webserver could be offline as well (i.e. intranet).
  • sneakernet (Score:4, Informative)

    by TWX ( 665546 ) on Thursday August 06, 2015 @04:17PM (#50265491)
    Ship encrypted files on flash with instructions for them to call when the media arrives. Provide phone support to walk them through the install process, where you provide the password to the files at that time. Once the patch is installed, walk them through formatting the flash media and mailing it back to you.

    If you really want to be fancy, make the installer check for something that is supposed to be on a legitimate customer system before it even prompts for credentials to decrypt the files, to make sure that it is being used on the correct machines and that it actually is the customer calling.
    • Ship encrypted files on flash with instructions for them to call when the media arrives. Provide phone support to walk them through the install process,

      Why bother with encryption? Why not have an automated process where you just put the USB stick in the USB hole and reboot, and the system finds the update and installs it? If you're worried about authenticating the update, sign it.

      where you provide the password to the files at that time.

      Free clue: if you don't want the user to get the update, don't send him the files. Why bother with a password?

      Once the patch is installed, walk them through formatting the flash media and mailing it back to you.

      The price for a USB stick is so low these days that it will cost more to manage the mailing and return than the stick is worth. And why do they have to format the media

      • I work at a bank that has airgapped network for credit card processing. Encrypt files because you do not want hackers having the code that is used to process cards. USB and CDRoms are disabled on all systems to keep users out. To physically access the server, I need a manager and security guard to open the door, and they both have to stay there with me the whole time. Upgrades are scheduled weeks to months in advance.
      • by Anonymous Coward

        Because I'm an asshole that is going to replace your patch/binary with malware. At least with a password you can quickly tell something went wrong when there is no password requirement anymore.

        • by Anonymous Coward

          Which can be done much better with signed software, distribute the public key on your software, and distribute software as an archive, that archive contains 4 files, a file containing a version number, a file containing the software installer/all files, and signatures for both files. To install you check the signatures and then compare the version number to the current version number, if the checksum is valid and the version number is newer then you install the software, Passwords are hackable/sniffable, an

        • That's why I said "If you are worried about authenticating the update, sign it." You replace my patch with malware. You put a password on the encrypted file that you got from some other customer after I told it to him. Bingo, you have an "encrypted" patch that has the right password. You win.

          You don't have my private key to sign it with. When the signature doesn't match, you don't win. No password needed. No encryption needed.

      • by tlhIngan ( 30335 )

        The price for a USB stick is so low these days that it will cost more to manage the mailing and return than the stick is worth. And why do they have to format the media? Just let them delete the file if they want to, and then use the stick.

        NO, you cannot reuse the stick.

        First off, the network is probably airgapped for a reason. There are many known attacks to airgaps, and using a USB drive is a great way to infiltrate and exfiltrate information.

        Think something like Stuxnet - it infected an airgapped networ

        • by TWX ( 665546 )

          The price for a USB stick is so low these days that it will cost more to manage the mailing and return than the stick is worth. And why do they have to format the media? Just let them delete the file if they want to, and then use the stick.

          NO, you cannot reuse the stick.

          First off, the network is probably airgapped for a reason. There are many known attacks to airgaps, and using a USB drive is a great way to infiltrate and exfiltrate information.

          Think something like Stuxnet - it infected an airgapped network, and for that to work, the creators probably did tricks to exfiltrate information to get a map of the network layout.

          The only safe way is after the USB stick is used, is to destroy it.

          That's part why I suggested formatting and then sending it back, if the method of couriering the media is secure then the originating party can inspect the flash (as formatting isn't terribly thorough) but formatting it might stop the malware from actually executing on a random third-party's computer if something happens and the media is lost. I suppose that is inadequate in some applications though. Maybe a way to preserve the USB media for forensic inspection would be to have a literal bin like hospital

          • That's part why I suggested formatting and then sending it back, if the method of couriering the media is secure then the originating party can inspect the flash (as formatting isn't terribly thorough)

            And just what good to the customer is it if you inspect a formatted USB stick and find nothing on it? I suppose you did a full byte-level dump on it before sending it and are going to compare to see if any of the blocks outside the filesystem management ones are different. And if you find something, does that just mean that the system wrote a file to it, or is it a sign of infection? Do you call the customer in a panic, just to have him tell you "yeah, we copied the system log files onto it since we were t

        • The only safe way is after the USB stick is used, is to destroy it.

          If you can't trust your vendor to not send you malware on a USB stick, you don't put it into the airgapped system in the first place, and then you find a different vendor. So I guess "USB stick" as a way of getting data into an airgapped system is not acceptable. That would seem to apply to other methods of entering an update except, perhaps, getting a printed sheet of paper with source code on it. You first have your programming experts look through the source code, then they type it in and recompile.

          Also, always assume that your airgapped network is infected.

          The

        • when the drive starts up have a message come up "Good[Morning|Afternoon|Evening] Mr %name% Your mission if you decide to accept it is to patch this system with the included files. As always if you or anyone on your team ..." and then when the patching program completes

          "This message will self destruct in 5 seconds"

          this will prevent reuse of the media

        • Why even have an airgapped network? They are all infected just like every other network. - That's the conclusion you should reach if you actually assumed airgapped networks are infected.
    • At the end of the day, the only way to update an airgapped machine is via sneakernet. USB, DVD, 3.5 floppy, it doesn't really matter. If you can beam an update into a machine via say, IR, it wouldn't be air-gapped.

      If you have an entire air-gapped network, then any normal package manager would work. Just have to update the server via sneakernet and push the patch out from there.

    • Re:sneakernet (Score:4, Insightful)

      by techno-vampire ( 666512 ) on Thursday August 06, 2015 @05:34PM (#50265989) Homepage
      Ship encrypted files on flash with instructions for them to call when the media arrives.

      No. Not on flash. Flash can be intercepted and modified. Send it on a CD/DVD that's not rewritable, and send a hardcopy of the MD5 hash in a second package. Then, before running the update, calculate the hash and compare it by eye with the hardcopy. I won't say that it's impossible for anybody to slip an infection past this, but it's not going to be easy, especially if you send the two parts of the message by different companies.
      • by TWX ( 665546 )
        I was assuming that this was more of a medium-security system. It's offline because it has no reason to be online, as opposed to being offline because it has a specific reason to not be online.

        If it has a specific reason to not be online, then I expect that it might not even have an optical drive, depending on what it's used for. If it's that important then it might not even have external USB ports either. A service technician would have to open up the computer to use an internal USB header to interfa
        • It does look like there's a shortage of modern read-only tech these days, optical disc appears to really be the only game in town and it comes with its own baggage.

          It's a buck for a blank CD-ROM, or less. It has no more baggage than a USB stick. If you don't have a USB port on the system and are going to have to open the box to plug a USB header onto the MB, then plug a USB CD into it. It's YOUR verified CD reader so there's no issue with it having malware.

          And if you say you can't trust the CD from the vendor to be malware free, then you need to explain why you would trust a USB PROM to be malware free from the vendor.

          • by whit3 ( 318913 )

            It does look like there's a shortage of modern read-only tech these days, optical disc appears to really be the only game in town and it comes with its own baggage.

            It's a buck for a blank CD-ROM, or less. It has no more baggage than a USB stick. And if you say you can't trust the CD from the vendor to be malware free, then you need to explain why you would trust a USB PROM to be malware free from the vendor.

            It gets a bit more complicated, trusting a USB PROM; a CD filesystem can be end-user inspected a

        • The only true way to protect an airgapped machine is write-only memory.
    • > make sure that it is being used on the correct machines and that it actually is the customer calling. I once read (my guess would be reddit's Tales From Tech Support) about people sending off sealed hacks with a rubber chicken rigged to the rack doors, with express instructions to NEVER OPEN THE RACK, and call if something happened. Any time someone called regarding a rubber chicken, support could tell right away their warranty was void, since they could only know this if the did the one thing they w
  • by wbr1 ( 2538558 )

    And if not, how do you distribute patches to air-gapped machines?

    Sneakernet + authorized tech + calendar/ticket tracking

  • Am I really getting so old that people find this to be a legitimate question?

    Have I really been doing this for so long that there are now people who don't remember a time when disconnected machines was the norm?

    Am I the only one left here who remembers (for example) dialling for hours to get into the local IBM-run BBS to download the 21 diskette images needed to update OS/2 2.1 to the latest patch level, digging into the cabinet for several boxes of diskettes, de-imaging each and every one of those diskette

  • by John Bresnahan ( 638668 ) on Thursday August 06, 2015 @04:31PM (#50265571)

    I worked on an airplane-based system, and we had removable hard drives which we swapped any time we had to update the software. This way, each upgrade also restored the system to a pristine condition.

    I've also done this with CD-ROMs. One nice thing about booting and running from a CD-ROM is that it's impossible for it to be "hacked" (short of creating a new version and sneaking it in to the physical machine).

  • For Windows machines, I use WSUS Offline [wsusoffline.net]. It also comes in handy when I'm at a customer site and their internet is so slow that I can't patch a single machine in a day. Yes, there are still areas of the world where DSL is sold less than a Mbps download.
    • by rjforster ( 2130 )

      Yep. WSUS Offline for the windows boxes (although most of my offline windows installations are XP VMs so they are already as fully patched as they are ever going to be). Then we have an Umbongo server that serves all the Umbongo patches to the various offline workstations that host the VMs. A download script and a bit of rsyncing and the update server stays fresh.
      The only issues are the rare times someone needs a MS patch that isn't covered by WSUS Offline, in which case they deal with it manually using MBS

  • Microsoft has a product called Microsoft Baseline Security Analyzer, when you combine it with the WSUS CAB file, it will output an XML file of all patches installed and (more importantly) not installed on your machine.

    With some small scripting (VBS, Powershell, etc), you parse the XML and find the needed patches in a patch repository.

    Then you can remotely push all of that out via PS-Remoting or PSExec, and your offline/air-gapped network can stay patched.
  • I feel like I must be missing something important...
    • I guess technically I should have said "dpkg/apt"
    • by sconeu ( 64226 )

      What part of offline and airgapped did you have a problem understanding?

      • First of all you can install .rpm and .deb packages without being "online". Secondly you can be "airgapped" (i.e. not connected to the internet), and still install software using yum and apt from a server on an intranet, and actually this is quite convenient as it allows you to install software updates on many different machines simultaneously.

        Maybe you should be more knowledgeable before you decide to act like a prick.

        • by sconeu ( 64226 )

          And look at the subject.

          It's implicit that GP is not talking about .rpms and .debs, but rather the yum and apt files. And yes, you can run an airgapped intranet (I've done it myself for classified data), but it pushes back the question one level.

          How do you get said updates onto the airgapped network?

          • It's implicit that GP is not talking about .rpms and .debs, but rather the yum and apt files.

            I have read this sentence 5 times, and I still have no idea what you are talking about.

            And yes, you can run an airgapped intranet (I've done it myself for classified data),

            congratulations

            but it pushes back the question one level.
            How do you get said updates onto the airgapped network?

            How do you get anything on to an airgapped network? You don't have many options which makes this easy. Write once optical media (e.g. CDR, DVDR, BDR), provides the most security, as it prevents one avenue for data to escape the airgapped network (presumably the reason for the airgap in the first place).

            To a person who has run an airgapped intranet for classified data, I would have assumed that part was o

            • by sconeu ( 64226 )

              Pardon me. Yum and apt applications, not files.

              How do you get anything on to an airgapped network? You don't have many options which makes this easy. Write once optical media (e.g. CDR, DVDR, BDR), provides the most security, as it prevents one avenue for data to escape the airgapped network (presumably the reason for the airgap in the first place).

              Which is what the story poster was asking. There was no need for yum or apt.

              • Which is what the story poster was asking. There was no need for yum or apt.

                If you read what the story poster wrote, he/she was asking about software packages that allow patches to be uploaded to servers where customers could download them, *and* how to distribute them on airgapped machines.

                the software packages I am recommending are versatile package management systems (dpkg, rpm), that also come with tools for distribution of packages (apt, yum).

                Once the software update package files are passed the airgap, they still need to be installed. If one were to set up an apt/yum server

                • by sconeu ( 64226 )

                  You know what, I may have misinterpreted the story poster's question. Let's let it be, because there's no point to arguing here.

  • It somehow jumped the air gap and ended up in the industrial controllers in the Qum nuclear facility in Iran. It will find a way to get to your customers.
    • Yeah, I was gonna suggest this. Just put your update on a USB thumb drive with a rootkit, leave a few out in the parking lot, and wait. Someone will plug it in, at which point your drive can take over the machine and update it!

      At that point, you might also consider installing something else to help dodge the air gap to make future updates for your customer even easier; for example, try data transmission via ultrasonic frequencies to another compromised^w updated machine with a network connection!

  • Hate to plug a windows product.. but we use shavlik protect with cloud.
    It allows you to control what is patched and handles laptop users who don't always connect via VPN often.

    The product needs work to work in larger environments. GUI is sometimes a little confusing too. Things are not where you would expect them for easy use.
    It also patches our ESXi environment.

  • You have to have someone upload the patches to the airgapped machines using the sneakernet.

    I'm not quite getting what we're talking about here? Operating system patches? or program patches?

    Program patches are frequently offered in an offline format and MS offers patches like that as well.

    So I don't get what we're talking about here. What are we patching?

    To paraphrase Samuel L... Be specific, motherfucker.

  • If you are patching at the code level, I believe that git is a very good option. Because the git repository can be moved as a standard directory and essentially contains all the patches and history. So even if the customer as custom patches, they can probably easily rebuild around it.

    And since a git repository is essentially a directory, you can simply put it on a disk or flash drive and send it by snail mail or courier.

  • You're OFFLINE, bust out the thumb drive, image it, and start making your rounds to ensure they are properly applied like a good IT guy.

  • > What [...] is generally the best way to distribute patches

    Patches for what? For PC operating systems? PC software? Embeded computers?

    > in a way so customers can download them, considering
    > that the machines are offline?

    Well you can't download anything when you are offline. You mean that customers download the patches, put it on removable media and install them on their machines...

    > Are there any software packages (open source preferred) that pretty much
    > allow engineers to upload a patch wi

  • I've only had to deal with air-gapped machines on one project, but we used to send out engineers/developers to upgrade those machines. They were security-critical, so we couldn't have customer support staff getting root access to do upgrades, and they were too operationally-critical to trust to automated update disks (not USB sticks -- those can be modified in shipping.)

    So we sent staff to do the installs and updates. Not that there was a huge client base, and it was a pricey project, so the cost was n

  • We see our customers using a few technologies routinely to update "air gapped" networks, to apply updates from standard sources: Anti-virus vendors, Microsft, etc. - no special software is used on the patch/update repository:

    (1) If safety is the goal (eg: power plant control networks, train control systems) people deploy a Waterfall FLIP. The FLIP unidirectionally replicates industrial servers to external networks so corporate users can see the data, and reverses direction on a schedule to pull updates.

"Everyone is entitled to an *informed* opinion." -- Harlan Ellison

Working...