How a Cheap Barcode Scanner Helped Fix CrowdStrike'd Windows PCs In a Flash (theregister.com) 60
An anonymous reader quotes a report from The Register: Not long after Windows PCs and servers at the Australian limb of audit and tax advisory Grant Thornton started BSODing last Friday, senior systems engineer Rob Woltz remembered a small but important fact: When PCs boot, they consider barcode scanners no differently to keyboards. That knowledge nugget became important as the firm tried to figure out how to respond to the mess CrowdStrike created, which at Grant Thornton Australia threw hundreds of PCs and no fewer than 100 servers into the doomloop that CrowdStrike's shoddy testing software made possible. [...] The firm had the BitLocker keys for all its PCs, so Woltz and colleagues wrote a script that turned them into barcodes that were displayed on a locked-down management server's desktop. The script would be given a hostname and generate the necessary barcode and LAPS password to restore the machine.
Woltz went to an office supplies store and acquired an off-the-shelf barcode scanner for AU$55 ($36). At the point when rebooting PCs asked for a BitLocker key, pointing the scanner at the barcode on the server's screen made the machines treat the input exactly as if the key was being typed. That's a lot easier than typing it out every time, and the server's desktop could be accessed via a laptop for convenience. Woltz, Watson, and the team scaled the solution -- which meant buying more scanners at more office supplies stores around Australia. On Monday, remote staff were told to come to the office with their PCs and visit IT to connect to a barcode scanner. All PCs in the firm's Australian fleet were fixed by lunchtime -- taking only three to five minutes for each machine. Watson told us manually fixing servers needed about 20 minutes per machine.
Woltz went to an office supplies store and acquired an off-the-shelf barcode scanner for AU$55 ($36). At the point when rebooting PCs asked for a BitLocker key, pointing the scanner at the barcode on the server's screen made the machines treat the input exactly as if the key was being typed. That's a lot easier than typing it out every time, and the server's desktop could be accessed via a laptop for convenience. Woltz, Watson, and the team scaled the solution -- which meant buying more scanners at more office supplies stores around Australia. On Monday, remote staff were told to come to the office with their PCs and visit IT to connect to a barcode scanner. All PCs in the firm's Australian fleet were fixed by lunchtime -- taking only three to five minutes for each machine. Watson told us manually fixing servers needed about 20 minutes per machine.
There's something about a clever hack... (Score:5, Insightful)
It cheers me up when I hear about a gnarly problem solved by somebody thinking out of the box a bit, and coming up with a clever solution.
Re: (Score:1)
I agree it's a very clever hack. However the fact that this works at all makes me uneasy... it sets off alarm bells. It feels like there's a horrible exploit just waiting to be found here.
Re:There's something about a clever hack... (Score:5, Informative)
It does make you feel that? Then here is something more for you: You can program the Arduinos with native USB to emulate a keyboard as well, and they can present _any_ USB vendor and device ID you like. Not even hard to do. And with that, a PC _cannot_ tell whether this is a real keyboard or that Arduino pretending to be the exact same keyboard.
Re: (Score:2)
That is just the class of the device, you need to be able to emulate any Product ID and Manufacturer ID in order to really be undetectable. Otherwise you show up as Arduino(R) USB Device. Microcontrollers like Teensy provide you this ability, though it is against the terms of the USB license to sell a device that doesn't use it's own Product or Manufacturer ID.
Re: (Score:2)
Yes, you cannot put the USB logo on the result and sell it. But this is not an "emulation" at all. This is the real deal.
Re:There's something about a clever hack... (Score:5, Informative)
I agree it's a very clever hack. However the fact that this works at all makes me uneasy... it sets off alarm bells. It feels like there's a horrible exploit just waiting to be found here.
it's just a USB barcode scanner -a USB HID KEYBOARD device at the hardware level. I have a Symbol LS2208 that I have used for years to enter strings of alphanumeric data accurately.
The horrible exploit for this has been around for years: a USB device configured to present itself as whatever it is and as a keyboard ...and then execute a series of commands on the console (just as if you had typed the commands) a little while after being connected. You can buy these things on various websites.
Re: Your signature. (Score:2)
Shouldn't that be "Why settle for the lesser Evil?"
(Or do you think Cthulhu wouldn't be the lesser Evil?)
Re: (Score:2)
The character limit of the sig incudes the url.... the longer version doesn't fit.
Re: (Score:2)
It's no different to loading an autoexec.bat file into a computer. Your well-founded fear is rather late: That feature/bug has existed for decades.
Re: (Score:2)
Oh there are exploits.
But first, barcode scanners for PCs come in two types - ones that emulate a keyboard (PS2, USB) or ones that use a serial port (or USB).
The USB ones typically support both a virtual serial port and a keyboard.
The exploit part is that you can often get them to do weird things by scanning certain barcod
Re: (Score:2)
However the fact that this works at all makes me uneasy...
It makes you uneasy that a barcode scanner does literally what it says on the box? Scan barcodes and enter that data into a computer as text?
Re: (Score:2)
Old news (Score:2)
At least for me and my IT group. We use this "hack" since 2016, gone thru iterations of printed barcodes with the asset name/tag, then via a live search to AD for more volatile systems. Helps with emergencies but as a non-Crowstrike user, we just watched the world burn while asked ourselves what can get us in such trouble. So far, our 100k Windows based devices have not failed that catastrophically.
Re: (Score:2)
So far, our 100k Windows based devices have not failed that catastrophically.
The way Microsoft and surrounding software makers have been going recently, it is only a question of time...
Re: Old news (Score:1)
Iâ(TM)m assuming you didnâ(TM)t apply the July patch which also caused BSOD or prior patches which had issues, or in the near future you may have the privilege of updating for the UEFI âleaked keys in my Secure Bootâ(TM).
Patching Windows generally these days is more and more cross your fingers, the fact there arenâ(TM)t more problems is pure luck, but nothing critical should run on Windows these days, at least many vendors are now starting to switch to Linux especially in the health
okay but (Score:2)
Interesting workaround, you look like heroes. But you picked Clownstroke.
Missed it by "that much" (Score:2)
The script would be given a hostname ...
Sounds like a good use for a barcode. :-)
Nice idea (Score:4, Insightful)
Always good to see there are still people in IT that actually deserve to be called "engineer".
An Arduino in keyboard emulation and fed via serial would have done the same thing, but would have used a bit less standard parts and required more skills.
Re: Nice idea (Score:1)
Don't insult engineers that way, every usb barcode scanner presents itself as a usb hid keyboard since usb barcode scanners have existed
Wow such 1337 hacker engineers using an off the shelf device in its default mode to do it's fucking default job entering data into a computer
Re: (Score:2)
You are the one insulting engineers. But I guess you are not smart enough to understand that.
Re: Nice idea (Score:2)
Re: (Score:2)
That is a legitimate question.
That approach is slow, error-prone, tedious and you typically have not many people that are authorized to reboot servers. Yes, you could technically just get other people you have or even hire some gig-workers to do exactly that, but it would be a security nightmare. For example, the non-authorized people then get access to data they should not have. They get access to keys they should not have. They get access to a data-center that they should not have. At the very least, this
Re: (Score:2)
I think you just nailed the point. Not only was it thinking of a solution when adrenaline and crisis is clouding clear thought, but doing so with readily available and scaleable parts. There was no time to wait for amazon prime. This had to be done by shopping at local retailers to get equipment in hand in a matter of hours. This isnt just engineering, this is big-picture command thinking. This is the sort of leadership skill we as a society should look for in our leadership. Instead we tend to focus on thi
Re: (Score:2)
Re: (Score:2)
And thank you as well.
Re: (Score:2)
Thank you. I completely agree. Yes, the solution is simple, but coming up with it under high pressure and then making it work fast is what the key elements here are.
Re: (Score:2)
Re: (Score:2)
Typically true. Unless you have one already lying around, getting one is a delay you do not need in this situation. Hence I applaud the fast and clear thinking under pressure that these people did.
Re: (Score:2)
Moreover, if I understand correctly, the bar code is displayed on screen when the machine boots? What is that? Print the admin password as a bar code as well while at it. How does bitlocker prevent access to the data on a stolen computer then if all you have to do is scan a barcode?
There must be something I don't understand I hope.
Re: "20 minute saving"? (Score:1)
Iâ(TM)m assuming a nearby laptop with the screen, type in the asset name and it shows the key. We solved it by putting a script on a USB flash drive with Windows PE that queries the AD for the BitLocker keys, unlocks it, deletes the file and reboots. Less than 30s for most machines to recover and fully automated. They may not have had the privilege of getting a network up and running in their recovery environment.
BadUSB (Score:4, Interesting)
Couldn't the whole process be automated via a BadUSB device?
Re: (Score:2)
No because the commands to send were different for each machine
No CueCat (Score:2)
I came here to see it was a CueCat that saved the day, but I'm leaving empty-handed.
Re: (Score:2)
Re: (Score:3, Funny)
Re: (Score:2)
Barcode (Score:4, Insightful)
I had that exact idea on the day I read people were having to hand-keying long/complex codes. I use a cheap barcode reader to enter serial numbers and MAC addresses and other things routinely.
Take your LibreOffice Calc spreadsheet of the codes and create a new column with the barcode equivalents of them. https://ask.libreoffice.org/t/... [libreoffice.org] then print it out for people to use.
Or you can use a code 39 barcode font, which is even easier because it just needs a "*" for start and stop. I believe it is the only full alpha-numeric 1D barcode that doesn't require computation of check digits. But it won't work if you need lowercase letters or certain symbols unless you use "extended" code 39, which I haven't tried before. Good reference on that here: https://www.barcoderesource.co... [barcoderesource.com]
Servers or PCs? (Score:2)
Re: (Score:1)
Well, for security reasons, they didn't want to email or SMS BitLocker decryption keys out to everybody.
When rebooting encrypted Windows PCs in safe mode there's an on-screen prompt that asks you to enter its 48 character BitLocker decryption key so that it can decrypt the boot volume and finish booting. The point of using a USB barcode scanner (that emulates a USB keyboard) is that it's a fast and error-free way to enter the decryption key for you: scan a barcode and the reader confirms its checksum and ty
Very cool (Score:1)
Not to downplay the brilliance here of actually knowing how barcode scanners work, but would it really add fifteen minutes to type the code in in manually from a screen? The screens were displaying the barcodes after all.
Re: (Score:3)
How often do you successfully type a 48-character string without error?
If copying directly from a screen/printout?
If being listening to someone read it out loud? And how often do they correctly read it?
Now apply that to doing the same thing over and over.
Re: (Score:1)
Well it's a 48-number string so you could go pretty fast with the numpad. Not saying it would be just as fast, but it seems like the article blows the difference a little out of proportion. certainly much less tedious though!
Re: (Score:2)
Seems excessive, but even shaving 3-5 extra minutes with worth when you talk about many machines, it's hours or days of work.
Clownstrike was a disaster before this (Score:3)
In one instance I had finished the installation and the customer's IT department came and installed clownstrike. Our software never worked again on that system, as clownstrike killed critical communications that are required to start our software. They couldn't find a way to change the configuration to make it work, and we were locked out of repairing our software install. After a day of fighting it - with their IT folks in the room with us - we opted to reinstall the OS and start over. We had to install clownstrike first, and then our software, and then we were OK.
Though in another instance we found that if clownstrike was installed first we couldn't install our software. Different customer, but still clownstrike. That system also had to be reformatted, but this time we had to install our software first.
I have yet to find a corporate IT department who really knows how to configure clownstrike. It makes a ton of assumptions for the customer on what to block, when to update, when to scan, etc. Often we'll find that different workstations on the same network are showing different behaviors even though they are supposed to be under the same clownstrike rules.
This seems to be another product that cannot meet its marketing claims.
And yes, I would love for our instruments to communicate through Linux. I can't ask our customers to learn Linux though. Some of our installations are over a million dollars, if it doesn't "just work" for the customer we're in for a lot of pain.
Re: (Score:1)
Re: (Score:2)
Honestly it sounds like the clowns are you and your IT department. You put a lot of effort into claiming that it can't meet its marketing claims while admitting that you don't know what you're doing and that the IT people you work with don't know either.
Re: (Score:2)
Expense Report Denied (Score:1)
Woltz went to an office supplies store and acquired an off-the-shelf barcode scanner for AU$55 ($36).
At most companies, there is no way this expense report would get approved! Barcode scanner!? Expense report denied! It's better to have a longer outage than expense an AU$55 item!
Bitlocker? (Score:2)
Not being a Windows user, I'd never come across Bitlocker before From the name I would have assumed it was some kind of ransomware.
Re: (Score:2)
BitLocker is just Microsoft's native drive encryption scheme, built right into Windows. In Settings or Control Panel, there's an object just for it. It has the dubious advantage that using Active Directory, the BitLocker recovery key can be backed up in the workstation's Active Directory account. Or you can print out the recovery key, save it to a thumb driver, or your organization might have an MBAM (Microsoft Bitlocker Administration and Monitoring) site to get the key from.
You mention ransomware, and tha
Fonts (Score:2)
Does it take 15 minutes to enter the Bitlocker pas (Score:2)
Does it take 15 minutes to enter the Bitlocker password!?!?!