Microsoft and Intel Project Converts Malware Into Images Before Analyzing It (zdnet.com) 45
Microsoft and Intel have collaborated on a new research project that explores a new approach to detecting and classifying malware. From a report: Called STAMINA (STAtic Malware-as-Image Network Analysis), the project relies on a new technique that converts malware samples into grayscale images and then scans the image for textural and structural patterns specific to malware samples. The Intel-Microsoft research team said the entire process followed a few simple steps. The first consisted of taking an input file and converting its binary form into a stream of raw pixel data. Researchers then took this one-dimensional (1D) pixel stream and converted it into a 2D photo so that normal image analysis algorithms can analyze it.
Like Graphs or Heat Maps (Score:3)
We will often take data and put it in Graphs to see trends that may be difficult to calculate. We use heat maps to figure out density of how things are distributed.
Humans are still much better than computers at catching trends and patterns especially visually. I have actually taken data and converted it to an image, then I did an algorithm on it and I was able to see a graphical representation of the image. Also I can spots patters much quicker. So a Malware embedded in an other program will look about the same way, even if it is spread across some extra static.
Re: (Score:3)
Really? How many images can you scan in an hour?
Re: (Score:2)
Yeah. But the technique that they appear to be using here just slices the 1D byte array of the code into equal length lines and then stacks them as the second dimension in a picture. But there is no guarantee that bytes in adjacent lines have any correlation. There are methods for representing program and data flow as multidimensional maps, which have graphical representations. But that processing is non trivial and I don't see it mentioned in TFA.
Ummm... (Score:4, Insightful)
What's the difference between a grayscale image and an array of bytes? This summary sounds like the worst of science reporting, where the reporter doesn't have the first clue what they're talking about...
Re: Ummm... (Score:2)
Computers, you see, are magic.
Once we convert from executable Evil Bit Format to Microsoft(r)/Intel(r) Safe Image AI Render(tm) format then we can apply our special machine learning AI algorithms to it.
The standard process I'm sure you are familiar with:
1: claim to use AI
2:
3: profit!
Re: (Score:2)
Re: (Score:1)
A grayscale image is an array of bytes. Basically this is a bit-matching analysis to see whether the code contains certain fragments that are known to be bad or closely related at least.
Re: (Score:2)
And that is different from the usual pattern-matching algorithm used so far to fingerprint sections of malware in what way exactly?
Oh. Right. We have converted it to an abstract greyscale picture first.
Transfer learning - Edge detection, shadows, etc (Score:3)
The advantage of treating it as an image is that they are able to use pre-existing training rather than training from scratch. Existing image processing models can already detect edges, basic shapes, etc.
For example, if a given malware family looks like an oval, the new model only has to determine that oval=Cryptolocker. It doesn't have to first learn how to identify ovals.
Re: (Score:2)
That's awfully convoluted, you do know that the image processing is mostly dependent on teaching a computer how to interpret the binary representation of shapes in pictures, right? And that computers are still not really good at it (that's why CAPTCHAs are still a thing).
Why not cut out the middle man and just teach it to interpret the binary representation of an executable binary? I can see why you want to use what picture interpretation software did as a starting point for your development, but turning a
Re: (Score:2)
> That's awfully convoluted, you do know that the image processing is mostly dependent on teaching a computer how to interpret the binary representation of shapes in pictures, right?
And that part is already done. You don't need a samples of the new malware to train the ML on because they are using ML that has already been trained on millions of "shapes in pictures".
Kinda like you learning a new word vs starting with a baby who doesn't know the letters. Starting with a pre-existing knowledge of letters
Missed a word - a million samples (Score:2)
I missed a word in what I wrote. It should be:
--
You don't need a *million* samples of the new malware to train the ML on because they are using ML that has already been trained on millions of "shapes in pictures".
--
Sufficient samples for training is an issue when trying to use ML and AI to recognize new malware. They are polymorphic, meaning they aren't the same each time. You're looking for subtle patterns. It's awfully hard to reliably find the subtle patterns when you have a total of five examples, if y
DAC to Shazam (Score:2)
> sounds a bit like pushing a signal through a DAC only to feed that into an ADC.
Yes. Where Shazam exists. Consider you have a large collection of mp3 files, eacha live recording of a song. Your task is to identify the songs. You also have the Shazam app, which recognizes songs exactly as you want to do. It takes analog input thriufh the microphone.
A reasonable plan is to pay the mp3s and let Shazam recognize them. It's inefficient to build Shazam for this purpose of you're starting at Genesis 1:1, bu
Re: (Score:2)
There are two reasons. The first is that while the underlying binary data between two similar malware files may change considerably, the image-equivalent may end up being extremely similar. This happens in actual images: you can take the same image and produce two files with completely different binary values (a jpeg vs a png, for instance) . Your binary-matching tool won't notice they're the same image, while an image-scanning tool will.
I can see why you want to use what picture interpretation software did as a starting point for your development, but turning a binary into an image to then use an algorithm to interpret it sounds a bit like pushing a signal through a DAC only to feed that into an ADC.
Actually, yes, that is in fact a very good way to perform song identi
Re:Ummm... (Score:5, Interesting)
The interesting thing would be that existing image analysis algorithms would be relevant to identifying malware vs. non-malware, which isn't obvious -- although probably not totally unexpected, because presumably there are patterns in malware code related to their similar behavior that are not present in non-malware. The mechanics of converting it into image data suitable for the existing algorithms isn't very interesting, as it sounds like they pretty much just use the bytes as pixel intensity values.
The thing which seems weird to me is that from a quick glance at the (very superficial) article, it sounds like they just naively chop up the stream of bytes into equal-sized lengths to create a 2D array ("picture"). In a real picture, there will generally be a high amount of correlation between pixels directly above or below one another, and that seems like it wouldn't apply in this case.
Re:Ummm... (Score:5, Insightful)
Re: (Score:2)
It's binary code. It's actually highly structured and not as random as you think. It looks random to you and me, but that's because we're not trained to spot the patterns (plus, hexadecimal isn't always the best way to view opcodes especially if it doesn't line up with the fields).
In fact, those who are experienced disassemblers can easily go and recognize patterns in the code - it comes with experience and you learn to recognize what compile
Re: (Score:2)
My skepticism is mostly aimed at the validity of using a convolutional net trained on natural imagery distributions. I suspect that a well-designed neural net model could be trained to find similar such patterns on run-of-the-mill binaries, but I'd expect it to use attention rather than 2D connectivity, and I'd expect it to need
Re: (Score:2)
Except that generally speaking the machine vision treatment of pictures would tend to process the image in very low precision. Machine vision would have a lot of problems if it would fixate on high variation among very nearby pixels, which in the real world would be noise to discard.
As far as I've been told, the AI field have had a lot of interest in image and voice samples, because they are complex and unstructured data sources that are hard to traditionally process, but they can and do apply their method
Specific malware family, not malware generally (Score:2)
> The interesting thing would be that existing image analysis algorithms would be relevant to identifying malware vs. non-malware, which isn't obvious
The technique doesn't really distinguish between malware and non-malware per se. The system can be trained to detect the Clop ransomware. It can be trained to detect the Maze ransomware, etc. It detects each specific malware family that it was trained on.
Re: (Score:3)
What's the difference between a grayscale image and an array of bytes?
A grayscale image is an array of numbers, of course.
The point is that there exists a large body of software to do image analysis, and the proposal is that some of this image analysis tech might be usable to analyze that array.
I'm not sure if I believe it, but it's an interesting approach.
Re: (Score:3)
An image encodes associations between values in two dimensions instead of just one. I don't see any reason why those associations would exist in a binary though. Which might mean that some intern at MS or Intel wanted to apply a pretrained 2D convnet used for image analysis instead of just training a 1D one.
Or perhaps they don't realize 1D convolution exists. I know that sounds strange, but I've met "data scientists" who think convolution is specifically an image analysis technique.
Re: (Score:3)
The problem is that the second dimension is completely arbitrary. Unless they found something rather spectacular that allows you to find malware by "diagonal reading".
Re:Ummm... (Score:5, Insightful)
Exactly. I've seen people try this before. The idea is that we've got all these powerful tools for image analysis, and encoding data as a visualization is really useful for humans, so it should be a good idea for computers too, right? People seem to like that style of superficial reasoning.
If the description of what they've done is accurate, I'd think you'd be able to fool it just by sticking a few random bytes in your binary to throw off the way the lines are stacked. Like loading an image with the wrong size fastest varying dimension.
Re: (Score:1)
Re: (Score:2)
Why don't you go use the store of all human knowledge and educate yourself enough so you can have a moderately informed opinion? In the meantime, the adults are talking. Ssh.
Re: (Score:1)
Re: (Score:2)
Lol
Blurry images (Score:2)
Reading the article - They're turning binary files into blurry images and building the hash from the blurry image.
Re: (Score:2)
if i do make some pixel data at some step of the process, it's so i can display it on screen for dramatic effect. i might add some high tech looking green targeting reticles to make it really look like it's doing something for the user. Then i'll intentionally slow my processing down because if t
Re: (Score:2)
Re: (Score:2)
What's the difference between a grayscale image and an array of bytes? This summary sounds like the worst of science reporting, where the reporter doesn't have the first clue what they're talking about...
In the first, you look at correlations between arr[X] and arr[X-Width] and arr[X+Width] and arr[X-1] and arr[X+1]. In the second, you only look at correlations with arr[X-1] and arr[X+1].
Well that fits (Score:2)
Microsoft has never known how to deal with viruses in any of its software products. I'm not surprised they try to turn them into animated GIFs this time around...
I guess research is research, but (Score:2)
I'm not sure I get the point of this beyond proving it can be done?
Isn't it much more computationally intensive to analyze 2D images (that you have to render the raw data into, in the first place) than to just directly look for matching pieces of code?
Re: (Score:2)
"Click here to make the magic eye picture dance"...
Reuse hardware, like GPGPU (Score:4, Interesting)
Their proposal is analogous to the early attempts at using GPUs for general-purpose programming. Before you had CUDA and OpenCL, some people noticed that the shaders and texture pipelines in programmable GPUs (once they received FP support) were, essentially, high-performance matrix operators. When the GPU frame buffer could be extracted back to the CPU, they adapted their algebra code such that one matrix would be considered as a texture, and the pipeline applied the required operation by considering data matrices as image components. This was circa early 2000's.
Today, they are doing the exact same conversion to exploit their already-existent HW-supported image analysis AI pipelines to identify malware samples.
Not new, but this particular approach is faulty (Score:2)
Using images to store byte arrays and then referencing them with linear algebra and other GPU-optimizable math is quite clever and a wonderful optimization. Support Intelligence [support-intelligence.com] has done this for several years. Using a CNN built for photos (they're using a pre-trained ImageNet model) is new though, and I can't say it makes any sense to me; why would the features be at all related? (SI uses SVMs.)
Take a look at Malware Detection by Eating a Whole EXE [arxiv.org] by researchers at UMD, nVidia, UMBC, et al. In section 7
That just sounds like slavery with extra steps. (Score:2)
Right, so you take data that is fundamentally 1D and you cut into slices of an arbitrary length because you need to fit it into a rectangle, presumably you also pad it with some extra 0s or something.
So the relation between adjacent pixels on the Y axis represents very little and is based on your arbitrary choices.
Then to scale the image to a more manageable size you average these unrelated values.
And then you run it through a neural net that has been trained on data that has completely different characteri
Computational Homeopathy (Score:2)
Oh wait, I get it, it's computational homeopathy, you keep diluting the data until it's indistinguishable from random noise.
Re: (Score:2)
I read the article, such as it is. They wanted to use a pretrained network. Most of the pretrained ones are for 2D images, so well, let's format our binary as a 2D image!
I imagine it works just as well as looking at a bitmap you've rendered with an arbitrary width.
Neural networks are extremely flexible though, so I'm sure they get something just by doing a bit of "fine-tuning" training. I expect they'd get better results training from scratch using a sensible model though.
Here's my evaluation (Score:1)
This is the dumbest of dumb ideas. Really dumb. Incredibly dumb. Opcode patterns are relevant. 2D representations of opcodes are not. Anyone who had the faintest idea of what they're doing would know this simply would not work.
No one read Snow Crash??? (Score:2)
Because this is how we get Snow Crash!!!