Facebook Unveils Superpack, a New Compression Technique (fb.com) 54
An anonymous reader writes: Facebook unveiled a new compression technique they call 'Superpack compression.' In a blog post written by software engineer Sapan Bhatia, they claim that their compression improves Android app size by 20% over the default Zip compression used by Android. The post gives an overview of the compression ideas. The basis of these ideas is called out to be a key insight in Kolmogorov Complexity, that any data can be represented in the form of programs that generate that data. Facebook's tool, Superpack, mines out such small programs and optimizes them using compiler techniques.
This plot seems familiar. (Score:5, Funny)
I would have never thought Richard would go work for Zuck. I mean, why not just go work for Gavin Belson again?
Re: (Score:2)
I thought they had already sold the Pied Piper tech to FB....
Improves or reduces the size? (Score:1, Funny)
Re: (Score:2)
Given a smaller size for binary data saves on disk space, "improving" the size of something is synonymous with reducing the size, you autist.
Maybe you have plenty of storage space, so lower CPU load compressing/uncompressing files "improves" the files by reducing time to read/write and using less power? Wait, who's the "autist?"
Re: (Score:2)
Basically a binary compressor (Score:3)
Hardly surprising you get improved compression with a domain specific compression algorithm.
That was how WinRAR also gots its claim to fame, with different binary compression algorithms.
Re: (Score:1)
It's also not a particularly impressive to compare it against Zip, an old algorithm that has been surpassed years ago.
Re: (Score:2)
To be honest if you read the article proper they also compare it with XZ which probably means LZMA2 like the one in 7-zip.
Re: (Score:2)
Impressive, no, but a useful data point since zip is still in use in the app compression domain and easily available to compare other approaches against
Re: (Score:1)
Re: (Score:2)
It's quite a clever way of compressing machine/byte code though, an improvement over more traditional executable compressors.
Pricing (Score:3)
What about the real problem? (Score:4, Interesting)
They could, you know, just develop a less bloated app.
Re: (Score:2)
To follow up, just look at Whatsapp on their chart. It seems that the apps originally developed in-house were far worse / required more compression while things they haven't meddled with for as long are a bit more lean. Instagram was sold to Facebook a couple years earlier, so they have a bit more bloat.
Re: (Score:2)
They could, you know, just develop a less bloated app.
That is silly.
Developing an app is expensive and time consuming.
Improving the app to make it 20% smaller is even more expensive and time-consuming.
Running a compression algorithm costs almost nothing.
It is ridiculous to use hours of programmer time rather than seconds of machine time.
Re: (Score:2)
Or they could employ more skillful developers who know when to do a minor re-factor. As a side benefit, the code becomes more maintainable.
Re: (Score:2)
Says the guy who is apparently NOT skillful enough to know when to do a minor re-factor and how the result would be relevant to this discussion.
Re: What about the real problem? (Score:2)
This, so much this.
Will be one of my new favourite questions when I hire someone, along with "how much weekly time do you usually devote to refactoring in a project you're developing from scratch"?
Hint: anyone who hasn't realized that usually by Friday they've acquired a deeper understanding of the problem they were working on since Monday, warranting at least a partial refactoring, is not in the top 25% of their field.
Re: (Score:2)
Re: What about the real problem? (Score:1)
Re:What about the real problem? (Score:4, Insightful)
Would you hire the 3 Stooges to wire your house? They're very affordable.
The compressor makes initial distribution smaller, but it still expands out in memory and doesn't do anything for maintainability. Also, developing the compressor took a lot of man hours.
Re: (Score:3, Insightful)
Bloated code is a problem for the end user - no other party. Apps gobble up space on the phone because of their size. Even downright simple small apps these days show their apk size in hundreds of MBs. The only reason I fathom is because of multi-device frameworks that are used for write-once, deploy on multiple devices. Why these frameworks don't discard unused code from their libraries at code generation time is beyond me.
Imagine how sinful this whole thing is. For saving some time of a developer they are
Re: (Score:2)
The compression only affects the download size though, not the amount of memory the app uses on your phone. So de-bloating it is a good idea too.
Re: (Score:2)
That strategy would, unfortunately, require some actual thought and skill.
Re: (Score:2)
That strategy would, unfortunately, require some actual thought and skill.
Indeed. That is why optimizing toward the Kolmogorov Complexity is sometimes seen as the ultimate Turing Test.
KC is proven to be noncomputable. There is no way to brute force it.
But you don't have to believe me. You can click here to see it explained by an Australian chick [youtube.com].
Re: (Score:2)
Yes, the ultimate Turing Test...of programmers!
Better programmers invariably write far fewer lines of code than worse programmers. And those fewer lines will invariable do more, with greater accuracy, than the more lines of code written with less skill.
Concise code is indeed one of the best measures of a talented programmer.
Re: What about the real problem? (Score:2)
I'm afraid that the developers of tbe Facecrap app will see the reduced size, and just use it as an excuse to bloat it up even further, with the added overhead of real time decompression.
Also, there is no fucking excuse for this to not be removable on non rooted phones. This is robbing the device owner of the memory he or she paid for.
Re: (Score:1)
We did. It's called not Facebook!
Just uninstall Facebook. There's only morons left there nowadays anyway. All it does, it make your life worse.
Re: (Score:2)
What's on everyone's mind... (Score:2)
Re: What's on everyone's mind... (Score:2)
I mean, it's Facebook... How could it be anything but middle out?
Solution without a Problem? (Score:2)
This is code size only.
You're telling me they've spent several man-years figuring out how to compress their code so that initial download and install takes less time?
Isn't the real problem with these apps that they pull video and autoplay it all day long to a bunch of digital junkies? The quantity of data pulled over the lifetime of the app is many thousand times larger than the app itself.
I got an idea, I could improve the code size by 100% if I just remove it from the phone entirely.
Re: (Score:2)
Isn't the real problem with these apps that they pull video
Optimizations based on Kolmogorov complexity can work with video too.
When Data is No Longer Data (Score:5, Interesting)
Is this a fixed corpus of code snippets selected by tokens in the data stream? Or is the "compressed" data stream actually a pile of tiny subroutines I'm expected to execute to reconstruct the data?
Because if the latter, there is absolutely no fscking way I'm touching this. "Oh, it'll be sandboxed!" Fsck you; that's what you nincompoops said about Java and JavaScript and ActionScript and MSWord macros, to name but a very few.
Re: (Score:2)
Since you didn't say "Don't get me started on CPU hardware vulnerabilities." Might I get you started on CPU hardware vulnerabilities?
Re: (Score:2)
and don't get me started on kidnapping the IT admin's child to extort access.
Re: (Score:1)
closed source, too bad (Score:2)
From the blog post: "We may someday consider open sourcing Superpack."
Too bad, i had good luck using zstd (https://github.com/facebook/zstd) on binary data. Since this leverages zstd, was hoping even better compression.
Re: (Score:3)
I wish they could have Superpack as part of zstd, because zstd is extremely useful. I mainly wind up using both with Borg Backup (a good deduplicating backup program), as well as btrfs (which is another thing Facebook has tamed and improved on.) Both have been quite useful in day to day use.
FB's bad stuff aside, these F/OSS contributions are quite useful in day to day use. It would be nice if they could come out with something that beat "xz -v9e" for compression, because often, there is plenty of CPU ava
"We may someday consider open sourcing Superpack." (Score:2)
Great. Once you do, assuming it's not patent-encumbered, maybe someone else will consider using it too.
And then maybe you won't have problems like this:
Since the diffing process implemented in the current tooling is not able to interpret Superpack archives, the deltas come out to be larger for apps containing such archives.
But this only shrinks the APK size, not the app (Score:2)
But this only shrinks the APK size, not the app. The app is still decompressed on the device and takes up as much space as it wants, regardless of how the APK is created.
Re: (Score:1)
Yes, and? Every picture on a website has to reside uncompressed in RAM aswell.
People always wonder why web sites take so much memory. Have you ever tried decompressing all the shit on a modern site? 1GB only gives you a square 16k image in 8-bit RGBA!
Facebook already was using extreme compression (Score:2)
Just look at the Facebots' brains with an x-ray machine.
A More Apropos Subject Line is (Score:3)
Sounds like what xz already does. (Score:1)
20% more than zip isn't exactly a high bar.
RAR and xz have been doing that for more than a decade.
Re: (Score:2)
Not really, compression is about as relevant as more pi digits at this point.
very nice (Score:2)