Meta Developing Compressed RAM 'CRAM' For Linux: Better Than ZRAM and Zswap? (phoronix.com) 94
Phoronix reports on a presentation this week by Meta engineer Gregory Price at the Linux Plumbers Conference in Prague:
With today's memory shortages, high prices, and ever increasing memory capacity needs, Meta engineers have been working on Compressed RAM ("CRAM") for Linux.
This hardware-offloaded compression still allows cacheline/byte access to compressed memory and in turn a better implementation than the likes of ZRAM and Zswap. Very promising is that there is near-native performance to raw DRAM speeds with CRAM... For end-users that may already be relying on the likes of Zswap or ZRAM, CRAM is offering near-native DRAM speeds. Read-only data is shown to be as fast as the raw DRAM performance. And for memory writes, CRAM is much faster than ZRAM or Zswap or plain swap.
"Most of the individual pieces needed to support CRAM are already mainline," the article points out. Tom's Hardware writes that "It uses a private NUMA node (essentially, a ghost CPU) instead of pretending to be a block device, and this lets Linux continue using all of its memory semantics, including migration and ballooning, to manage CRAM." Because CRAM is stored in RAM and treated as RAM, with full cacheline/byte access, it can be accessed in a read-only fashion with little delay; just the cost of hardware-offloaded compression... Even when you enable writes, CRAM is still much faster than ZRAM; 5.4x in the worst tested case of 20% writes. That's a huge drop from the 452x read-only case, but keep your context; a 5.4x speedup is still titanic. The massive performance cliff when writes are involved comes down to the need to page fault and migrate folios back to the original NUMA domain, as you can't write directly to compressed data; you'd corrupt everything...
While the obvious target of the work here (given its origin at Meta Platforms) is big Linux servers, ZRAM and Zswap are used all over the Linux ecosystem, even on machines as constrained as the Steam Deck. Many distributions enable one or the other by default. CRAM seems like it could offer a considerable speed-up for some of these machines, so hopefully it finds its way into the kernel once the remaining implementation questions are answered.
This hardware-offloaded compression still allows cacheline/byte access to compressed memory and in turn a better implementation than the likes of ZRAM and Zswap. Very promising is that there is near-native performance to raw DRAM speeds with CRAM... For end-users that may already be relying on the likes of Zswap or ZRAM, CRAM is offering near-native DRAM speeds. Read-only data is shown to be as fast as the raw DRAM performance. And for memory writes, CRAM is much faster than ZRAM or Zswap or plain swap.
"Most of the individual pieces needed to support CRAM are already mainline," the article points out. Tom's Hardware writes that "It uses a private NUMA node (essentially, a ghost CPU) instead of pretending to be a block device, and this lets Linux continue using all of its memory semantics, including migration and ballooning, to manage CRAM." Because CRAM is stored in RAM and treated as RAM, with full cacheline/byte access, it can be accessed in a read-only fashion with little delay; just the cost of hardware-offloaded compression... Even when you enable writes, CRAM is still much faster than ZRAM; 5.4x in the worst tested case of 20% writes. That's a huge drop from the 452x read-only case, but keep your context; a 5.4x speedup is still titanic. The massive performance cliff when writes are involved comes down to the need to page fault and migrate folios back to the original NUMA domain, as you can't write directly to compressed data; you'd corrupt everything...
While the obvious target of the work here (given its origin at Meta Platforms) is big Linux servers, ZRAM and Zswap are used all over the Linux ecosystem, even on machines as constrained as the Steam Deck. Many distributions enable one or the other by default. CRAM seems like it could offer a considerable speed-up for some of these machines, so hopefully it finds its way into the kernel once the remaining implementation questions are answered.
Think smaller (Score:4, Interesting)
ZRAM and Zswap are used all over the Linux ecosystem, even on machines as constrained as the Steam Deck
The technique is especially useful on even more constrained systems, like embedded controllers in 3d printers and such. The only places I've ever needed ZRAM were on a phone with 512MB and on my 3d printer with 256MB.
Re: (Score:1)
Seems like that would be entirely false. Embedded and constrained systems are where a proper amount of RAM is easiest to provide directly. Your perspective is after the fact, you apparently desire to use systems beyond their design points. More flexible memory would be most advantageous in programmable systems, not fixed function ones.
How does one know where they "need ZRAM" and where they don't?
Re:Think smaller (Score:5, Insightful)
you apparently desire to use systems beyond their design points
Everyone wants to use systems beyond their design points. Or rather, more accurately, the design point is *always* the cheapest way to accomplish a goal, and that's a moving target.
If you are a company that provides an embedded system with a known functionality that requires a specific amount of RAM, and you realize you can ship that system with less RAM because you of CRAM or ZRAM, you just decreased your costs and increased your margin. There is literally no embedded provider that wouldn't be looking at that.
Re: (Score:2)
That said, Meta clearly isn't targeting 68HC05s here, so the distinction is largely meaningless.
Re: (Score:2)
Seems like that would be entirely false.
Sure, if you have no experience and want to share your feelings about it.
Embedded and constrained systems are where a proper amount of RAM is easiest to provide directly.
This is a matter of perspective. Where you're shipping many units and want to save money, it can easily be false, as it was in both of my examples.
How does one know where they "need ZRAM" and where they don't?
man free
Re: (Score:2)
I think the man pages need a graphical UI update and at least 1 AI function in order to attract the "modern" programmers attention.
/sarcasm
Re:Think smaller (Score:5, Insightful)
> you apparently desire to use systems beyond their design points
Who doesn't include cost in design?
If my CPU/SoC is plenty fast for my task and the DRAM is faster than I need it to be and a CPU instruction to compress takes less power than a write out to the bus - it would just be wasteful to manufacture and purchase twice as much DRAM as the application needs.
Either the end user gets a better price (and all money is the result of expending energy, so we save again) or the manufacturer makes a larger profit (in a constrained market).
Now then - all you lunatics embedding Electron and costing me 400MB to send an 80 character text message - we need to talk about this.
CRAM seems like a great idea but it's not the solution to not caring about the Craft.
Re: (Score:2)
Why would I buy compressed RAM when I can just download a RAM doubler? lol.... Welcome time travelers to the 1990s?
Re: (Score:2)
What kind of printer is that?
Mine runs on 16 KB plus the RAM of the Klipper host.
Re: (Score:2)
It's a FlashForge Adventurer 5M. This is the Klipper host I'm talking about. I don't know how much RAM is in the other parts, it's not that relevant. It's really not enough memory so ZRAM is a necessity. There are mods for the printer, I am using ZMOD. It also offers a Klipper upgrade, fluidd, lower-memory camera streaming, and a better touch screen interface.
The AD5X has 512MB, the AD5M and AD5M Pro have only 256 and it's not enough. It causes timer too close errors. The controller hardware is just general
Re: (Score:2)
I see. My klipper host is a repurposed thin client (Dell Wyse 3040) which is way overpowered for the task.
Re: Think smaller (Score:2)
If I were to build my own printer I would do something similar. Experience with this printer has taught me that it is the way to go. For my next printer I plan to build a tool changer.
Re: (Score:2)
Yep, my printer is of mostly my own design, somewhat inspired by the LH stinger.
data || GTFO (Score:5, Insightful)
it can be accessed in a read-only fashion with little delay; just the cost of hardware-offloaded compression..
This reminds me of the days when we were using hardware RAID controllers...even though it was all done in hardware, it seems like they were somehow always slower than the Linux software RAID controller. I never figured out why. What I am saying is, I don't trust the hardware-offloaded compression to be faster without seeing the data.
Re: (Score:2)
It reminds me the days I needed DoubleSpace to fit DOOM on my 20 Mb hard drive (or whatever it was).
Re: (Score:2)
it seems like they were somehow always slower than the Linux software RAID controller.
That's your confirmation bias. Back in the day people specifically used hardware RAID precisely because it was faster, measurably so. Also I/O performance wasn't the only metric. Software RAID consumed considerable CPU resources back when we weren't running threadrippers with half their cores idle. Not only was I/O performance better on hardware RAID but your CPU utilisation was lower allowing for more cycles to be dedicated to the application (quite critical for database servers of old).
Re: (Score:2)
But on Linux? The software RAID controller was faster. The job is I/O bound, not CPU bound.
Re: (Score:2)
If you go old enough, you have to take into account that memory bandwidth was terrible.
On a Pentium 1, I believe RAM can do somewhere around 100MB/s. And sure, it gets better later, but still. If you consider that a serious RAID-5 for instance, you're reading at a minimum 3 drives, and computing parity, then yes, you can easily take most of your memory's bandwidth just on that. It makes a lot of sense then to offload stuff to a RAID card.
These days hard disks are laughably slow compared to the rest of the c
Re: (Score:2)
One of the key benefits of a RAID controller was having a battery backed write cache. Writes could complete instantly into the cache and then be written to the drives later. If the power failed data was held in the battery backed memory and written out when the power for the drives came back.
Re: (Score:2)
Yep, though if you had UPS, this wasn't really an issue. Though obviously corporations like to minimize risk, though RAID cards dying was a nasty issue, sometimes had to use special software to access the data if you couldn't buy a new compatible RAID controller. Or worse, there were some that stored the RAID configuration in NVRAM so if the RAID died, it was non-trivial to rebuild even with correct replacement card.
Re: (Score:2)
Remember when Microsoft wrote a storage driver for MS SQL Server? And how it interacted with the 'native' MS Server driver?
Writeback was the killer feature (Score:2)
But on Linux? The software RAID controller was faster. The job is I/O bound, not CPU bound.
The good RAID controllers had a battery-backed writeback cache, which would make transactional workloads (lots of fsync) much faster by reordering across write barriers. It's an order of magnitude faster in many cases.
You can't do that without battery-backed memory to replay the transactions after a power loss. Memory-only cards did exist, but they were rare and proprietary. In practice 99% of people who needed this just used a hardware RAID card.
Re: (Score:2)
Even today we have SSD with capacitors and DRAM cache for the same reasons - instantly return that the write is committed, and if there is a power loss, the capacitors have enough juice to commit to flash.
Re: (Score:3)
Around Pentium era and before, running RAID5 in software was quite cpu bound. But after that CPU became much faster than the i960 processor used on many hardware RAID cards, so using software RAID on Linux was becoming a great option. It also didn't suffer the problem of hardware failure making the RAID array unaccessible. I recall many cases where my customers lost data or we had to hunt for identical used RAID controller as the one that died.
The CPU load became basicly non-issue once we got multicore CPUs
Re: (Score:2)
The earliest "consumer" grade RAID cards, by contrast, were basically just softRAIDs with a built-in IDE controller - And that's still true to a large degree; the "RAID-mode" in your gaming PC's BIOS is still merely offloading the hard work to the CPU.
I'm not claiming Linux' softRAID implementation was or wasn't faster than cheap RAID hardware, but I have no problem believing it could at least be comparable.
Re: (Score:2)
Those cards rarely did RAID5, which is the only one that really consumes CPU cycles, requiring lots of XORing. Hardware RAID1 and RAID1/0 was faster than software as it could duplicate the writes instead of software writing the data twice. Though once you saturated the PCI bus, it really didn't matter that much.
Re: (Score:2)
So, at first when they were introduced they were faster, because they used cores comparable to your CPU. That's why they were expensive. Then cheaper ones were introduced, they did less of the work, they were slower. In that case more of the "RAID" magic is happening in the driver instead of on the card. But expensive RAID controllers with full offloading were still fast. Eventually our CPUs got to be fast, and now software RAID is usually faster.
In the case of hwraid vs swraid, there is another reason not
Re: data || GTFO (Score:3)
We measured the same thing: a hardware raid controller performed better software raided, and the vendor would tell us "were were measuring wrong, here is a super synthetic benchmark that is faster".
My best guess is that the hardware raid inevitably collapsed all the disks into a single IO queue, so workloads with independent IO could be more independently scheduled and serviced in software raid. Alternatively, the software stack has more visibility to the topology and can default to better aligned data lay
Re: (Score:2)
Sorry, you can't see the data, it's compressed.
Re: (Score:3)
I'm reminded of the arguments had over SCSI RAID controllers, and turning off read-after-write-verify... Since, at the time, SCSI drives did RAWV in hardware, that was redundant on the controller. And in software.
This came to a head, for me, when I queried a Novell Labs fellow about the 'non-media defects' I saw deactivating SCSI drives on servers. They finally gave in and explained, it was a controller defect. Which I challenged. And they finally stated that a drive hardware write error (RAWV fail) would
Re: (Score:2)
Throughput Vs latency. Hardware RAID tends to be fast for processing large volumes of data, but the overhead increases the latency.
It's a bit different with DRAM though because the controller is complex and built into the CPU. AMD, for example, already transparently encrypts data in RAM with no performance loss, all in hardware.
turns out you can (Score:3, Funny)
download more ram.
The return of Ram-Doubler (Score:4, Interesting)
Back in MacOS 8 days or thereabouts.
It didn't have the benefits of hardware compression though.
Re: The return of Ram-Doubler (Score:2)
Re: The return of Ram-Doubler (Score:3)
Re: (Score:2)
Windows? Even DOS had a RAM doubler. lol.
Re: (Score:2)
We also need better programmng (Score:3)
Better programming means fewer computing resources for the same task, RAM included.
C or Rust to the rescue
Re: (Score:2)
C is pretty good at that, but I haven't seen statistics for Rust. I suppose all the complex stuff could be optimized away during compilation, but I don't know that it is.
Re: (Score:2)
Turn off Rust's safety features (cargo run --release), when finished debugging.
Re: (Score:2)
OK.
But that's not the actual reason I don't use rust or go. The actual reason is that I don't trust downloading stuff off the net from sites that I don't feel are sufficiently monitored. I prefer stuff that resides in, e.g., a debian repository. (Also programming in rust feels like programming with one arm tied behind my back. Safety measures that only make sense when one is doing multi-threading being applied everywhere.)
Re: (Score:2)
Too many layers of abstraction. You're running an OS inside a slice of a processor in a virtual machine. Inside that sits more containers like docker and even python now with virtual envs. Every layer ads more cpu instructions to simple tasks.
Re: (Score:2)
Too many layers of abstraction. You're running an OS inside a slice of a processor in a virtual machine. Inside that sits more containers like docker and even python now with virtual envs. Every layer ads more cpu instructions to simple tasks.
Unikernel to the rescue?
https://github.com/hermit-os/h... [github.com]
Re: (Score:2)
Virtual environments in Python are literally just path manipulation.
What the hell.
Re: (Score:2)
Too many layers of abstraction. You're running an OS inside a slice of a processor in a virtual machine. Inside that sits more containers like docker and even python now with virtual envs.
Just to correct one part of this, Python's virtual environments are in no way a virtual machine - they just alter the environment variable PATH and the search paths used by the Python interpreter to find installed modules. There's no additional hardware virtualization.
Actually, containers are similar. Normally, they use the same kernel as the host operating system. There is no virtualization unless you specifically need to run a different kernel (such as running Linux containers on Windows). Running a Li
Re:We also need better programmng (Score:5, Insightful)
Better programming means better programmers. We are currently going the other way with bad programmers being replaced by even worse slop.
Re: (Score:2)
I agree we need better programmers.
We have better programmers.
The frontier models _are_ better programmers than all but the very best.
Look at the AI related posts from ESR and DHH on twitter. They are _all in_ on AI.
(DHH has been showing benchmarks of web apps being machine-re-written in Rust. Unsurprisingly, the apps get much faster)
A rough estimate is that the best LLMs do software engineering about as well as IQ 150 software engineers do it. That makes them better at it than the vast majority of peopl
Re: We also need better programmng (Score:2)
Re: (Score:2)
I agree we need better programmers.
We have better programmers.
The frontier models _are_ better programmers than all but the very best.
A rough estimate is that the best LLMs do software engineering about as well as IQ 150 software engineers do it. That makes them better at it than the vast majority of people who write code.
People see an AI do something they dont like and then conclude "the AI sucks". But lets use honest weights and measures: Human programmers make worse mistakes, and make them more often, and iterate 1000x more slowly.
I wish everyone saying LLMs are not even AI and everyone saying LLMs are as good as "software engineers" with 150 IQ would lock themselves in a room and beat each other senseless until they both started making sense.
Re: (Score:2)
Donald incited insurrection, pardoned violent assaults on 140 police then fired agents for bringing attackers to justice
I saw your comment out of the corner of my eye, and immediately wondered what Donald Knuth had to do with this.
Re: (Score:2)
Don't worry. LLMs are complete crap at writing secure software but are pretty good at attacking software. Hence the people using them for actual coding will not be around anymore in a few years.
Re: (Score:2)
Perhaps you should try using an LLM.
Or simply join some news groups / mailing lists.
Most modern coding LLMs can code from random Jira (inert your issue tracker) tickets, better or on par with a human.
I guess once long ago you learned how to use a compiler, it is time to learn how to use an LLM, or stay using a compiler.
Re: (Score:2)
Re: (Score:2)
Those are two examples.
You can also go read what Carmack says about it. (example: x.com post from 2026/9/11)
I hope you won't be so stupid so as to try dunking on Carmack.
I apologize for not knowing you or your accomplishments, but just playing the law of averages, both ESR and DHH have significantly larger bodies of successfully and daily-used code than you do, with bigger impacts on the field and the industry, than you do.
You probably don't have the right to shit on them, given your own lack of contributi
Re: (Score:2)
I can say that both of them certainly eclipse _me_ so greatly that I can't be seen on any map that is zoomed out enough to show both of them. And while I am a relative nothing, I've had a handful of patches accepted in my life... and I've secured a few software patents in my day. That puts me in above-average territory. But there's more accomplishment gap between them and I, than there is between me and a person who cannot code at all.
There's no harm in admiring outstanding people. A particular fault of our age is that men do not know how to truly compliment and admire greatness when they find it.
ESR is a great man. DHH is a great man. My world and our industry is better for both of them. When they say that LLMs have completely changed the way they do the thing that they are better than almost all other human beings at... you should listen.
Please just put a sock in it. You keep spewing inane rhetoric about AI even in places where TFA has nothing to do with AI. You can't support your statements objectively and so all you have left are lame ass appeals to authority and generalized claims lacking any useful specificity.
There are a lot of people here who spent decades working various domains you've never even heard of it doesn't fucking matter that one person is more popular than another or their code is used by more people than someone else's.
Re: (Score:2)
>Please just put a sock in it.
No.
The OP complained about low quality programmers and "slop"; clearly a reference to AI generated code.
My retort was to challenge that - to point out that fronteir LLMs are now much better at writing software than most programmers. That is objectively true. It doesn't "feel true"; it _is_ true.
Then someone replied to me to try and snipe at ESR and DHH, as if they weren't both excellent programmers. Both of them are. That wasn't a person arguing against my point; that wa
Re: (Score:2)
My retort was to challenge that - to point out that fronteir LLMs are now much better at writing software than most programmers. That is objectively true.
If it is objectively true provide your evidence. Given such an extraordinary claim it shouldn't be that difficult.
It doesn't "feel true"; it _is_ true.
Your assertion _is_ absurd on its face.
There are some bright and accomplished people here. That's part of what makes the place interesting and valuable. There are also a lot of drive by haters.
You sound like the goonies in our marketing department.
Re: (Score:2)
I was very impressed by ESR in general until he supported the OSI OSS terminology origin story myth, now I'm only impressed with him as a programmer. How long before the software makes him obsolete in that area as well?
Re: We also need better programmng (Score:2)
Better programming IS done by better programmers only.
Even AI agents rely on actual programmers skills written in training code.
And the choice of the programming language is another key point.
Re: (Score:2)
Yes. LLMs cannot exceed their training. They can sometimes make statistical connections that humans overlooked, but that is it and that is not insight. Usually it is just hallucination, sometimes it is something of limited worth that then gets blown all out of proportion by the propaganda machine feeding the stupid.
And the choice of the programming language is another key point.
I used to think that, not anymore. Good programmers can write good software with any language. Bad programmers can do it in no language. These days I am firmly convinced language choice is a matt
Re: (Score:2)
Re: (Score:2)
The FOMC [wikipedia.org] has been manipulating [federalreserve.gov] the traditional (stock, bond, commodity) markets to prevent market crashes (especially since 2008). The side effect is that it's become more difficult to get the large returns from market swings (ie crashes and rebounds). So now speculators (let's not call them investors) are searching outside the traditional markets for "big profits".
RAM is simply the latest victim.
Re: (Score:1)
C'mon man. The RAM crisis will be over in 4 weeks.
We're only the seeing Super-Intelligence dominate the news cycle leading up to mid terms.
AI slop will dissipate once the alliance has guaranteed the house and the senate for another 2 years.
US President Eisenhower sat and warned the American People about the growing threat of the American Military Industrial Complex, over 65 years ago.
What makes you assume that entity will simply walk away from the next nameless, faceless, war on (AI) terror, and the trillions it will make for them, using nothing more than “cold” threats and DDoS sims, with every politician being serviced with a big, fat PAC.
Four weeks? If AI collapsed entirely tomorrow, how easily would all of society recover at
That old idea again (Score:2)
It seems enough time has passed since the last catastrophic failures in this space that people do not remember. No, this does not work except in some very specific scenarios. What Meta is probably doing is get hardware done for their very special and very irrelevant elsewhere use-case. That does not even merit a story.
Re: (Score:2)
I'd be interested to see the power budget on this. Regular RAM's power cost mostly scales with installed capacity, compressed RAM's power cost is going to scale with usage. It might be a big win for specific applications that need to have huge amounts of fast memory on-hand but aren't hammering it constantly. Doesn't look like Meta published any power figures.
If the compression power cost is low enough, it might be a win for everyone. Otherwise, it's still a win for the hyperscalers: they get to exchang
Re: (Score:2)
Right, both of you.
If you compress a harddrive or stuff a RAR file full at max compression, it costs power (CPU cycles which adds up to more electrical power) to access that data at a slower throughput VS just accessing the data pre-compression.
Haven't read the whole article, but I bet their aiming for t he compression processor on the RAM stick, along with the ECC chip... so any request for data runs through (and is slightly slowed by) ECC, then to the compression processor (which, I assume, maintains a ta
Re: (Score:2)
Depends on the performance of the RAM...
RAM is much slower than the CPU itself or the cache, if you can read half as much data from RAM because it's compressed and the decompression is handled by dedicated hardware then potentially you get a speedup.
Re: (Score:2)
I suppose the real-world benefit would depend on how you implement it... if it's a whole new RAM stick design that handles the (de)compression on the stick itself VS an interposer that any old RAM clips into and the in-between board handles the compression VS a new motherboard design that has a chip onboard to do all the compression jazz VS the CPU doing the heavy lifting.
But, I still don't see this as being for us home users.
Re: (Score:2)
It seems enough time has passed since the last catastrophic failures in this space that people do not remember.
Citation needed. Can you mention some cases?
Re: (Score:2)
Those "very special" use cases are actually fairly common. I routinely deal with data structures in software I write that are read-mostly: large data tables that are accessed in the inner loop (so need fast access) but whose data is updated infrequently (anywhere from minutes to days between changes, and often the changes are in batches so many changes can be applied in one operation). Being able to put them in compressed memory would free up a lot of normal memory for things that do actively change, which
Re: (Score:2)
This is exactly the wrong way around. On reading, you cannot avoid the latency. On writing you can by buffering. Hence this is most suitable for data structures that are primarily written but rarely read.
Re: (Score:2)
No, if you read it says that read performance is pretty much equal to uncompressed DRAM so you don't have to take a latency penalty.
Re: (Score:2)
That claim is total nonsense. Stop believing in magic.
CRAM may show promise... (Score:2)
... but most find LEMBAS far superior.
Re: (Score:2)
Re: (Score:2)
Both are a crumby substitute for the meat and potatoes of real RAM.
Hardware sold separately (Score:2)
The Steam Deck speculation gets ahead of the hardware a bit. CRAM needs hardware that can serve compressed memory directly, and a kernel update won't supply that. The 452x figure is against ZRAM in a read-only test, which makes considerably more sense than the headline suggests. Being able to read compressed data without first hauling an entire page through software decompression is a useful improvement. I'd happily take more usable memory at close to normal read speeds. Writes requiring pages to move back
Re: (Score:2)
As far as hardware like Steam Deck goes, a future SoC could include the hardware needed if it was cheaper somehow.
Is this a software or hardware implementation? (Score:2)
Re: (Score:2)
Signetics (Score:2)
We need lossy RAM compression (Score:2)