A Newly Proposed Linux Filesystem Can Fix Corrupted Data (itsfoss.com) 68
BeamFS creator Aurelien Desbrieres is not calling for a merge, but is requesting feedback on an 18-patch proposal that takes the place of FTRFS, an earlier filesystem he proposed in April.
The Linux/Open Source blog It's FOSS reports: Filesystems that can repair damage, like Btrfs and ZFS, usually do it by falling back on a spare copy of the data. BeamFS skips that and stores Reed-Solomon error-correcting codes inside every block. Every read gets checked against those codes, and small amounts of damage are corrected before the data reaches the application. If the filesystem can tell the damage is beyond repair, the read fails with an error instead of returning the wrong bytes...
In his technical report, ext4 and ext3 returned wrong data without any error, while Btrfs refused to read the file. BeamFS returned the correct file every time except once, when a flip hit a pointer to the file's data and it refused the read.
"Aurelien says he used Anthropic's Claude to write the code, the tool that checks the patches, and the cover letter..."
The Linux/Open Source blog It's FOSS reports: Filesystems that can repair damage, like Btrfs and ZFS, usually do it by falling back on a spare copy of the data. BeamFS skips that and stores Reed-Solomon error-correcting codes inside every block. Every read gets checked against those codes, and small amounts of damage are corrected before the data reaches the application. If the filesystem can tell the damage is beyond repair, the read fails with an error instead of returning the wrong bytes...
In his technical report, ext4 and ext3 returned wrong data without any error, while Btrfs refused to read the file. BeamFS returned the correct file every time except once, when a flip hit a pointer to the file's data and it refused the read.
"Aurelien says he used Anthropic's Claude to write the code, the tool that checks the patches, and the cover letter..."
The physics is measured, not supposed. (Score:4, Informative)
Oh FFS. I hope the linux mailing list doesn't start accepting these claude jargon filled technical discourses.
"atmospheric neutrons alone upset avionics at altitude (Normand [9]); and the medium is not exempt"
??????
Re:The physics is measured, not supposed. (Score:5, Interesting)
Just because you don't understand, doesn't make it not true.
Cosmic radiation indeed causes higher storage failure rates in avionics systems. However, because we can do things like redundant storage, the problem usually isn't in the file system, but the SSD firmware.
It's incredibly common to replace an LRU (Line Replaceable Unit), take the SSD out of the replaced LRU and reflash the firmware on it, and reinstall and test to find out that the device works perfectly again because it was a bit-flip in the SSD controller firmware.
Source: I am an avionics engineer.
Re: (Score:1)
Re: (Score:2)
And "the medium is not exempt" is just weird.
Re: (Score:2)
The cited paper uses the phrase "atmospheric neutrons" in the abstract. It's from 1996.
Re: (Score:2)
Re: (Score:2)
"Single-event effects in avionics," from the IEEE Transactions on Nuclear Science. https://ieeexplore.ieee.org/do... [ieee.org]
Abstract:
The occurrence of single-event upset (SEU) in aircraft electronics has evolved from a series of interesting anecdotal incidents to accepted fact. A study completed in 1992 demonstrated that SEUs are real, that the measured in-flight rates correlate with the atmospheric neutron flux, and that the rates can be calculated using laboratory SEU data. Once avionics SEU was shown to be an ac
Re: (Score:2)
Re: (Score:2)
Because Bit-Flips due to atmospheric radiation is a use case of the filesystem. How often do you get bitflips in your desktop PC? Either blocks are dying or they are not, but random flips are rare. In a airplane things are different.
Re: (Score:2)
I'm a little surprised nobody makes an aviation grade SSD that has the firmware burned into a ROM, and with some RS coding to correct errors in the NAND. I guess there is probably some cost issue, as in it's cheaper to just deal with failing SSDs than to develop such a device.
Re:The physics is measured, not supposed. (Score:5, Informative)
That's exactly the calculation. Use redundant storage in-flight, and make LRU swaps very quick on the ground. If you catch a failure in-flight, you have a tech with a replacement LRU waiting for the plane when it lands (for various values of "where" - obviously you don't have techs at every airport).
It's really not that different from anticipating disk failure in a server by having online redundancy, and a service callout to replace the part. Especially if you can quickly refurbish and recertify the LRU and put it right back into the replacement inventory.
Re: (Score:2)
Thanks, that's interesting.
Re: (Score:3)
It also depends on the safety certification level. If you're in navigation or fuel management systems (DAL-B certification) and such, the rules are very different from DAL-E certified things like the in-flight entertainment / connectivity systems.
Much more rigor is applied to DAL-A / -B / -C as you can imagine, because they are safety critical. And thus the price goes far higher, because active operation must be maintained.
Re: (Score:3)
I'll let the entire IFEC industry know that you don't approve of using commodity hardware in non-safety critical systems, when redundancy techniques that have been proven for decades completely mitigates the problem, and everything is designed for line-fit replacement in under 10 minutes while the aircraft is on the ground. Add in ground operations telemetry and predictive failure analysis and we can have the replacement LRU provisioned and ready when the aircraft arrives.
I'll similarly let the technicians
Re: (Score:2)
The cited article is E. Normand, "Single-event effects in avionics," in IEEE Transactions on Nuclear Science, vol. 43, no. 2, pp. 461-474, April 1996, doi: 10.1109/23.490893 https://ieeexplore.ieee.org/do... [ieee.org]
I don't have access to the full text, just the abstract, but the abstract does use the phrase "atmospheric neutrons".
The article is from 1996, so think it is unlikely that the phrase is "claude jargon" rather than an actual term of art.
Vibe coded huh? (Score:3, Insightful)
A vibe coded *file system*. Really.
Gonna stay away from this one.
Re: (Score:3)
It's also completely the wrong thing to use. RS-coding will deal with small amounts (bits/bytes depending on the layout) of data corruption, not general filesystem corruption. And the overhead is enormous, unless you're streaming data through hardware that can correct and reassemble on the fly it's not practical, and for filesystems even useful.
This proposal is a bit like the hypothetical "why didn't we use the space shuttle to rescue the Apollo 13 astronauts" question. Yeah, you just go ahead and do that
Re: (Score:2)
Re: (Score:3)
Re: Vibe coded huh? (Score:2)
I really don't think most RAID implementations check data as it's read so won't catch random bit flips. It protects against known failed devices, which seem to be the majority of issues we see in practice.
RAID Z does check checksums if you are on ZFS.
Re: (Score:2)
RS is well suited to this problem space. The overhead is a fixed percentage of the filespace. Error detection is computationally simple. Error correction uses more compute but it's only invoken when there is an error.
If you want reliable storage, using forward error correction is an entirely sensible thing to do. I have my own file level FEC using A concatenated Golay & RS with data reordering that I use on stuff I don't want to fall to bit rot. It's neither complicated nor computationally prohibitive.
Re: (Score:2)
If it's not clear, the data reordering spreads block errors across many blocks, so long runs of errors turn into multiple individual bit errors that are handled efficiently by the code. The Golay part of the concatenated code mops up these. The RS part deals with larger blocks.
Re: (Score:2)
Is there any advantage over RAID5 though? Doesn't ZFS do this at a block level?
Re: (Score:2)
I'm finding that ZFS is stupid, and I'm thinking about switching to normal dmraid.
Notably, if you really load it up for a while such that it has a chance to fill up its buffers, it doesn't save any I/O for anything else. The whole system just gets punched right in the fucking balls and becomes completely unusable while the cache empties. Maybe there's some way to tune this tendency out of it, but...SANE DEFAULTS!!! If your project doesn't have them, you're a fuckup! Allowing one process to make a system unu
Re: (Score:2)
Is there any advantage over RAID5 though? Doesn't ZFS do this at a block level?
RAID5 is at the disk level. What about files that lay around on USB sticks or cloud drives? Those need some FEC love too.
Re: (Score:2)
This proposal is a bit like the hypothetical "why didn't we use the space shuttle to rescue the Apollo 13 astronauts" question. Yeah, you just go ahead and do that then.
Well, dude did go ahead and do that. Whether it's a useful approach is certainly debatable, but since there's a working prototype, anyone can test it and find out.
They say the secret to how it works is... (Score:2, Interesting)
I hear it works using GenAI to rebuild an approximation of the data, so close a human can't distinguish it.
Unfortunately, this also means Linux will not boot without a gemini api key added to Grub.
Needs stress tests. (Score:1)
Sound like a sometimes reasonable approach (Score:2)
That would have been really useful for some mag tape systems I used to use. They had too many errors for their error correction system.
OTOH, this kind of thing is very likely to have bugs. Test it THOROUGLY before folks start depending on it. Perhaps, after it passes initial tests, make it an optional choice in the installer, and require extra steps to do the install (with warnings: CAUTION: BETA FILE SYSTEM HAS BEEN CHOSEN).
File systems need more thorough testing that most software choices.
Re: (Score:2)
It's a very reasonable approach, so reasonable that the drives already store ECC data. Why do you think it reports "corrected" and "uncorrected" errors separately in SMART data?
Is there really a need to store multiple copies of ECC codes? At some point, why not just rely on backups. Checksumming is good, wasting diskspace because of a neutrino from space is not.
Re: (Score:2)
You should do it for backup systems. It's why PAR2 exists. You run it over your data and it creates the Reed-Solomon parity files. Each file contains a checksum of every file in the set, and you need only enough parity file to fix the corruption. E.g., if you have a 1MB block that's corrupted, you just need a 1MB parity file. The files often come in varying sizes so you just
Re: (Score:2)
There's lots of programs able to handle the PAR2 format and it's been around for decades.
Used to use this way back to fix incomplete binary downloads from newsgroups.
Re: (Score:2)
The tapes I was talking about were 800 bpi reels even parity. (The odd parity ones had better error recovery.)
TFS is a bit incorrect (Score:1)
In fact, ZFS checksums every block, and then checksums those checksums so it knows which copy of the data is correct one (in the case of a 1:1 mirror, for instance).
Re: (Score:3)
To be clear for those that don't know, with the defaults and a simple 2 way Mirror, ZFS will have 4 copies of standard Metadata. 2 per Mirror, and 2 way Mirror is 4 copies, (of standard Metadata).
So that case o
Re: (Score:2)
" it stores 2 copies of standard Metadata"
That's the problem. 2x data is larger than data + error correction for no benefit.
Re: (Score:2)
nah, a RS code can correct a tiny bit of loss but two metadata copies can correct a bad sector or even track if you're lucky.
Different requirement spaces.
The idea this guy has might make embedded devices slightly more long-lasting.
They usually have quantity one of a cheap storage part and a bad block in the wrong place could brick the whole thing.
It would only be useful in the case where maybe the build is 12MB and the chip is 16.
Most vendors won't ever pay extra for this.
Re: (Score:2)
why do you need this? If you have RAID1 and one file gets corrupted you have a file with correct checksum and a file with incorrect one. ZFS checksums are not just parity like RAID5, but true file hashes.
Stored in the block? (Score:3)
Seems like you would want to store the error correction data somewhere besides right at the data, if it is your recovery plan - like an off site backup. Use a fast checksum for the initial validation and only read the error correction when necessary.
Re: (Score:2)
No, that's when you want a RAID system with multiple disks.
This is only for small corruption, like a failing drive might cause, or due to cosmic rays/bitflips.
This is just EC for a non-redundant drive.
Re: (Score:2)
No, you're confusing backups, redundancy and error correction. The latter, ECC, is *always* stored locally and exists to address single unit errors. It's usually stored in a way that can be read as part of a sequential read operation as every read needs to be checked so having ECC information non-locally would lead to a huge performance cost.
The scenario here is bit flip or bad block or some in flight data corruption. Not a failure of a storage medium.
Why ? (Score:5, Interesting)
I don't see an explanation of why you would need a whole new filesystem.
Re: (Score:2)
Re: (Score:2)
Why do you need BTRFS subvolumes (or even RAID) when there are LVM / device-mapper solutions to this? Because some things are better handled on file basis than on block basis. With BTRFS you can put a 4 TB disk, 2 1TB disks and a 2 TB disk into a RAID and be sure every file is stored on two disks, even though a block-layer mapping between this disks would be a nightmare to maintain. Why does it work? Because BTRFS knows what a file is and the device mapper only knows block positions.
Re: (Score:2)
One of the more subtle dangers of vibe coding is that the genAI system will generally drown you in praise and at no point tell you that it's already been done or that there are already ways to achieve what you're doing.
That's why this project got to the coding stage.
stupid idea at the basic level (Score:3)
Any descent storage hardware will already do this, in hardware. How do they think HDDs and SSDs keep track of the number of corrected and uncorrected errors?
Re: (Score:3)
No not quite. The HDD / SSD writing checksums onto the platter only cover the write operation itself. It will only confirm what bits were written to the platter along with its immediate ECC codes are correct, and it does not confirm any failure outside the immediate write operation. An example of corruption that it would miss is a power failure mid-write. The in flight bits would be written along with the ECC code before the power goes out (thanks to capacitors, especially true for SSDs which need to then n
Re: (Score:2)
Are you trying to argue that HDDs and SSDs don't use full ECC? You do realise even the humble, no criticality, audio CDs used full blown ECC, right?
Re: (Score:2)
No not at all, but I was arguing for a point of confusion. I was talking about a file system level checking can be larger in scope but the summary does say this is block level so it is actually completely redundant.
My main point was on scope. Something like ZFS is still necessary because the HDD doesn't know what a file is, it only knows what a short sequence of bits are, and while those bits may be correct a file may be corrupt if they are part of a missing sequence. So filesystem level error correction is
Re: (Score:2)
ECC is heavily used everywhere in telcos. We have quality long distance data links because of hardware ECC. It's just a basic part of modern digital fabric.
Re: (Score:2)
ECC is heavily used everywhere in telcos. We have quality long distance data links because of hardware ECC. It's just a basic part of modern digital fabric.
Indeed, and I'm not refuting the importance of ECC. I'm refuting the importance of ECC doing it twice (once in hardware, and again in the filesystem). It's a question of scope.
If a sector on a disc has ECC let that deal with it. The error may be frequent and correction of an individual bit is important.
A file system on the other hand doesn't suffer from individual bit flips, that's already handled by the disc's ECC. ECC isn't important here, checksumming of the entire file to ensure all the data is intact i
More info needed (Score:5, Funny)
I refuse to adopt any Linux filesystem without proof of life evidence from the founder's wife.
Re: More info needed (Score:2)
Cold.
Re: (Score:2)
I do believe you reised to the occasion here.
No free lunch (Score:2)
Dumb, but... (Score:2)
That's such a dumb, smoothbrained description I doubted the capabilities of the person who wrote it - at first.
Then I realized he was right. There isn't a filesystem that does RS or, really, any sort of single-device EC. This is a genuinely good idea, at least for spinning disks, where sector corruption is more likely.
This may be personal anecdote only, but I've only seen 1 SSD get any sort of data corruption as a failure mode, and it was a very early iteration - maybe a 8 or 16GB one. SSDs seem to simply f
Re: (Score:3)
There isn't a filesystem that does RS or, really, any sort of single-device EC.
There's a reason for this:
a) the ATA spec has error correction to maintain data in flight.
b) single-devices (e.g. your HDD) has error correction as part of its normal operation.
You can see how well this works by looking up the SMART attributes for any HDD and reading:
195 - Hardware ECC Recovered
184 - End-to-End error / IOEDC (not all drives report this one)
and
204 - Soft ECC Correction
and when shit hits the fan you get:
187 - Reported Uncorrectable Errors
RAID-6 (Score:2)
"Reed-Solomon error-correcting codes"
So you literally re-invented RAID-6? UUuuuhhhh, congrats, I guess?
Re: (Score:2)
You don't need RAID anything. Reed-Solomon error-correcting codes are used on every drive compliant with the ATA specification. There's not a 1-1 relationship between bits on the platter, and storage capacity of a drive.
So yeah RAID-6 re-invented RAID-5, which reinvented the HDD that already existed, which reinvented the CD, which reinvented the mechanical tape... do you see how stupid the "literally re-invented" part of what you said actually is?
Not sure why OP thinks storage is corrupt (Score:5, Interesting)
This worked reasonably well. A few times a year, I got a corrupted file in the media. Pretty straight forward to restore from backups. Not really a problem. Mostly video files because they are large, and would statistically be higher percentage of used space.
However, whence I implemented ZFS scrubs every 2 weeks, the corruption stopped, COLD. I have not had to restore any file, (due to on disk corruption), in over 6 years!!! And yes, I really mean that non-redundant media pool that has 2.1TBs used. My conclusion is that reading the used storage blocks caused the mSATA SSD & laptop HDD to find any marginal blocks. Then apply their internal ECC, before bad data could be supplied to ZFS.
Eventually I wrote about this here:
Arwen Star's blog - Self correcting storage [wordpress.com]
Now do I believe this is a workable solution for production data?
Of course not. Any serious data need should have some form of Mirroring or RAID.
Nor do I believe this is something others should do...
Re: (Score:2)
Then apply their internal ECC, before bad data could be supplied to ZFS.
Can you expand on how you think this is supposed to work? ECC is ECC. When it is read is irrelevant especially in the case of HDDs. An unrecoverable error is not blindly passed anywhere, it will be logged by the kernel which would throw an error, and logged in SMART attributes (and is virtually always a precursor to a drive failure). Recoverable errors are just logged in SMART, but do not get passed to the kernel. Critically reading from a HDD does not in any way give you info about its marginal status, nor
Re: (Score:2)
However, whence I implemented ZFS scrubs every 2 weeks, the corruption stopped, COLD. I have not had to restore any file, (due to on disk corruption), in over 6 years!!! And yes, I really mean that non-redundant media pool that has 2.1TBs used. My conclusion is that reading the used storage blocks caused the mSATA SSD & laptop HDD to find any marginal blocks. Then apply their internal ECC, before bad data could be supplied to ZFS.
This is strange. SSDs are supposed to have scrubbers that periodically scan for this and rewrite leaky blocks internally.
Cute... and not caring. (Score:3)
As a PoC, I'm all over this. But BeamFS supports neither CoW nor journaling, both of which I consider essential for a modern FS. So... yeah. "Heck, no." Maybe someday it will acquire those features, or its "one neat trick" will be merged into another FS.
Forward Disk Error Correction is Interesting (Score:2)
Let the hardware deal with it (Score:2)
I think it best to let the hardware deal with error correction. It is way better positioned in the stack to judge and handle these sorts of failures at least cost. At best all you need are simple checksums to alert the operator to hardware failure.
Not remotely for normal end users. (Score:2)
AFAICT, this is explicitly for fixing flips in transmission between the disk and the page cache. On the storage medium, data is almost certainly already encoded using Reed-Solomon codes or Hamming codes or something of that nature, so you won't receive a block with embedded errors, you will receive a failure to read the block. And you can get ECC for you rmemory. But there may not be end-to-end ECC for every step of transmission. So instead of imagining an entirely new hardware ecosystem (which will ne