Secure File Storage Over Non-Trusted FTP? 384
hmckee writes "Does any software exist that enables me to store/backup/sync files from my local computer to a non-trusted FTP site? To accomplish this, I'm using a script to check timestamps, encrypt and sign the files individually, then copy each file to an offsite FTP directory. I've looked over many different tools (Duplicity, Amanda, Bacula, WinSCP, FileZilla) but none of them seem to do exactly what I want: (1) multi-platform (Windows and Linux), stand-alone client (can be run from a portable drive). (2) Secure backup (encrypted and signed) to non-trusted FTP site. (3) Sync individual files without saving to a giant tar file. (4) Securely store timestamps and file names on the FTP server. Any help or info on alternative solutions appreciated."
Really is a pity (Score:5, Informative)
Re:Errr (Score:3, Informative)
No, it's more like someone running around naked outside holding a frosted pane of glass in front of them wondering if maybe they should also build a tool to hold a second pane of frosted glass behind them.
rsnapshot (Score:1, Informative)
rsnapshot?
http://www.rsnapshot.org/howto/ [rsnapshot.org]
Re:Really is a pity (Score:5, Informative)
I'd translate "wasn't possible" to "couldn't be bothered". Once SSH installed (and it is there by default in most *nix distros), you have but one 'user' file to configure (to 'jail' you within a certain hierarchy). Ta-da! Change your host and use SFTP.
TrueCrypt (Score:5, Informative)
Does it have be to ftp? (Score:2, Informative)
Re:Really is a pity (Score:5, Informative)
It is ENTIRELY possible to provide that on any host, regardless of the number of users. All you are asking (correct me if I am wrong) is that the connection between you and the FTP server is secured through SSH or TLS.
That is trivial. Sounds like they cannot be bothered to enact rudimentary security. As a policy in my own systems, and any systems that I pay to use, I demand that any connections that go over untrusted networks be encrypted. There are so many products that help you do this it just makes their refusal all the more ridiculous. I have a product that does not support encrypted connections and I just stunnel to protect it.
Anything less is just reckless. Tell them to protect your connection or you will get another provider. Simple as that.
Manent fits the bill perfectly. (Score:5, Informative)
Manent is an algorithmically strong backup and archival program. It features efficient backup to anything that looks like storage. Currently it supports plain filesystems ("directories"), FTP, and SFTP. Planned are Amazon S3, optical disks, and email (SMTP and IMAP). It can work (making progress towards finishing a backup) over a slow and unreliable network. It can offer online access to the contents of the backup. Backed up storage is completely encrypted. Backup is incremental, including changed parts of large files. Moved, renamed, and duplicate files will not require additional storage. Several computers can use the same storage for backup, automatically sharing data. Both very large and very small files are supported efficiently. Manent does not rely on timestamps of the remote system to detect changes.
Check it out: http://freshmeat.net/projects/manent [freshmeat.net]. It's under active development (the UI and the setup are currently in fetal stage) but the basic functionality is there and is well tested.
Disclaimer: I am the author.
Re:Working On Something Similar (Score:3, Informative)
GPL3 might be why. He would have to open what he does on his servers to make it work with the GPLv3. That might not be an option for this type of webservice.
If it was GPLv2, it might be a little better of an option except that Rsync wouldn't be able to do incremental backups unless it could decompress/decrypt all the files and then re-encrypt them without damage and when accessed. Storing information in a file index with mapping to the encrypted files would open the encryption to hacking and wouldn't be a good idea unless you could verify that something didn't change the stored file without updating it. So the server or host side has to be able to open the stored file, send something that says what we have already, then close it. And when your dealing with 100 changes in an employee handbook or something, that can start to take a lot of time and CPUs.
Re:I knew a guy who always had headaches (Score:5, Informative)
What about Portable Python [portablepython.com]?
If I understand your problem, you want the remote image encrypted, right? In which case SFTP/FTPS is redundant overhead (and whatever data is sent is stored in its plaintext). This is something that might be possible with FUSE (e.g., use the Python-FUSE bindings to construct an FTP client that passes stuff through GnuPG first).
Heh, you'd be surprised how many people around here lack a sense of humour.
Re:I knew a guy who always had headaches (Score:4, Informative)
I should have stated that the data wasn't THAT important since it's already backed up in two other places.
I was initially using Amazon S3 to do the backups, but since I had 20 GB of spare storage on my hosting site, I figured someone else must have tried doing the exact same thing because it's the cheapest solution. It didn't take me long to write a small script to encrypt files and send them to the FTP server, another reason I figured someone else may have done this. So, rather than extending the script, I thought I'd "Ask Slashdot" to see if anyone else had completed the exercise.
If it were REALLY important for me to have this storage, I'd go back to using S3 or spring an extra $10 a month to get my account upgraded to use SSH/SFTP.
As it stands now, I may just get a kick out of implementing the project for fun.
Re:Manent fits the bill perfectly. (Score:3, Informative)
Re:I knew a guy who always had headaches (Score:4, Informative)
You are correct, the access from Windows is really of secondary concern. Still, it would be nice to access the data from work or the wife's computer.
I should have also added that I asked the question to see how much flaming and ridicule I could draw by asking about such a cheap-ass, overly complex solution that is simply solved by SSH/SFTP. :)
Well, I've also looked into it and found nothing (Score:5, Informative)
Re:I knew a guy who always had headaches (Score:4, Informative)
The real problem is not knowing about rsync since it's designed for exactly his problem.
Re:A slight oxymoron here. (Score:3, Informative)
After thinking about this, I realize that I never should have written "non-trusted" in the description. I was thinking about FTP in the sense that it's inherently non-secure, which would make "non-trusted" redundant. I actually trust the FTP server as much one can be trusted.
This made me think about side question to which others in this thread have alluded: When is FTP as secure as SFTP? When you're on a non-trusted computer. It's a minor issue, but if you're trying to access an encrypted file from a non-trusted computer, a compromised SFTP client will do you no good if you use the same password on the SFTP server and your encrypted data.
It's been educational reading all these posts and thinking about various security issues with any of the solutions. I'm glad I submitted this question.
A solved challenge, mind you (Score:4, Informative)
Rsyncrypto
http://rsyncrypto.lingnu.com/ [lingnu.com]
Encrypts while making the above sentence untrue - a small change in the file results in a (relatively) small change in the cypher text.
It's stable, been around a while, and is written by me :-)
Shachar
P.s.
It is, in fact, written by me for the purpose of a commercial backup service. The software itself, however, is free.
Re:Working On Something Similar (Score:3, Informative)
I've looked at it a while back (I'm the one who wrote rsyncrypto). When compared with rsyncrypto, the main thing I didn't like (aside from the fact there appears to be no implementation... ) is the amount of state stored. Esync actually needs access to the old plain text file in order to work (or a substantially similar state). Rsyncrypto, on the other hand, needs just a few pieces of state per file, that once created never change. These include the symmetric session key and such stuff, and are about 68 bytes in length.
I really don't remember the details any more, but I think that there were also performance implications to the above.
If want, there is also a program called "murk" http://murk.sourceforge.net/ [sourceforge.net], which implements the same principle as rsyncrypto. As far as I can tell, it is much less mature, and no longer maintained.
Shachar
Re:I knew a guy who always had headaches (Score:3, Informative)
Re:I knew a guy who always had headaches (Score:4, Informative)
Re:I knew a guy who always had headaches (Score:3, Informative)
From http://rsyncrypto.lingnu.com/index.php/Algorithm [lingnu.com]: "The entire rsyncrypto can be summarized with one sentence. Rsyncrypto uses the standard CBC encryption mode, except every time the decision function triggers, it uses the original IV for the file instead of the previous cypher block."
So you're basically dividing each file into chunks and encrypting them separately using the standard algorithm. Seems pretty safe to me. The only obvious leakage is that an attacker can tell if two files are substantially identical. In this, it is similar to ZIP files whose members are encrypted.
The decision function is that used to periodically reset gzip: the sum of the previous 8196 bytes is evenly divisible by 4096. Personally, I'd have used the same Adler-32 algorithm that rsync uses internally.
Re:I knew a guy who always had headaches (Score:1, Informative)
A rolling block cipher that does not use previous data as a seed for future blocks will be 'secure' while reducing the extent of the changes in the crypt to the blocks that were touched. The information that is made available to attackers is the relative location of changes in your source, with the granularity of the block size of the crypto. This may be useful in some circumstances, perhaps.
Re:I knew a guy who always had headaches (Score:5, Informative)
Rsyncrypto is insecure. By resetting to the IV, it opens an information leak similar to the one with ECB mode [wikipedia.org] (see the picture of the penguin on that page).
To see why CBC with occasional reset-to-IV is insecure (regardless of trigger function), consider a long repeating pattern of the same bytes (e.g. the white spaces in the penguin picture). CBC won't encrypt them to the same value (like ECB does), but every time the IV resets the same sequence of encrypted bytes will appear. This pattern is detectable and further the places where this pattern is disrupted is detectable. So going back to the penguin picture, the non-background portions will have a shadow that disrupts the repeating background pattern and revealing the content of the file.
Re:I knew a guy who always had headaches (Score:5, Informative)
Note - I'm the one who designed and wrote rsyncrypto.
Well, no. Two files will likely be encrypted using different session (read - AES) keys, and therefor will not be even remotely similar. One file having two significantly similar areas will show up, however. This case was deemed somewhat remote in my analysis. You are free to perform your own, of course. If you do, please feel free to email me.
The only place where actual data leakage may happen are due to the fact that a persistent attacker can compare cipher texts, and know where the decision function triggered. This is a good point to ask "how much information is gleaned about the file".
I think current rsyncrypto is ok on that front, and future plans include improvements. These improvements, alas, will cost in performance.
Shachar
Re:Really is a pity (Score:1, Informative)
I fully agree with this provider. Providing shell access to a shared machine is madness and you cannot provide security for your users this way.
The mostly recent version of OpenSSH can be configured to auto-chroot, without you having to set up a new environment for each user:
http://undeadly.org/cgi?action=article&sid=20080220110039
http://it.slashdot.org/article.pl?sid=08/02/20/1637259
Jails or Solaris zones is another option as mentioned. Both Solaris and FreeBSD can run Linux binaries under emulation, otherwise POSIX is POSIX. The underlying OS shouldn't matter too much if you install the GNU environment (which isn't that difficult with Ports (and it comes with Solaris 10 by default)).
How many people care what the OS is if all the proper PHP/Perl/Ruby/SQL modules are installed?
Re:A slight oxymoron here. (Score:3, Informative)
Lol.. If your going to pay for insecure redundancy, why not consolidate the expense and use a secure solution from the start?
That's assuming a "secure solution" exists. I think I adequately proved that such a "secure solution" is at least as impractical -- it pretty much involves a machine that you've configured running in the apartment of someone you trust not to let it be physically tampered with.
Your missing the point, FTP isn't secure. You transmit the user name and passwords in plain text.
Yeah, I got that. So what?
And even though crypto is pretty good, it isn't fool proof nor it is impossible to break.
Fine then -- so suppose I use your "secure solution" from the start, and I'm using rsync over ssh to a machine in my friend's apartment.
What's to stop them from cracking the SSH?
After checking your credentials, making sure you as well as your server is in the US and under the US's jurisdiction, yes, I probably would. I'm not against a remote backup,
Neither am I.
Suppose for a moment that you wouldn't. Does encrypting the files start to make sense?
If you're already encrypting the files, SFTP doesn't buy you a lot over FTP.
Well, there are some things that you have to trust that ethic laws and plain old laws will stop someone from doing something.
Unless my files are encrypted. Then I don't actually have to trust "ethic laws" or "plain old laws" -- short of being myself kidnapped and waterboarded, I'm safe.
AES 256 is something like 10^(76) possible answers which is huge. But by breaking down the possibilities, you have a better chance of getting lucky.
Right. But let's do the math. Assume a million computers:
10^76 / 10^6 = 10^70.
Is it really that much more likely that you will get lucky with 10^70 possibilities than with 10^76?
when you divide the pool down to the number of nodes, you could end up with it being within the first day of calculation.
Right. Still isn't going to happen.
See, Quantum Physics also predicts that it's entirely possible that every molecule in your body could simultaneously jump a foot to the right. It's just so ridiculously improbable that, in fact, no one has ever seen it happen -- and no one ever will.
Let's assume, for the moment, that each compromised machine has a dual-core 2 ghz chip -- likely far less on most, but let's just assume. Assume 1.5 million of them. Assume that it takes only one cycle to attempt one key -- it doesn't, but pretend it does. And assume that threading is perfect -- you can keep both cores warm -- you can't, but pretend you can.
Note that all of these allowances are in your favor.
So, that's 2 ghz = 2 billion operations, times 2 cores per machine, times 1.5 million machines. By my count, that's 6 quadrillion tests per second. That's a lot, right?
Let me give you some more numbers: 60 seconds per minute, 60 minutes per hour, 24 hours per day, 365 days per year. So, a little over 31.5 million seconds in a year.
It will take 52849653306274315810851830343929408752938685099409408 years to search that keyspace.
That also means that the probability of a hit in the first 100 years, assuming that the key could be anywhere in the keyspace, is approximately 0.00000000000000000000000000000000000000000000000000189216 -- percent.
Adjusting for Moore's Law, and assuming that we'll have 2^(2*50) = 2^100 times more power by then -- and then, to make the math simpler, assuming we have that kind of power from now until then -- it's still about 0.0000000000000000000023986 -- again, that's a percentage.
I keep trying to make your odds better, and I still can't get anything remotely realistic to suggest that even if a botnet were devoted full-time to cracking my personal backup, that there's even a reasona