Slashdot Log In
Secure File Storage Over Non-Trusted FTP?
Posted by
kdawson
on Mon Aug 18, 2008 01:26 AM
from the beeping-sounds-while-backing-up dept.
from the beeping-sounds-while-backing-up dept.
hmckee writes "Does any software exist that enables me to store/backup/sync files from my local computer to a non-trusted FTP site? To accomplish this, I'm using a script to check timestamps, encrypt and sign the files individually, then copy each file to an offsite FTP directory. I've looked over many different tools (Duplicity, Amanda, Bacula, WinSCP, FileZilla) but none of them seem to do exactly what I want: (1) multi-platform (Windows and Linux), stand-alone client (can be run from a portable drive). (2) Secure backup (encrypted and signed) to non-trusted FTP site. (3) Sync individual files without saving to a giant tar file. (4) Securely store timestamps and file names on the FTP server. Any help or info on alternative solutions appreciated."
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
I knew a guy who always had headaches (Score:4, Insightful)
This guy was always complaining about headaches. He would constantly be pounding his head into his fist and whimper to me that he felt like his head would split open. He took pain killers all the time, and for a long duration was addicted to a certain prescription pain medication. But none of that helped because as soon as the medication started to wear off, the pain would come right back again.
Finally, I had had enough of his complaining. I told him to stop pounding his head with his fist. Whaddayano! His headaches went away in a day.
Moral of the story: Don't try to find workarounds for your problem. Fix the problem.
Re:I knew a guy who always had headaches (Score:5, Funny)
The problem is asking questions to Slashdot?
Parent
Re:I knew a guy who always had headaches (Score:4, Informative)
The real problem is not knowing about rsync since it's designed for exactly his problem.
Parent
Re:I knew a guy who always had headaches (Score:5, Insightful)
The real problem is not knowing about rsync since it's designed for exactly his problem.
No, rsync isn't a very good solution for a couple of reasons. First, unless there's some capabilities that I'm not aware of, rsync has no encryption capabilities. Given an unencrypted file tree and an encrypted version of the file tree, rsync has no way to compare the two for changes. The only solution to that which I see is to maintain a local encrypted mirror of your file tree. So then you need twice as much space, since you're maintaining two local file trees, and you need a tool to update automatically sync the local file tree and the local encrypted version of the file tree. If you have that tool, then it may work or be hacked to work with a remote file tree, completely removing the need for rsync. Even supposing that you found a tool to do that which won't work with a remote file tree, you're nullifying the primary advantage of rsync.
rsync is designed to do incremental updates. If you have a text file and change one word, rsync doesn't transfer the whole file. It only sends enough info to correctly update the remote file so that it matches the new local file. (Or vice versa, of course.) But when you change a single word and reencrypt a text file, the whole file changes. So rsync will have to transfer the whole file. So will any other solution, of course, but it does mean that rsync loses much of the capability which makes it so valuable.
You could do something like unencrypt the local file tree mirror, rsync with the working file tree, reencrypt the file tree and then rsync the local encrypted tree with the remote encrypted tree mirror, but that's a lot of work and processing power and hardly matches the clean, integrated solution that the article is asking for. It's probably more cumbersome than whatever it is he's doing now.
Parent
Re:I knew a guy who always had headaches (Score:4, Informative)
Parent
Re:I knew a guy who always had headaches (Score:5, Insightful)
Interesting. Things like this are why I always hedge my bets and say things like "...unless there's some capabilities that I'm not aware of, rsync has no encryption capabilities..."
That being said, I'd be extremely leery of this program. The website says: "Rsyncrypto does, however, do one thing differently. It changes the encryption schema from plain CBC to a slightly modified version. This modification ensures that two almost identical files, such as the same file before an after a change, when encrypted using rsyncrypto and the same key, will produce almost identical encrypted files." I'm far from an expert at crypto but I know enough to be extremly suspicious of that claim. A "slight change" in an encryption algorithm can be enough to transform an algorithm from highly secure to trivially crackable. And I strongly suspect that making similar files produce similar encrypted files means that there's a great deal of info about the unencrypted file suddenly available from examining the encrypted file. I wouldn't trust this without extensive review from some heavy weights in the crypto field.
Parent
Re:I knew a guy who always had headaches (Score:5, Informative)
Rsyncrypto is insecure. By resetting to the IV, it opens an information leak similar to the one with ECB mode [wikipedia.org] (see the picture of the penguin on that page).
To see why CBC with occasional reset-to-IV is insecure (regardless of trigger function), consider a long repeating pattern of the same bytes (e.g. the white spaces in the penguin picture). CBC won't encrypt them to the same value (like ECB does), but every time the IV resets the same sequence of encrypted bytes will appear. This pattern is detectable and further the places where this pattern is disrupted is detectable. So going back to the penguin picture, the non-background portions will have a shadow that disrupts the repeating background pattern and revealing the content of the file.
Parent
Re:I knew a guy who always had headaches (Score:5, Informative)
Note - I'm the one who designed and wrote rsyncrypto.
Well, no. Two files will likely be encrypted using different session (read - AES) keys, and therefor will not be even remotely similar. One file having two significantly similar areas will show up, however. This case was deemed somewhat remote in my analysis. You are free to perform your own, of course. If you do, please feel free to email me.
The only place where actual data leakage may happen are due to the fact that a persistent attacker can compare cipher texts, and know where the decision function triggered. This is a good point to ask "how much information is gleaned about the file".
I think current rsyncrypto is ok on that front, and future plans include improvements. These improvements, alas, will cost in performance.
Shachar
Parent
Re: (Score:3, Informative)
Re:I knew a guy who always had headaches (Score:4, Insightful)
The real answer to your problem is use a secure protocol like SSH which does everything you just asked for natively.
Does it encrypt and sign the files one-by-one so that the admin of the remote site (who you don't trust) can't read, alter or share them on you?
Parent
Re:I knew a guy who always had headaches (Score:4, Insightful)
What's wrong with using a public key for the backups? That's what I do.
If you're using purely symmetric encryption for your backups, you have to store the keys somewhere, and that somewhere has to be online when the backup is generated. Then you have to physically move it somewhere that it's not reachable. It's a manual process.
With a public key system, you can store your private key offline all of the time, and not have to deal with symmetric key management. GPG does that for you.
Where is the downside?
Parent
Re:I knew a guy who always had headaches (Score:4, Insightful)
The problem is FTP. It is an old deprecated protocol that is inherently insecure and even FTP w/ SSL is simply a work around to a broken problem.
Wow. It might be better to understand the problem before you make suggestions. FTP isn't the problem. FTP is just a way to move files from here to there. It's unsecured and untrusted but, in this case, SO IS THE REPOSITORY. Exactly what benefit do you get from using SSH to securely transfer files to an unsecure location? That's like using an armored truck to move your valuables to the QuickStorage down the road. What's wanted is an automated way to encrypt the files locally, then transfer the encrypted files to an untrusted site. If the files are encrypted, then it doesn't matter that FTP is unsecured.
Parent
Re:I knew a guy who always had headaches (Score:5, Insightful)
Yeah, I don't get this guy. First, he says he wants it for his home computer. Then, he says it has to be multi-platform (Windows and Linux) plus stand-alone that can be run from a portable drive.
And I say why? Let's assume for a moment that this guy has two computers at home, one that runs Linux and one that runs Windows. He doesn't need an app that does everything perfectly on both platforms. He just needs an app that does it perfectly on one, and either one is fine really. If he prefers to use his Linux box to coordinate the secure backup to an untrusted FTP site, then he just needs to have his Windows machine send the data unencrypted over to his Linux box -- then his Linux box can just do the bulk of the job. Or if he prefers to do it the other way around and use his Windows machine to do the secure backup to the untrusted site, he can just use that and have his Linux box send the data unencrypted to his windows machine.
And of course, why does it even need to go onto FTP instead of SFTP? Instead of wasting valuable man-hours reinventing SFTP from scratch, or finding someone else that has, he could just pay a few dollars to a provider who will give him SFTP. And if his current Provider won't do that, get an other additional provider that will do it. If backing up is really as important as he seems to make it, then spending a few extra dollars each month shouldn't be a problem.
Parent
Re:I knew a guy who always had headaches (Score:4, Informative)
I should have stated that the data wasn't THAT important since it's already backed up in two other places.
I was initially using Amazon S3 to do the backups, but since I had 20 GB of spare storage on my hosting site, I figured someone else must have tried doing the exact same thing because it's the cheapest solution. It didn't take me long to write a small script to encrypt files and send them to the FTP server, another reason I figured someone else may have done this. So, rather than extending the script, I thought I'd "Ask Slashdot" to see if anyone else had completed the exercise.
If it were REALLY important for me to have this storage, I'd go back to using S3 or spring an extra $10 a month to get my account upgraded to use SSH/SFTP.
As it stands now, I may just get a kick out of implementing the project for fun.
Parent
Re:I knew a guy who always had headaches (Score:4, Informative)
You are correct, the access from Windows is really of secondary concern. Still, it would be nice to access the data from work or the wife's computer.
I should have also added that I asked the question to see how much flaming and ridicule I could draw by asking about such a cheap-ass, overly complex solution that is simply solved by SSH/SFTP. :)
Parent
Re:I knew a guy who always had headaches (Score:5, Funny)
You appear to have misspelled "perl".
Parent
Re:I knew a guy who always had headaches (Score:5, Funny)
Parent
Re:I knew a guy who always had headaches (Score:5, Funny)
#10,407's got you by 1,132,922 membership points, #1,143,329.
#10,407 and #44,513 both want you off HIS lawn right now.
Sincerely,
Membership Police
Parent
Re:I knew a guy who always had headaches (Score:5, Funny)
go home kids. You can come back and play in the morning.
Parent
Re:I knew a guy who always had headaches (Score:5, Funny)
In my day we tied an onion to our belts..
Parent
Re:I knew a guy who always had headaches (Score:4, Funny)
get offa my lawn!
Parent
Re:I knew a guy who always had headaches (Score:5, Informative)
What about Portable Python [portablepython.com]?
If I understand your problem, you want the remote image encrypted, right? In which case SFTP/FTPS is redundant overhead (and whatever data is sent is stored in its plaintext). This is something that might be possible with FUSE (e.g., use the Python-FUSE bindings to construct an FTP client that passes stuff through GnuPG first).
Heh, you'd be surprised how many people around here lack a sense of humour.
Parent
Re:I knew a guy who always had headaches (Score:4, Interesting)
Like pretty much every SMB I have ever worked on. It is really a shame there isn't an easy native way to run VB code in Linux,because every single SMB I walk into has some damned VB3-6 app that holds that place together. Worse is it always seems to be written by a guy named Chuck that worked there ages ago and didn't know WTF he was doing. I swear I opened up the last VB4 one to convert it to VB6 and try to fix the crashing and was like "OMFG! Tap dancing Christ what was this guy thinking! Was he high?".
The ENTIRE first page was nothing but a giant mess of GOTO calls. I'll admit I'll throw in about one GOTO per app just to do a quick function call,but DAMN. The entire first page was GOTO here,then GOTO there,then GOTO somewhere else. Luckily like a good 75% of VB apps out there it was a GUI connecting to a database so I was able to whip off a replacement. But it would be nice if just once they brought in an actual coder to write a program instead of clueless chuck. But as always this is my 02c,YMMV
Parent
Re:I knew a guy who always had headaches (Score:4, Funny)
Worse is it always seems to be written by a guy named Chuck that worked there ages ago and didn't know WTF he was doing.
The best part is everyone loved Chuck and thought he was some tech god. The second you bitch about the beautiful VB Chuck wrote you are now seen as incompetent.
I feel your pain.
Parent
Really is a pity (Score:5, Informative)
Re:Really is a pity (Score:5, Informative)
I'd translate "wasn't possible" to "couldn't be bothered". Once SSH installed (and it is there by default in most *nix distros), you have but one 'user' file to configure (to 'jail' you within a certain hierarchy). Ta-da! Change your host and use SFTP.
Parent
Re:Really is a pity (Score:5, Informative)
It is ENTIRELY possible to provide that on any host, regardless of the number of users. All you are asking (correct me if I am wrong) is that the connection between you and the FTP server is secured through SSH or TLS.
That is trivial. Sounds like they cannot be bothered to enact rudimentary security. As a policy in my own systems, and any systems that I pay to use, I demand that any connections that go over untrusted networks be encrypted. There are so many products that help you do this it just makes their refusal all the more ridiculous. I have a product that does not support encrypted connections and I just stunnel to protect it.
Anything less is just reckless. Tell them to protect your connection or you will get another provider. Simple as that.
Parent
Working On Something Similar (Score:4, Interesting)
I'm working on a backup solution that allows people to back up their data to a remote server securely and efficiently. For "efficiently", think rsync: only the differences are sent (and some information necessary to identify what the differences are). For "securely", think assymetric cryptography: your backup is stored in encrypted form, so that only someone who possesses your private key can use it.
All this is currently in very early stages of design. I'd welcome any suggestions for protocols or software I could use. Currently, I am thinking to implement a transactional network block device protocol, and implement the backup protocol on top of that. I still need to decide on a programming language I can use for parts I need to write myself, too (something safe (no buffer overflows, please), yet with byte level access...and no Java or .NET, please).
By the way, this is going to be a commercial product, but the code and the protocols will be open. I'll charge for the storage and bandwidth. :-D
Re: (Score:3, Interesting)
Have you checked out rsyncrypto [lingnu.com]?
Re: (Score:3, Informative)
I've looked at it a while back (I'm the one who wrote rsyncrypto). When compared with rsyncrypto, the main thing I didn't like (aside from the fact there appears to be no implementation... ) is the amount of state stored. Esync actually needs access to the old plain text file in order to work (or a substantially similar state). Rsyncrypto, on the other hand, needs just a few pieces of state per file, that once created never change. These include the symmetric session key and such stuff, and are about 68 byt
Re: (Score:3, Informative)
GPL3 might be why. He would have to open what he does on his servers to make it work with the GPLv3. That might not be an option for this type of webservice.
If it was GPLv2, it might be a little better of an option except that Rsync wouldn't be able to do incremental backups unless it could decompress/decrypt all the files and then re-encrypt them without damage and when accessed. Storing information in a file index with mapping to the encrypted files would open the encryption to hacking and wouldn't be a g
A solved challenge, mind you (Score:4, Informative)
Rsyncrypto
http://rsyncrypto.lingnu.com/ [lingnu.com]
Encrypts while making the above sentence untrue - a small change in the file results in a (relatively) small change in the cypher text.
It's stable, been around a while, and is written by me :-)
Shachar
P.s.
It is, in fact, written by me for the purpose of a commercial backup service. The software itself, however, is free.
Parent
TrueCrypt (Score:5, Informative)
duplicity + ftplicity (Score:5, Interesting)
duplicity combined with ftplicity:
"Anyone storing data on an unfamiliar FTP server needs to encrypt and sign it to ensure reliable protection against prying eyes and external manipulation. duplicity is just the tool for this, and the ftplicity script from c't magazine makes working with it child's play."
http://www.heise-online.co.uk/security/Backups-on-non-trusted-FTP-servers--/features/79882 [heise-online.co.uk]
http://duplicity.nongnu.org/ [nongnu.org]
Re:duplicity + ftplicity + Portable python (Score:3, Insightful)
Then you should have put this as a requirement in your query. But I would ask WHY you want a gui? Backups should be set-and-forget! My USB sticks have multi-platform autorun scripts to execute my backup. I only need an interface if I choose to expand or shrink the backup set--I can edit a text file that has the list of what to exclude.
Python is pretty easy to put on a portable hard drive and there ar
vanilla ftp: your password will be in the clear. (Score:4, Insightful)
Re:vanilla ftp: your password will be in the clear (Score:5, Insightful)
This may well mean that despite whatever you do, encypt etc, someone can sniff the password and then simply come in and delete all your files.
i.e, whatever other steps you take, this is inherently worthless.
Hardly. As long as the data is encrypted well enough to stop people from stealing or modifying the data in ways that could have serious privacy and financial implications this is a net gain in data availability.
Even if the chance of someone doing this was as high as 5% over the period in question, it would still mean that there was 95% chance of you having a good off site backup. That is better than nothing as long as you realise that there is still a 5% risk and don't act like it is totally secure.
As a simplified example; if your PC at home is 95% sure of retaining all of its data in the period and your portable USB hard drive is 95% sure of retaining all of the data, the chance of you losing any data at all is 0.0125%. Even with exaggarated risk factors, this is not bad.
Parent
Manent fits the bill perfectly. (Score:5, Informative)
Manent is an algorithmically strong backup and archival program. It features efficient backup to anything that looks like storage. Currently it supports plain filesystems ("directories"), FTP, and SFTP. Planned are Amazon S3, optical disks, and email (SMTP and IMAP). It can work (making progress towards finishing a backup) over a slow and unreliable network. It can offer online access to the contents of the backup. Backed up storage is completely encrypted. Backup is incremental, including changed parts of large files. Moved, renamed, and duplicate files will not require additional storage. Several computers can use the same storage for backup, automatically sharing data. Both very large and very small files are supported efficiently. Manent does not rely on timestamps of the remote system to detect changes.
Check it out: http://freshmeat.net/projects/manent [freshmeat.net]. It's under active development (the UI and the setup are currently in fetal stage) but the basic functionality is there and is well tested.
Disclaimer: I am the author.
Re: (Score:3, Informative)
Re: (Score:3, Informative)
No, it's more like someone running around naked outside holding a frosted pane of glass in front of them wondering if maybe they should also build a tool to hold a second pane of frosted glass behind them.
Re: Errr (Score:5, Funny)
Am I the only one thinking this is like someone saying they want privacy then running around butt naked then wondering how they can keep their privacy at the same time.
And the answer of course is security through obscuring. Wear a mask ;-)
(I guess you're not completely naked, but hey, close enough)
Parent
Re: Errr (Score:5, Funny)
A rabbi and a minister went swimming naked at a lake when some children came. Both ran out of the water to their clothes.
The minister covered his 'private parts' while the rabbi covered his face. When the minister asked why the rabbi did not cover his genitals for the children, the rabbi said : "Hey they recognize me by the face."
Parent
Re:Errr (Score:5, Insightful)
Even if his userid/passwd are compromised, his data wouldn't.
So if someone used his userid/passwd to delete his archive or overwrite it, his data wouldn't be compromised?
Or has the data no value, so the archive can be deleted/corrupted without loss? Then what is the use of archiving it at all?
Parent
Re:A slight oxymoron here. (Score:5, Insightful)
"secure" and "untrusted" don't go hand in hand. If you want security, don't put things in untrusted spaces. Period.
Are you sure about that? I consider my SSH connections secure even tho' they traverse untrusted links. Same goes for my encrypted mails, https connections to my bank, etc.
Anyway, to the submitter - is areca [sourceforge.net] close to what you want?
Parent
Re:A slight oxymoron here. (Score:4, Insightful)
"secure" and "untrusted" don't go hand in hand. If you want security, don't put things in untrusted spaces. Period.
I disagree. Everywhere you can store your files should be considered "untrusted". And "securing" the files is what we do to mitigate that reality.
Parent
Re:A slight oxymoron here. (Score:5, Insightful)
If you want security, don't put things in untrusted spaces. Period.
Completely, utterly incorrect. It's a sad comment on the ambient understanding of data security that this got modded insightful.
Trust is seldom a good approach to security. Good security is when you can trust nobody and still sleep at night. That means strong encryption. That is exactly the approach implied by the article and it is exactly the right thing to do.
I think it is very unwise to ever assume any level of trust in the storage of backups, certainly offsite backups. The whole idea of backups is that you keep them around for a long time, in several copies and several locations. The more valuable your data, paradoxically, the more copies you need and the more widely dispersed they should be. This is antithetical to maintaining trust. The right way, indeed the only way out of this paradox is strong encryption.
Parent
Re: (Score:3, Interesting)
It depends on what you put behind the word "security".
"Backup" is also "security". And a cheap of-site backup is better than no off-site backup at all.
I have the same need as the submiter as my ISP provides 10 GiB of public web space available only through FTP (r/w), HTTP (r) or HTTP+PHP (r/w). I have the storage, I need the software to use it while hiding backup content from my ISP and from other web eyes.
Re:A slight oxymoron here. (Score:4, Insightful)
All encryption can be broken. The solution then, is to ensure that the encryption cannot be broken within a useful timeframe. I really don't care if you manage to decrypt my credit card number if the card has already expired. If I'm having a secret meeting this time tomorrow then the encryption only needs to last just over 24 hours, since by the time you work it out it will be too late.
I actually think you've got it backwards. Encrypt them strongly and you can put the data on a billboard in the centre of picadilly circus and no one will be able to work it out in a useful timeframe. Ever seen Kryptos? http://en.wikipedia.org/wiki/Kryptos [wikipedia.org] The data is public, there are many thousands of people attempting to break it, and yet the hardest section remains unresolved. The acceptable risk is related to the minimum amount of time that you can allow for the code to be broken, which determines how strong you need your encryption.
Parent
Re: (Score:3, Informative)
After thinking about this, I realize that I never should have written "non-trusted" in the description. I was thinking about FTP in the sense that it's inherently non-secure, which would make "non-trusted" redundant. I actually trust the FTP server as much one can be trusted.
This made me think about side question to which others in this thread have alluded: When is FTP as secure as SFTP? When you're on a non-trusted computer. It's a minor issue, but if you're trying to access an encrypted file from a n
Well, I've also looked into it and found nothing (Score:5, Informative)
Parent