Dreamhost FTP/Shell Password Database Breached 123
New submitter Ccmods writes "Below is a snippet from an email Dreamhost sent to subscribers early Saturday morning, describing an intrusion into the database storing FTP and SSH usernames and passwords: 'We are writing to let you know that there may have been illegal and unauthorized access to some of your passwords at DreamHost today. Our security systems detected the potential breach this morning and we immediately took the defensive precaution of expiring and resetting all FTP/shell access passwords for all DreamHost customers and their users. ... Only the FTP/shell access passwords appear to have been compromised by the illegal access. Web panel passwords, email passwords and billing information for DreamHost customers were not affected or accessed.'"
Not a big deal (Score:5, Informative)
As a Dreamhost customer, I watched this unfold in real time. Apparently the passwords were hashed, and there's no indication that they were compromised, other than the fact that it was technically possible. So they changed the passwords because it's cheaper, PR-wise, than being wrong.
There's a big warning up on the panel, which has a password stored in a different, non-compromised DB. Between the panel and the email, I doubt anybody's confused as to what's going on.
In other words, it's really not that big of a deal. The database shouldn't have been compromised, and I'll expect a full postmortem of how they screwed that up, but in terms of damage (or even inconvenience), there really isn't any to speak of.
Re: (Score:3, Insightful)
Re:Not a big deal (Score:5, Funny)
it's good to see some rationality being injected somewhere.
You mean as opposed to SQL being injected somewhere, of course.
Re:Not a big deal (Score:4, Funny)
Re: (Score:3, Insightful)
Re: (Score:3)
Well, several years ago, before they moved their servers, I was in the same datacenter with them. My cage was almost next to theirs. On several occasions, I talked to them. All of their folks knew their stuff, and showed me around the inner workings a good bit. I was impressed. I highly recommended them at the time. Unless someone made some horrible decisions, I strongly suspect they're still worth the praise.
Now, what happened? Hell if I know. I'm on the other side of
Re: (Score:2)
I'd be hard pressed to believe you've dealt with anyone other than DreamHost. When I failed to renew my service with them it was because their hosting was glacial. It could barely keep up with a lightly used PHP image gallery. That was years ago. When I migrated clients away from DH last year it was because of chronic downtime. "Oops we fucked up" is great, and it's honest. It's also not something you want to keep seeing [dreamhoststatus.com]. "Oops we fucked up, but your worthless blogs about your kittens' trousers are s [dreamhost.com]
Re: (Score:3)
We're suffering a DDoS... no... wait... we don't know how to fix our Cisco equipment... is not something you want to see ever.
I think off-hand - though I don't deal with Cisco kit - that is indeed a failure mode that shouldn't happen ever.
Re: (Score:2)
They also have a twitter feed for their status, @dreamhoststatus, I think.
Phone support is available, but at a fee. Email / control-panel- based support is (IME) excellent
Re: (Score:2)
status.dreamhost.com = dreamhoststatus.com = down when they bork one of their core routers. After last year, I'd sure hope that they've put the status blog and corporate e-mail on a separate network. Pretty sure that they've been too busy [dreamhost.com] posting pictures of cat anuses [dreamhost.com] to have bothered changing anything tho. Phone support /is/ available, but you've got to plead for a call back via e-mail. That's all fine and dandy except for the fact that it's all on one network. A single point of failure is a single p
Re: (Score:3)
As a Dreamhost customer, I watched this unfold in real time. Apparently the passwords were hashed, and there's no indication that they were compromised, other than the fact that it was technically possible. So they changed the passwords because it's cheaper, PR-wise, than being wrong.
There's a big warning up on the panel, which has a password stored in a different, non-compromised DB. Between the panel and the email, I doubt anybody's confused as to what's going on.
In other words, it's really not that big of a deal. The database shouldn't have been compromised, and I'll expect a full postmortem of how they screwed that up, but in terms of damage (or even inconvenience), there really isn't any to speak of.
It's good to see they took the matter seriously, even with the circumstances you describe. Bad that it happened in the first place, but it sounds like the situation was nicely handled.
Re: (Score:1)
>Apparently the passwords were hashed.
If you choose the "forgot my password" option for the dreamhost webpanel it automatically emails you your current password in plain text form (not a new password or a way to reset).
Given that webpanel password is stored in platintext by dreamhost I have little confidence that ftp passwords have stronger protection.
Re: (Score:1)
Re:Not a big deal (Score:4, Informative)
Just because you can get it emailed to you does not mean that it is stored plaintext.
Re: (Score:1)
Re: (Score:2)
That is probably true in this case, but is not necessarily true in all cases. Imagine, for example, that the password is encrypted with the email address as its key, and the email address is hashed. Upon login, a hash lookup is done for the email address, and the encrypted password is decrypted and compared to the one sent. Or alternatively, the password is stored both encrypted as mentioned, and hashed, so that logins are done by 2 hash checks.
Either scenario would allow the user to retrieve the passwor
Re: (Score:2)
That is probably true in this case, but is not necessarily true in all cases. Imagine, for example, that the password is encrypted with the email address as its key, and the email address is hashed. Upon login, a hash lookup is done for the email address, and the encrypted password is decrypted and compared to the one sent. Or alternatively, the password is stored both encrypted as mentioned, and hashed, so that logins are done by 2 hash checks.
Either scenario would allow the user to retrieve the password without the host or an attacker being able to see what the associated email or password is.
I'm pretty sure the hosting provider would require an unencrypted/unhashed copy of your email address.
Re: (Score:2)
No, they would not. This is simple. They store only the email hash. Your account id is tied to that hash. Upon login, you are required to provide your email address and password; the email is used to try to decrypt the password, and is then hashed and compared to their stored hash. If the provided password matches the decrypted password, and the hashed email matches the stored hash, you get logged in.
For password recovery, all you provide is the email, which is again used to perform a hash lookup, and
Re: (Score:2)
I understood the concept. Just pointing out that they need to have your contact info somewhere which would naturally include your email address
Re: (Score:2)
Email address could be stored in a cookie upon login. The site would indeed be unable to send you unsolicited email, but would be able to email you upon login or password reset.
Re: (Score:1)
Re: (Score:2)
That is probably true in this case, but is not necessarily true in all cases. Imagine, for example, that the password is encrypted with the email address as its key, and the email address is hashed. Upon login, a hash lookup is done for the email address, and the encrypted password is decrypted and compared to the one sent. Or alternatively, the password is stored both encrypted as mentioned, and hashed, so that logins are done by 2 hash checks.
Store the password hashed in the "production" database, and keep an encrypted copy in a separate database hosted on a service that is responsible for sending out emails relating to password reminders (or resets). All the frontend services can do are to validate that a supplied password matches (through hashing) or request a reminder or reset for a particular user; nothing else should be possible since the encrypted version is kept out of reach.
Not that I expect anything so sensible in this case. SQL injecti
Re: (Score:2)
Why store the password in a retrievable fashion?
Sending the actual password to the end-user via email in clear-text is stupid. The end-user will likely go "ohh, right" and keep using it. Much better to send them a random one-time use password or a link that allows them to reset the password once.
Re: (Score:2)
If they can email you the plain text then they are not hashed.
Re: (Score:2)
Re: (Score:3, Interesting)
>Where? I've been a DH customer for 5 years...
The "forgot my password" link on the webpanel login page (discovered today by virtue of needing to log in to set user passwords again).
You are right that for users within your webpanel account there is no email reset option - you log into the webpannel to set these passwords.
But the webpanel account itself - passwords are emailed in plaintext.
Re: (Score:2)
Yes, it does. Those are stored in plain text anyways, at least the mysql ones (your config files). The passwords in question are for shell/FTP accounts, as the title says.
Re: (Score:1)
This has been a problem for at least 3 months, any dreamhost user can upload a php based rootkit and download the password database.
I'm speaking from experience in removing rootkits from dreamhost hosted wordpress sites.
Re: (Score:2)
Uh... what do WordPress user databases have to do with shell account user databases?
Re: (Score:1)
Not a wordpress database. Upload a rootkit to DH, and you can read any file on the system as if you were root. It has nothing to do with wordpress other than that being the mechanism by which the rootkit was dropped into DH's servers. The rootkit runs with the privileges of the web server, not the user. You can read all the files in /etc
Re: (Score:2)
Re:Not a big deal (Score:5, Insightful)
I actually think it's a big deal, but not for the reason most people are crying about.
It's a bit deal that they have been open, honest, & cautious about the intrusion. Having seen so many high profile companies take the opposite stance lately, the DH intrusion should be made a big deal of, if anything, to show other companies how you react to being hacked without losing face with customers.
For me, there is only one chance when it comes to security to get it right. If you try to hide intrusions, lie to customers, or stonewall tech sites trying to get more information, you aren't a company to be trusted with my data.
Re: (Score:2, Informative)
Let me second that. I got the email, checked into my dreamhost account, used the excuse to call my sister (and will have a conversation with someone else). and then I was done with the *protective* aspect. Actually, the protection happened right away because dreamhost locked the possibly-compromised accounts immediately, as I understand it. The *recovery* aspect, then, just took a few minutes, and involved an enjoyable family chat.
I don't think of dreamhost as "less secure" than I thought it was.
Re: (Score:2)
Had I found out months later, that hackers had compromised dreamhost, and that dreamhost had kept it quiet, I would have been an unhappy customer. As it is, I'm a happy one.
I'm happy for all the DH customers that this's turned out to be little more than "HEADS UP", and it appears DH went out of their way to handle the damage correctly. Great! Bravo!
However, as an IT guy interested in system security, I *hate* that this happened in the first place, and seems to happen far too often regularly. Why is FTP still being used, and why don't you guys know how powerful a shell account can be in the hands of a master (or a gifted amateur, for that matter)?
This !@#$ shouldn't happen.
Re: (Score:3, Informative)
Re: (Score:2)
FTP is still a useful protocol because:
1.) few people upload sensitive data to a web hosting service.
2.) it requires less CPU overhead.
3.) FTP transfers, while better with a client like Filezilla/Cyberduck/xFTP, don't *require* a client since both Windows and OSX support it natively.
Re: (Score:3)
The problem with FTP isn't that it transmits data as cleartext (though it does that too). The problem is it transmits passwords as cleartext. Anyone snooping on your FTP session will know your username and password.
Re: (Score:2)
1. User names and passwords are sensitive.
2. CPU is cheap.
3. Time to force end users to use a real ftp client and/or have MS or Apple implement a modern protocol.
Re: (Score:2)
Re: (Score:2)
"Shell account" is why I'm with DH. I upload everything using rsync-over-ssh. It's not like those other options aren't there.
Yeah, but stuff that was determined more than a decade ago to be inherently insecure should not be supported and ought to be turned off. Make the users happy, sure, but try to keep them from shooting themselves too, yes?
DH customer: "Why can't I get to my ftp login?!?"
DH Tech Support: "Because we can't know whether your box is infected with a keylogger trojan (or worse) or if your network connection's being sniffed. Please use the much more secure ssh & etc. You'll prefer it once you get to know it,
Meanwhile under the radar (Score:4)
A few weeks ago Telstra Bigpond, one of Australia's largest ISPs, was caught out with the utterly stupid situation of having their customer list of username, plain text password, email address and mailing address out there naked on the internet. Outsourced workers in call centres needed the information but some idiot decided instead of them having to log in somewhere to get access that they should simply be able to use a URL with the customers username on the end of it. The site with the passwords was still up ten hours after it hit the mainstream news.
Now that's the sort of thing I expect when I see something like the article summary above, but instead it's the opposite - full disclosure early instead of being caught out by the press and not plain text passwords.
Re: (Score:2, Interesting)
It's a bit less trust-inspiring than you represent it.
Brian H. from Dreamhost initially posted on the Dreamhoststatus page that FTP/SSH passwords are only stored hashed. Later he deleted that statement. Why?
Web panel passwords are definitely stored in a retrievable way, because when you forget your web panel password they mail it to you. Not a nonce key that allows you to set a new password, they mail you the actual password. According to Dreamhost CEO Simon Anderson [dreamhost.com], they're now evaluating if they could
Re: (Score:3)
Well, it *IS* a big deal, but only for people who are using the same password on dreamhost and other services. Obviously, people shouldn't do that, for reasons that are now obvious, but people do. Whoever got this password list is likely to start looking at facebook and other sites for accounts with similar names and use any passwords they can crack from this database.
The compromise is sometimes not the obvious one... For example, I had an account on a service that was recently compromised, and that acco
Re: (Score:2)
Except that they found a symptom, and not the actual problem. Someone has unauthorized access to their servers. Until they figure out how they've gotten in and closed the door, it's pointless to scramble passwords. This also wasn't a "quick response" as people have been complaining about their accounts getting hacked and their WP configus and .htaccess files getting modified for months.
Re: (Score:1)
Well, my sister's company uses Dreamhost, and they were hacked. They do use ftp (instead of sftp) to upload their files, so I'm guessing that's the likely culprit. I've since set them straight.
I've been a loyal Dreamhost cusomter since 1998 and I'm happy with their response.
Cleartext passwords (Score:1)
As a dreamhost customer (who doesn't store anything overly sensitive there), I've noticed that they also have a tendency to send me mail with my real password in the past, which indicates that it's stored somewhere in cleartext (which is BAD).
Re: (Score:1)
I'm not sure that's quite true across the board. According to a blog comment here [dreamhost.com] by Simon Anderson (dreamhost CEO):
Re: (Score:1)
In other words, it's really not that big of a deal. The database shouldn't have been compromised, and I'll expect a full postmortem of how they screwed that up, but in terms of damage (or even inconvenience), there really isn't any to speak of.
I run dozens of web sites on DreamHost with almost as many shell accounts associated with them. Going through all those account and assigning new passwords to them, then reconfiguring my development tools with the new passwords is a major undertaking. I think that qualifies as an inconvenience (especially since the password change sync seems to be taking much longer than usual right now).
However, given what had happened, I wouldn't have it any other way. I'm glad they recognized the problem and responded
FTP? (Score:5, Insightful)
If the passwords are used for FTP they should be considered comprimised anyway.
Re: (Score:2)
Very valid comment, this deserves to be modded up. Since FTP authenticates in cleartext, anyone capable of sniffing the transaction gets the authentication credentials in full.
That's why it's never, ever safe to attach FTP credentials to anything else.
I believe Dreamhost handles this by issuing a separate password for FTP.
Re: (Score:2)
I believe Dreamhost handles this by issuing a separate password for FTP.
They should handle it by only supporting SFTP.
I'll see your SFTP and raise you... (Score:5, Insightful)
I'll see your SFTP and raise you disabling password authentication entirely, and using SSH public key authentication only.
If your SSH server is visible over the Internet, you should use public key authentication instead of passwords if at all possible. If you don't think it's important, try logging all of the malicious login attempts you get for the next week.
-- https://help.ubuntu.com/community/SSH/OpenSSH/Keys [ubuntu.com]
Re:I'll see your SFTP and raise you... (Score:4, Informative)
I'll see your SFTP and raise you disabling password authentication entirely, and using SSH public key authentication only.
I do this on my own servers but I don't use plain file transfer at all. Instead I use a distributed version control system (mercurial) and I push to the server. Mercurial lets me define a hook to update the remote copy to the repository tip when new changesets are pushed to it. Working this way I have a full version history at the local and remote end. Additionally I only have to manage the directory tree locally. The remote end is taken care of.
Another advantage is that mercurial hashes the whole repository so if anybody does fiddle with any files, I hear about it as soon as I touch the repository.
Re: (Score:2)
The stolen laptop problem renders that insane (Score:2)
I think you've misunderstood some good advice of using BOTH a key AND a password/passphrase. The quote you've used has left out the point that typically when you generate a key it also prompts you for a password/passphrase. Passwordless keys are useful within internal networks but are a very bad idea fo
Reading comprehenion? (Score:2)
Nobody implied that you shouldn't encrypt your private key with a strong passphrase.
This setup is absolutely perfect for laptops, because it's two-factor authentication. The thief will need both the key from the laptop and the passphrase.
No, caught out with bad advice instead (Score:1)
Apart from some guy under the handle of "sakdoctor"?
It's not a "reading comprehension" thing if you state the opposite of what you now say you intended.
My point stands - as written it's very bad advice to do it all with keys and no password authentication with the keys.
What the fuck do you expect readers to think you mean by "disabling pa
Re: (Score:1)
People who store keys on their disks SHOULD be taught to use full-disk-encryption, specially in case of laptops.
Re: (Score:2)
Can sftp and scp do resume downloads and uploads yet?
Re: (Score:2)
Use rsync over ssh.
Re: (Score:2)
Ah cool, so it can resume upload and download? I will check it out later. Currently, I use old school sz and rz through SecureCRT and PuTTY's zmodem hack/mod.
Re: (Score:2)
That'd be nice, but for some reason it still seems to be the standard to support FTP for web hosts. My host, fatcow,com, does so as well. Why, I don't know; I'm guessing it's probably because a bunch of people use shitty website-authoring tools that don't support SFTP. I wouldn't be surprised if MS FrontPage is the big reason; my host supports FP and FP extensions, even though last time I checked, FP is defunct and unsupported, and has been replaced by some other MS tool with a totally different name. B
Re: (Score:2)
Re: (Score:2)
rsync only works if the hosting service supports it, IIRC. My hosting provider, fatcow.com (owned by Tucows I believe), does not seem to support it from everything I could find in their help pages, only FTP and SFTP. rsync would be much better because it's really fast and works really well, but I guess most hosting account customers are clueless about it, probably mostly being Windows users using some sort of commercial web authoring software, and rsync being mostly a Unix-only tool it seems. I ended up
Re: (Score:2)
Re: (Score:2)
Unfortunately, no there's ssh access on my webhost, so sftp (w/ password) is the only secure way.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:1)
Content management software from commercial vendors lack it. I guess they assume you have your own ftp servers in your big corporate office. They forget not everyone has the luxury of 10 t1s for a good network connection and their own IT management and servers in the same building. Most small to medium businesses use an ISP.
Re: (Score:2)
Is that a serious question? If so, how about /usr/bin/ftp just for starters? And I imagine c:\windows\system32\ftp.exe as well, though I don't use toy operating systems myself. Those are by far the most common ftp clients.
Only since last June.. (Score:3, Interesting)
It got so bad at one point that I recommended that readers of my blog . [dynamoo.com]
Re: (Score:3)
Re:Only since last June.. (Score:4, Informative)
This has been going on since last June [dynamoo.com]. Dreamhost were completely unresponsive to reports that their services were being abused. Hey, it only took 'em half a year to figure out there was a breach..
Probably because that has all the hallmarks of a software PHP vulnerability web-hack of a site, NOT an FTP compromise. I've seen plenty of those, they use some vulnerability to gain access, then upload a file (through the web software) that gives them what's basically a PHP web-based shell. There's no need for the FTP account password to be compromised (and it usually isn't).
All web hosting companies get a lot of that type of attack because their customers don't all update and/or secure their sites properly. WordPress is a particularly popular target.
Re: (Score:2)
The entire point behind using a hosting provider and not just running your own server is for them to deal with shit like that.
No it's not. It's like saying it's a car rental companies fault if someone steals a rented car and crashes into a school bus. All a hosting provider does is provide the hosting environment, after that it's up to the customer to deal with things unless they pay for some specific service.
Re: (Score:2)
In all case except where the customer has manually instal
Were they hashed? (Score:1)
Were these passwords hashed?
If not, sweet mercy, Dreamhost, what could you possibly have been thinking?
If so, sweet mercy, Slashdot, could you be troubled to include this little detail in the summary?
Re: (Score:3)
Those passwords aren't stored in a database, but in the shadow file.
Not necessarily. SSH can validate the passwords using PAM, and PAM can use a database (e.g. Postgres or LDAP) as backend.
Re: (Score:2)
Any admin that has to manage a large number of machines with common authentication is INSANE to use /etc/passwd or /etc/shadow. It's FAR too much work, FAR too much trouble, and will cause FAR too many problems.
I'd bet the farm they're using some combination of Kerberos + LDAP. Kerberos+LDAP is the standard for holding the authentication and user information - even Microsoft uses it in their Active Directory.
PAM = Pluggable Authentication Modules. This means you just use a Kerberos+LDAP plugin for authentic
Painless (Score:1)
Re: (Score:2)
I'm not sure what the password for my account was before, and I'm really not sure what it is now. I use an SSL key for authentication anyway, so this doesn't matter much....
What would be really cool would be if DreamHost allowed you to inject an SSL public key into the account for login purposes, and stopped having a password database entirely. Just my $0.02.
Re: (Score:2)
Err... s/SSL/RSA/g.
Re: (Score:2)
Re: (Score:1)
Found a planted perl script (Score:2)
Conversation probably went like this (Score:1)
"The passwords are hashed though, right?"
"Yes."
"So they can't reverse the passwords out of it?"
"Virutally impossible."
"Is there any advantage they gain from having the hashed passwords?"
"Well, it would allow them to brute force their guesses against it without fear of being slowed down or blocked."
"But wouldn't they need to know our salting algorithm for that to be useful?"
"Um..."
"Let's go
Why a Database of Password? (Score:2)
Re: (Score:1)
Didn't they stored passwords crypted?! (Score:2)
As the Unix does since ... 1969?
Seriously, WTF?!
- Hubert
Re: (Score:2)
Why do you just assume the passwords are cleartext? Do you not understand that even hashed or crypted passwords are easy things to crack?
There are numerous programs specifically designed to crack passwords. They start with the obvious things (dictionary), then adds in misspellings, moves on to simple substitutions (1=l, 4=a, @=a), and finally to more difficult things, like multiple words+substitutions. Many crackers combine clustering with GPUs, which makes for highly effective cracking.
They are terrifyingl
Love Dreamhost (Score:2)
Their support is very responsive, and the occasional technical hiccups with hosting packages are handled quickly and professionally. Although this break-in is a bit scary, it seems like they're playing "better safe than sorry" by res
I think they do store clear text passwords (Score:1)
At least in the past they sent me my web panel password when I asked them to.
How else can they send the password back?
argh! (Score:1)
More info on what happened from CEO (Score:1)
Simon Anderson Says:
January 21st, 2012 at 11:55 am
some more detail – our systems have stored and used encrypted passwords for a number of years, however the hacker found a legacy pool of unencrypted FTP/shell passwords in a database table that we had not previously deleted. We’ve now confirmed that there are no more legacy unencrypted passwords in our systems. And we’re investigating further measures to ensure security of passwords including when a customer requests their password by email (this was not the issue here, though). Re your shell accounts, I’d suggest that you select a new password just to be sure.
Search for "January 21st, 2012 at 11:55 am" at this link [dreamhost.com]
Also, due to the number of customers changing their passwords, the password sync time is very slow right now. More info here. [dreamhoststatus.com]