Slashdot is powered by your submissions, so send in your scoop

 



Forgot your password?
typodupeerror
Businesses EU Networking IT

Ex-Admin Deletes All Customer Data and Wipes Servers of Dutch Hosting Provider (bleepingcomputer.com) 215

An anonymous reader quotes BleepingComputer: Verelox, a provider of dedicated KVM and VPS servers based in The Hague, Netherlands, suffered a catastrophic outage after a former administrator deleted all customer data and wiped most of the company's servers. Details of what exactly happened aren't available, but according to posts on various web hosting forums [1, 2, 3], the incident appears to have taken place Thursday, when users couldn't access their servers or the company's website.

Verelox's homepage came back online earlier Friday, but the website was plastered with a grim message informing users of the ex-admin's actions. Following the incident, the hosting provider decided to take the rest of its network offline and focus on recovering customer data. Verelox staff don't believe they can recover all data.

Saturday night the web site was advising customers that the network and hosting services "will be back this week with security updates," adding that "current customers who are still interested in our services will receive compensation."
This discussion has been archived. No new comments can be posted.

Ex-Admin Deletes All Customer Data and Wipes Servers of Dutch Hosting Provider

Comments Filter:
  • Good (Score:5, Insightful)

    by 110010001000 ( 697113 ) on Sunday June 11, 2017 @10:38AM (#54596019) Homepage Journal
    Maybe people will start realizing that the Cloud is just "someone elses servers" and you have no idea how they manage them or back them up.
    • What is particularly idiotic is that everyone didn't understand it from the beginning. But clearly, they don't.

      • Hardly surprising. People look at the bottom line. Ohhh, we can save $$$ by outsourcing nearly all of our IT to some cloud company!
        • Executives also read the press release, though. The mighty Cloud was supposed to mean much easier administration so we didn't need to handle most IT matters in-house.

          In actual $$$ terms, at both the low end and the high end the Cloud often works out more expensive than self-hosting, often by quite a wide margin. There's a zone in between where that doesn't always seem to be the case, but I'm not sure how wide it really is, and it's usually based on TCO rather than the hardware and connectivity expenses alon

    • That's a bit condescending of a position. Most cloud users do know that; you pay for the convenience not to fund and support a datacenter yourself.

    • Re:Good (Score:4, Insightful)

      by XXeR ( 447912 ) on Sunday June 11, 2017 @11:16AM (#54596179)

      Maybe people will start realizing that the Cloud is just "someone elses servers" and you have no idea how they manage them or back them up.

      Hosting was around long before cloud, and for some reason never garnered the amount of haters that cloud currently endures. VPS is hosting, not cloud...please direct your hate appropriately.

      • by dissy ( 172727 )

        Hosting was around long before cloud

        That is very likely not true, unless you count being best friends with one of the admins at uni who gave you a shell account for free on a server.

        "Cloud" is a term from the 1970s.
        "Hosting", as performed by a company specifically offering such a thing, came about in the 1990's after the commercialization of the Internet.

        The 90's did not come before the 70's

        • Hosted solutions started back in the 60's, typically mainframe. The earliest I encountered it was mid-80's, one of our clients used a hosted financials application.
      • by iCEBaLM ( 34905 )

        Hosting is Cloud. It's like saying cars were around long before automobiles.

        VPS is hosting is cloud. There is no difference. It's a marketing term to synergize the verticals.

    • Maybe people will start realizing that the Cloud is just "someone elses servers" and you have no idea how they manage them or back them up.

      You're barking at the wrong tree. Customers just buy a service based on price, uptime and retention, as advertised.

      If you don't feel respected due to underpayment, lack of resources or management, it's your job to step up and take a stand.

      In any case, the customers are most likely not at fault so don't fuck them. It looks bad on all of us.

      If you can't deal with responsi

    • 'the Cloud is just "someone elses servers" and you have no idea how they manage them or back them up.'

      The Internet is is just "someone elses networks" and you have no idea how they manage them or back them up.

    • That's why you keep local copies.

  • Did they not remove the ex-admin's credentials, or what? I mean, regardless of how the ex-admin gained access to the data to wipe it, it's a crime. But I'd like to know if Verelox was stupid enough to not remove his credentials, or this happened some other way. (Like, did he do this his last day of work as a "fuck you" to management for firing him?)

    • Re:So... (Score:5, Insightful)

      by Kjella ( 173770 ) on Sunday June 11, 2017 @10:59AM (#54596099) Homepage

      Did they not remove the ex-admin's credentials, or what?

      They should... but if you're sitting with the keys to the kingdom you might have the domain administrator account password, root passwords, various service accounts set up for particular purposes including but not limited to integration with external access... Yes, all could be done with the proper procedures in place. But very often the responsible for such IT procedures is the admin and the admin is the one keeping tabs on what everyone else has access to. Plus you often have the rights to create undocumented loopholes that you might reasonably excuse as being a test account and an oversight if discovered. Not to mention the setting you'd bring this up, either you're basically questioning the loyalty of one of the most trusted men in the system or it looks like you're setting him up to be fired.

      • Then it's a matter of strong procedures. There should never be a single point of failure, and all those passwords should be written down and sealed away somewhere accessible by the appropriate people.

        The CEO/President may not *need* the passwords, but if he/she (and/or his/her admin) have the passwords saved somewhere then should the Admin get hit by a bus the company can keep moving forward. Any sane company would have the hand-off of the keys to the kingdom as part of the out-processing procedures. This i

        • and all those passwords should be written down and sealed away somewhere accessible by the appropriate people.

          And your procedure for updating the locked-away passwords is?

          And your procedure for checking that the password hasn't been changed without notification is ?

          Conceptually, it's simple. In practice, it's not so simple. That's why screens have bezels - for sticking the post-it to.

    • They may well have done this, but an admin is well placed to create a backdoor that nobody else knows about.
  • At least these two stories are from different perspectives: https://m.slashdot.org/story/3... [slashdot.org]

    The story stays the same - don't fuck over your admins and have proper procedure and backup.

    • In a certain sort of movie (e.g. Mad Max, The Crow) the difference between good guys and bad guys is the order in which they commit their atrocities. In these two stories, good guys delete the data and then get fired, bad guys get fired and then delete the data.

      • In a certain sort of movie (e.g. Mad Max, The Crow) the difference between good guys and bad guys

        There was a good guy in Mad Max? That's not the impression I got from seeing bits of it. Oh, I see - that's your point. OK : quod erat demonstratum.

    • The story stays the same - don't fuck over your admins and have proper procedure and backup.

      There's a lot of lessons that management somehow never seem able to learn. Like, never have only 1 sysadmin, even if having more seems like overkill. Or, how about nurturing a relationship based on mutual respect and trust? The sysadmin has after all been trusted with one of the most important resource in any company, in most cases: the data. Yet, in a previous job I had an experience that I think is not uncommon: I had been there for over 10 years, I was regularly praised by my colleagues, but I was 'old'

  • Why no secure backups? Idiots.
    • Re:Backups? (Score:4, Insightful)

      by QuietLagoon ( 813062 ) on Sunday June 11, 2017 @11:17AM (#54596195)

      Why no secure backups?...

      The article(s) seem to indicate that most, but not all, customer data can be recovered. So it seems there were working backups. But in a hosting environment, not everything is backed up continuously, and that may be where some of the data will be lost.

  • by UnixUnix ( 1149659 ) on Sunday June 11, 2017 @11:58AM (#54596365) Homepage
    Love and coddle your Admin -- or else!
    • The BOFH has a cattle-prod with your name on it. The PFY will hold you down and line up the welding rods. It's a helluva cattle prod (unsurprisingly).
  • by v1 ( 525388 ) on Sunday June 11, 2017 @12:05PM (#54596405) Homepage Journal

    and this is obviously one of them. Criminals come from all walks of life, sysadmin isn't a position immune to containing the occasional bad apple.

    So many questions of course, a lot of which boil down to "They must have had some serious lapse in procedure to have allowed this to happen." That's not really the case though. Back doors and logic bombs are serious threats when a person has been a trusted system administrator. Done "right", they can be extremely difficult to detect. It's a bit like the widely accepted advice of "Server was hacked? Don't try to clean it, you might miss something. You must wipe and reinstall it." (same really applies even to a home desktop) A departing admin (on bad OR good terms) is basically the exact same issue, a compromised system, but we only very rarely see such an extreme response. It's much less practical to nuke-n-pave when it's your entire network that is basically now classified as "compromised." Is this how we should respond? When you really stop and think about it, it starts to show itself as a really difficult question to answer. Rebuilding everything when an admin leaves when your system is large is just really hard to justify. But if your system is big, it's also more difficult to review it all and proclaim it "clean". It's just a bad position to be in, and that's why admin departures are such a headache. If you're big enough you have several admins and better compartmentalization of access, more robust isolation of systems, better logging, security software that's under the control of the CIO and not the admins, etc. They have a better chance, but it doesn't look like this one was big enough to have those benefits.

    The lack of backups is the most troubling though. That's what stung the other recent post on the cleanout-from-inside. There's just no excuse for that.

  • I was with a one-man ISP for 12 years, starting off with a dial-up UNIX account to hosting my websites. Unfortunately, in 2007, the two lines from different providers to the out of state data center got cut in separate backhoe accidents (what are the odds?). It took ten days for the providers to restore the lines and him to get a third line with a different provider installed. I've already moved my websites to a dedicated hosting provider by day eight. I haven't suffered an outage since then.
    • Yeah, I gotta admit I saw the, "current customers who are still interested in our services will receive compensation." and thought, "Yeah. The both of them."

  • Can't recover? What did he do, dd if=/dev/zero of=/dev/fs ? Or were they using something like NTFS? Or most likely: storing the data in the CLOUD.

    Pretty well every linux filesystem has recovery tools. There's a reason the POSIX term for "delete file" is "unlink". Because you aren't clearing the data, you're just unlinking from the table.

    Since pretty well every file has a MAGIC at its start, it becomes fairly dooable to recover.

PLUG IT IN!!!

Working...