Ex-Admin Deletes All Customer Data and Wipes Servers of Dutch Hosting Provider (bleepingcomputer.com) 215
An anonymous reader quotes BleepingComputer: Verelox, a provider of dedicated KVM and VPS servers based in The Hague, Netherlands, suffered a catastrophic outage after a former administrator deleted all customer data and wiped most of the company's servers. Details of what exactly happened aren't available, but according to posts on various web hosting forums [1, 2, 3], the incident appears to have taken place Thursday, when users couldn't access their servers or the company's website.
Verelox's homepage came back online earlier Friday, but the website was plastered with a grim message informing users of the ex-admin's actions. Following the incident, the hosting provider decided to take the rest of its network offline and focus on recovering customer data. Verelox staff don't believe they can recover all data.
Saturday night the web site was advising customers that the network and hosting services "will be back this week with security updates," adding that "current customers who are still interested in our services will receive compensation."
Verelox's homepage came back online earlier Friday, but the website was plastered with a grim message informing users of the ex-admin's actions. Following the incident, the hosting provider decided to take the rest of its network offline and focus on recovering customer data. Verelox staff don't believe they can recover all data.
Saturday night the web site was advising customers that the network and hosting services "will be back this week with security updates," adding that "current customers who are still interested in our services will receive compensation."
Good (Score:5, Insightful)
Re: (Score:2)
What is particularly idiotic is that everyone didn't understand it from the beginning. But clearly, they don't.
Re: (Score:2)
It's not just the about $$$ (Score:3)
Executives also read the press release, though. The mighty Cloud was supposed to mean much easier administration so we didn't need to handle most IT matters in-house.
In actual $$$ terms, at both the low end and the high end the Cloud often works out more expensive than self-hosting, often by quite a wide margin. There's a zone in between where that doesn't always seem to be the case, but I'm not sure how wide it really is, and it's usually based on TCO rather than the hardware and connectivity expenses alon
Re: (Score:2)
If your loading is very uneven, it can be cheaper to offload to something like AWS than to keep enough servers around to handle peak loads.
That is true, but how many web apps really have such highly variable demands that this works out in their favour?
That sort of thing happened long before we started calling it the Cloud.
True enough, but that doesn't make the current Cloud-related incarnation any better.
Re: (Score:2)
Easy - e-commerce does. Every person consumes a certain percentage of server resources and you can handle the average load pretty easily.
But on special days like Black Friday (or Boxing Day or others), your server load and customer count can easily be 2-5x as much, or more. Just for a single day of shopping. And maybe if there's a big day of sales, it may peak at 2x or so. While a 2x peak might b
Re: (Score:2)
That is a good example. If you're running a large online store and have very seasonal buying patterns, sure, the on-demand scalability of Cloud-style architecture makes sense.
I think it's also fair to say that not all e-commerce is like that. I've worked with a few smaller or niche market stores, and they'd love to have enough demand that hosting everything on a handful of servers and delegating payment processing to an external service wasn't sufficient. If you're a big national chain and dealing with thou
Re: (Score:2)
Of course it's not much cheaper. At the low end, if you only need a handful of servers, it's far cheaper to just buy them and colocate them somewhere, and the management overheads are low anyway. At the high end, it's still cheaper to just buy the dedicated hardware and arrange your own facilities and IT team, because that's all you're effectively doing with Cloud hosting, you're just paying someone else's profit margin on top. In between, where there is a chance to make savings on things like rapid scaling
Re: (Score:2)
At the low end, if you only need a handful of servers, it's far cheaper to just buy them and colocate them somewhere, and the management overheads are low anyway.
You can get a very small (t2.micro) Linux AWS server with 1GB storage for $106 a year. Is that really more expensive than colocation (let alone far more expensive as you claimed)? Then consider that you can set up an identical server on the opposite coast that costs almost nothing until you need to spin it up for disaster recovery. There are also ways to set up replication and automatic failover though I'm less familiar with that.
Re: (Score:2)
You can get a very small (t2.micro) Linux AWS server with 1GB storage for $106 a year. Is that really more expensive than colocation (let alone far more expensive as you claimed)?
That's quite an apples-to-oranges comparison, don't you think? A single t2.micro instance is a far lower specification than even the most basic server you'd colo, and surely no-one is running a substantial online service on a single t2.micro instance and nothing else even in AWS world. I confess that I've never got my head around all the modern AWS options, but it looks like even a moderately loaded web server could exceed the bursty CPU behaviour of such a small instance, and you need to allow for storage
Re: (Score:2)
t2.medium is $412 per year, plus about a buck per gig of storage, and $0.09 per GB data transfer out up to 10 TB per month (up to 1 GB is free). How does that compare? For apples to apples it should be somewhere with redundant everything since I'm sure Amazon data centers have that. Also, if t2.micro (which isn't even the smallest available) is smaller than anything you would colo, then that is a plus for AWS. That means something that would be run off some developer's desktop machine, or shoehorned ont
Re: (Score:2)
It's tough to do a fair comparison, because I'm in the UK so pricing for a lot of things will be slightly different, but I can have a go in general terms.
As an example, a colo package we've looked at for one of my projects would place boxes in major London data centres with proper redundancies etc. The entry level has 1TB/month of data transfer included and enough power and rack space for an entry-level server, and would cost about the same as the data transfer costs alone with AWS, with a similar rate for
Re: (Score:2)
Laugh ?
Re: (Score:2)
That's a bit condescending of a position. Most cloud users do know that; you pay for the convenience not to fund and support a datacenter yourself.
Re:Good (Score:4, Insightful)
Maybe people will start realizing that the Cloud is just "someone elses servers" and you have no idea how they manage them or back them up.
Hosting was around long before cloud, and for some reason never garnered the amount of haters that cloud currently endures. VPS is hosting, not cloud...please direct your hate appropriately.
Re: (Score:2)
Hosting was around long before cloud
That is very likely not true, unless you count being best friends with one of the admins at uni who gave you a shell account for free on a server.
"Cloud" is a term from the 1970s.
"Hosting", as performed by a company specifically offering such a thing, came about in the 1990's after the commercialization of the Internet.
The 90's did not come before the 70's
Re: (Score:2)
Re: (Score:2)
Hosting is Cloud. It's like saying cars were around long before automobiles.
VPS is hosting is cloud. There is no difference. It's a marketing term to synergize the verticals.
Re: (Score:2)
You're barking at the wrong tree. Customers just buy a service based on price, uptime and retention, as advertised.
If you don't feel respected due to underpayment, lack of resources or management, it's your job to step up and take a stand.
In any case, the customers are most likely not at fault so don't fuck them. It looks bad on all of us.
If you can't deal with responsi
Re: (Score:2)
'the Cloud is just "someone elses servers" and you have no idea how they manage them or back them up.'
The Internet is is just "someone elses networks" and you have no idea how they manage them or back them up.
Re: (Score:2)
Re: (Score:2)
That's why you keep local copies.
So... (Score:2)
Did they not remove the ex-admin's credentials, or what? I mean, regardless of how the ex-admin gained access to the data to wipe it, it's a crime. But I'd like to know if Verelox was stupid enough to not remove his credentials, or this happened some other way. (Like, did he do this his last day of work as a "fuck you" to management for firing him?)
Re:So... (Score:5, Insightful)
Did they not remove the ex-admin's credentials, or what?
They should... but if you're sitting with the keys to the kingdom you might have the domain administrator account password, root passwords, various service accounts set up for particular purposes including but not limited to integration with external access... Yes, all could be done with the proper procedures in place. But very often the responsible for such IT procedures is the admin and the admin is the one keeping tabs on what everyone else has access to. Plus you often have the rights to create undocumented loopholes that you might reasonably excuse as being a test account and an oversight if discovered. Not to mention the setting you'd bring this up, either you're basically questioning the loyalty of one of the most trusted men in the system or it looks like you're setting him up to be fired.
Re: (Score:2)
Then it's a matter of strong procedures. There should never be a single point of failure, and all those passwords should be written down and sealed away somewhere accessible by the appropriate people.
The CEO/President may not *need* the passwords, but if he/she (and/or his/her admin) have the passwords saved somewhere then should the Admin get hit by a bus the company can keep moving forward. Any sane company would have the hand-off of the keys to the kingdom as part of the out-processing procedures. This i
Re: (Score:2)
And your procedure for updating the locked-away passwords is?
And your procedure for checking that the password hasn't been changed without notification is ?
Conceptually, it's simple. In practice, it's not so simple. That's why screens have bezels - for sticking the post-it to.
Re: (Score:2)
Slashdot wipes memory of recent posts? (Score:5, Insightful)
At least these two stories are from different perspectives: https://m.slashdot.org/story/3... [slashdot.org]
The story stays the same - don't fuck over your admins and have proper procedure and backup.
Good guys and bad guys (Score:2)
In a certain sort of movie (e.g. Mad Max, The Crow) the difference between good guys and bad guys is the order in which they commit their atrocities. In these two stories, good guys delete the data and then get fired, bad guys get fired and then delete the data.
Re: (Score:2)
There was a good guy in Mad Max? That's not the impression I got from seeing bits of it. Oh, I see - that's your point. OK : quod erat demonstratum.
Re: (Score:2)
The story stays the same - don't fuck over your admins and have proper procedure and backup.
There's a lot of lessons that management somehow never seem able to learn. Like, never have only 1 sysadmin, even if having more seems like overkill. Or, how about nurturing a relationship based on mutual respect and trust? The sysadmin has after all been trusted with one of the most important resource in any company, in most cases: the data. Yet, in a previous job I had an experience that I think is not uncommon: I had been there for over 10 years, I was regularly praised by my colleagues, but I was 'old'
Re: (Score:2)
The first story does not identify the company, other than saying that it's fairly large ("40+ devs"), but if that story is actually true, then it seems likely that the two are the same. Otherwise I think we would have heard from the customers of a second company affected by an outage.
I can't imagine any company being honest enough to put out a press release saying "We lost our customers' data because we put the root credentials of the production database in a training document, and we didn't have working ba
Re: (Score:2)
Whether or not these are the same story - the CTO should be fired (in both cases - this case).
Re: (Score:2)
The story stays the same - don't fuck over your admins and have proper procedure and backup.
Or alternatively -- don't trust admins, they can be stupid and / or assholes, and don't care about your business.
How do you "not trust" your admin?
You have to put a second admin there, who knows as much, and then trust THEM.
How do you trust the second admin? Maybe that person is the problem instead!
You could put a complicated system of checks and balances in place and reward following draconian rules and reward informing on someone who calls out incorrectness... But you still have to trust them.
The only practical way to increase the reliability of the meat based "admin system" is to keep the meat informed, happy,
Re: (Score:2)
So, Juvenal is a candidate for the Patron Saint of InfoSec? Since he was aware of this problem around 100AD. (He's the guy who came up with the question "Quis custodiet ipsos custodes? [wikipedia.org]" ("Who watches the watchers themselves?")
Backups? (Score:2)
Re:Backups? (Score:4, Insightful)
Why no secure backups?...
The article(s) seem to indicate that most, but not all, customer data can be recovered. So it seems there were working backups. But in a hosting environment, not everything is backed up continuously, and that may be where some of the data will be lost.
Takeaway (Score:3)
Re: (Score:2)
some admins belong in prison (Score:5, Insightful)
and this is obviously one of them. Criminals come from all walks of life, sysadmin isn't a position immune to containing the occasional bad apple.
So many questions of course, a lot of which boil down to "They must have had some serious lapse in procedure to have allowed this to happen." That's not really the case though. Back doors and logic bombs are serious threats when a person has been a trusted system administrator. Done "right", they can be extremely difficult to detect. It's a bit like the widely accepted advice of "Server was hacked? Don't try to clean it, you might miss something. You must wipe and reinstall it." (same really applies even to a home desktop) A departing admin (on bad OR good terms) is basically the exact same issue, a compromised system, but we only very rarely see such an extreme response. It's much less practical to nuke-n-pave when it's your entire network that is basically now classified as "compromised." Is this how we should respond? When you really stop and think about it, it starts to show itself as a really difficult question to answer. Rebuilding everything when an admin leaves when your system is large is just really hard to justify. But if your system is big, it's also more difficult to review it all and proclaim it "clean". It's just a bad position to be in, and that's why admin departures are such a headache. If you're big enough you have several admins and better compartmentalization of access, more robust isolation of systems, better logging, security software that's under the control of the CIO and not the admins, etc. They have a better chance, but it doesn't look like this one was big enough to have those benefits.
The lack of backups is the most troubling though. That's what stung the other recent post on the cleanout-from-inside. There's just no excuse for that.
This won't go will for hosting provider... (Score:2)
Re: (Score:2)
Yeah, I gotta admit I saw the, "current customers who are still interested in our services will receive compensation." and thought, "Yeah. The both of them."
Re: (Score:2)
But really, thanks for jumping in with your irrelevant anecdote, Captain Obvious.
You're welcome.
I'd laugh my ass off if your waste of perfectly good electrons disappeared from the internet.
The ad revenues I get from Slashdot readers visiting my websites over the last three months is enough to pay for six months of web hosting. But I'm putting the money towards three months of The Wall Street Journal instead.
Re: (Score:2)
Re: (Score:2)
Slashdot must really be going down hill then. I remember when a link from Slashdot meant a windfall. A HUGE windfall.
If I was doing this ten years ago, ad revenues from Slashdot traffic would have been in the hundreds of dollars.
Re: (Score:2)
Aren't there public libraries where you are?
I read the WSJ on my iPhone on the express bus to my tech job in Palo Alto.
Can't Recover (Score:2)
Can't recover? What did he do, dd if=/dev/zero of=/dev/fs ? Or were they using something like NTFS? Or most likely: storing the data in the CLOUD.
Pretty well every linux filesystem has recovery tools. There's a reason the POSIX term for "delete file" is "unlink". Because you aren't clearing the data, you're just unlinking from the table.
Since pretty well every file has a MAGIC at its start, it becomes fairly dooable to recover.
Re: (Score:2, Informative)
Hey dumb motherfucker, ever heard of a logic bomb? Or backdoors? If this guy went and deleted everything, what exactly makes you think that he didn't also backdoor everything or planted a logic bomb to delete it all.
Sounds like you are just as stupid as the guys who work at Verelox who think that just removing a account/passwords solves all security issues related to firing a sysadmin.
Firing a sysadmin is perhaps one of the most dangerous things a company can do.
Re: (Score:2)
And so we come to the crux of the problem: FEMALES.
This has nothing to do with male vs female admins. If anything, it's the other way around. In my experience, females 'accept' more abuse than male admins. And by that I mean, that the women I've worked with let it slide more often and longer and only decide to look for another job when really needed.
This whole case is nothing more than amateur hour of a three-man "hosting shop" without proper procedures. Just look at the whois information for verelox.com. The phone number listed is a Dutch cell phone numb
Re: (Score:2)
You know, /. really aught to prohibit modding and posting AC in the same article. Really? Modded down for sticking up for a female? I hope you and your other butt-buddy AC have a fun time tonight while packing each other.
Re: And... (Score:2)
Re: And... (Score:2)
Re: (Score:2)
Re:Not a big deal (Score:4, Insightful)
Nobody with a brain stores important data on someone elses server.
...without a backup.
Re:Not a big deal (Score:5, Interesting)
Nobody with a brain stores important data on someone elses server.
...without a backup.
You can wipe every single VM I have and I can restore everything within an hour because they are all configured using salt. The databases are snapshotted every hour and backed up using tarsnap as well as an rsync down to a NAS at my house.
I know I can do it in an hour because when Digital Ocean was having trouble at one of their data centers a few years back, I spun up new VMs and migrated everything to another data center.
Re: (Score:3)
You can wipe every single VM I have and I can restore everything within an hour because they are all configured using salt. The databases are snapshotted every hour and backed up using tarsnap as well as an rsync down to a NAS at my house.
You're the kind of guy I'd want running my IT department.
Re:Not a big deal (Score:5, Insightful)
I wouldn't hire a guy who copies all my data to his house.
Re:Not a big deal (Score:5, Informative)
I wouldn't hire a guy who copies all my data to his house.
Funny, it's data from *my* company. I'm the guy who *owns* the data. So why shouldn't back a copy up to my 12 TB storage array at my house?
If I worked for *your* company, I would back it up wherever *you* wanted it.
Re: (Score:2)
Re: (Score:2)
"In my ideal system, you have incremental backups to a disk, and routinely (weekly, monthly) do a full backup to tape, "
Why? A decent tape backup system(*)(**) will let you do nightly incrementals/differentials etc and ONLY restore the files you need if the time ever comes to do a full restore in anger.
Being able to do synthetic full backups is also handy when you can't afford the backup window on a particular server.
(*) Decent tape backup systems are backed with a database that keeps hashes of the files -
Re: (Score:2)
If you really own the company, what you do with it is your business. If you are working for someone else, moving data to your own home is potentially unethical and an option of last resort after you've presented better options for data archiving.
Agreed.
The instance for the source article is a former admin was the one who wiped all the data. So the better question is, at your company, are you alone completely capable of wiping all of your data, backups, and archiving? What is the company's procedure for data security if you leave or are terminated?
At my company (that I started and own), no one is capable of wiping all the backups.
If I leave or get 'terminated' (not sure how that works in a private business...death maybe?), it's probably in a situation where I've sold the company. It's up to the new company to implement their own backup and security policies and procedures.
This wasn't a regular employee, it was an admin with access privileges which weren't revoked in time, or had set up a killswitch prior to leaving. It's not the first time it's happened, and won't be the last.
Yup. A company I worked for 20 years ago had the admins start locking you out the moment you were pulled in to the HR meeting. They made sure the meeting lasted at lea
Re: (Score:2)
Depends on the guy and what "his house" actually means.
In my case it means another data center. Yes, I live in one. The air condition is awesome!
Re: (Score:2)
Re: (Score:2)
Because C level guys' behavior is always beyond repoach. True upstanding citizens? :)
LOL
Re: (Score:2)
An hour, if you happen to be awake and available.
Re: (Score:2)
An hour, if you happen to be awake and available.
Perhaps you've never heard of 'automation'? It allows you to perform actions 'automatically'. You can do wonderful things like launch backup jobs...
Re: (Score:2)
They're talking about the restore job. I'm not sure I'd want to automatically launch that...
Re: (Score:2)
They're talking about the restore job. I'm not sure I'd want to automatically launch that...
Restoring from tarsnap is a bit slow at times, but that's an "everything else has failed" contingency plan. I have monitoring and alerting, so if the DB mysteriously went away, I'd wake up, coordinate with the other ops guys, and start a restore.
Re: (Score:2)
Believe me, if I need that data, he WILL be awake and available! I do know where my coworkers live. And there isn't a single point of failure, meaning that there are always at least 2 people capable of doing any job, and one of them IS within reach.
No, we don't simply subject our workers to such conditions. We pay handsomely for that privilege.
Re: (Score:3)
there isn't a single point of failure
We pay handsomely
Are you hiring?
Re: (Score:2)
Actually we are. You'd be surprised, though, how hard it is to find suitable people. "No police record" and "years of IT security experience" is a combo that seems to be really rare...
Re: (Score:2)
You can wipe every single VM I have and I can restore everything within an hour because they are all configured using salt.
Any competently run IT department should be able to largely recover from the malicious actions of an external actor. But that's not really the question now, is it?
The question is: if you chose to destroy all of that data - including, I assume, your salt configurations - could someone else recover and rebuild those VMs - including their (reasonably recent) data?
Re: (Score:2)
You can wipe every single VM I have and I can restore everything within an hour because they are all configured using salt.
Any competently run IT department should be able to largely recover from the malicious actions of an external actor. But that's not really the question now, is it?
The question is: if you chose to destroy all of that data - including, I assume, your salt configurations - could someone else recover and rebuild those VMs - including their (reasonably recent) data?
Yeah--exactly. In my case, our salt configs are checked in to a git repo, so it's as 'simple' as spinning up a new salt master, cloning the config, configuring the master master from the salt config, then spinning up the other hosts and kicking off the config. Then restore the databases from backups. In my case they should be ~2 hours old at most. As long as you have access to the git repo, you can do it. Finding a competent admin that knows salt (or puppet or chef) with the skills required to spin up
Re: (Score:2)
I can't delete anything anyway. But nobody really can. You see, our backups cannot be deleted by the admins. "Deleting" here means that you mark it for deletion which is executed at a later moment by the storage ... thingamajig (don't ask me, storage really isn't my strong side). Now, marking a recent backup for deletion pretty much instantly hits some of the storage upper echelons in the face because that isn't proper procedure and he'll ask 5 minutes after the mark gets set and about 5 days before it actu
Re: (Score:2)
Re: (Score:3)
The databases are snapshotted every hour and backed up using tarsnap as well as an rsync down to a NAS at my house..
So, you only have one backup at one place? You're flirting with desaster.
Re: (Score:2)
The databases are snapshotted every hour and backed up using tarsnap as well as an rsync down to a NAS at my house..
So, you only have one backup at one place? You're flirting with desaster.
Nope. Backups happen in two ways. ZFS snapshot combined with a snapshot pull to my off-site NAS, and an 'autopostgresqlbackup' "snapshot" that gets backed up via 'tarnsnap' as well as rsynced to yet another off-site NAS.
;)
So there's a copy on the actual DB server, a snapshot on the DB server, a snapshot on my local NAS, a copy on another NAS in a different location, plus a tarsnap backup. I'm confident in my ability to restore. I've tested it.
Re: (Score:2)
Can you really guarantee that you'll be able to do that if another admin with equal rights to yours maliciously wipes data?
Because sure you have snapshots but a couple lines of Powershell/BASH and all of them are gone in 5 minutes. And you might have tape or cloud backups but another few commands and the tapes get zeroed overnight while cloud storage can be deprovisioned in seconds.
Re: (Score:2)
Can you really guarantee that you'll be able to do that if another admin with equal rights to yours maliciously wipes data?
Because sure you have snapshots but a couple lines of Powershell/BASH and all of them are gone in 5 minutes. And you might have tape or cloud backups but another few commands and the tapes get zeroed overnight while cloud storage can be deprovisioned in seconds.
Yes, because I have three personal policies related to this:
1. Only hire admins I feel comfortable absolutely trusting.
2. Follow the principle of 'least privilege' (I have backups on my storage NAS at home, and I am the only one with access to the data. A friend of mine has a similar storage setup at his house and he also has backups of the database that only *he* has access to.
3. Keep backups of your salt config somewhere where other malicious actors don't have access to it. (Salt config is stored o
Re: (Score:2)
Wait. So, now two people have full backups at their houses? I really, really hope that you don't deal with any data or servers that touch PCI-DSS, HIPPA, FERPA or other controlled data sets. I also hope you and your employee are never in a situation where you get laid off or fired. Having ALL that data in a personal residence will cause nothing but problems and could be extremely damaging to the company.
Re: (Score:2)
Wait. So, now two people have full backups at their houses? I really, really hope that you don't deal with any data or servers that touch PCI-DSS, HIPPA, FERPA or other controlled data sets. I also hope you and your employee are never in a situation where you get laid off or fired. Having ALL that data in a personal residence will cause nothing but problems and could be extremely damaging to the company.
Nope. I'm not a doctor, I don't run any sort of medical clinic, and I don't touch PCI-DSS data. Remembver the original topic? Wiping and spinning up new VMs in the cloud? Why would someone be storing PCI-DSS data 'in the cloud' on virtual machines?
Re: (Score:2)
I know people I absolutely trust. I also know they're going to screw up now and then. I'm also aware that I can be mistaken in my absolute trust, but I'm willing to take the chance.
Re: (Score:2)
"And you might have tape or cloud backups but another few commands and the tapes get zeroed overnight "
1: The backup server should be its own physical and administrative space, with its own security policies and be entirely separate from the backup clients (ie: only the backup admins needs access to the backup server.)
2: Tapes should be rotated frequently enough that the last full backup is ALWAYS in a data safe along with as close to current as you can get on differentials/incrementals. it's hard to zero a
Re: (Score:2)
Could you explain "salt?" It's new to me but from your context I should know it. (A link would be fine).
Hey, did you try googling 'salt'? ;)
I kid ... try salt devops though - you'll get what you need.
Re: (Score:2)
https://saltstack.com/ [saltstack.com]
Without going into the details SaltStack is similar to Ansible, Chef or Puppet.
Re: (Score:3)
Could you explain "salt?" It's new to me but from your context I should know it. (A link would be fine).
Yup--as others posted below, 'salt stack'. It's pretty much like 'Puppet', 'Chef', or 'Ansible'. Set up a 'salt mater' and point all your 'salt minions' to the master. Then you can define the way you want your systems to be configured from the master. i.e. things like disabling SSH password auth, deploying authorized SSH keys, configuring firewalls, cron jobs, packages installed, etc...
Re: (Score:3)
So it is enough to blow up your house.
Only if you didn't read and understand the entire comment.
Re: (Score:2)
They have backups and although they haven't recovered yet completely, the servers that are recovered haven't lost data.
Re: (Score:2)
You did a restore test, right?
Re: (Score:3)
without a backup
Actually, when I worked at a.... major transportation organization, I once accidentally deleted the entire database. It wasn't my fault, my code worked and was tested through dev -> stage -> test and all that.. but at the last minute my boss was like "Hey, you didn't use this cutting edge new ORM technique, refactor now" WHILE I WAS PATCHING IT TO PRODUCTION!!!!! So I bowed my head and said "yessir..." Well, what was supposed to delete one record ended up cascading to every related model and.... BAM.
Re: (Score:2)
Bossman made it break with his "strongly suggested" refactoring.
Sometimes you have to tell your boss "no". That's what written policies are for, to give you the power to say no. If he insists, you ask for it in writing (at least an email trail). I have told a boss no, but I explained why. He didn't insist so it didn't get to the "in writing" demand.
Re: (Score:2)
Years ago, I had a website on a shared hosting environment at a company. One day, their servers all went down and their support lines weren't responding. Eventually, they came back enough to explain that they had gotten hit with a worm that was going around and that their servers should be back up in a week or so. This raised major red flags with me because I knew that the fix for this work was 1) take the servers offline, 2) apply the patch, 3) reboot the server, 4) done. Even a large data center doing thi
Re: (Score:2)
Hang on ... think that through for a second. That suggests either that they're routinely using senior IT Admins on their tier-1 phone lines ("Hello customer, what can I help you with? ... HAND!"), OR they're using tier-1 phone line staff to do important admin work, OR ... does anyone else smell bullshit? (From the company involved, not the OP
Re: (Score:2)
And now my buddy just quit. So now I'm sysdamin, dba, developer, tester, etc.
Be very careful when you tell me I need to train my H1-B replacement.
Re: (Score:2)
Seconded.
And never forget: NEVER accept counteroffers!
Re: (Score:2)
Nope.
Anytime a counteroffer is on the table, you are faced with two choices:
1. A new job, with a brand new basis for all future raises.
2. The same old job, where you just 'extorted' a raise they didn't want to give you.
The best case for scenario #2 is they won't give you another decent raise until the next time you have an offer on the table. The things that really made you want to quit won't change.
Re: (Score:2)
The thing is that it seems they haven't lost any data. I bet the cloud did no harm.
Re: (Score:2)
Yes they did. Almost everything is recovered.
Re: (Score:2)
Libel laws are generally much stronger in Europe compared to the US. So yet another Anonymous idiot writing crap. So surprising...
Re: (Score:2)