Slashdot Log In
Massive VMware Bug Shuts Systems Down
Posted by
CmdrTaco
on Tue Aug 12, 2008 08:49 AM
from the at-least-it-only-shut-down-the-virtual-ones dept.
from the at-least-it-only-shut-down-the-virtual-ones dept.
mattmarlowe writes "Imagine if Red Hat released a version of Linux, and after it was deployed, customers noticed that any processes with a start date of today would refuse to run? Well, that's what happened to VMware — a company that wants nearly all server applications running in virtual machines within a matter of years." Supposedly a fix will be available ... in 36 hours.
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
License Management Software!? (Score:5, Insightful)
If someone wants to run virtual machines at home or in a small business, they're likely going to be more than satisfied with VMWare Virtual Server (formerly GSX) and wouldn't even consider the much more complex ESX.
In a major corporation, fear of massive fines and prosecution is enough to stop them from pirating your software. Hardware dongles, software license managers and the like only hurt your paying customers.
Re:License Management Software!? (Score:5, Interesting)
The call home variety is extremely infuriating. On top of whatever nonsense key/activation crap you have to go through, you have to put up with it trying to call home or deactivating itself. MS isn't the only guilty party in this, but those bastards certainly made the situation much worse.
Parent
Re:License Management Software!? (Score:5, Insightful)
Good god do I hear you, brother. I work IT for a legal firm. So many little apps no one else in IT has ever even heard of. And most of them, you're talking to the same guy for support that developed it, and filled the sales order. Out of his basement or garage. Multi-million dollar a year law firm, and it can be brought to its knees if one of our obscure applications goes down and needs support, and the one guy that can support it is out taking his kids to soccer practice.
I'm looking at you North Winds Software. I'll BUY a support contract! If you offered such a thing. If you answered the phone.
I need to go back to bed. :(
Parent
Re:License Management Software!? (Score:5, Insightful)
I'm looking at you North Winds Software. I'll BUY a support contract! If you offered such a thing. If you answered the phone.
There's an Ask Slashdot for you. Is there something out there that can replace this magic bit of software? Is anyone interested in writing an Open-Source equivalent?
Parent
Re:License Management Software!? (Score:4, Interesting)
The more important variant of that question is does the parent want to share enough of the details of operation (clean room style) to get someone to want to write an OS equiv.
Don't misunderstand me, I like to write code, but if I don't know what the hole looks like, I can't carve a peg to fit it...
Parent
Re:License Management Software!? (Score:5, Insightful)
The problem is that every industry has a few of these super-specialized vertical apps that come from one-guy software companies. Most of them are fairly simplistic Access/FoxPro type things, the hard part was implementing all of the business rules.
I've worked with a few companies that recreated their software package in-house (because they needed specific customization the author wouldn't provide), and it's never as cheap or easy as it might seem superficially.
Parent
Re:License Management Software!? (Score:4, Informative)
What is a "disadvantaged business" anyway, and why would someone actually use that as a sales point?
Government work. Some government contracts require a percentage of the work to be done by minority/women/veteran/disadvantaged owned businesses.
Parent
Re:License Management Software!? (Score:4, Funny)
Government work. Some government contracts require a percentage of the work to be done by minority/women/veteran/disadvantaged owned businesses.
There aren't enough minority/women/veteran/disadvantaged heart surgeons out there! I demand that we establish hiring quotas for all! There needs to be ethnic & cultural diversity within the heart surgeon community!! :)
Parent
Re:License Management Software!? (Score:5, Insightful)
and as far as everyone else in the business is concerned, any failings in the product is the IT department's problem not theirs
This is true, and particularly frustrating. We recently have converted from an (old, but very functional and stable) 20+ year old COBOL program to a new Windows application in our organization. This is a Visual Basic application that if I'm being kind I'd say is a kludge held together by the electronic equivalent of duct tape and glue. The thing is junk and crashes ALL THE TIME. IT didn't pick this app though - we just get stuck supporting it. However, no amount of explanation can convince these people that the program crashing is not IT's fault. We can reinstall it as many times as they ask for it. We can update everything on their computer. We can buy them a new computer. But the basic fact is the program you bought is crap and full of bugs and nothing IT does is going to make it stop crashing and screwing up data.
Sadly, this is a hard fact to make users accept.
Parent
Re:License Management Software!? (Score:5, Funny)
I'm looking at you North Winds Software. I'll BUY a support contract! If you offered such a thing. If you answered the phone.
North Winds Software? Just a WILD guess... is this 'software' based on MS Access? I wonder where they got the company name from...
Parent
Re:License Management Software!? (Score:5, Informative)
I'm looking at you North Winds Software. I'll BUY a support contract! If you offered such a thing. If you answered the phone.
Um, isn't North Winds the name of the company that comes with the sample Access database? They're not real, you know... ;)
Parent
Re:License Management Software!? (Score:5, Funny)
Well, that explains the lack of support.
Parent
Re:License Management Software!? (Score:5, Interesting)
I'm on the other side of the divide---a tiny company that's not too much more than a guy in his garage (just a few of us), and frankly, I agree with you. I'm astonished at the way we do things, even though we sell to huge firms (including big law firms, like yours). Part of it is just size---we don't have the people or skills to do all the safety, security and support steps a big corporation would. Still, freaks me out that the crap I wrote is out there being used to do important things by important people who don't realize how dumb the guy who wrote their software actually is.
Parent
Re:License Management Software!? (Score:4, Insightful)
Parent
Re:License Management Software!? (Score:4, Insightful)
Exactly. Most large companies usually have an entire person, and sometimes multiple people dedicated to nothing but license management.
What a colossal waste of money.
Parent
Re:License Management Software!? (Score:4, Insightful)
I really don't think the fines would keep large corporations in line. look at all the stuff you see big business doing that they know is illegal and that they know will land them big fines if they get caught. Software piracy is no different. In fact it's probably easier to use a pirated piece of software than it is to dump illegal chemicals or defraud investors. You can manage the exposer.
Parent
Re:License Management Software!? (Score:4, Insightful)
I really don't think the fines would keep large corporations in line. look at all the stuff you see big business doing that they know is illegal and that they know will land them big fines if they get caught. Software piracy is no different. In fact it's probably easier to use a pirated piece of software than it is to dump illegal chemicals or defraud investors. You can manage the exposer.
Having acted in an advising capacity on a software license management project currently underway at one of the worlds largest financial institutions (400k employees), I disagree. Purchased software is an asset on the books and needs to be tracked. Pirated software is a risk and even the largest companies will occasionally be brought to court for "over implementation."
The main hurdle with Software Asset Management (SAM) is the complexity of the licenses involved, and the multitude of way in which it can be obtained. Some examples: is the license perpetual or subscription based; is it a "named user" license or is it assigned to the org; does it include maintenance (upgrade rights); if it includes maintenance is the maint co-termed with the other licenses that the org owns; if it includes maintenance, what was the most current version at the time the maintenance expired; does the current version allow for "downgrades" and how many version prior can be downgraded; what previous versions qualify for an upgrade license and which would need a full new version; can the licenses be transferred within the org; can they be transferred globally; does the license allow for home use; does the license allow for portable device use; just to name a few.
If large corporations were willing pirates, you would not see them making their annual multi-million dollar payments to Microsoft for their Enterprise Agreements. You wouldn't see them spending millions on risk management/mitigation consultants or conducting their own software audits. There are people out there getting paid piles of cash to implement a working SAM system.
It's unavoidable that a large corporation will be under-licensed. However, they spend big bucks to mitigate the risk that this opens them up to.
Parent
Re:License Management Software!? (Score:5, Interesting)
Having administered ESX, I can say the license management is useful for one thing: it helps you ensure you aren't exceeding what you're licensed for. For example, if you aren't licensed for multi-processor boxes, it will complain until you get a valid license. If nothing else, it gives you some confidence that you will pass an audit.
License management is also useful for things like MATLAB and OPNET that are licensed per concurrent user: you can install on as many machines as you like, but they need to be able to talk to your license server (not that this is _your_ license server on your network - it isn't "calling home") to ensure that the number of concurrent users is below the maximum allowed. That way, if say, everyone needs to be able to run OPNET occasionally, but not very often, everyone can install it, but you only need to pay for a few licenses. You know you aren't exceeding your licenses because it won't let you launch more instances than you're allowed simultaneously. If your users regularly complain that they can't fire up OPNET due to lack of licenses, you pay for a few more seats.
On the other hand, I can't stand software that calls home to ensure that it's "genuine" a la Windows Vista, or those stupid CD copy protection schemes. That's bullshit. Things like that make more work for a sysadmin, not less. I only like license management when it helps me, the admin; I don't care what it does or doesn't do for the software vendor. I'm a selfish pig, I know.
Another thing I can't stand is things like Rational Purify where they attempt to count your "activations" at their end: when you install Purify, it increases the installed count in IBM's system, and decreases it when you uninstall. If the IBM server thinks you're using all your licenses, you can't install. Too bad people always forget to uninstall Purify before wiping their computers for a clean OS install (or scrapping the computers)! And don't get me started on how bad it is to deal with IBM's phone support. This is one copy protection scheme that I do bypass: I install Purify in a VMware virtual machine, snapshot it, uninstall Purify, and roll the virtual machine back to the snapshot. That way, Purify will work in the virtual machine, but IBM's servers will think I haven't used any of my licenses. Also, I can make copies of the virtual machine for multiple people to use. It's easier for me to track the licences than put up with a crap license management scheme.
Parent
Re:License Management Software!? (Score:5, Insightful)
Actually its quite a common policy in MegaCorps to reject software that require machine specific or expiring license keys for use in "Mission Critical" applications.
The backup server not having the correct licenses is one of the biggest risks in a Disaster Recovery.
Migration to newer better hardware also becomes a nightmare where license keys are involved -- what do you mean the new server doesnt have centronics port for the dongle?
Its also screws up the companys virtualisation strategy as you have no idea whether a given license scheme will work in inside a VM or not.
Do like the Fortune 500 and just say no to runtime licenses.
Parent
Re:License Management Software!? (Score:4, Informative)
Not true. I worked for a smallish software companies that had their software replicated in at least one large customer installation well beyond the number of seats that were actually paid for. When confronted, the reaction was "so sue us..." We eventually settled for about 1/10 of what we would have made if they had obeyed the license terms because the cost of litigation coupled with the delay tactics they could have used would have meant that we would be out of business long before the court case was over. Size just means that they have more resources to defend their slimy actions.
Parent
Can't start processes? (Score:5, Funny)
any processes with a start date of today would refuse to run? Supposedly a fix will be available... in 36 hours.
Good thing the fix will be available tomorrow, because if it was available today nobody would be able to run the update process
what do you expect? (Score:5, Insightful)
I stick to virtualbox. I'm not going to pretend I've audited the source code, but if I need to, I can.
Say YES to freedom.
Re:what do you expect? (Score:4, Interesting)
Then give me USB support in VirtualBox. Cause I kinda need that the most.
Parent
Re:what do you expect? (Score:4, Insightful)
I'd rather have better video support than USB support so that you play games in a virtual machine (and by games, I mean games beyond 2D games from the early 90's). If a virtual machine would support something like DirectX or OpenGL so that I could have the kids running their games in a virtual machine (and being able to install them, etc.) I would have them set up with a locked down OS with a virtual system for their games.
There are some options, but they haven't been successful for me yet. But I'm sure the technology is getting closer.
Layne
Parent
Re:what do you expect? (Score:5, Informative)
USB license dongle for the application software running on the VM.
Seriously. Last week.
Parent
NOT the *OpenSource Edition* (Score:4, Informative)
Support for USB, iSCSI and RDP (along with USB-over-RDP) are only available in the closed source variants of VirtualBox.
The opensource edition of Virtual Box doesn't have them.
Also the USB support may lock the system when in fast emulation/patching/ring-2 mode, and only works flawlessly when using the slower mode with virtualisation CPU extensions (my brother tried using it to get old USB hardware accessible when moving to Vista 64 but since then he ended up buying newer hardware)
Parent
Re:what do you expect? (Score:4, Insightful)
http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf [cmu.edu]
What if you can't even trust your compiler? At some point, even with fully open, GPL-compliant software, there is some point you just have to trust someone else to not jack you.
Parent
Re:what do you expect? (Score:4, Insightful)
> What if you can't even trust your compiler?
You are referring to "Reflections on Trusting Trust" I assume. That is not really a practical attack in the real world.
> At some point, even with fully open, GPL-compliant software, there is some point you
> just have to trust someone else to not jack you.
A supplier of Free Software can never be sure that someone he doesn't even know about let alone control will decide to review his source code.
Parent
Workaround available (Score:4, Informative)
Re:Workaround available (Score:5, Informative)
Parent
Re:Workaround available (Score:5, Interesting)
Troubleshooting that one was fun.
Parent
Re:Workaround available (Score:5, Informative)
1) edit the vmware config file to include the line:
host.cpukHz = XXXXX
Where XXXXXis your CPU in kilohertz
2) enable time sync via vmware tools
3) modify Type in HKLM\System\CurrentControlSet\Services\W32Time\Parameters from "Nt5DS" to "NoSync"
That's always taken care of clock skew issues on DCs for me.
Parent
No! Don't set the time back! (Score:5, Informative)
VMware is suggesting setting the system time backwards to work around their license manager problem. That's a desperation move. Not only will it mess up everything from Kerberos to CVS to "make", if you're running certain licensed software, in particular software licensed via FlexLM, that software will stop working. FlexLM will disable your licenses if the clock goes backwards by more than 24 hours. Now your expensive high-end software protected by FlexLM (Rational, Avid, Matlab, National Instruments, ANSYS, Cisco Unity, Clearcase, Nokia network management, etc.) will stop working. Setting the clock forward again may not re-enable it, either; there's tamper detection.
Also, if you have server/client licensing with FlexLM, or multiple license servers, and the clocks disagree significantly, FlexLM gets suspicious and turns licenses off.
Parent
My head hurts. (Score:5, Funny)
My head hurts reading that article. Who the fuck wrote it? A ten year old mental retard?
It's like ............... this and VM's this VM's that (Yes, notice the spelling?). Ooooh and the cyberwarfare boogeyman! You can't even find this much Hollywood scenario fear mongering from Hollywood themselves. Oh noes! Our entire infrastructure will be killed by evil cyber terrorists because it runs on VMware!
Oh and and lovely parts like 'w/' instead of 'with'. Hey douchebag, this is not SMS, is it so hard to hit another 2 keys on your keyboard? Oh and for the love of $DEITY$, please learn basic HTML and use links so I don't have to copy paste text into the address bar.
As for Slashdot editors, why the fuck did they pick the worse possible article from the Firehose when plenty others look *WAY* more professional?
Re:My head hurts. (Score:5, Funny)
Even worse, he got the meme wrong. The title of the blog post should have been "All your VM are belong to us". Idiot.
Parent
Re:My head hurts. (Score:5, Funny)
for the love of $DEITY$
That's either $DEITY or %DEITY%, please learn basic shell scripting for your platform :)
Morale: if you're gonna rant, make sure you do not make the same mistakes as the target of your rant
Parent
Re:My head hurts. (Score:5, Funny)
That's moral, which is a lesson to be learned. Morale refers to high spirits, or lack thereof, as in "his morale was crushed when he realized his error in verbiage".
Parent
Yes, it is a bug (Score:5, Insightful)
But the real bug is license enforcement in the first place. Why would you run the risk of making your business depend on the whims of someone else's IP policies and enforcement?
Now, I'm somewhat realistic. I know that there isn't (yet) an adequate replacement for every piece of closed proprietary software out there. But for my own business (admittedly small) I am building with nothing but GPL/BSD/Apache license code. And it is working. I don't trust closed code. Of course my software will have bugs, some of them serious. But I won't have stuff shutting down because of "license" issues. Why do people go quietly into enforced licenses? Why do people accept remote kill switches on their servers? Why doesn't this strike everyone as a crazy thing to do?
Re:Yes, it is a bug (Score:4, Insightful)
And when that happen? Who cares, I'll just sue their asses, like I do whenever OTHER problems come up, and it works to recover losses
No you won't. For essentially any software product available on today's market, during installation you agree to waive your rights to recover any losses beyond the purchase price.
Parent
Utility computing w/o virtualization (Score:4, Insightful)
VMWare licenses for ESX server cost something like $5k apiece. My company uses VMWare and I don't quite get it. We pay for expensive blade hardware ($8k each for those, not to mention the chasis), then we pay $5k per virtual server. And for what? Adding virtualization overhead to the runtime cost.
Meanwhile, in articles like this [markturansky.com], people are showing how to run many applications and different versions within a single container. A single node in the cluster can run any application. There are always busy, keeping the hardware fully utilized. Isn't that the promise of utility computing? Rack up a bunch of cheaper (but not cheap/shoddy) servers and let your cluster go to town.
So, my question is, why are we (as an industry) embracing virtualization when apps written for a smart container (like OSGi) give the same benefits without all the additional co$t and runtime overhead?
Re:Utility computing w/o virtualization (Score:4, Informative)
Isolation and easy management.
Isolation of applications in OSGi containers is leaky, one bad-behaving application can bring down the whole containers.
Lightweight containers (OpenVZ, Virtuozzo) have almost no overhead and allow cool features like load-balancing of ALL applications between cluster nodes. However, all lightweight containers use the same kernel, and one kernel bug can bring down all virtual nodes.
XEN/KVM have a bit more overhead but with even more isolation (each node has its own kernel).
Parent
Re:Utility computing w/o virtualization (Score:5, Informative)
Simple...power. Right now our datacenter is strapped for power, and power isn't cheap. Neither is cooling. For 10U and 8000 watts I can install a fully loaded blade chassis with 128 CPU cores and 1 Terabyte of RAM, attach it to a SAN and run 150 VMs in it. Or I can install 150 rack and stack servers at taking up 4 racks and 75000 watts. Let me think here...
And while I'm thinking about it, let's also remember that using VMWare gives you options like DRS and VMotion that you don't get with physical hardware. Or you can replicate your SAN to another SAN at your DR site and have a VMWare cluster waiting there for recovery. Then instead of having to do a bunch of restores to bare metal hardware, you could potentially get your servers back up and running in minutes instead of hours.
There are many, many benefits to virtualization. If there weren't then people wouldn't have been using for decades in one form or another.
Parent
"License management code..." (Score:5, Insightful)
...Says it all, I think. Perhaps you should reconsider the ramifications of making your business critically dependent on software that contains code specifically design to make it stop working.
Consider this: to a proprietary vendor the only safe failure mode for "license management code" is one where everything stops.
Patch Tuesday (Score:5, Interesting)
Um, isn't today Patch Tuesday? [wikipedia.org] This could be worse than we thought.
Re:Patch Tuesday (Score:5, Informative)
Rebooting a host doesn't power down the VM.
The licence checking is done at VM power up, apparently.
Parent
License Management Server (Score:4, Informative)
KVM and XEN (Score:5, Interesting)
The Open Source Model gets a leg up again after this nonsense. A client of mine just ported all their VMs and said good bye to VMware. That's 280 VMs by the way. Thank God we had a contingency plan for switching VM providers for a DR exercise a year ago and here we go.
Management is pretty upset and I doubt we will be switching back any time soon to VMWare products after this.
On a side note this scenario did prove one thing:
Having a VM-agnostic storage makes migration easy. We changed a mount point, powered on the alternate VM host and we were off and running just that quick. We lost the ability to do live migrations for now but beyond that is was a good opporunity to see just how important an VM-agnostic disk storage array is. (I'm not the admin of those machines but I believe we are using iSCSI).
On my side though I had about 50 scripts tapping VMWare via PERL but I guess I can start building workarounds now... No more batch submission and dynamic routing for a week or two... The part I hate the most was I had a nice script to take a batch submission and if necessary migrate a utility node to bigger hardware to accomidate the batch... pisses me off but what can I do, thank you Vmware, that aquisition seems to be improving your product as much as when Symantec aquired Ghost Corp!
Re:Ummm... How? (Score:5, Informative)
If you read the article, you'd know it's the license-management code. Licenses expire.
Parent
Re:Ummm... How? (Score:4, Insightful)
Parent
Re:Nice.. (Score:4, Funny)
You know that you should read ./ before you do any actual work. Don't you?
:)
Parent