Failed Win XP Upgrade Wipes Out UK Government Agency 731
Lurker McLurker writes "The BBC and the Register report that the UK Government's Department for Work and Pensions attempted to upgrade seven PCs from Windows 2000 to Windows XP, and ended up with BSODs on over 60,000 machines. I wonder if the National Health Service is regretting awarding Microsoft a £500 million contract now." The Guardian also has a good story.
Too slow. (Score:4, Informative)
It's certainly bad PR for Microsoft though, perhaps this will serve as a wake-up call to other governments that "other options" are out there.
EDS again (Score:5, Informative)
For those not reading the article... (Score:5, Informative)
Jon.
RTFA! (Score:5, Informative)
Brilliant work on the part of EDS, trying to patch the wrong systems, lord only knows what can happen then.
You could force an XPSP2 onto a 2k machine... would you still blame Microsoft for it? That seems to be the case here, EDS screwed up, and of course it's Microsoft's fault in the eyes of
Wrong! (Score:5, Informative)
the UK Government's Department for Work and Pensions attempted to upgrade seven PCs from Windows 2000 to Windows XP, and ended up with BSODs on over 60,000 machines.
In actual fact, the Register quotes:
According to one, a limited network upgrade from Windows 2000 to Windows XP was taking place, but instead of this taking place on only a small number of the target machines, all the clients connected to the network received a partial, but fatal, 'upgrade.'
and then below it:
Another source says that the DWP was trialing Windows XP on a small number ("about seven") of machines. "EDS were going to apply a patch to these, unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 per cent of the Win2k desktops.
So, by merging them you get the following story:
There was a trial of seven PC's, instead of patching only those seven, the request to roll it out was accidently performed and every computer attempted to install a botched version of XP.
Somewhat slightly different to the Slashdot version wouldn't you say?
In addition, I'm pretty sure that if you accidently deployed a botched version of the linux kernel then it too would probably have a similar effect.
Corruption (Score:1, Informative)
The companies that get contracts (e.g. EDS and Capita) produce poor software and have a dismal record when it comes to requirements elicitation. This latter failing is a good cash cow. When inadequacies in a system are exposed the tax-payer has to fork out more.
Exactly what government ministers and senior civil servants are getting out of this mess is not clear. It's worth noting that board-level jobs after retirement is a favourite in the defence sector.
We need to educate the decision makers (Score:5, Informative)
I have found that many MPs when questioned on anything related to technology simply say that "it is a complex issue", which to me isn't good enough when such huge amounts of money and significant impact on people's lives is involved.
There is a huge contract that'll be up for grabbs soon - EDS are preparing themselves to manage the UK national identity database and identity card scheme [itsecurity.com]. This is one we could lobby our representatives on to ensure they do it right..
Where to have the debate where it might be read by those who mater:
Free service to fax your MP [faxyourmp.com]
Boris [boris-johnson.com]
Richard Allan [richardallan.org.uk]
Tom Watson [tom-watson.co.uk]
Shaun Woodward [shaunwoodward.com]
Citing the recent and ongoing failures such as that cited in the article, and the UK Child support agency's computer failure [bbc.co.uk]. as well as the NHS computer system UK [bbc.co.uk]
Re:TCO costs rise scarily with Windows XP failures (Score:3, Informative)
Re:We need to educate the decision makers (Score:5, Informative)
Free service to fax your MP
Can I take the opportunity to point out that faxyourmp is for UK citizens ONLY and should only be used to fax your own MP. It is not for international write-ins or mass lobbying.
Best wishes,
Mike.
Re:umm.. (Score:5, Informative)
The question about all of this that I am left with is, how did the patch even install? Microsoft has had sanity checking on their patches for ages, checking not only the Windows version, but even service pack levels and any other prerequisites. Ever tried installing a patch intended for IE6-SP1 over plain IE6 for example? I'm assuming that this is some custom patch rolled by EDS, rather than an official Microsoft one downloadable by all and sundry. Still, the story appears to have made it onto UK prime time news, so no doubt more details will emerge...
Re:Local Government Mod parent up! (Score:1, Informative)
It has the entire staff (that has been there the longest and knows networking) wishing they could just trash the MS button powered cruft and go back to a good old simple and faster vax unix system. The simple user friendly Unix guis are still yearned for by those who are in the know! Where do you want to go today Has become a slogan for suckers and foolish IT admins.
Re:We need to educate the decision makers (Score:5, Informative)
A Member of Parliament, i.e. a member of the UK's primary legistlative body. Each represents an individual constituancy (area), and the government is formed by senior MPs of the party which has a majorty in Parliament (usually).
-Chris
Come on! (Score:4, Informative)
1. The patch they tried to update with wasn't a complete one for an OS upgrade.
2. Then they deployed it to their entire network by mistake.
This interesting piece of information can be gathered by RTFA.
I wonder what would happen to, say, Linux boxes if they had 60,000 and they applied an incomplete kernel patch?
Maybe some... thing... would panic?
Re:We need to educate the decision makers (Score:2, Informative)
Re:You guys are amazing! (Score:5, Informative)
As for managing large networks of desktops, that's another very different matter. Not many people have high-level experience doing that.
My network, for example, is only thirty machines. Hardly huge. In fact, it gives me the opposite perspective on a lot of issues, because I find many of the large-site friendly features of Windows networks utterly useless for a small site, and no small-site friendly managability features to compensate.
Personally, I've trialled XP at work as a possible upgrade for our 9x machines, and come to the conclusion that it's not worth the pain. It might be good if you have the management tools, a dedicated test network, and an admin team dedicated to designing and rolling out updates. For small sites, however, it's pure hell. Even controlling how the clients update themselves is hard without an extra server to do the job. I also found accessible information for small-site management to be very thin on the ground.
We're now using thin clients for some of our network, and seeing very good results. Yes, they're Linux based - MS looked good until we figured in the CALs and the isssues with NT-based terminal server security. I'm far from floored by the results with Linux - the bugs, oh, the bugs, I'm drowning in stupid f***ing bugs. There's also more than a little totally retarded design, and the classic issues with no two apps having the same open/save dialog.
That said, for our basic users the results have been very good. They need little support, hardware and software costs are both low, and things generally run very smoothly. Trials with more demanding users aren't going as well (see above rant about bugs and bad design), but current development in the OS is addressing most of the issues I've run into and I expect to be able to move the 9x users across to the thin clients mid-late next year.
I do agree with you that managing a large collection of Linux desktops would probably be pure hell. It's awful to even think about, frankly, especially upgrades. *shudder*. My solution would be to simply not use desktops, but instead move most users to department level thin client services hanging off a redundant set of beefy servers. I'd use LDAP to store user and sytem information (yes, much like AD) as I currently do on my network. For many users, such a setup can be expected to work very well, and dramatically reduces the admin nightmare compared to Linux desktops. I also wouldn't even try to migrate all users to Linux - only basic users for whom it would work well, such as those who only need email, a browser, a word processor, and access to a couple of specific in-house apps.
As for migration - I can't possibly imagine how it could be done in a sane way. I suspect a lot of custom tools would have to be written, the migration would need to be a rolling one, and there would need to be a lot of staff on hand to handle glitches. That doesn't sound like fun to me.
The worst part of moving my users over to the thin clients was migrating their data and settings. That despite the fact that almost all of it was already on the servers, and their systems were pretty basic and very uniform. Doing it in a large company wouldn't be nice.
Re:TCO costs rise scarily with Windows XP failures (Score:3, Informative)
This sounds like they were pushing out the upgrade via SMS. Checking that the upgrade was on an appropriate system here would not have mattered since the upgrade path from win2k to WinXP is legitimate. This sounds more like sysadmins instead of applying to a custom collection applying to the "All Systems" container. The real question here is why are so many systems under one system and even better why did the sysadmins who did this application not check to ensure the advertisement was sent to the proper container.
EDS takes the blame for this, not MS.
Keyboard Infantry since 2002
Re:We need to educate the decision makers (Score:3, Informative)
Re:All systems are prone to failure (Score:3, Informative)
You can bollux up ANY operating system so it can't boot if you work hard enough.
Re:Hope they all loose their jobs tomorrow (Score:1, Informative)
This cock up is in fact down to a US co
Re:EDS managed upgrade--Altiris? (Score:1, Informative)
You -cannot- accidentally send software to all the machines on the network (by the way there are 160,000 plus, not 80,000. Article inaccuracies
And no, I will not be posting under my usual
Re:FAT CLIENT (Score:3, Informative)
Upgrades usually work (Score:2, Informative)
I say this as someone who has done hundreds, if not thousands of windows upgrades and Windows installs.
Re:Educated Chimps (Score:1, Informative)
Okay it is a big improvement on what went before in the Microsoft world, but these things are all way over the top. Heck NIS+ is way over the top for most admins and most networks, which is why many Unix sites stuck with NIS for distributed authentication for so long despite known major security flaws (its internal methods were a joke) - in practice it worked and was simple enough for admins to grasp fully what was going on. Not least we knew its limitation clearly and could design our networks to allow for its weaknesses.
Indeed it was possible to design variants that address the weaknesses, because the protocols were simple enough for admins to understand.
Sure Microsoft networking has great security features now, but it reminds me of that quote by Spafford about SSL (checks written in crayon on park benches, sent in security vans....). The whole things is way too complex for what 99% of the users wish to achieve, which is single sign-on, and delivering a small set of parameters.
Interestingly one of the people I worked with always pointed out the term "transparent" is used back to front in computing. The old systems were transparent because you could see the inner working, some modern systems are opaque because you can't.
A great example of what I mean is POP3 authentication. The old system used a login and password sent in plain text. It became popular, it worked (and works), and was easy for the admin to eavesdrop problems. Every admin knew the weakness - plain text passwords - and could address this in a variety of ways.
Some opted for POP3 over SSL (which Microsoft implements VERY badly BTW), this addressed the main weakness (plain text passwords, and the lesser one, no authentication of the server) and also gave some privacy to the messages, and there were still ways for the admin to intercept it server side and debug.
Others opted for complex authentication schemes that use zero information protocols, in general these merely managed to make the systems more difficult to "debug", and in one implementation the server side needs to keep a plain text copy of EVERY password, thus making compromise of such a system a gold mine for the cracker.
TRANSPARENCY matters - too much complexity can blind you to the obvious flaws in the approach - I'm afraid ADS is way too complex, and even where it isn't it is obfusicated by being a closed or proprietary technology.