Failed Win XP Upgrade Wipes Out UK Government Agency 731
Lurker McLurker writes "The BBC and the Register report that the UK Government's Department for Work and Pensions attempted to upgrade seven PCs from Windows 2000 to Windows XP, and ended up with BSODs on over 60,000 machines. I wonder if the National Health Service is regretting awarding Microsoft a £500 million contract now." The Guardian also has a good story.
Uh-oh... (Score:5, Interesting)
Another nail? (Score:1, Interesting)
When a government ends up with BSODs on 60000 computers, it can't be good for Microsoft.
On another note, How did upgrading seven machines to XP BSOD 60000?
umm.. (Score:2, Interesting)
If this was a Linux/Oracle/$flavourOfTheMonth upgrade, would you be just as scathing?
Go on, Slashdot, you know you want to. (Score:2, Interesting)
EDS managed upgrade--Altiris? (Score:5, Interesting)
It's a great set of tools--we own it at work and managed our own Win2k -> WinXP upgrade using the PC Transplant and Deployment Server tools, but can massively bone you if you don't do enough testing. PC Transplant, in particular, can hurt if you--that's the application that lifts your profile off of one PC and slaps it down on another, so that you don't have to re-configure your Exchange settings, Office personalizations, backup documents and application settings and bookmarks, and a whole mess of other things. When doing an OS migration, if you don't design your personality transplant template correctly, you can end up with all kinds of Win2k-specific settings stuffed into your WinXP profile, which can lead to all kinds of crazy-ass problems.
Not a nail for Microsoft. (Score:5, Interesting)
If you read the register article, it says that they were attempting to only push the update out to 7 PCs, but it actually went to all 60,000.
I would imagine they were using something like Microsofts SMS services or Bigfix to push out packages, and simply selected push out to all instead of a test community.
I don't think this is a nail in Microsofts coffin, I have seen similar things happen in the mainframe world where patches intended for dev hit live production systems with similar bad consequences. It has to count as a bad day at the office for the person pushing the button though.
It also highlights the difficulty in pushing out big updates to major networks of PCs, be they running Windows or Linux. The complexity of moving from Win NT to XP has proved so complex in my organisation that for the future Longhorn upgarde and beyond we are now looking to Citrix to allow the migrations of applications across servers and essentially use the PC as a thin client for all but core office and email apps.
TCO costs rise scarily with Windows XP failures? (Score:5, Interesting)
Assume £8/hr employee. 40 hours of work a week. 60,000 unusable systems.
=> TCO increased by £19.2m for the 8 PCs they upgraded (before costs incurred fixing the problem)! £2m TCO per system for Windows XP eh? A clear example that Windows TCO can increase rather horribly if something goes wrong, and this was a standard upgrade. It's £320 per PC if you count all 60,000 systems - that's still horrendous.
Re:EDS again (Score:5, Interesting)
Yet the government keep awarding them [EDS] contracts. Why?
I don't know, but I do recall an article about IBM refusing to tender for UK.gov contracts: apparently it was too costly, and too risky - you could spend millions only to not get the tender, and IBM felt that the chance of getting the tender awarded to IBM was too small. So... I'd suggest either it's too costly to play so players are dropping out (the reasonably answer), or someone in government really loves EDS, and IBM know it (the tinfoil hat answer).
Living in the UK, I'm minded to go for option 2.
what the hell went on??? (Score:3, Interesting)
Re:EDS again (Score:5, Interesting)
There is a huge hue and cry (outrageous exclamation of disgust and anger) over mismanagment and eggregious spending in government contracts. Having worked in the sector, I'm somewhat familiar. The contractor I worked for made sure there was no waste, fraud, or abuse. However, it spent 10 times as much as the job required, just to do this. The obvious choice for our firm was it would have been far cheaper to run things by GAAP (Generally Accepted Accounting Practices)(the private sector accounting standards), and have both a nice large internal audit division and "internal affairs" watchdog enforcement. Alas, most governments are not run this way, and if they are, they devolve into the current format due to political expediency.
I have friends that work for EDS and they comment on the kinds of hoops they have to jump through just to do simple stuff. They've built up a rather large experience pool in doing this hoop-jumping, so they can do contracts cheaper than some other companies.
EDS also tends to run things according to CMM levels whenever they're developing things, so at least if there's a mess-up (as there obviously was here), there will be some kind of follow-through to improve the process of doing this kind of work. EDS's management doesn't want the black eye any more than the government or Microsoft do, but they'll spend the money to make sure it doesn't happen the same way again. There is, after all, no way to prevent all errors, but I give them credit for trying most of the time.
Re:Another nail? (Score:4, Interesting)
You'd probably be retired now! Pity you chose long hair, and have another 40 years of work to go.
Re:Another nail? (Score:3, Interesting)
You'd probably be retired now! Pity you chose long hair, and have another 40 years of work to go.
I get to do cool stuff with UNIX nowadays. 40 years of cool stuff is better than becoming an EDS pointy-hair for 4 years and having to learn IBM JCL.
Knoppix (Score:2, Interesting)
Re:TCO costs rise scarily with Windows XP failures (Score:5, Interesting)
Re:RTFA! (Score:1, Interesting)
One of two things would happen:
- Nothing. Glibc since libc6 is forward and backward compatable.
- You would have to reboot the machines in single-user mode and, using all those shiny static binaries in
/sbin, you'd simply recover the correct Glibc.
I get your drift though. Just wanted to argue the details.Re:EDS again (Score:2, Interesting)
From this weeks Private Eye:-
"The government seeks outline bids with no detailed specifications for most PFI or PPP projects and then chooses its "preferred bidder". It is only then that the detailed contract is worked out. But with a "preferred bidder" identified most of the other contractors decide its not worth wasting any more money and drop out. The preferred bidder can then hold a gun to the government's head
(Subscription to Private Eye is only £21)
The most disturbing thing about EDS is that if David Blunkett ever gets ID cards through parliament EDS will be one of the front runners to run the scheme. Be afraid, be very afraid.
Really minor quibble with article title (Score:1, Interesting)
From the article (Score:2, Interesting)
Is it possible that someone noticed that the updates were going to 60,000 machines instead of just 7, said 'oh shit', and pulled the plug without thinking?
EDS is also thought to be flying in fire brigades.
Yeah, to put out the fires from their smouldering backsides.
Re:This is typical of our government. (Score:5, Interesting)
Only theregister appears to talk about Win2k and XP, so lets see what they're saying.
So if this is true then EDS pushed out a partial upgrade. Now come on, if you installed 75% of a new distro over an old one then rebooted would you blame Redhat because it didn't work?
Or there's the other version
So again EDS pushed out XP patches, overwriting Win2k files and the machines crashed
Not really surprising if you overwrite parts of an OS with files from a different OS that there is a mass crash, but folks, this is an EDS fuckup not really a problem with Windows.
Of course theregister could be wrong. It might happen. Heh.
Re:We need to educate the decision makers (Score:4, Interesting)
Is there another MP who's taken a clearer anti-ID card stance, and is prepared to discuss their positon so openly?
FAT CLIENT (Score:3, Interesting)
This is what happens when you have a fat client. There's a lot in a fat client. A lot to go wrong, a lot to be insecure. It therefore needs a lot of looking after. Many updates, many risks. Multiply by many desktops and it only becomes manageable by central updates. Central updates means lots of automation. Lots of automation means someone presses the wrong button and.. BANG.
But for the whole thing to go BSOD... now THAT is bad. It means you can't even back out. The reports I have seen imply that they had to nuke Windows or install stuff manually using some kind of recovery diskette... It's a disaster whichever way up you put it.
Would it have happened if they used Linux? Who knows. Linux is a complicated beastie too.
However, if they had used web apps or thin client for eveything then the issue might not have even come up.
It does make an interesting academic exercise to consider what would happen if the same screw-up hit other installations with many thousands of windows clients. Yes I am referring to the recently announced UK NHS (900,000 nodes) and US AirForce (500,000) Microsoft "wins".
I have seen NHS and DWP apps. Pretty basic stuff. Running these things on XP or W2000 is a bit of a hammer to crack a nut. The only earthly reason I can think of is the MS upgrade machine says they have to.
As a former subcontractor ... (Score:1, Interesting)
EDS (UK) is a true management culture, its all about statistics, statistics and damn statistics, regardless of the consequences.
To quote a former manager:
I know that if we replace these PCs now they won't work until we have the power cables next week, but we need to get them out
I guess as usual their development department had the 50% required sucess rate before launching something live.
It will be easy for them to roll back to the original install, I still have one of the ghost cd images around here somewhere if they need to borrow it.
Devil's Advocate (Score:1, Interesting)
Re:Not Microsofts fault, in this case (Score:5, Interesting)
My co-worker and I spent the next period AMAZED that Windows networking even worked at all. The system of domain controllers and WINS servers and browse lists and host files... it's too byzantine to be believed. There is, without doubt, a corporate network somewhere that could be comopletely undone by someone opening a wireless laptop in the wrong place at the wrong time. Add Windows XP and the attendant SP2 fun they're having and you get chaos.
Yes, those delightful folks at EDS are the chimps in this scenario, but Microsoft's products are definitely the defective Uzi. And I note that the BBC News article studiously avoided mentioning either of them. Hmm... Microsoft wouldn't be doing everything it can to tamp down this PR disaster, would it?
Naaah!
Long long time ago (Score:2, Interesting)
Re:With Linux, it would be harder to do ... (Score:3, Interesting)
Second, The machines wouldn't boot, therefore there is no way to run any kind of script to fix the problem, thus your third solution is likely what happened in this case as well. However, it takes some time to manually go to 60,000 machines and fix them, even if it only takes 5 minutes per machine.
The exact same thing would have happened with any OS, including Linux, had EDS decided to bypass the normal version checking tools and do things themselves, creating an unbootable system.
Re:All systems are prone to failure (Score:2, Interesting)
Oh, I must have missed just that commercial...
In all *nix systems I've seen, the root user has the ability to run "rm -rf
Although I agree on that the design of the particular infrastructure seems a bit unsecure when you that easily can break 60000 systems. What if some bored hacker gained access to that main account...
Re:All systems are prone to failure (Score:2, Interesting)
Microsoft builds very powerful software. Their Active Directory and Group Policy can do amazing things over huge numbers of computers. But if the guy in charge gives admin rights over the directory to some monkey that doesn't know how it works it's no different than giving root to the same monkey on your big Unix file server.
I'm quite familiar with the Microsoft recommendations for big projects like this. Claiming Microsoft would "sell a complex system [of this magnitude] with the claim idiots can administer it" is flat out false. They recommend highly trained proffesionals, testing in an offline environment and strict change control procedures. Don't let your personal prejudice cloud the facts: MS sells big systems to big customers with the understanding and recommendation that they'll have tallented staff to run them.... Just like every Unix house does.
TW
62 $ / per year / per pc (Score:1, Interesting)
I wish we could reverse the time and seeing this happening.