Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
Windows Operating Systems Software Bug

Failed Win XP Upgrade Wipes Out UK Government Agency 731

Posted by michael
from the automatic-upgrades-deemed-harmful dept.
Lurker McLurker writes "The BBC and the Register report that the UK Government's Department for Work and Pensions attempted to upgrade seven PCs from Windows 2000 to Windows XP, and ended up with BSODs on over 60,000 machines. I wonder if the National Health Service is regretting awarding Microsoft a £500 million contract now." The Guardian also has a good story.
This discussion has been archived. No new comments can be posted.

Failed Win XP Upgrade Wipes Out UK Government Agency

Comments Filter:
  • by Anonymous Coward on Friday November 26, 2004 @10:31AM (#10924662)
    They wanted that new version of Internet Explorer with the fancy built-in pop-up blocker.
    • by mishmash (585101) on Friday November 26, 2004 @10:57AM (#10924946) Homepage
      We've got to stop this happening again, we've got to educate the people spending our money on huge computer systems which are prone to failure.

      I have found that many MPs when questioned on anything related to technology simply say that "it is a complex issue", which to me isn't good enough when such huge amounts of money and significant impact on people's lives is involved.

      There is a huge contract that'll be up for grabbs soon - EDS are preparing themselves to manage the UK national identity database and identity card scheme [itsecurity.com]. This is one we could lobby our representatives on to ensure they do it right..

      Where to have the debate where it might be read by those who mater:
      Free service to fax your MP [faxyourmp.com]

      Boris [boris-johnson.com]
      Richard Allan [richardallan.org.uk]
      Tom Watson [tom-watson.co.uk]
      Shaun Woodward [shaunwoodward.com]
      Citing the recent and ongoing failures such as that cited in the article, and the UK Child support agency's computer failure [bbc.co.uk]. as well as the NHS computer system UK [bbc.co.uk]

    • by b00m3rang (682108) on Friday November 26, 2004 @12:00PM (#10925522)
      Upgrades NEVER work! Not for Windows 95, 98, ME, 2000, XP, Longhorn, whatever! It will never be a good idea to try and replace a MS OS without doing a clean install.

      This is first day stuff.
  • Uh-oh... (Score:5, Interesting)

    by Dynamoo (527749) on Friday November 26, 2004 @10:32AM (#10924671) Homepage
    You know that sinking feeling when you've just pressed the wrong button...

    ..of course, it seems to be our friends EDS behind it, who are just great at making a mess of government contracts.. and then, the government just gives them another one.

    • Re:Uh-oh... (Score:5, Insightful)

      by Coryoth (254751) on Friday November 26, 2004 @10:56AM (#10924932) Homepage Journal
      Everyone needs a second chance. And a third. And a fourth. And a fifth. And a ...

      I'm sure the government has perfectly good reasons for continuing to hand contracts to EDS. It's just probably not a reason they want to tell you because it involves (bribery|nepotism|stupidity|all of the above)

      Jedidiah.
    • by dackroyd (468778)


      = The ohno second - That minuscule fraction of time in which you realize that you've just made a BIG mistake.
  • by bairy (755347) * on Friday November 26, 2004 @10:32AM (#10924676) Homepage
    If something is actually working right (and it's rare), change it!
    I can imagine it now
    Intern: "Sir, Microsoft have bought out Windows XP Service Pack 2. It's had numerous bug reports of dying pcs and software not working anymore. THIS is the time to upgrade to Windows XP, then upgrade to SP2 because windowsupdate won't stop bugging the hell out of us until we do!"
    Boss: "You mean we could cock something up, and it might not even be our fault for a change?! Lets pay someone vast amounts of money to do it!"

    The Gaurdian reports it was a week long outage. Now, I may be completely wrong here, but surely all they had to do was restore those pcs back to their previous Windows 2000 state using the daily backups they do... I mean, it's only common sense to do backups on such a critical syst...oh, wait, nevermind.

    </cynical>

    • by hattig (47930) on Friday November 26, 2004 @10:44AM (#10924801) Journal
      So ... 5 working days, 60,000 PCs (= 60,000 employees?)

      Assume £8/hr employee. 40 hours of work a week. 60,000 unusable systems.

      => TCO increased by £19.2m for the 8 PCs they upgraded (before costs incurred fixing the problem)! £2m TCO per system for Windows XP eh? A clear example that Windows TCO can increase rather horribly if something goes wrong, and this was a standard upgrade. It's £320 per PC if you count all 60,000 systems - that's still horrendous.
      • by turgid (580780) on Friday November 26, 2004 @10:56AM (#10924930) Journal
        The bean-counters will find a way of "writing off" this debacle so it doesn't show up in TCO. Not that I'm bitter and cynical or anything....

        I once knew a bean-counter (quite senior) on nearly 3 times my engineer's salary. He was sat there in front of a spreadsheet adding up a column of numbers on a pocket calculator.

        Welcome to the UK Public Sector. That was your tax money.

      • Read the article. EDS applied a patch intended to update 7 Windows XP boxes to 60,000 Windows 2000 machines. The TCO here applies to the contract to EDS, not the software. It's like saying that a prison guard intending to open one gate to let someone out accidentally opened all of the gates and then they blamed the door manufacturer.
        • by ultranova (717540) on Friday November 26, 2004 @11:11AM (#10925071)

          Read the article. EDS applied a patch intended to update 7 Windows XP boxes to 60,000 Windows 2000 machines. The TCO here applies to the contract to EDS, not the software.

          Yes. It's not like the upgrade could detect the version of the program it's being applied to, and only install if the version matches the version it is intended for. That is completely unheard of, and would be impossible technically.

          This was sarcasm, FYI.

          It's like saying that a prison guard intending to open one gate to let someone out accidentally opened all of the gates and then they blamed the door manufacturer.

          This situation is more analogous to a wrong signal causing the door to open and then jam. And yes, such a door manufacturer deserves to be blamed.

          • Read the article. EDS applied a patch intended to update 7 Windows XP boxes to 60,000 Windows 2000 machines. The TCO here applies to the contract to EDS, not the software.

            This sounds like they were pushing out the upgrade via SMS. Checking that the upgrade was on an appropriate system here would not have mattered since the upgrade path from win2k to WinXP is legitimate. This sounds more like sysadmins instead of applying to a custom collection applying to the "All Systems" container. The real question her
          • by malkavian (9512) on Friday November 26, 2004 @11:55AM (#10925475) Homepage
            They're probably using something like Novadigms's Radia. And instead of linking the correct 7 PCs, they linked to all of them (misconfigured group). In that case, it's not a case if installing a patch that is installed using the new mechanisms, the "Patch Manager" simply dumps the files to all the machines that connect up using it's client, and force an overwrite.

            Given, they should actually have an install script that checks the OS before it actually dumps the install package on there, but hey.

            Not normally an MS apologist, but this isn't really Microsoft's problem. It's the contracted company that made the update package failing to ascribe it to the right download group.

            So, the analogy. It's like some perfectly good system being installed, and someone presses the button marked 'open all doors' instead of simply open door 7.
            I don't see anyone really blaming the door manufacturer here (Microsoft or the contractors), although I'd hazard a guess that the person who skipped over the part of the process that said 'double check the groups you assign this patch to' will be sorely chastised...
            • Radia is a bloomin' pain in the neck, the last place I worked used Radia and it was horrible ( Radia it's self is possibly very nice and useful, it's the way it was implemented that was annoying ).

              Company polict stated that everyone should always turn off there PC's when they left for the day and you'd get moaned at if you didn't. The Radia team told everyone they must keep their PC's on at all times but this was never company policy.

              Every morning it would take 20mins or so for Radia to install all the ni
      • by speed-sf (721339) on Friday November 26, 2004 @11:06AM (#10925028)
        Something that makes me curious, you hear Ballmer lament about the lower TCO of windows. You hear the linux community shriek about it's lower TCO. The bottom line is really this, if your sysAdmins are less than competent and bugger up something like this which system would have a lower cost to recover? This is a really good thing to know when you are considering any enterprise system. Call it, TCCR (total cost of catastrophic recovery). Ballmer, Linux communities answer me this!
        • This is an interesting question. Most large companies have at least a few gloriously incompetent people who really should have gone long ago but for whatever reason haven't.

          However, I don't know any reports which consider Total Cost of Ownership Assuming Your IT Department Is A Bunch of Blathering Idiots. Most seem to assume a certain degree of competence.

        • FAT CLIENT (Score:3, Interesting)

          by carldot67 (678632)
          Some interesting views here, but I would contend that this was a screw-up waiting to happen because screw-up potential was built in to the setup. A sysadmin has pressed the button here for sure but I wouldnt be too hasty to point the finger.

          This is what happens when you have a fat client. There's a lot in a fat client. A lot to go wrong, a lot to be insecure. It therefore needs a lot of looking after. Many updates, many risks. Multiply by many desktops and it only becomes manageable by central updates. Ce
          • Re:FAT CLIENT (Score:3, Insightful)

            by adolfojp (730818)
            I personally like web services with rich clients. The business logic stays centralized and the people have a "better" experience on their desktops. I still don't understand the advantage that one would get by upgrading from W2K to WXP in a work environment.

            Cheers,
            Adolfo
    • by blowdart (31458) on Friday November 26, 2004 @11:32AM (#10925238) Homepage

      Only theregister appears to talk about Win2k and XP, so lets see what they're saying.

      According to one, a limited network upgrade from Windows 2000 to Windows XP was taking place, but instead of this taking place on only a small number of the target machines, all the clients connected to the network received a partial, but fatal, 'upgrade.'

      So if this is true then EDS pushed out a partial upgrade. Now come on, if you installed 75% of a new distro over an old one then rebooted would you blame Redhat because it didn't work?

      Or there's the other version

      DWP was trialing Windows XP on a small number ("about seven") of machines. "EDS were going to apply a patch to these, unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 per cent of the Win2k desktops.

      So again EDS pushed out XP patches, overwriting Win2k files and the machines crashed

      Not really surprising if you overwrite parts of an OS with files from a different OS that there is a mass crash, but folks, this is an EDS fuckup not really a problem with Windows.

      Of course theregister could be wrong. It might happen. Heh.

  • by Anonymous Coward on Friday November 26, 2004 @10:33AM (#10924683)
    OH SHI-
  • Too slow. (Score:4, Informative)

    by Lostie (772712) on Friday November 26, 2004 @10:33AM (#10924684)
    If only they had reached the conclusion hinted at in this BBC News article [bbc.co.uk] a year or two ago, this would not have happened.

    It's certainly bad PR for Microsoft though, perhaps this will serve as a wake-up call to other governments that "other options" are out there.
    • Re:Too slow. (Score:5, Insightful)

      by Apathetic1 (631198) on Friday November 26, 2004 @10:44AM (#10924803) Journal
      It was EDS that screwed it up. I can't say I'm surprised. For once I find it hard to blame Microsoft - rolling an XP patch out onto a Windows 2000 machine (or 60000) will have the predictable effect of hosing the system. Given what I know about EDS (I worked there for two summers) I don't think running Linux would have helped.
      • Re:Too slow. (Score:3, Insightful)

        by Sxooter (29722)
        Really, I'd expect Microsoft to have designed the two very different operating systems to NOT take each other's patches. It couldn't have been that hard to do, just toss some identifier in there somewhere in the file and if XP sees a win2k id or vice versa, refude to install the update.

        It's still Microsoft's fault, because they designed a system that accepts updates for the wrong system, and after that update is installed, it's damned near impossible to back it out. EDS has fault here too, but let's face
        • Re:Too slow. (Score:3, Insightful)

          by Apathetic1 (631198)
          Microsoft can't help it if you force an XP update onto a Win2K machine using an automated tool. Trying to manually install a patch onto the wrong operating system WILL fail as it should.

          Somebody else in the thread mentioned this - if you overwrite your Linux kernel with a botched version, your system's hosed. If you didn't keep a backup, it's damned near impossible to back it out.

          Nobody can protect an incompetent admin from him / herself.
    • Re:Too slow. (Score:5, Insightful)

      by Zocalo (252965) on Friday November 26, 2004 @10:52AM (#10924894) Homepage
      Actually, I tend to see this as potentially an opportunity for Microsoft to gain some excellent, and totally undeserved, *good* PR. The root cause of the problem seems to be that EDS erroneously pushed a Windows XP update out to Windows 2K desktops - hardly Microsoft's fault. Having got completely out of their depth (which isn't especially far out of the shallows given EDS' track record to date) EDS decided that it couldn't fix the problem and called in Microsoft.

      Now, assume Microsoft bails EDS out, and there is no reason why not, because you can bet they'll send a bunch of temps to every DWP office at EDS' expense if they have too. In a nutshell, Microsoft gets a PR coup: "We've just bailed out out a leading *cough* solution provider! Now imagine that had been, say, a Linux deployment... Who could EDS have called then?" Given the excellent grasp of PR, spin and FUD Microsoft has, I don't think this is going to help break the Microsoft stranglehold at all.

      • Re:Too slow. (Score:5, Insightful)

        by Asic Eng (193332) on Friday November 26, 2004 @11:04AM (#10925017)
        Now imagine that had been, say, a Linux deployment... Who could EDS have called then?

        They could have called Novell or IBM.

        Apart from that though - any setup can be screwed-up by an admin, no currently available OS can protect you from that. So for a TCO estimate at least we would have to look at the total loss due to screw-ups like this, and weigh them with the number of installations. Using a single data point can't be valid. That said, my gut feeling is that Linux provides considerably better TCO.

      • by Skuld-Chan (302449)
        Now imagine that had been, say, a Linux deployment... Who could EDS have called then?

        This is actually a really good question. One thing I've found in Linux support is much of the software (the new software raid as an example) isn't clearly documented and when you do run into serious problems beyond a few simple things to try people generally seem clueless - even very experienced people. I blame a lot of this on constantly moving support targets (the day you document one issue, and its solution there have
  • by Lispy (136512) on Friday November 26, 2004 @10:34AM (#10924692) Homepage
    But still I have to say it: "HAHA!"
  • EDS again (Score:5, Informative)

    by Hieronymus Howard (215725) * on Friday November 26, 2004 @10:34AM (#10924694)
    Every time I hear about a big government IT fuck-up it seems to be caused by EDS. Yet the government keep awarding them contracts. Why?
    • Re:EDS again (Score:5, Interesting)

      by I confirm I'm not a (720413) on Friday November 26, 2004 @10:44AM (#10924804) Journal

      Yet the government keep awarding them [EDS] contracts. Why?

      I don't know, but I do recall an article about IBM refusing to tender for UK.gov contracts: apparently it was too costly, and too risky - you could spend millions only to not get the tender, and IBM felt that the chance of getting the tender awarded to IBM was too small. So... I'd suggest either it's too costly to play so players are dropping out (the reasonably answer), or someone in government really loves EDS, and IBM know it (the tinfoil hat answer).

      Living in the UK, I'm minded to go for option 2.

    • Re:EDS again (Score:5, Insightful)

      by supersnail (106701) on Friday November 26, 2004 @10:47AM (#10924832)

      Because Accenture is the other choice!

      This sort of cockup would have been impossable with the ex Arther Anderson crowd. They would still be struggling to get the shrink wrap off the CDs without wrinkling thier suits.

      Seriously the problem is government procurement procedures. The contract goes to the lowest bidder and a record of past f****ups is not taken into account.

    • Re:EDS again (Score:5, Interesting)

      by justanyone (308934) on Friday November 26, 2004 @10:47AM (#10924834) Homepage Journal
      EDS is one of very few companies that will accept government contracts. US Gov'mt accounting requirements are onerous (hard to comply with) by any standard, so in order to compete for the contract, you have to have a huge team of accountants that know how to produce the kind of records and reports that the Government accounting office(s) expect.

      There is a huge hue and cry (outrageous exclamation of disgust and anger) over mismanagment and eggregious spending in government contracts. Having worked in the sector, I'm somewhat familiar. The contractor I worked for made sure there was no waste, fraud, or abuse. However, it spent 10 times as much as the job required, just to do this. The obvious choice for our firm was it would have been far cheaper to run things by GAAP (Generally Accepted Accounting Practices)(the private sector accounting standards), and have both a nice large internal audit division and "internal affairs" watchdog enforcement. Alas, most governments are not run this way, and if they are, they devolve into the current format due to political expediency.

      I have friends that work for EDS and they comment on the kinds of hoops they have to jump through just to do simple stuff. They've built up a rather large experience pool in doing this hoop-jumping, so they can do contracts cheaper than some other companies.

      EDS also tends to run things according to CMM levels whenever they're developing things, so at least if there's a mess-up (as there obviously was here), there will be some kind of follow-through to improve the process of doing this kind of work. EDS's management doesn't want the black eye any more than the government or Microsoft do, but they'll spend the money to make sure it doesn't happen the same way again. There is, after all, no way to prevent all errors, but I give them credit for trying most of the time.

      • Re:EDS again (Score:3, Insightful)

        by mpcooke3 (306161)
        I love the goverment tendering process it fills me with much amusement.

        The idea that you pay one company to come up with a box of requirements then send it out to tender, and get several boxes back from a few large companies like EDS. Then these get send off to the company contracted to deal with the subcontracting/tendering process. A haggling process commences between bunches of lawyers on both sides resulting in usually only one or two possibilities the cheapest one is then selected and fucks it up. Now
    • Re:EDS again (Score:3, Insightful)

      by SonicBurst (546373)
      I don't know anything about EDS, but I do know this: They could have done 100,000 projects/upgrades flawlessly and you'd never know it, but let them screw up once (or however many times this company has) and you'll never hear the end of it.

      Please realize that I'm not defending them. I'm just pointing out that, as someone who works in IT, management never sees it when things go flawlessly, but they will not hesitate to throw your ass to the wolves should something go wrong.
  • *sigh* (Score:3, Insightful)

    by turgid (580780) on Friday November 26, 2004 @10:34AM (#10924697) Journal
    The thing is, this sort of thing is expected and accepted by the UK public sector. They'll just find a scapegoat and keep on buying Microsoft. The sad thing is, that's my tax money.
  • Come on now (Score:5, Insightful)

    by Mick Ohrberg (744441) <mick.ohrberg@gma i l . com> on Friday November 26, 2004 @10:35AM (#10924704) Homepage Journal
    Incompentent admins can turn any minor upgrade to a catastrophic failure. Don't blame M$ for this one unless there are irrefutable proof that the admins did everything by the numbers.
    • by orasio (188021) on Friday November 26, 2004 @11:02AM (#10924994) Homepage
      Microsoft sells itself as easy to administer, what in management terms means that the systems are so /user friendly/ that any moron can administer them.
      So, admin stupidity can also be blamed on MS, it's part of the TCO studies that make the decision to buy MS.

      Aside from that, a point-and-click update cannot fail so miserably. A script made by the admin, of course should, because you can assume that someone smart (and bold) enoguh to make a little script should be responsible for their decisions. Some guy clicking checkboxes shouldn't be allowed by those means to break 60000 computers, through a /user friendly/ GUI program.
      GUIs for dummies should have enough checks to prevent such underiable effects, they have a sufficiently constrained domain to be able to do so. If the guy wanted to do a legal task that the tools dosnt' allow, he could always write some Visual Basic Script, and then he would be on his own. Bringing down an organization by mis-clicking checkboxes is responsability of the guy that provided the checkboxes, too.
  • ...is to fire the IT manager, or whoever sanctioned this method of implementation. How do you fall into such a trap?

    Next, they should setup a similar Linux environment and see how a similar upgrade would have gone. This should be done before the Linux zealots declare that theirs would have been any better.

    In the meantime, let M$ and its cronnies be prepared for the wrath of Linux promoters.

    Cb..

  • by BlackHawk-666 (560896) <ivan.hawkes@gmail.com> on Friday November 26, 2004 @10:35AM (#10924708) Homepage
    Every Desktop Shutdown.
  • Nooo! (Score:5, Funny)

    by Turn-X Alphonse (789240) on Friday November 26, 2004 @10:36AM (#10924722) Journal
    It's like a thousand solitaire players suddenly cried out in frustration and then silence...
  • by eltoyoboyo (750015) on Friday November 26, 2004 @10:37AM (#10924740) Journal
    From the Guardian article: "At this point there is no known solution or ETA"

    I RTFA and all I see is a money discussion, not a technical discussion. I would speculate that an SMS or Zenworks push or somthing similar which was supposed to be restriced to the 7 PC's went almost everywhere. It might be a fair bet that the remaining 20,000 might have been upgraded too if those people had been at work and turned on their computers. IT Computer management tools give the department much power, which could do plenty of damage in the wrong hands.
  • by JKR (198165) on Friday November 26, 2004 @10:38AM (#10924747)
    Apparently EDS attempted to do a test upgrade on a small network of 7 machines, but accidentally deployed it to all 80,000 machines instead. It's not clear that they'd tried it on any target machine, so it's entirely possible that EDS is to blame here...

    Jon.

  • by willith (218835) on Friday November 26, 2004 @10:39AM (#10924759) Homepage
    The BBC article mentions that EDS is responsible for the ugprade. They're partnered with Altiris [altiris.com], so I'd be willing to bet that the upgrade was carried out using the Altiris Client Management Suite [altiris.com].

    It's a great set of tools--we own it at work and managed our own Win2k -> WinXP upgrade using the PC Transplant and Deployment Server tools, but can massively bone you if you don't do enough testing. PC Transplant, in particular, can hurt if you--that's the application that lifts your profile off of one PC and slaps it down on another, so that you don't have to re-configure your Exchange settings, Office personalizations, backup documents and application settings and bookmarks, and a whole mess of other things. When doing an OS migration, if you don't design your personality transplant template correctly, you can end up with all kinds of Win2k-specific settings stuffed into your WinXP profile, which can lead to all kinds of crazy-ass problems.
  • RTFA! (Score:5, Informative)

    by DaHat (247651) on Friday November 26, 2004 @10:39AM (#10924761) Homepage
    From the article: Another source says that the DWP was trialing Windows XP on a small number ("about seven") of machines. "EDS were going to apply a patch to these, unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 per cent of the Win2k desktops. This patch caused the desktops to BSOD and made recovery rather tricky as they couldn't boot to pick any further patches or recalls. I gather that MS consultants have been flown in from the US to clear up the mess." EDS is also thought to be flying in fire brigades."

    Brilliant work on the part of EDS, trying to patch the wrong systems, lord only knows what can happen then.

    You could force an XPSP2 onto a 2k machine... would you still blame Microsoft for it? That seems to be the case here, EDS screwed up, and of course it's Microsoft's fault in the eyes of /.
  • Contractor (Score:3, Insightful)

    by HogGeek (456673) on Friday November 26, 2004 @10:41AM (#10924776)
    I know we all like to blame Microsoft when these types of things happen, bu this appears to be a major fubar by the EDS people.

    The installation and update of operating systems is so easy any more, a blind one armed monkey masturbating could do it.

    I've worked with EDS people, and the one armed monkey would be a godsend compared to most of them that I've had the "fortune" of working with...

  • by Waffle Iron (339739) on Friday November 26, 2004 @10:42AM (#10924793)
    Obviously these sysadmins were incompetent. Everybody knows that a BSOD is impossible under Windows XP. If they had simply upgraded the other 60,000 machines to XP first, and then updated these 7 problem systems, this whole problem would easily have been avoided.
  • Wrong! (Score:5, Informative)

    by Mr_Silver (213637) on Friday November 26, 2004 @10:44AM (#10924810)
    Bad Slashdot reporting again? Quote Slashdot:

    the UK Government's Department for Work and Pensions attempted to upgrade seven PCs from Windows 2000 to Windows XP, and ended up with BSODs on over 60,000 machines.

    In actual fact, the Register quotes:

    According to one, a limited network upgrade from Windows 2000 to Windows XP was taking place, but instead of this taking place on only a small number of the target machines, all the clients connected to the network received a partial, but fatal, 'upgrade.'

    and then below it:

    Another source says that the DWP was trialing Windows XP on a small number ("about seven") of machines. "EDS were going to apply a patch to these, unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 per cent of the Win2k desktops.

    So, by merging them you get the following story:

    There was a trial of seven PC's, instead of patching only those seven, the request to roll it out was accidently performed and every computer attempted to install a botched version of XP.

    Somewhat slightly different to the Slashdot version wouldn't you say?

    In addition, I'm pretty sure that if you accidently deployed a botched version of the linux kernel then it too would probably have a similar effect.

  • by swordgeek (112599) on Friday November 26, 2004 @10:45AM (#10924817) Journal
    They upgraded seven machines and 80,000 died? That sounds weird, but maybe they were the AD servers. Why then, on a small number of such critical boxes, didn't they just restore from backups?
  • by Lee_in_KC (816490) on Friday November 26, 2004 @10:48AM (#10924842)

    I wish I could take one of you Linux "experts" up on your idea. "Here, upgrade these 2000 PCs, all of which are from different manufacturers and different configurations, to Linux. I need it done in the off hours and I need everything to work like it did before.".

    *crickets*

    Of course someone will reply and say "ok!" knowing it won't happen. It's not because I don't have the ability to make that decision but it's because I know better than to get real information/insight about IT from most /. posters.

    It's painfully obvious that a scant few here actually have a clue about running a business that relies on IT. It's more than ripping CDs and DVDs kids. Sure, the company that did the mistake is at fault but the problem is not in the chosen OS, it's in the chosen technicians and management.

    • by rpozz (249652) on Friday November 26, 2004 @11:03AM (#10925004)
      different manufacturers and different configurations

      You know that (re)installing Windows on a large number of systems of different types, for example when an upgrade fails, is a total fucking nightmare, yes?

      At least Linux comes with 99% of drivers pre-installed. With Windows you have to find them on the net first, then find some way of getting them to the target system (because you don't have a NIC driver, remember?).
    • by Craig Ringer (302899) on Friday November 26, 2004 @11:42AM (#10925333) Homepage Journal
      Frankly, I think "a scant few" is pushing it ... despite the number of clueless morons, many here do have at least some idea what's going on and how to sensibly address some IT problems.

      As for managing large networks of desktops, that's another very different matter. Not many people have high-level experience doing that.

      My network, for example, is only thirty machines. Hardly huge. In fact, it gives me the opposite perspective on a lot of issues, because I find many of the large-site friendly features of Windows networks utterly useless for a small site, and no small-site friendly managability features to compensate.

      Personally, I've trialled XP at work as a possible upgrade for our 9x machines, and come to the conclusion that it's not worth the pain. It might be good if you have the management tools, a dedicated test network, and an admin team dedicated to designing and rolling out updates. For small sites, however, it's pure hell. Even controlling how the clients update themselves is hard without an extra server to do the job. I also found accessible information for small-site management to be very thin on the ground.

      We're now using thin clients for some of our network, and seeing very good results. Yes, they're Linux based - MS looked good until we figured in the CALs and the isssues with NT-based terminal server security. I'm far from floored by the results with Linux - the bugs, oh, the bugs, I'm drowning in stupid f***ing bugs. There's also more than a little totally retarded design, and the classic issues with no two apps having the same open/save dialog.

      That said, for our basic users the results have been very good. They need little support, hardware and software costs are both low, and things generally run very smoothly. Trials with more demanding users aren't going as well (see above rant about bugs and bad design), but current development in the OS is addressing most of the issues I've run into and I expect to be able to move the 9x users across to the thin clients mid-late next year.

      I do agree with you that managing a large collection of Linux desktops would probably be pure hell. It's awful to even think about, frankly, especially upgrades. *shudder*. My solution would be to simply not use desktops, but instead move most users to department level thin client services hanging off a redundant set of beefy servers. I'd use LDAP to store user and sytem information (yes, much like AD) as I currently do on my network. For many users, such a setup can be expected to work very well, and dramatically reduces the admin nightmare compared to Linux desktops. I also wouldn't even try to migrate all users to Linux - only basic users for whom it would work well, such as those who only need email, a browser, a word processor, and access to a couple of specific in-house apps.

      As for migration - I can't possibly imagine how it could be done in a sane way. I suspect a lot of custom tools would have to be written, the migration would need to be a rolling one, and there would need to be a lot of staff on hand to handle glitches. That doesn't sound like fun to me.

      The worst part of moving my users over to the thin clients was migrating their data and settings. That despite the fact that almost all of it was already on the servers, and their systems were pretty basic and very uniform. Doing it in a large company wouldn't be nice.
  • by t_allardyce (48447) on Friday November 26, 2004 @10:50AM (#10924862) Journal
    Im pretty embarrassed for my country right now. How the fuck did we go from technological pioneers to this? And its only the tip of the iceberg, what with Ken Livingstone's numerous stupid ideas, David Blunkett's insanity and the incompetence of 100's of 'IT' projects (hint: if its called an IT project it means its run by incompetent MCSEs and it will fail catastrophically leaving millions of people without a service or having planes crashing into the ground, time and time again) with tax money falling out of their pockets, fuck them! Why do these idiots get the contracts? What happened to all the competent people??
  • by WindSword (596780) on Friday November 26, 2004 @10:50AM (#10924869)
    was the government spokesperson. After the intro to this piece on Radio 4 this morning, her opening sentence was "Let me correct you, 20% of our workstations are functioning". Talk about a positive spin.
  • by joshsnow (551754) on Friday November 26, 2004 @10:52AM (#10924888) Journal
    from the reg article;

    "This patch caused the desktops to BSOD and made recovery rather tricky as they couldn't boot to pick any further patches or recalls. I gather that MS consultants have been flown in from the US to clear up the mess."

    So, even more of the money I pay in tax is being diverted to M$ then...
  • by erroneus (253617) on Friday November 26, 2004 @11:00AM (#10924969) Homepage
    I'm guessing that they were attempting to use some fancy MS too to perform some automatic upgrades/updates and while they intended for a limited number of machines to recieve the updates, they went and installed on all the machines that used the service... just a guess mind you but it would fit the circumstances as layed out in the articles. And I would suspect when you add WinXP components to a Win2k installation that bad things would happen.

    As much as I would like it to be, it doesn't seem like a "Microsoft" problem exactly and were a parallel Linux situation have happened I'm not sure anything less would have happened... well I guess it would have to depend on a number of things -- for example, if it were an RPM-using distro on the desktops and the wrong RPMs were sent to ALL machines instead of the select few, the machines for which the upgrades were unsuitable would have simply failed due to dependencies unless the --force option were used... okay I'm rambling now but basically, I don't see it as a Microsoft problem as much as I see it as a misuse of tools.

    The TCO of their MS installation just went up though... and they shouldn't exclude the cost of firing, hiring and retraining either.
  • Come on! (Score:4, Informative)

    by Jugalator (259273) on Friday November 26, 2004 @11:33AM (#10925243) Journal
    Jeez, sometimes Slashdot readers are blind and zealous like headless chickens...

    1. The patch they tried to update with wasn't a complete one for an OS upgrade.
    2. Then they deployed it to their entire network by mistake.

    This interesting piece of information can be gathered by RTFA.

    I wonder what would happen to, say, Linux boxes if they had 60,000 and they applied an incomplete kernel patch?
    Maybe some... thing... would panic?
  • by emrysk (787256) on Friday November 26, 2004 @11:34AM (#10925248)
    ...but is there any actual evidence is was a Microsoft error? I like bashing Windows as much as the next guy, but it seems this is at least as likely to be a huge fumble by the admins.
  • Windows? Or EDS? (Score:3, Insightful)

    by reverendslappy (672515) on Friday November 26, 2004 @12:06PM (#10925581)
    Without any specific details on the failure or what exactly happened, it seems like this is a huge admin error. My guess is they're using something like Altiris to do their builds, and if an admin were to accidentally "drop" the package meant for the the test group on to the production group, wham-o... every PC starts installing a build that probably isn't meant for them, and won't work. And you can imagine how that would go.

    As much as I'm sure the zealots among us would like to make this seem like a Windows failure, it looks like it's more of an example of how outsourcing leads to disconnected, incompetent, and unmotivated IT staff. And that, of course, leads to mishaps like this.

    Either way, if you work for a company that brings EDS in house in any way, drop your shit and run. And don't look back. The flash could be blinding.
  • by PingPongBoy (303994) on Friday November 26, 2004 @12:23PM (#10925736)
    She added that the emergency payments system was "working perfectly."

    Jones agreed, "I still have plenty of blank cheques. My pen is at room temperature."
  • by dunstan (97493) <dvavasour AT iee DOT org> on Friday November 26, 2004 @01:01PM (#10926049) Homepage
    Interesting set of threads ... "it's not Microsoft's fault that EDS pushed the update out wrongly".

    The fundamental error here is deep seated and architectural - they have 80,000 user interface devices which are stateful. By putting the wrong device on the desktop they have set this situation up.

    In the olden days when clerks in government agencies used green screens this problem wouldn't happen. If a green screen failed, it would be replaced as a FRU. Today's equivalent is something like a SunRay - the user interface device holds only enough configuration to bootstrap itself and, again, is a FRU.

    The situation at the DWP is different: the user interface device is a stateful device which holds configuration itself, and requires this configuration to be consistent before it gets enough connecticity to be remotely managed. The toolkits discussed, which are used to push config around these UI devices, are probably most excellent, but there should be no need for this sort of mularky.

    So while I don't necessarily blame Microsoft for this incident, I do blame them for creating a monoculture where this sort of architecture is deployed. I expect the trials underway in government using SunRay devices as the user interface will be watched with more interest after this debacle.

    A final question - how on earth do DWP recover 60,000 unbootable PCs?
  • by iminplaya (723125) <iminplaya.gmail@com> on Friday November 26, 2004 @04:53PM (#10927478) Journal
    It's much more reliable to back up your data and do a fresh install. I experimant with upgrades, but even(or especially) with linux, I prefer to clean the disk and start fresh. Apple on the other hand(before OS X anyway, don't know if it still is) was great. It would just create a clean new system folder. With the old one still there, I could just "bless" it if necessary. Oh, well...There's still nothing more trustworthy than pen and paper, and a good ol' mimeograph machine(the hand crank variety) for makin' copies...And they smell great.
  • by horza (87255) on Friday November 26, 2004 @07:59PM (#10928393) Homepage
    It's well known that EDS are incompetant and unprofessional, costing UK taxpayers hundreds millions of pounds. Examples include tax [computerweekly.com], welfare [greatreporter.com]and air safety [computerweekly.com]. In fact they seem to be awarded contracts by default despite not a single success with projects running hundreds of millions over budget and those that aren't a couple of years late are junked as a massive write-off.

    It's well known that the UK government are in the pocket of EDS and Microsoft. The worst thing is that it's not intentional. The people in charge of making these decisions are complete non-techies and haven't heard of any IT company that aren't a regular in the new headlines of the FT. It's not corruption, it's basically a lack of education.

    Phillip.

No user-servicable parts inside. Refer to qualified service personnel.

Working...