The London Stock Exchange Goes Down For Whole Day 792
Colin Smith writes "TradElect, the Microsoft .Net based trading platform for the London Stock Exchange, was offline for about seven hours, meaning that their 5-nines SLAs are shot for approximately the next 100 years. The TradElect system was launched back in June of 2007 and was designed for increased speed and system capacity."
Oh, my. (Score:4, Interesting)
So what happens when this happens again?
Re:Good lord, they're running on Windows? Why? (Score:3, Interesting)
Perhaps the bit you're missing is that windows isn't quite as bad as the /. crowd likes to say it is. Especially if its an older (translation: fixed & stable) variety like win2k or even nt4.
A) Yes, in fact, it is quite that bad (just not as bad as when it was first released) and
B) There is no "fixed and stable" version of .NET yet. At least none I would hinge my mission critical business on.
Re:Oh, my. (Score:5, Interesting)
Actually this is "again".
The LSE used to run on HP-NonStop (w/ Cobol and C as far as I can find) but still managed to take itself down for 8 hours in 2000.
If they're going to go down for a day every 7-8 years it might as well be cheaper and faster. (Articles quote the CTO as citing 10x performance increases).
(All based on a quick google search)
So before the hounds descend upon Microsoft it would seem the LSE has a history managing to bring down whatever system they run on.
ketan (Score:5, Interesting)
Re:Good lord, they're running on Windows? Why? (Score:3, Interesting)
Re:What do Brits say when stuff like this happens? (Score:4, Interesting)
You've seen the first scene of "four weddings and a funeral", surely?
Re:Misleading summary (Score:5, Interesting)
The Johannesburg Stock Exchange, which uses the LSE's trading platform TradElect, also suspended trading.
Hmm. Smells like a new version to me.
Comment removed (Score:2, Interesting)
Virtual 100% uptime? I call BS... (Score:2, Interesting)
What the hell are you smoking??!? I worked for one of the top switch/router manufacturers for 7 years and this is FAR from true in their shop and pretty much everyone else's. Talk to any technical call-center rep for the top 5 or so router manufacturers and I am sure they can tell you many horror stories...
What We Do Know (Score:2, Interesting)
Is that this is good news for most of /. readers. A lot of big corporatations are being reminded of how important it can be to not cut corners when hiring programmers and IT.
Cheaper programmers (Score:1, Interesting)
Perhaps .NET is not directly at fault here.
Interestingly, the main reason .NET (specifically, C#) was being adopted in our company is that "it gives access to far cheaper programmers than your legacy C++ types."
I guess programmers are not only more expensive because they have a "legacy language" on their resume after all.
Re:Oh, my. (Score:3, Interesting)
How the hell can you 'complete' a trade in 3 ms?
Is there a hard drive out there that can even complete a write in 3ms?
Re:Good lord, they're running on Windows? Why? (Score:3, Interesting)
Re:Misleading summary (Score:5, Interesting)
Yep, I remembered and laughed so hard I had to put the images next to each other:
http://tipotheday.com/2008/09/08/microsofts-foot-in-mouth-london-stock-exchange/ [tipotheday.com]
Why would you even do that? (Score:3, Interesting)
Mainframes, AS/400 - pricey but high reliability with 2-3 decades of baseline.
PC's / Windows - cheap, "good enough" for home use and light business use, new as hell and subject to unknown problems- and with a known history of issues too.
I use a pc daily... but there is no way I would put any 99% uptime application on it where huge amounts of money or lives were at stake. It's fine as a client.
Re:Good lord, they're running on Windows? Why? (Score:3, Interesting)
but it does have a lot to do with the development. They chucked the old, stable (but "obsolete" and slow) systems for something shiny and new. In this case, Windows and .NET 1.1 written by a consultancy with Indian developers.
I doubt reliability really factored much into it, using the newest coolest stuff came first. Possibly for marketing reasons. Remember this was 4 years ago, .NET had pretty much just come out (we ignore v1.0 which was practically a preview release), you know it couldn't have been as good as the MS marketing man said it was.
Re:Oh, my. (Score:4, Interesting)
I think its all about network latency - ie the marketing machine says 3ms, but they are referring to the time taken to get the message to the stock exchange's switch.
A Computer Weekly [computerweekly.com] article (and its first link) explains it - basically, they replaced the old networks with new fibre-based ones and colocated servers for brokerages.
Re:You have your causes and effects reversed. (Score:3, Interesting)
A network problem which simultaneously affected the Johannesburg Stock Exchange, which uses the same software as per TFA? That's some interesting network topology.
Re:Misleading summary (Score:3, Interesting)
Follow that logic rigorously and all mission-critical systems would be running on abacus.
Re:Good lord, they're running on Windows? Why? (Score:3, Interesting)
You have hit the nail right on the head.
I'd like to suggest another factor: Stability-conscious developers -- those that know about race conditions, memory leaks, atomic transactions, and the like -- tend to gravitate towards operating systems that make it easy to put their ideas into practice.
That isn't to say Windows is inherently unstable, it just means that it is more difficult to write a stable and reliable application on that platform. And even if you think you got all your bases covered, you can still get blindsided by depending on poorly written code churned out by some .NET developer who was happy enough to ship something that appeared to work most of the time.
The good developers then shrug and say Windows is not suitable for critical computing, and go back to UNIX-ish platforms or whatever they are more comfortable with. Rushing into that void are legions of Windows developers who are also happy enough to ship something that appeared to work most of the time, and the cycle continues.
Re:Get The Facts (Score:5, Interesting)
Interesting since they haven't been "running on Microsoft technologies" for "the past six years"...
Re:Vietnam outperforms London (Score:4, Interesting)
Ok, so here's the tally I've seen so far:
- LSE today (7 hours downtime)
- Ho Chi Minh City stock exchange (3 days downtime)
- Brazil futures, BM & F, aug 26, 2008 and Bovespa Nov, 30th, 2007.
that I've heard of.
It's incredible! This looks systemic and widespread.
I guess it's a great marketing achievement for Microsoft.
When will people in the financial sector wake up and learn they've been duped?
Re:100 years? (Score:5, Interesting)
In business, generally it means that solution provider (software + hardware) bears direct responsibility for all unplanned downtimes.
If solution cannot provide such service availability, the solution provider has to be ready to cover all the damages. And it is often planned that way from day one: some downtimes are covers by the "5 nines", some are covered monetarily by solution providers.
That's why 5 nines solutions cost as much as they cost: on one side to allow providers to bring quality of solution to desired level, on another side, in case of emergency, to let them to cover some downtimes with money.
But covering seven(!) hours(!!) can be lethal to the solution provider. But again, it all depends on their support contract. Some (cheaper) 5 nines are delivered without any guarantees: they only theoretically 5 nines and provide only "best effort" service availability.
Re:Good lord, they're running on Windows? Why? (Score:1, Interesting)
But it says a lot that NYSE runs on Linux and not Microsoft. It seems SOMEONE did listen to the techies.
Not really. I work for an IT sales company. There are plenty of times when we try to sell Linux solutions to our customers and management is all for it but it gets shot down by the "techies" because they want a Windows solution because that is what they are used to.
Re:Uhhh, what? (Score:3, Interesting)
Haha. Having worked for a Super-Platinum-Alpha MS partner before I'll tell you how it works:
You pay your enormous Partner fee every year. Occasionally MS will send someone out with a powerpoint presentation on a 6 month old MSDN article. If you are doing one of these projects, MS will never ever touch it, or help in any way, beyond paid support for a particular product (like anyone else can pay for). They will put an article in MSDN Magazine like "Microsoft and Fagware collaborate to make Some Awesome System". There will be a photo, with the MS rep shaking the clients hand, with your boss half cropped out. They will mention a whole bunch of MS tech that you (probably didnt) use.
Its not like they ever see a single design doc, let alone line of code. They don't do a damn thing beyond telling everyone how they are collaborating with you to build Awesome-X Plus for Important Client.
Also you get to put "Gold Partner" on your website, and MS occasionally refers clients that need someone to implement stuff.
Re:Oh, my. (Score:3, Interesting)
Re:To be fair (Score:2, Interesting)
Re:Using Microsoft for a 5-nines SLA? Is that a jo (Score:5, Interesting)
WTF did a moderator mark this as flamebait? The poster was right, HA is a) hard and b) expensive.
I designed some of the HA stuff many years ago for Eurex [eurexexchange.com]. We used OpenVMS and had two clusters (over 40Km apart) for the main and standby with the standby system also being used for development with a flick of a switch the standby cluster could take over in production. We had no SANs in those days but used Digital's Hierarchical Storage Controllers. These days it runs with SANs but the host systems still run VMS and there are now product specific clusters.
The next level down there are access points containing communications servers providing connectivity to member systems and routing to the hosts which are scattered around the globe. A member normally has connectivity to two access points. The only single point of failure for a member is where both lines come together for the last few metres into their building and some idiot digs a hole in the road.
The real question: Who benefits? (Score:1, Interesting)
I don't believe that this was a technical glitch.
I think people should investigate whether or not certain parties benefited, financially or otherwise, from such a prolonged suspension of trading. I think analyzing the trades that most rapidly transitioned to alternative trading exchanges, and had the highest volumes or dollars, would be very interesting and revealing...
This news story makes me think of a fictional "singularity" themed story I read yesterday:
http://www.ssec.wisc.edu/~billh/g/mcnrsts.html
Is it happening?
Not getting it (Score:3, Interesting)
MOST of your systems, that says it all. If you design a system you must be a little bit more certain then that "most" of it will be up for the availability your are promising.
I got a consumer HD that so far lasted for 6 years, non-stop. By your logic therefor consumer HD's are fine for a server enviroment because they last for 6 years running 24/7.
If 100% of you systems run 99.999 availabibilty, THEN you can come back. The acceptable error-rate is 0.001% FOR ALL YOUR SYSTEMS. Lets say that you have 10 2003 systems and MOST is 9, then you only got 90% availabilty, you fail, back to the drawing board.
It is hard to get true HA. MS can't do it, doesn't mean you can't use it for little projects. I have seen many a succesfull project launch on PC's thrown together at a local computer store and shoved into a rack. Worked fine, but only an idiot would guarantee any availability on it. Prove me wrong, put in your contracts that you guarantee HA for your clients. See you how your boss/lawyer reacts.