The London Stock Exchange Goes Down For Whole Day

Colin Smith writes "TradElect, the Microsoft .Net based trading platform for the London Stock Exchange, was offline for about seven hours, meaning that their 5-nines SLAs are shot for approximately the next 100 years. The TradElect system was launched back in June of 2007 and was designed for increased speed and system capacity."

The London Stock Exchange Goes Down For Whole Day

...now if only my wife would do that! /rimshot!

Re:The London Stock Exchange Goes Down For Whole D

Oh, she does... just not with you.

nudge nudge, wink wink.

That's okay

most of the american stock exchanges have been going down all year.

Re:That's okay

most of the american stock exchanges have been going down all year

My wife did that once. Nearly killed me. Come to think of it, it was just after we signed up for the life insurance ...

99.9967% Uptime if up the next 100 years

Assuming 8.5 hour trading day (0700-1530) and 250 trading days/year. Maybe a squirrel caused the problem ... ;-) [komar.org]

Re:99.9967% Uptime if up the next 100 years

Yeah, because they turn it on when trading starts and turn it off when trading ends.

Ugly Day

It was an ugly day of finger-pointing and near-fixes, but in the end, it just left all the financial firms standing there staring at the Exchange. Definitely was a big deal--and it seemed like a lot of volume spilled over to US markets, creating volume related issues here.

MS should hurry up and patent....

.... a method of controlling the market.

Patch Tuesday

But Patch Tuesday is tomorrow?

Reliability?

Looks like someone needs to brush up on their buzzwords, specifically "mission critical" and "services no longer required".

single page

I wish people would get into the habit of linking to the single page version of the FA [reuters.com].

Misleading summary

The summary implies that TradElect was responsible for the shutdown, but according to the stock exchange itself, it wasn't [itworld.com] the case. They say instead it was a network problem.

Re:Misleading summary

Why the heck they were using MS Windows for this type of environment is stunning... Transactional processing which is the bulk of this type of setup is where Solaris and Linux excel. Any company that builds a system like that on .Net should be thown out on the street.

In short.. Not to rock on Windows, but different platforms always offer different strengths..

Re:Misleading summary

if it was a network problem, then they're in more trouble than the summary implies. It's relatively simple to get 100% uptime (minus a dropped packet or two) in a network. The key here is redundancy. If you throw enough hardware at it, yes, it will not break.

Internal? Dual(+) homed servers, redundant switches, redundant AC, redundant power.
External? BGP on 2 or more transits on separate physical runs.

What, you say that you need to account for natural disasters? Then get a second site, at least a few hundred miles away, and repeat.

Virtual 100% uptime is a solved problem in the networking world.

performed as expected...

"and was designed for increased speed and system capacity"

and see - it went down far faster and more completely than the previous system would have been able to. So that's progress. It's all in how you present it.

5 nines?

So their 9.9999% uptime is screwed?

Nothing taxes can't fix

After the malfunction, TradElect was immediately bought by UK's government for $200 billion and all its debts waved. In an unrelated story, medicare tax was raised yet again because of an unexpected shortfall.

What, no ads?

Does anyone else remember the "The london stock exchange chose windows 2003 for reliability, they didn't choose linux" ad banners that used to run all over the place, including slashdot if i remember?
Funny how it's all come crashing down...

"The london stock exchange chose windows, but after 7 hours of downtime wishes they had chosen linux".

5-nines SLA

"5-nines SLA"

I had to look this up, so I imagine other people didn't know it either (I thought was was a stock exchange term). First Google search result reveals the answer,

The Battle With "3 Nines" and The Goal of "5 Nines" [cubiccompass.com]

Re:That's some strange math...

Since when is 7 hours even close to "a whole day"? Maybe you meant "almost a whole business day"?

It's a whole trading day--and that's all that really matters when it comes to a major market.

Re:Still don't know why...

Agreed. It's a bit of flame-bait mentioning them in the summary when the exchange is being tight-lipped about what the root-cause is (if they even know at this point.) I do a lot of .NET stuff and, like other platforms eg. Java, there's many things that could cause problems, like plain old programming bugs.

Re:Good lord, they're running on Windows? Why?

Oh please. Persuasive marketers can get Windows installed just about anywhere including US war ships.

While it is commonly accepted by many techies (and strongly denied by others) that Microsoft Windows is not a suitable platform for that level of computing, sales people often bypass the techies who know better and sell to managers and executives who still believe "you can't get fired for using Microsoft."

With all this said, it will be quite some time (and possibly never) that we will ever know for certain what is at the root cause of the failure. You can be sure that Microsoft is all over this problem both technically and P.R.-wise. They won't let the facts get out if they are damaging. Recall the major power outage that many still believe was caused by a worm attacking Microsoft servers? As far as I can see, the true cause of that failure has yet to be revealed.

But if this was a planned event, or an unplanned disaster resulting from a planned event gone bad (updates, upgrade, other maintenance), you would think they would have provided for mishaps in some way or another.

But as this news story is all I have to go on, there is no indication of cause and so I will not presume this is a Microsoft problem. But it says a lot that NYSE runs on Linux and not Microsoft. It seems SOMEONE did listen to the techies.

Re:Good lord, they're running on Windows? Why?

Perhaps the bit you're missing is that windows isn't quite as bad as the /. crowd likes to say it is. Especially if its an older (translation: fixed & stable) variety like win2k or even nt4.

I'm not sure if you're serious or not, but surely you aren't trying to compare NT4 uptime with the 5 9s of a solid System z platform?

Re:Oh, my.

The same thing that happened this time?

Re:How many failures before..

When I worked in academia I used to collaborate on a research project with a data architect from one of the major electronic exchanges. His whole shop is MS and .NET. I asked him why he didn't run Linux / Unix. He said that with competent guys the MS boxes had great uptime. Wall Street can afford to pay the top salaries so they attract guys who really know their stuff. Not just semi-competent people who managed to sit through an MSCE exam. [his words not mine]

Also he said support was crucial for his company. If something went down, he wanted to be able to call someone immediately. He couldn't afford to just post a question on a message board and hope someone replies. He wanted contracts with 3rd party support that had experience with similar huge enterprise systems that he had.

When I said there were companies who could provide excellent Linux support, he said his ass was on the line if something broke so he wanted to be able to justify his software choice to the the C-level guys. And those guys knew the name Microsoft. So he didn't see anything else as an option.