The London Stock Exchange Goes Down For Whole Day 792
Colin Smith writes "TradElect, the Microsoft .Net based trading platform for the London Stock Exchange, was offline for about seven hours, meaning that their 5-nines SLAs are shot for approximately the next 100 years. The TradElect system was launched back in June of 2007 and was designed for increased speed and system capacity."
100 years? (Score:3, Insightful)
5 nines does not mean what you think it means.
Re:That's some strange math... (Score:5, Insightful)
Since when is 7 hours even close to "a whole day"? Maybe you meant "almost a whole business day"?
It's a whole trading day--and that's all that really matters when it comes to a major market.
Re:Still don't know why... (Score:5, Insightful)
Re:Good lord, they're running on Windows? Why? (Score:3, Insightful)
Perhaps the bit you're missing is that windows isn't quite as bad as the /. crowd likes to say it is. Especially if its an older (translation: fixed & stable) variety like win2k or even nt4.
Potentially misleading summary (Score:3, Insightful)
The exchange insists the problem was connectivity, not the trading platform.
Not to sound overly cynical, but I'd hardly expect them to acknowledge the problem if it were the trading platform that was the issue. That'd kind of be business suicide.
Re:Good lord, they're running on Windows? Why? (Score:5, Insightful)
Perhaps the bit you're missing is that windows isn't quite as bad as the /. crowd likes to say it is. Especially if its an older (translation: fixed & stable) variety like win2k or even nt4.
I'm not sure if you're serious or not, but surely you aren't trying to compare NT4 uptime with the 5 9s of a solid System z platform?
Re:Good lord, they're running on Windows? Why? (Score:5, Insightful)
Oh please. Persuasive marketers can get Windows installed just about anywhere including US war ships.
While it is commonly accepted by many techies (and strongly denied by others) that Microsoft Windows is not a suitable platform for that level of computing, sales people often bypass the techies who know better and sell to managers and executives who still believe "you can't get fired for using Microsoft."
With all this said, it will be quite some time (and possibly never) that we will ever know for certain what is at the root cause of the failure. You can be sure that Microsoft is all over this problem both technically and P.R.-wise. They won't let the facts get out if they are damaging. Recall the major power outage that many still believe was caused by a worm attacking Microsoft servers? As far as I can see, the true cause of that failure has yet to be revealed.
But if this was a planned event, or an unplanned disaster resulting from a planned event gone bad (updates, upgrade, other maintenance), you would think they would have provided for mishaps in some way or another.
But as this news story is all I have to go on, there is no indication of cause and so I will not presume this is a Microsoft problem. But it says a lot that NYSE runs on Linux and not Microsoft. It seems SOMEONE did listen to the techies.
Re:Misleading summary (Score:5, Insightful)
Why the heck they were using MS Windows for this type of environment is stunning... Transactional processing which is the bulk of this type of setup is where Solaris and Linux excel. Any company that builds a system like that on .Net should be thown out on the street.
In short.. Not to rock on Windows, but different platforms always offer different strengths..
.NET clearly to blame...... (Score:3, Insightful)
.....I mean, there couldn't be any other possible cause for the problem.
Re:Good lord, they're running on Windows? Why? (Score:0, Insightful)
My goodness! You believe in God? ("Good lord") That's just nuts. I don't understand the rationale for that at all, in this day and age
Re:Still don't know why... (Score:5, Insightful)
Wait! Are you suggesting that downtime can be caused by application problems, network problems, hardware problems, dumbass systems administrators and a whole slew of other things completed unrelated to the platform on which it is running?
I am *shocked*! *Shocked* I tell you!
Re:Oh, my. (Score:5, Insightful)
Re:Misleading summary (Score:5, Insightful)
As is normally the case M$ threw lots of money at the exchange to get it to switch unix/linux base to windows net so that M$ can tout that a major exchange is running windows.
Full page ads touting the switch and the reasons they cited were better through put and better up time.
They even had ads touting it here on /.
It's official. (Score:4, Insightful)
Let me explain computers to you. See, the developer uses a set of platforms, languages, integration components, etc.. to deliver his functionality to the end user. A failure at any level can cause the application to fail. It could be application logic, network issues, hardware issues, integration with third party systems, a dipship systems administrator, etc...
And yet the 90-105 IQ SlashDweeb set comes out in numbers with no data and says "lolz Windoze! .NET haha!". Crikey.
Re:Misleading summary (Score:5, Insightful)
No... Actually I deal with this everyday. Windows is great for places where you need desktop apps or such. It also does well when you must have generic developers for web development.
Where Unix/Linux/BSD truly shines is on back office type transactional processing. There are many reasons for this, and have a long history at doing exactly this. Meaning, mainframes may not have every been considered sexy, but they ran critical systems in companies for decades with very little problems... Actually they built such a reputation that when they failed most instantly assumed it was a hardware failure... Working on them, however, takes a more polished developer...
Re:Oh, my. (Score:4, Insightful)
Ah. Some blamecasting, after which everybody pretends it had never happened?
Re:Oh, my. (Score:5, Insightful)
Re:Misleading summary (Score:4, Insightful)
The point is you shouldn't be running mission critical systems on new and shiney (it's bound to have bugs) you should be running it on old and reliable (or at least where the bugs and workarounds are well known)
Re:It appears high load/usage crippled the system. (Score:4, Insightful)
No different then what can happen on a unix box I suppose.
Note that the current system is built around a large cluster of 2.2GHz servers, while the unix-based system it replaced (which coped perfectly happily with a substantial portion of the same traffic) ran from a smaller cluster of much slower servers.
The primary purpose for the new system, introduced less than a year ago, was to expand capacity. For it to have failed within a year due to lack of capacity basically means that it has failed in that objective.
Re:How many failures before.. (Score:5, Insightful)
In other words, he used the "no one ever got fired for buying IBM" defense.
Re:Good lord, they're running on Windows? Why? (Score:0, Insightful)
Tee Hee (Score:5, Insightful)
Re:It appears high load/usage crippled the system. (Score:5, Insightful)
No, actually the Windows system (10 ms per transaction) was a 13x speedup over the older system (135 ms per transaction), followed quickly by an addiditonal 50% speedup (6 ms per transaction). The Windows system was just recently updated to double performance again (3 ms per transaction), so it's now 45 times as fast as the unix-based system it replaced.
You may be able to fault it on reliability (though the olde system wasn't perfect either), but you can't fault it on performance.
Re:Potentially misleading summary (Score:1, Insightful)
Not really. In an environment like this they will probably be subject to an independant audit. If they claim one thing and the audit finds otherwise...
Basically they planned on X capacity and overloaded this morning. Probably some tech said they needed X, but the powers that be said build me Y as we need it cheap.
Re:Oh, my. (Score:5, Insightful)
I couldn't disagree more. Although automatic garbage collection is nice, this doesn't mean that you'll get "five nines uptime" systems by working with "less experienced" coders.
If you're building a system that must guarantee 999.99% uptime, you wait until your best professionals become available, because it doesn't only involve code. You DON'T give the job to the less experienced ones, no matter how great the programming language. Five nines uptime requires a very robust design and very solid code quality running on a very solid platform which is running on a very solid OS on a very solid infrastructure. You'll want everything to be tested by unit tests, integration tests, regression tests, and whatnot. That involves a whole lot more than 'just' coders, but whoever works on it, they better be good at it.
Re:How many failures before.. (Score:1, Insightful)
"justify his software choice to the the C-level guys. And those guys knew the name Microsoft. So he didn't see anything else as an option."
IBM?
Re:Oh, my. (Score:3, Insightful)
yeah, 'cos that's what we really need - less experienced coders.
Look on the bright side, they're probably much cheaper.
Re:How many failures before.. (Score:3, Insightful)
In other words, he used the "no one ever got fired for buying IBM" defense.
Yes, he did and as a matter of fact it's a valid defense and it's not a chicken or egg problem. Almost everything that ever wants to run mission-critical systems have to work their way up from missing-trivial through mission-sensitive and mission-important. Customers didn't just one day decide IBM was overpriced and threw out all their computers, they tried and tested clones and nothing bad happened. Can you point to any smaller exchanges that use Linux? Or do you expect that suddenly Linux should go from nowhere to running the biggest, most critical markets in the world economy? No offense, but it doesn't happen to anything else either, walk before you run. That, or bribe before you run but expect some trip-ups...
Re:5-nines SLA (Score:3, Insightful)
Yeah, me neither! If only we also had this "Google" thing.
Re:Misleading summary (Score:4, Insightful)
I thought unfair advantage was the whole point of capitalism...I have it you don't! what kind of communists run the place?
Re:It's official. (Score:4, Insightful)
Well, no, it's just that Microsoft shouted long and hard about how reliable the LSE would be now it was running on Windows Server System 2003. So it's deliciously ironic that after all this trumpet tooting, it still fell flat on its face, regardless of the reason...since Microsoft's ads were obviously to get everyone to believe that the system would be highly reliable.
Re:Oh, my. (Score:3, Insightful)
This is a funny ha ha right? Because it's like saying we should do away with new glass displays in airplanes because they make pilots lazy.
Apologies if it's a joke but every time someone makes a coding error someone on here has to pull out the "back in my day we were men and coded without errors because the system exploded!" card. That's nonsense. Garbage in garbage out. A tool is only as good as it's user. It's the responsibility of the developer to use the tool properly.
Re:It could be .. but wasn't :) (Score:5, Insightful)
Why did the upgrade fail, I guess is what an intelligent person would ask. You haven't asked that. You've hilariously assumed it's .NET or Microsoft's fault.
As a matter of like for like, I'm going to assume it was because some Linux dweeb walked in and tripped over a network cable. Ergo, I now claim Linux dweebs are clumbsy oafs who should be banned from computer rooms.
Re:Tee Hee (Score:5, Insightful)
I also, long ago, used to believe that language features could improve software reliability. Nowadays the idea just makes me cackle
Why? Certain languages have features that eliminate large classes of errors. Whilst its possible that programmers will find other ways to screw up, I'd have though that reducing the set of errors that are actually possible would go some way to improving reliability.
Out of curiousity, what languages are you familiar with? Have you worked much in languages with very tough compile-time checks, like Haskell?
Re:That's some strange math... (Score:1, Insightful)
Yeah. Isn't that ironic???
He probably meant USA, though; AFAIK she's a Canadian.
/. is like the Keith Olbermann of the IT world (Score:3, Insightful)
I fully appreciate they have an agenda and a cause. No problem. But for such a popular site, if they're going to throw stones they should do at least wait until the target is verified.
A critical system shoud be RELIABLE! (Score:5, Insightful)
It's about the same thing when people say that "XP does not crash, it's faulty device drivers that crash".
If a system should be reliable, then it should be reliable, no excuses accepted. It does not matter if it's system bugs, application bugs, hardware failures or power outages, a system that pretends to achieve 99.999% availability should take all that into account.
The operating system is not at fault if the power goes down, of course, it's a sloppy engineer that designs a system without redundant power supply. But, likewise, a sloppy engineer will prefer a system that lets him configure and operate it by click-and-drag, instead of a carefully designed and tested set of procedures.
A critical system should NEVER depend on an operating system that does not have a proper batch language. That should be a compact and powerful script language, using TEXT files for configuration that can be hand edited if needed, that can be stored and archived in a version control system, so that bugs can be tracked.
Re:Oh, my. (Score:3, Insightful)
No, but there are likely plenty of programmers with 20+ years of experience who have learned .Net at some point in the last 5 years. The language experience isn't as important as the programming experience, despite what the headhunters think.
Re:Get The Facts (Score:5, Insightful)
Right from your article "and be cheaper to manage"
sounds like the LSE fired expensive. knowledgeable admins and went for 'cheaper' ones, there is your problem right there. windows server isn't perfect, but clearly they had good hardware, were running mission critical apps, but went with cheaper less experienced admins.
also, your fine article specified there were 'no production outages', they don't claim the system ran 24/7/365 with no reboots or glitches, but that there was no production outages for six years. there is quite a bit of difference. the former states that admins and hardware were able to offer the specific services needed at the time it was needed for 6 years, but not on the amount of redundant hardware, etc required to accomplish everything.
so given everything i've read here, under experienced windows admin approves an under tested system upgrade that epic fails, and takes down the production server for the first time in 6 years. no shock here, they wanted to cut corners on admin costs, they brought the epic fail on themselves.
Re:Good lord, they're running on Windows? Why? (Score:2, Insightful)
I love to bash windows as much as the next guy, but I don't see the connection.
Developing an application on .NET does not transfer all your responsibility to Microsoft.
Latency is kinda pointless for this kind of stuff (Score:5, Insightful)
I mean, that might be what they worked on, but it's kinda pointless; what's interesting is the # of transactions per second, and that can usually be improved at the expense of individual latency. For example, databases can be configured to wait a few milliseconds to group transactions, so as to write several to disk in one single write/sync.
Re:Bad upgrade (Score:5, Insightful)
this same kind of thing( replace *nix with Windows ) is what took out the LAX comm system a few years ago and left dozens and dozens of airplanes in the air and on the ground at/over LAX without communications.
What blows me away is that for years, UNIX systems were one of the defacto standards for mission critical OSs. Along comes a marketing company, Microsoft, and people are saying it is capable of mission critical use even when there are constant disruptions from virus attacks, Ctl-Alt-Del and BSoD are a well known features, and any of a hundred other reasons it is NOT ready for mission critical systems.
What kinds of morons are running the show anyways? And it is about time people start getting fired for this junk. From my experience on operating systems, UNIX was the one OS where when you wrote code, you dealt with the business logic/code and not OS issues. Only once in a blue moon did an OS patch or structure tweak get in the way of coding the application(s). OS/2 was pretty good but not as good as UNIX and Windows was the worst. Gawd, I still hear people complaining about that little Windows Mobile OS crashing. They can't even get a small chunk of code working properly let alone the behemoth that is the Windows desktop and server OS.
LoB
Choice quotes (Score:4, Insightful)
Nick Illidge [onwindows.com] Financial Markets Sales Manager at Microsoft UK "We are delighted that the London Stock Exchange has selected the Windows platform to base a significant part of its business on. This is further evidence of the enterprise scalability of the Windows franchise. We see our relationship with the Exchange and Accenture as a strong partnership. The Exchange is bold in its technology vision, Accenture provides the capability to deliver this vision, and Microsoft is providing the core technology to help provide the business benefits that the Exchange is looking for."
David Lester CIO at the LSE says [advancedtrading.com] ... that the LSE "is the only exchange in the world not to have had a single outage in six years."
"This is all about the question, 'How are we going to take over the world?'" says Lester, "... I believe this system -- because it's fast, agile and reliable -- will help us compete better. Our current system has to go down for four hours every evening to get ready for the next day's trading," he says. "The batch processing is '80s and '90s technology. You can't run a global market with a system that has to be down for four hours."
Here [londonstockexchange.com]'s a great factoid
Before joining the Exchange in 2001, David worked for Thomson Financial and Accenture.
Re:Misleading summary (Score:1, Insightful)
It depends on the situation. If you have a situation where you can break a problem into parallel, independent, and redundant tasks, then Quantity has a quality all its own [wikiquote.org]. That's why the space shuttle has 3 computers and it's the principle behind all those _____@Home distributed computing projects. It's the principle behind using redundant clusters. Quantity doesn't necessarily deliver quality, but it can be used that way.
Heck, it's the key point in the advantages of market competition. If you're willing to spend the money to have 3 or 4 teams of .NET developers work on independent implementations of a system and then you pick the best one, you've got a reasonable chance of getting a good system if you let the teams know that quality, not feature bloat, will be the success factor for who gets the completion bonus. Theoretically, software markets would externalize that process but, unfortunately, quality ranks too low in the purchasing criteria of most software customers.
Back in the day - Not only London (Score:5, Insightful)
IIRC, Brazil Bovespa had a small glitch last month or two.
Back in the day when Wall Street and financial markets ran on Solaris systems (AFAIK), this shit wasn't common.
Now it's probably going to become *acceptable* for stock exchanges and aviation reservation software to crash.
Apparently, there's a new generation of a-holes on the system administration markets who grew up with Windows and the Blue Screen of Death, that thinks it's acceptable for operating systems to crash, once in a while. Is it evolution?
Re:Battery-backed write through cache (Score:5, Insightful)
Oh, yes.. battery backed write cache. With batteries produced by the lowest bidder. The warranty is for 3 years, and the battery lasts just that long before silently failing. When the power goes, well you really didn't need that data written to disk on your database server, did you?
We now do not allow any server to be put into production with any kind of write cache on it. Ever.
that said, (Score:2, Insightful)
Picking Windows is a really bad way to start out.
Your aunt probably runs it, but is it really enterprise ready?
Re:Misleading summary (Score:1, Insightful)
Not necessarily. It depends on barriers to entry. There's a lot fewer players in professional sports leagues than in college or high-school leagues within the same geographical area, but if you can only hire one team for a contest (like the Olympics), you're usually better off with the people with the greater combination of raw talent, experience, and intense training schedule, so you would probably prefer to use the professional sports players. The Harlem Globetrotters don't lose too often when they tour high schools.
If your UNIX gurus have 20 years of average experience in successfully delivering highly fault tolerant and redundant applications, they have a better chance of success at producing that kind of software than a team 10 times that size of Indian .NET programmers with a maximum of 5 years experience.
On the other hand, if the cost differential between .NET and UNIX programmers was as great as the cost differential between college and professional players, it might behoove you to set up 10 competing .NET projects instead of 1 UNIX project. But I don't think people outsource development with that in mind.
Or VAX/VMS. (Score:3, Insightful)
Windows is just consumer junk, and not even very good consumer junk.
Kickbacks are almost certainly at work in a deployment like this.
Re:Tee Hee (Score:3, Insightful)
With a general purpose programming language, the number of ways to screw up is effectively infinite. If you take another infinite set, say, the integers, and eliminate a large subset, say the even integers, you still have an infinite set left over.
By the same token, the number of ways to befall a terrible accident is effectively infinite. Does that therefore mean we should not bother putting railings on bridges, or seatbelts in cars, or fuses in sockets?
I demand equal representation! (Score:1, Insightful)
Re:Misleading summary (Score:3, Insightful)
I certainly understand your point, but quality of developers are only half the picture. The OS design structure of Windows and Unix is just different, and both have their strengths and weaknesses. I am excluding OSX in this argument for several reasons...
The Unix development approach is allot better at appliance like implementations. Meaning, you build it for a specific job (batch transactional processing in this case), and you treat the hardware as an appliance. You also limit ways a user can interact with the system, so patching is rare to never. Video interaction is rare, and even shell interactions are console menu based and/or web based. The prevents users from tinkering, and prevents the ability to exploit any vulnerabilities that may exist.
The Windows development approach is more more flexible in nature. Meaning, primarily keyboard and mouse user interaction is guaranteed. The possibility of somehow getting a virus must be taken into consideration. Most interactions is RDC and/or VNC based. Web based servicing sometimes occurs, but Windows is impractical to build in an appliance fashion typically...
I hope this Helps..
Re:Tee Hee (Score:3, Insightful)
The average driver can't take out the whole bridge. The average programmer can. Software is just not understood as well as structural engineering, and most of the people we allow to do it couldn't even get into an engineering program.
Re:Tee Hee (Score:3, Insightful)
People actively AVOID getting into an accident, so while it's still possible to die horribly in all kinds of ways, a person will not likely to get to the point of actually getting into a fatal accident unless something is seriously broken (say, railing is unexpectedly missing on a bridge).
Inexperienced and stupid programmers produce bugs at nearly constant rate -- give them something that makes some bugs impossible, and they will either improve their productivity (still with $deity-awful rate of bugs), or start making different kinds of bugs.
Re:Tee Hee (Score:3, Insightful)
This is just why the average programmer shouldn't be writing software that controls a stock market. When you're dealing with something that mission-critical with that much money on the line, you damn well better be pulling from the same pool as NASA.
Re:Tee Hee (Score:3, Insightful)
Re:Good lord, they're running on Windows? Why? (Score:3, Insightful)
"you can't get fired for using Microsoft."
Maybe not but you sure can get fired when your system epic fails and stops trading for a day.
Seriously though given the length of the downtime, I doubt this was caused by an operating system problem. If that were the case, you'd just reboot the affected system and hope it doesn't go down again before you can get a patch, no? Even Vista doesn't take that long to boot.
Re:Using Microsoft for a 5-nines SLA? Is that a jo (Score:3, Insightful)
Re:Latency is kinda pointless for this kind of stu (Score:4, Insightful)
Based on this description, seems to me that "arbitrage" is a nice word for inserting yourself into a trade which has nothing to do with you for the purpose of bleeding both the seller and the buyer out of some profit without producing or contributing anything of value. Making it more difficult would make the actual productive parties in the trade better off, and likely help economy as a whole.
Or, to put it even more bluntly: arbitrage, as described by you, is a nicer name for parasitism.
Re:Using Microsoft for a 5-nines SLA? Is that a jo (Score:3, Insightful)
Why modded troll? It is possible to get high uptime figures with a lone system. You can't take it offline, but hell, I could probably run my PC for a year end to end without issue. The problem occurs when I try to scale that and make, say, 200 PCs all run for a whole year without issue.
Re:100 years? (Score:5, Insightful)
It's called framing and it is making public debate in western society increasingly difficult.
Re:Tee Hee (Score:5, Insightful)
I also, long ago, used to believe that language features could improve software reliability. Nowadays the idea just makes me cackle
Why? Certain languages have features that eliminate large classes of errors. Whilst its possible that programmers will find other ways to screw up, I'd have though that reducing the set of errors that are actually possible would go some way to improving reliability.
Out of curiousity, what languages are you familiar with? Have you worked much in languages with very tough compile-time checks, like Haskell?
Y'know, I agree with the grandparent. On my first coding job there was a guy (Chris Burton) who'd worked on the Manchester Mark One [computer50.org]. He was retirement age when I met him. We had a new model of inkjet printer, which had a new processor none of us had ever seen before. It printed characters, we needed it to print bitmaps.
Chris took the datasheet for the printer and the datasheet for the processor home on the train with him, and came back next morning with new code for the printer PROM written out - in opcodes, not assembler mnemonics - in longhand on a pad of paper. That code was blown into the PROM and worked first time, and continued to work without any errors reported for the three years I was on that project.
Programmers like that just don't seem to exist any more. Automatic memory allocation, bounds checking, type checking, etc. are great technology, and I wouldn't choose to live without them. But they mean we are all sloppy and careless, because we can get away with it, and when humans can, they do.
Re:Using Microsoft for a 5-nines SLA? Is that a jo (Score:5, Insightful)
I have a feeling that the 'normal' IT situation was to blame for this.
Preamble: Technical Expertise provided a wonderful architecture that was HA and robust, fast, and scalable.
Bean Counters looked at the cost and said "You Tech guys spend too much money."
IT architects: "How much is your data worth?"
Bean Counters: "Not this much. Look we don't really need all of these systems. My home system has been working for 4 years with no problems. And I've talked with Microsoft Execs and they will cut us a deal for their platform. Now go away, I've just decided how the architecture will be done. Why did we hire you anyways?"
Re:Using Microsoft for a 5-nines SLA? Is that a jo (Score:4, Insightful)
Um, wrong - Ever heard of the Mono Project?
Mono provides the necessary software to develop and run .NET client and server applications on Linux, Solaris, Mac OS X, Windows, and Unix.
http://www.mono-project.com/ [mono-project.com]
Glad you fell for Microsoft's marketing campaign. There is a reason they don't crush mono. It gives a illusion that there is choice. Name me
Re:Using Microsoft for a 5-nines SLA? Is that a jo (Score:1, Insightful)
At one point around 2005 I applied for a job at LSE. They had just brought on a new CEO (whose name escapes me) and were in the process of building this system. They were about to replace a Tandem machine that distributed about 500 stock advice messages a second with a cluster of 120 (!) servers running a .NET system. That implies a load of around 4 messages per second per server.
When the architecture was described to me I remember thinking 'that's brave'. I did express an opinion that .NET wouldn't have been my first choice for something like this. Apparently the decision was driven by the new CEO wanting to modernise the LSE and offer services that could not be built on the back of the Tandem platform. It had the feel of a technical decision being driven by non-technical management and didn't inspire confidence in the LSE.
It was being built by one of the big-5 firms (Accenture IIRC) and who insisted that the platform would work and provide the uptime. I would have thought 4 messages per server per second on modern wintel server should allow headroom for substantial spikes in volume but evidently it didn't.
Re:that said, (Score:1, Insightful)
I work for big oil... All of our million-dollar a minute apps run on AIX, Solaris, or Linux.
The 'most of us don't give a damn' apps run on Windows.
Right tool for the right job.
No National TV News Coverage? (Score:2, Insightful)