More Airline Outages Seen As Carriers Grapple With Aging Technology (reuters.com) 145
An anonymous reader writes: Airlines will likely suffer more disruptions like the one that grounded about 2,000 Delta flights this week because major carriers have not invested enough to overhaul reservations systems based on technology dating to the 1960s, airline industry and technology experts told Reuters. Airlines have spent heavily to introduce new features such as automated check-in kiosks, real-time luggage tracking and slick mobile apps. But they have avoided the steep cost of rebuilding their reservations systems from the ground up, former airline executives said. Scott Nason, former chief information officer at American Airlines Group Inc, said long-term investments in computer technology were a tough sell when he worked there. "Most airlines were on the verge of going out of business for many years, so investment of any kind had to have short pay-back periods," said Nason, who left American in 2009 and is now an independent consultant. The reservations systems of the biggest carriers mostly run on a specialized IBM operating system known as Transaction Processing Facility, or TPF. It was designed in the 1960s to process large numbers of transactions quickly and is still updated by IBM, which did a major rewrite of the operating system about a decade ago.
Dumb (Score:5, Insightful)
"Most airlines were on the verge of going out of business for many years, so investment of any kind had to have short pay-back periods,"
You really only see this type of thinking in the West. Most sensible companies know that when times are good, you build a war chest, when they are bad you invest the war chest to grow your business and be competitive. The problem wasn't that times were bad. You can always say times are bad. The problem was that they didn't make the best of things when times were good, and therefore deserve the cluster fuck situation they are in now.
Re: (Score:1)
Re:Dumb (Score:5, Informative)
I worked on the first big web-enablement for AA's Sabre system, back in 97-99. Saabre was the key to inter-airline reservation scheduling. Travel agents used this as their main system, and some other folks around here may remember the gateway with CompuServ. eAAsySaber. LOL.
It was unlike any dotcom experience I had around that time. Super legacy. Impossible to change anything - and grave uncertainty that changes were even possible!
The Sabre core compute and data storage stack was built on a series of different mainframe and mid-range systems, back when instead of writing new business functions, you instead attached new business systems.
The glut of stuff crossed vendors occasionally. Mostly IBM. Parts dated to the 70's and through the 90's. I never met anybody who had a "mastermind" view of how it all worked. Instead, lots of analysts with diagrams - mostly from vendors and "big five" firms. Any proposed change had to be run through an exercise that called on the various experts in different parts of the system. Most were not so much expert, as "acquiring some expertise". ;-)
Our work became the basis for travel services like Expedia, and customer offerings by American Express Travel, etc.
I'm sure that this may have changed only somewhat. Saber was sold off, and became a core to Travelocity - who in turn were finally bought by Expedia, who consumed Saber information. Behind it all, there are a 360 and some front-end processors ported to AS400 systems, I'm nearly certain.
Re: (Score:2)
There are no AS/400 systems in there at all. The front end processors were Solaris and then ported to Linux (NOFEP), these replaced the legacy VAX/VMS front end systems (OFEP). Sabre is an independent company, but Travelocity was sold off to Expedia.
Re: (Score:2)
The VAX to IBM SNA mainframe is already an ungodly mix!
How early are we talking. I saw this stuff in about 98 and we were bringing some of the first *nix in the mix. We had pod in different data centers than the legacy, but ran Stronghold Apache and ATG Dynamo in 1999.
That's greed for you (Score:4, Insightful)
Always short sighted and thinking tomorrow will be the same as today.
What I'm afraid of is this business / investment / management continues to infect the rest of the world. I can't wait until all of the stock markets are controlled by algorithmic trading, with the next quarter's number the sole goal.
Re: That's greed for you (Score:2)
LOL
They are. Supercomputers do a check and both buy and sell a stock at the same time to set the price at high frequency volumes. What you think a person actually trades today?
Only 90 days count for the CEOs to beat the computer to get their bonus
Re: (Score:2)
Re: (Score:1)
Re: (Score:1)
Have your heard of Fukushima and Chernobyl? Do you mean east of Japan?... yes, perhaps Pacific Islanders can teach us something.
Re:Dumb (Score:5, Insightful)
Re:Dumb (Score:5, Insightful)
From what I have read, this was not an obvious WTF moment. Delta apparently has a complete disaster recovery facility with duplicate hardware. But they had a single point of failure in their infrastructure, which caused them to lose power to the entire datacenter, and everything went down. That part might be a WTF. But once they got everything booted up again, they had to contend with trying to get a system restarted that simply wasn't designed to ever fail completely. So it took hours to get all the pieces back up and communicating again.
Then their are the real world problems - flight A feeds into flight B, but flight A was late, meaning all those connections were missed and passengers have to be rebooked. And flight B can't fly anyway, because the plane is still sitting 500 miles away because the flight that would bring it to this airport was cancelled as well. And the flight crew that was supposed to bring flight B to this airport technically went on duty the moment they reached the airport, and now they have reached the max allowed hours in the day, so a new crew is needed. But that crew is in a different city...
This incident will span some fascinating failure analyses, and no doubt people will get fired and lawsuits will be filed. And like most DR scenarios, it is way harder in real life than it seems in planning and exercises. I wouldn't be surprised if this causes a big project to deal with outages and restarts, so that this doesn't happen as easily next time.
Re:Dumb (Score:4, Informative)
Well, mainframe computers have such excellent uptimes (you almost never reboot one) because everything is hot-swappable. CPU failure? Remove the CPU module, insert new one, and continue - all while powered up. The OS takes care of suspending the failed one and scheduling around it. Ditto all other components. Effectively, you should never reboot them.
Of course, the thing is, when you eventually do reboot them, they take hours to boot all the way up as they perform comprehensive integrity checks (who knows why it was rebooted?).
This is, IMHO, the far bigger issue. Airlines are scheduled tight - if the plane's not flying, it's costing money. Ultra-low cost carriers have very right schedules to ensure the planes are always in the air.
Getting the crews and equipment all prepositioned in the right place and ready to fly is a delicate balance at the best of times and a complete nightmare when you have to start from scratch.
Re: (Score:2)
Of course, the thing is, when you eventually do reboot them, they take hours to boot all the way up as they perform comprehensive integrity checks (who knows why it was rebooted?).
Not inherently. The AS/400 (IBMs minicomputer) could do this, because of some bizarre disk optimization requiring effective a full FSCK (or RAID rebuild - either is a sloppy analogy) on an unclean shutdown just to become coherent, but that was an extreme corner case.
A mainframe won't boot as fast as a OC because there's a lot more hardware to POST, but that's a few minutes, not hours.
Getting the crews and equipment all prepositioned in the right place and ready to fly is a delicate balance at the best of times and a complete nightmare when you have to start from scratch.
Any sane DR strategy would ensure all flights get completed (albeit quite late) without the need for recovery of the main sy
Re: (Score:2)
Thgis is important. They should distribute the key information well in advance to local servers that can handle validating tickets and which flight they go on and simply communicate back confirmation that a given passenger boarded the flight. It may result in less efficient operation and doesn't allow for scheduling new flights or selling additional tickets, but as you say, it allows them to remain functional at some level so they will have fulfilled already promised tickets.
Re: (Score:2)
Well, mainframe computers have such excellent uptimes (you almost never reboot one) because everything is hot-swappable. CPU failure? Remove the CPU module, insert new one, and continue - all while powered up. The OS takes care of suspending the failed one and scheduling around it. Ditto all other components. Effectively, you should never reboot them.
That's interesting. My recollection from working on them a bunch of years ago was that our mainframes were IPLed on a regular, scheduled basis, because the folks responsible for them were disciplined about it and wanted to make sure there would be no surprises when one needed to be restarted. The fault tolerance you mention was used to make sure that the scheduled IPL's were the only times they went down though, and I never saw any unplanned downtime on them.
Re: (Score:3)
I'm going to go out on a limb, and guess that the people responsible for that mainframe transitioned into the role from conventional rack & stack servers. While that sort of discipline is otherwise admirable; the whole reason that you give IBM the kind of money it takes for them to deliver and support a mainframe is to make that sort of discipline unnecessary. IBM was guaranteeing uptime of 20 *years* back in the S/390 days. And I'm pretty sure it's only gone up from there. Basically, the only time
Re:Dumb (Score:4, Insightful)
No, the WTF is not that the datacenter had a single point of failure. If their IT setup had been designed properly, that would have been a minor inconvenience. The WTF is that they didn't have at least three datacenters in geographically isolated locations with hot failover and regularly test the hot failover to ensure that it worked reliably and quickly in the event of a sudden, catastrophic loss of their primary datacenter.
Re: (Score:2)
One problem with a global business is WHEN do you test the failover?
Early Sunday morning in Atlanta? Well, it's the middle of the day on Monday in Japan, Australia, and a lot of other countries. Of course, because of business travelers, Monday is probably one of the worst days of the week for an airline to run a test - despite it being Sunday somewhere else.
Sometimes, there is no good solution, and you end up continuing with what you have because anything else is an even worse idea.
Re: (Score:2)
Then they should have switched. The cost of switching was probably a fraction of the direct and indirect costs of what happened.
Re:Dumb (Score:5, Interesting)
No one credible would count duplicate equipment in the same data center to be any kind of DR plan at all. That's like confusing RAID with backup. And just like you don't have a backup unless you've tested it, you don't have a DR plan unless you've tested it.
But a "disaster plan" needn't be limited to IT in any way. Air France had some sort of computing disaster recently, a similarly total outage, but they completed all their scheduled flights (not on schedule, but still). They had a disaster plan involving everyone behind a counter at an airport on the phone to a massive call center, where everything was verified "manually" from offline backup systems (and possibly print-outs). "Is Joe Slashdotter booked for flight 123?" "Give me a minute - yup, let him on the plane." Low tech, but it worked.
Re: Dumb (Score:2)
Or you fire the IT guy and put the blame on him. Our former company cut back and demanded a 99.7% uptime to a 100% after a budget cut even.
IT is a cost center which yields 0 return on investment compared to planes
Re: (Score:2)
IT is a cost center which yields 0 return on investment compared to planes
The proper response to someone who says that is, "How efficiently to you think you can schedule flights, book passengers for them, process them onto a plane, and keep track of all of the above if the IT infrastructure were to all disappear?"
Re: Dumb (Score:2)
Then you'd better make damn sure that doesn't happen or you're fired!
Re: (Score:2)
Exactly! IT is a profit center as much as anything else since it enables the business to operate. It's a lot cheaper than buildings full of people with adding machines.
Re: Dumb (Score:1)
They paid out dividends instead of having a pile of cash to sit on, if they need more money they should ask investors for more money.
Sitting on cash is good for the company, but in the mean time the investors could have their dividends, and invest in something else, which is what is best for the investor.
Re: (Score:2)
Re: (Score:1)
And if they don't invest in their reservation systems they obviously don't see that as a feature that sets them apart from their competition. That means they could decide to share the cost. Start building an open source airline reservation system and have each airline contribute some developers, for example. But it seems that a large part of the corporate world has become so competitive that this kind of cooperation doesn't even occur to them, especially in the US.
Re: (Score:2)
Re: (Score:2)
When times are good the executives buy new yachts. When times are bad the executives whine about needing a new yacht.
Investing in your business is for losers.
Re: (Score:2)
The problem was that they didn't make the best of things when times were good, and therefore deserve the cluster fuck situation they are in now.
Oh, I'm betting that a lot of airline execs made the best of things when times were good. Nice fat bonuses and a golden parachute.
What was the name of that game that we used to play as children? "Pin the Tail on the Donkey?" This game is called "Pin the Bill on the Stockholders and Consumers." The airline execs say, "See you on the Caymans!"
self servingly lies (Score:3)
continuous complaint about times are bad or union is rendering the business unprofitable has never stopped their officers from drawing ever larger compensation packages, nor has it prevented their board from approving those compensation packages. to claim that there's no money to reinvest into company infrastructure is but a self serving lie.
Re:Dumb (Score:4, Insightful)
You really only see this type of thinking in the West.
While there is certainly some component of that, it's not the major reason why things are as they are. Airlines in the USA are not owned or managed by the government. If it really came down to this, the US government might let all them go out of business and let new airlines be built out of the ashes. Switzerland did that. Plus, some airlines are actually owned by the government in the countries where they are based or the relationship is not all that independent. Air France, for example, is theoretically an independent company, but if they were going out of business the French government would surely step in and save them despite it violating EU laws to do so. The US airlines know that the government may not have their backs like they did the automobile industry. Plus, being publicly traded on the US stock market is probably for them a bad thing. This causes them to make decisions for short term profit to keep the stock price high. Finally, in the USA most customers who fly in coach only care about price and literally everything else is negotiable. They will make decisions on price alone. This puts pressure on the majors US carriers to compete at perhaps unrealistic price levels with the smaller airlines, which no doubt reduces money available to spend to upgrade old computer systems.
Re: (Score:1)
Re: (Score:2)
This assumes a couple of things:
1. That you will consistently have cycles of "good times" and "bad times".
2. That you will know when you are having a good time or a bad time.
If you master #2 in particular, you will be a very rich man. This is analogous to "timing the market" in investing, and it is a very difficult art.
Re:Dumb (Score:4, Insightful)
You really only see this kind of simpleminded thinking in people who don't know what they're talking about - but who can repeat something they read somewhere like a parrot.
The problem is, times have never really been good for the airlines for any extended period. An airline is a capital intensive business, and runs on paper thin margins. Historically, as soon as they get their head above water, they get pushed under again. The shift to jets in the 50's, the fuel shocks and deregulation in the 70's, recession in the early 80's, another recession in the early 90's, a race to the bottom in fares sparked by the rise of internet ticketing, the post 9/11 drop in business, the Great Recession of 2007... (just to hit the high spots) There's a reason why practically every major airline has ended up bankruptcy court at least once.
Re: (Score:2)
Well said.
On top of what you said. In times of good, companies do not (and should NOT) build up reserves or a war chest. Such things do not operate like most people think. These are publically traded and that changes things from privately owned. If a company builds up a ton of cash or cash equivalent reserves, it is looking to be bought out. When times are good (and there were very few for the airlines, mostly via asset acquisition from bankruptcies) companies tend to try to expand or take a risk into
'Strategic bad times? (Score:1)
Re:Dumb (Score:4, Insightful)
That's because in the West, the post-1980 business philosophy has been "Efficiency to the Max!"
You are expected to give "110% percent". Everyone is running Big Data analytics. The bean-counters scour the numbers and do cherry-picking to get the big profits and lemon-dropping to discard the losers. JIT inventory. On-demand elastic clouds. And all in the context of the next quarter's earnings report.
No sane general would commit troops without maintaining reserves for the inevitable unforeseen, but modern western businesses cannot stand a moment of wasteful "idle time" or resources sitting around unused.
And so, when the inevitable happens - train wreck. There's no spare parts, no idle people to put to work, nothing. No reserve capacity.
As they used to say back when computers were expensive objects of reverence, "Never before in human history has it been possible to screw up so badly on so large a scale so quickly". Such is the 2-edged sword of modern technology.
Re: (Score:2)
Sounds like a ripe opportunity to write some 'cloud based' scheduler that can handle all of this, open source it, and charge a consulting fee.
Re:Dumb (Score:4, Interesting)
The problem is that if all of your competitors are willing to go bankrupt every 15 years its' really hard to not go bankrupt before they do. It doesn't do you any good if they'll be going bankrupt in 8 years if you go bankrupt next year because you're 10% more expensive.
We see that in my industry all the time. Lots of people undercutting sustainable rates. They inevitably go bankrupt but if you don't match prices you'll go bankrupt waiting for them to go first. And since they're offering products at under cost they can also appeal to investors with fat grosses and rapid growth.
Imagine for instance you were trying to take on Amazon. Amazon hasn't really ever made money. But they can point to their rapid growth for long term investors. If you're an airline you probably won't see growth and it's hard to say "look we're losing a lot of money now but in 8 years when our competitors hit a hard time we'll make some money then until someone else comes along and promises to do what we do but cheaper and never go out of business." You see that with Jetblue and Virgin America. Jump into the industry with lots of investment. Offer a product at razor thin margins and capture a ton of market. But their business plan isn't tested to survive a big recession. So it's a gamble. They'll either do great or it'll reveal they were built on sand.
When your competitors are playing with fire it means they capture all of the revenue when times are good leaving you nothing to save for the "bad times" and then you only prosper when the market is crappy anyway and they can write off their debt in bankruptcy. It's a lose lose.
Re: (Score:3)
The problem with your example of Amazon is that Amazon invests every penny they earn back into the business. Companies like Delta don't. So when there are bad times, Amazon will be much better poised to do well because they've diversified and built up their business to handle it. All it takes is a generator to malfunction and Delta could be out of business forever (yes, a bit of a stretch, but still).
Delta has a virtual monopoly for a large swath of the nation. You have no choice to fly delta in the mid
Re: (Score:2)
^ This, so very much this. It seems like every week or so, I notice something new in the AWS console. A new feature, or service, or region...
Just today, I noticed that they've recently added a new class of ELB that works at the application level instead of the TCP layer. Device farm also caught me by surprise when it went online. I've a buddy who does QA on mobile who'd not seen it before either, and the look on his face when I showed it to him was priceless.
(Though I'm still a bit annoyed that EFS stay
Re: (Score:3, Interesting)
Were the airlines really in that tough a shape for that long of a period of time?
If they have only recently returned to profitability and actually experienced extended times of economic uncertainty, how do you explain Boeing outperforming the S&P 500 and gaining 8000% in value since 1978? Overseas sales explain some of it, but not all of it and an extended depression in American airline business you would associate with some decline in Boeing's business, but it's been continuous growth.
And airports and
Re: (Score:2)
Airline accounting in the US is like Hollywood accounting. If you don't get yourself a percentage of the gross, not the net, you're screwed. They manipulate the books to appear bankrupt so they can break pilot and stewardes unions and raid their pensions, while nickel-and-diming passengers to death, so the execs can get their multi-million golden parachutes.
Meanwhile, you can actually fly on Singapore Air... which trades the "best airline in the world" title back and forth with Emirates every year or so..
Re: (Score:2)
Didn't have time to look up the others, but Singapore Air is supported by the government of Singapore. They own over 40% of the airline, according to Wikipedia.
Not taking away from them, I know they're good and have a great reputation. But you're not exactly comparing apples to apples if you try to compare Singapore with, say, Delta or United.
Re: (Score:2)
You do understand that Boeing is not an airline? As always, the money isn't in panning for gold - it's in selling the pans. Not to mention drawing a straight line between 1978 and 2017 misses some deep valleys in bet
MOD PARENT UP (Score:2)
The parent is spot on.
And just to add to that, until their recent run of profitability, the last time the airlines as a whole were consistently profitable was in the 1990s, before the dot-com bubble popped. Between roughly 2001 and 2011, they cumulatively lost money (the one bright spot was 2006, but of course the Great Recession hit).
http://web.mit.edu/airlines/analysis/analysis_airline_industry.html [mit.edu] (apologies for the tiny image, but historical data more than 5 years out is typically paywalled).
It wasn't
If it isn't broke... (Score:2)
If it isn't broke, don't fix it. .... Wait. It is breaking, but you aren't fixing it?
Anyway.... Why is it breaking to begin with? Age isn't a problem in IT, are they adding features that can't integrate with this system? Are there too many transactions? What gives?
Re:If it isn't broke... (Score:5, Insightful)
The Delta outage was caused by a power outage. Seems like TPF is not the problem.
Considering how well this 1960s tech seems to be working replacing it may and doing it better may not all that easy.
Re: (Score:2)
Re: (Score:1)
Re: (Score:2)
I read somewhere that one of the airlines' problem was simple lack of redundancy of a power transmission switch. Not an IT problem at all, unless IT was responsible for the power.
Re: If it isn't broke... (Score:2)
It most certainly is an IT and business process failure. No redundancy? If system is too old to be clustered then I disagree with grandparent if it ain't broke don't fix it. Problem is it is!
There is a book out there called if it ain't broke break it! The death of any incompetent company is the one who follows if it ain't broke don't fix it. Successful turnarounds are never from companies who fear change and only do what works. It is those who change mindsets and actions.
Re: (Score:3)
Age can be a problem in IT if your system was designed for 1960's travel habits/workload and now has to cope with 2016 travel habits/workload.
Re: (Score:2)
But it was overhauled a decade ago (after 9/11 so it should account for those changes), the vendor is still in business and in reasonably good condition, and new hardware to run the software is still being made. When it comes to legacy systems, this one is practically a luxury item.
No, the problem is finances. The solutions exist, they just don't want to pay for them. How's that race to the bottom doing for you guys now?
Re: (Score:2)
One of the things that has changed is that back before the internet and sites like Kayak, Orbitz, etc the query rate for free seats and ticket prices was in the 1000s per hour.
With automated scrapers, those queries have gone into the millions per hour and it keeps climbing.
What you see now is TPM based mainframes with lots of middleware systems acting as buffers to handle the requests without querying the mainframe.
Re: (Score:2)
There's some merit to that as an architecture, even without the legacy problem. Having the mainframe dump transactions and schedules to a web front-end is probably the next-best thing to an air-gap.
Aging? (Score:5, Insightful)
What's wrong with aging tech? If most airlines are on TPF and TPF works and TPF is still maintained by IBM, what's the problem with TPF?
Something being old doesn't mean it's bad. Quite often, the reverse is true. The mainframe is still the king when it comes to reliability and transaction integrity, for example.
Re: (Score:3)
What's wrong with aging tech? If most airlines are on TPF and TPF works and TPF is still maintained by IBM, what's the problem with TPF?
Something being old doesn't mean it's bad. Quite often, the reverse is true. The mainframe is still the king when it comes to reliability and transaction integrity, for example.
It's not "mobile first, cloud first" as my boss would say .................
Re: (Score:2, Flamebait)
Re: (Score:2)
It wouldn't be a problem if nothing changed, but as the posting pointed out new features are constantly being added and if those features weren't considered when the system was first designed they set up being kludges. When you're piling kludge on top of kludge for 50 years, it can get really bad. So bad that one day you have to reboot and it doesn't work and no one knows why.
Sometime you just have to refactor. You can still keep any code that still works well (usually because it was refactored more recentl
Re: (Score:2)
Sounds like they just rewrote the operating system. Necessary to keep it running on newer hardware, but probably just a faction of the code.
Re: (Score:2)
Re: (Score:2)
They have gone a level above, test-driven-development to user filed critical defects driven development. Why pay for QA when the users will do the testing for free?
Re: (Score:2)
"What's wrong with aging tech?"The documentation isn't written in Hindi.
In order to pacify the "old code is still perfectly usable!" crowd I write all my documentation in cuniform
Re: (Score:1)
Re: (Score:2)
Its monstrously expensive to manage and maintain for one. Its essentially impossible to change, so adding new features also becomes monstrously expensive. I know some companies cut checks for millions per month. Consider that. Any industry run through oligopoly is charging huge premiums for the right. Making a new res system would probably cost upwards of 100mil, and you're not even guaranteed of a successful project in the end.
Re: Aging? (Score:2)
It's not webscale [youtube.com].
Southwest (Score:1, Interesting)
Southwest airlines reservation system is run off a IBM System/360 mainframe they inherited from Pan-Am. I'd be surprised if there was another functioning unit anywhere else in the word. You could probably emulate the whole damn thing on cell phone.
Automation has it's price after all (Score:2)
Who's surprised by this? In the quest for the lowest fare possible, who has money for preventing something that "might* happen that keeps aircraft on the ground, say like a power outage in your computer center? Apparently NOT Delta.. I'm guessing most of the other carriers too, they've just not been lucky enough.
Makes you wonder about all that expensive aircraft maintenance really getting done...
Think of that next time you strap one of their aircraft on for a few hours..
interfacing with DHS (Score:1)
The Sky is Falling! The Sky is Falling! (Score:1)
I call BS (Score:5, Insightful)
This is bullshit. Software does not "age" the same way that a car or a washing machine ages. The hardware can age, but the hardware can be replaced, and in this case we are talking about IBM software and hardware, which has a long-standing reputation for reliability and for maintaining backwards compatibility.
I think the more likely story is that the interfaces to these systems are being compromised. That's why it's happening, first at one airline, then another. Someone, somewhere is fucking around with the airlines' reservation systems.
I think these stories about "fires" and "aging" software is covering up for the fact that these systems are getting hacked. If people start to lose confidence in the systems they'll fly less or stop flying altogether.
Re: (Score:2)
The worst thing a res system can do is stop you from boarding a plane, any maybe to allow bad actors to avoid mandatory pre-screenings.
Ignorant managers don't recognize their ignorance. (Score:2)
Also, managers are dominant. They hire low-pay employees and don't train them so that they can make more money. Yesterday's Delta story: Delta Air Lines employees mistake New Mexico for Mexico [koat.com] (Aug 11, 2016)
So, which is it? (Score:2)
Re: (Score:2)
While it's true that the technology itself is "dated", so is Unix. Also, as TFS mentions: " is still updated by IBM, which did a major rewrite of the operating system about a decade ago." Of course, they could do a RAD/SCRUM/No-SQL/Other-Buzzword-compliant technology rebuild and achieve the same results, with no downtime and seamless transitioning, right? Right?
I guarantee you they can duplicate Monday's results. That's about all I can guarantee.
Re: (Score:2)
I can tell you that they try that with every major change in technology, and it usually fails.
One of the problems is that those old systems got to grow with the business, getting fine tuned along the way.
The new systems have to work right, and work right starting with drinking from the firehose.
I work in Dallas, Airline software is cancer. (Score:1)
Every time I have been looking for a job, 3 or 4 recruitroaches would call me daily about 3 month contracts with Southwest!!! great place to work!!!!111!!!.
you know you only get the primo talent when you are only willing to budget for 3 months of work at a time.
I used to work
Re: (Score:1)
I don't know about Southwest itself, but fare forecasting is basically a white rhino, it doesn't exist. 8 years ago I worked at a fare management company who's job was basically a simpler view of what the fuck the company (and competitors) were doing with the industry, because some airlines didn't even do that much (fire anf forget fare management.
Some business leaders and I (programming) demo'd predictive/trending fares into the product, but it went nowhere because nobody was asking for it, and nobody want
Ugh (Score:5, Informative)
Do reporters even read these stories as they are writing them?!? "Airlines will likely suffer more disruptions like the one that grounded about 2,000 Delta flights this week because major carriers have not invested enough to overhaul reservations systems based on technology dating to the 1960s... [TPF] is still updated by IBM, which did a major rewrite of the operating system about a decade ago."
Big, complicated system, written by a big, experienced company, still maintained... Do they think we'd be better off if it were rewritten from the ground up as a Ruby on Rails app or something?
Psst, I don't want to cause a panic, but I heard that large, important chunks of the Internet run UNIX, which also dates back to the '60s.
important things often neglected (Score:1)
This is what you get with offshored support. (Score:2)
Combine the worst qualities of offshoring & agency labor with the worst of legacy systems and you get this disaster.
Why old systems should be bad? (Score:2)
Not Defending Delta, But ... (Score:2)
Maybe they do need to upgrade their systems. And actually, Delta is making a profit right now. Maybe they have the money, maybe not. I don't know.
Final disclaimer: I don't know the details of what caused Delta's meltdown. But I'll share my own, much smaller-scaled personal experience to let you know why I shall at least hesitate before pointing fingers at the airline.
I work for the best company in US radio broadcasting. (Personal opinion, but there you go.) (Heh.) We are willing to spend the money on new eq
Focusing on the wrong systems (Score:2)
"But they have avoided the steep cost of rebuilding their reservations systems from the ground up,"
As somebody who had family affected by the Southwest outage 3 weeks ago, the reservation system was one of systems that remained up the longest. Southwest still could happily take your money even if nobody was going anywhere. (I suspect they manually took it down later it became clear the day was lost.)
Focusing on the reservation system sounds like a contractor lobbying to sell something...
Cost of upgrades vs biting the bullet (Score:2)
"Henry Harteveldt, founder of the travel consultancy Atmosphere Research Group, said some airlines are choosing to risk outages that might cost them $20 million to $40 million rather than invest, for example, $100 million on technology upgrades. He believes investors and the general public will apply increasing pressure on airlines to avoid outages at any cost."
How much did this cost Delta? Both directly and indirectly.
How much will preventing this single point of failure cost?
Re: (Score:1)
Re: (Score:3)
Re: (Score:2)
Based on my experience in the late 90's, to "webify" the access to these systems, you pretty much have it.
Re: (Score:2)
Re: (Score:2)
...and 60 years later most companies still can't get this shit right.
Re:is it that complex? (Score:5, Informative)
Reservation system could be implemented in chapter 10 of your first programming book. It seems trivial thing?
It's actually really really complex.
It's not just a "reservation system" where you lock out a ticketed space for X seconds until someone completes a transaction. You really have to view it as "The Company" when you're talking about airlines. Let's say a pilot has been in the air too long due to a delayed departure in New York. He hits his max flight time for the next 24 hours but he was scheduled to fly from his destination to another leg. So now you need to replace the pilot. Which pilot? Well there is a plane coming into the airport around the same time as the NY flight. But is that pilot rated to fly that same aircraft? Ok he is, great. But because 30 of the 300 passengers are going to miss their connections now because of the delayed arrival they need to be moved to different flights. But those flights are maxed out. So you have to bump some passengers on a scheduled flight and move them to a later flight as well. Because the plane is getting in late it's also going to depart late. So you also need to either arrange all of the passengers at the next destination to be on different flights and set of a chain reaction or you need to pull in a different plane at the 2nd destination to short circuit the chain reaction. But where can you get a plane from for the cheapest? And how much will it cost to put people up in a hotel vs flying an extra crew in on overtime?
This is all simple enough to calculate with like 1-2 planes. But when you have 1,000 aircraft and all of the seat assignments effectively being interdependent along with business interests (profit/loss of changes), customer service interests such as ticket class... and you have to stay up to date instantaneously with dozens of terminals all trying to do the same thing manually in addition to the automatic callbacks for unexpected events... it's big engineering effort to not create some sort of automatic-trading style feedback loop that accidentally sets off a chain reaction that cancels every flight in the country.
Every change has a cost. No human can orchestrate thousands of interdependent variables with millions of passengers manually. You have to have a central director system which instantaneously handles all of the callbacks and dependencies for a change throughout the entire graph.
It's actually very cool when you stop and think about how well it does at keeping everything relatively straight.
Re: (Score:2)
The UN once wanted a single simple reservation system. The concept was simple, and the requirements were very simple compared to what most people would assume. Users were equivalent to a small international employee base... all could do English. Vendors were very few and contracts were simple.
Except it had ONE requirement that made it impossible: There should be ONE standard.
Simple enough, except each party said "Yes, we demand one standard, as long is its ours."
Never underestimate the shear stupidity of
Re: (Score:2)
Re: (Score:2)
The issue isn't simple number crunching and database retrieval, it's the fact that it's global.
Complete and utter bullshit (Score:2)
A casual glance, for example, at Delta's 2015 FY statement shows total operating revenue of $28,898 million, and net income of $4,526 million -- that's 11% profit, inclusive of cargo. Looking at passenger traffic specifically, they show an operating cost per available seat
Re: (Score:2)
What does that have to do with using TPS on a mainframe?
Wrong fucking cover sheet was used. That's what's it has to do with it!
Re: (Score:2)