Forgot your password?
typodupeerror
Bug Transportation

Computer Glitch Friday Grounded US Airways Flights 140

Posted by timothy
from the when-will-southwest-come-to-knoxville dept.
mschaffer writes "A computer glitch Friday night snarled the travel plans of US Airways customers, as reports flooded in of flights grounded around the country." As someone stranded for several hours yesterday by this outage, "glitch" seems like quite a euphemism. With outgoing flights blocked, and new ones arriving full of passengers expecting to meet connections, the atmosphere got a little heated. Customers could see nice weather, and planes lined up outside, but "The System Is Down" trumps all. The E concourse at Charlotte (a US Airways hub) was packed full of customers ranging from livid (a handful) to merely angry (most) to calmly resigned — which means those of us with seats, snacks, and books or computers. It was disheartening to see how brittle is the infrastructure the airline employs; with the part of the system visible to airline employees down, customers thought they might get more information, or even rebooking, through the US Airways website. But that was down, too, and all the desk staff could do is shrug.
This discussion has been archived. No new comments can be posted.

Computer Glitch Friday Grounded US Airways Flights

Comments Filter:
  • by Kensai7 (1005287) on Saturday June 11, 2011 @11:32AM (#36411704)

    OK, let's count to three and blame COBOL! :p

  • umm... (Score:3, Interesting)

    by datapharmer (1099455) on Saturday June 11, 2011 @11:33AM (#36411708) Homepage
    that sucks. No backup paper system in place? Can't they just read what the tickets say such as flight and seat number? They know where the flights are going as most are routine. It seems they should have been able to get *some* flights in the air.
    • Re: (Score:3, Interesting)

      by linest (157204)

      You can't fly unless you can prove your aircraft has had all required maintenance done. There are also rules about the number of hours per day crew members are allowed to be in the air. I suspect these records could be printed and used if it were a planned outage but this wasn't.

      • by Anonymous Coward

        they were faxing the necessary paperwork. Was supposed to pick up a friend @ SFO coming from Charlotte. They stated the systems were down and they had to fax all the pilot's info in order for the plane to take off.

    • Re:umm... (Score:5, Informative)

      by DesScorp (410532) <<DesScorp> <at> <Gmail.com>> on Saturday June 11, 2011 @12:08PM (#36411940) Homepage Journal

      I work in airport IT, so I'll describe what I see airline crews doing during trouble. If the system at one gate or terminal is down, yes, then they'll send the plane on it's way. This is called "boarding manually". They simply hand collect tickets, hand count bags, etc, and send the flight off. After they've gathered all of the info thats been collected manually, they'll send it to their local office or front desk and process it at working terminals that have a connection to airline systems. It's a pain, but do-able. But if EVERYBODY is down, then the whole thing grinds to a halt. If no one has any access to all the schedule info, weight and baggage, manifests, etc.... then it's simply impossible to board manually on a massive scale.

      • by FlyingGuy (989135)

        Not to mention hoping the pilot gets the weight & balance correct, not to mention fuel consumption ( which none of them have had to calculate in years ). Most times the plane datalink has the correct information but I have seen a commercial jet parked at the gate because the feed from the computer was down.

        Making assumptions based on the standard FAA passenger w/standard baggage is tricky business not to mention dangerous since that has killed more then a few passengers and crew especially stops at reg

    • Re: (Score:3, Informative)

      by DerekLyons (302214)

      No backup paper system in place?

      One of the reasons they went to computers in the first place is because paper systems could no longer handle the workload... And that was back in the 60's when air traffic volumes were a fraction of what they are today. I.E. having to maintain a duplicate paper system would actually slow things down likely without actually providing sufficient backup.

      Can't they just read what the tickets say such as flight and seat number?

      That isn't much help with getting the luggag

      • by jd (1658)

        I know Slashdotters don't always have the best manners, but this isn't Kuro5hin. We still have standards.

        Whilst I agree that paper backup is probably out of the question, most computers are quite capable of handling multiple ethernet lines and most routers are capable of supporting hot standby configurations. Even cold standby is a 30 second failover. The same goes for backend servers - it doesn't take much to add a checkpoint/failover system (cold standby) and it's quite possible to configure most servers

        • Re:umm... (Score:5, Insightful)

          by digitig (1056110) on Saturday June 11, 2011 @03:03PM (#36412960)

          This simply isn't about the problem. It's about whether the solution has been implemented. Nothing more.

          And that is down to whether it is cost-effective to implement the solution. You will never be able to get the probability of failure down to zero. and the cost skyrockets the closer you get to zero. How often do outages like this happen, and how much would it cost to prevent them at every airport worldwide? And to prevent every other conceivable scenario? Yes, it could have been prevented, and lots of other possible outages that didn't happen could have been prevented, but the cost of air tickets would be prohibitive.

        • I know Slashdotters don't always have the best manners, but this isn't Kuro5hin. We still have standards.

          They vary wildly between 'unreasonably high' and 'ludicrously amusing', but yes, Slashdot has standards. (And very occasionally, the have some relevance to the real world.)

          Asteroid takes out a data center? Well, then you've probably got bigger issues, but co-locating across the country is Standard Practice for most instustries.

          If 'most' industries had a system even half as complex - you'd have a

          • by jd (1658)

            Having worked in places with hundreds of remote offices, for that matter having worked at CERN on data collection for nuclear accelerators, I think I might, just might, have an idea of what it takes to keep large numbers of systems in sync over continental distances.

      • by egburr (141740)

        Overall, I agree with what you said. Only one part didn't make any sense, though: "With the system down they don't even know when/where plane 'A' is in order to get butt 'X' onto it."

        Couldn't they just look out the window? Maybe that big thing outside the window that a whole bunch of people recently walked off of just happens to be the plane everyone at the gate is waiting to board?

        Yeah, just loading it up and taking off would make a huge mess of the paperwork, but don't tell me they can't find the plane.

        • Couldn't they just look out the window? Maybe that big thing outside the window that a whole bunch of people recently walked off of just happens to be the plane everyone at the gate is waiting to board?

          Maybe, but probably not.

          If you land a plane at an airport, and go and park at a gate, what are the odds that the people waiting for a flight at that gate are supposed to be on the same plane that you just parked there?

          Even if you get lucky, how is anyone supposed to even know what gate they are supposed to be

          • by egburr (141740)

            "If you land a plane at an airport, and go and park at a gate, what are the odds that the people waiting for a flight at that gate are supposed to be on the same plane that you just parked there?"

            Pretty good odds, actually. Every time I've been waiting at the gate when a plane arrived and disgorged passengers, that was always the same plane that we eventually got on. Except for one time they had to replace it due to mechanical problems.

            It doesn't make any sense to pull up to one gate, empty the plane, and t

          • by egburr (141740)

            "Even if you get lucky, how is anyone supposed to even know what gate they are supposed to be at?"

            It's printed on my ticket. Except for the rare last minute gate change, it's been pretty accurate.

            With the computers down, it can't be perfect, but for *most* of the flights, the available data is still good.

        • by timothy (36799) * Works for Slashdot

          One of the things I saw yesterday was gate switches; the desk staff don't know until they get the (apparently very sporadic) updates about things like that, so people are often directed to distant gates rather than their originally scheduled ones. And when it happened yesterday, I saw two different flights (to different cities!) both being sent to the same gate, and I'm pretty sure they weren't both right. Glad I just had to wait, and it was my final leg.

          timothy

        • Maybe that big thing outside the window that a whole bunch of people recently walked off of just happens to be the plane everyone at the gate is waiting to board?

          Yeah, just loading it up and taking off would make a huge mess of the paperwork, but don't tell me they can't find the plane.

          Assuming the plane is at the gate - what about an hour before when it's still a couple of hundred miles away? What happens if it's delayed, or never even takes off? How do they route the luggage since it doesn't hav

  • sometimes its a pain

    • by Anonymous Coward

      I'd just like to ask timothy to revisit this post after he's had some months or years. Being stranded for several hours is pretty frickin' minor as far as bad things that can happen when flying or in life in general; calling it a 'glitch' isn't much of a euphemism, although I might call it a 'major glitch'. Bad things seem much worse when they happen to you, but when you're an adult you're supposed to be able to get some perspective on it.

      • by timothy (36799) * Works for Slashdot

        Yep -- I was fine.

        Like I said: snack, seat, plenty to read ... my Slashdot work for the day was done, too, so no one else was being much inconvenienced by my travel delays in particular. (It's the other people, with connections etc, who had bigger hassles.)

        This is so far from the worst thing that could happen even among modern travel disruptions that I hope you take my account with the same viewpoint I had: it's a 21st century problem / first-world problem, and that's the best kind of problem to have. Once

      • by russotto (537200)

        I'd just like to ask timothy to revisit this post after he's had some months or years. Being stranded for several hours is pretty frickin' minor as far as bad things that can happen when flying or in life in general; calling it a 'glitch' isn't much of a euphemism, although I might call it a 'major glitch'. Bad things seem much worse when they happen to you, but when you're an adult you're supposed to be able to get some perspective on it.

        No, when you're an adult you're supposed to be able to take whatever

  • Ive had more problems with us air and united than any other airline. Theyre incompetent
    • by oneiros27 (46144)

      And that would match with J.D. Power's customer satsfaction reports for 2011:

      http://www.jdpower.com/travel/ratings/airline-ratings/traditional/ [jdpower.com]

      Although, when you compare the "Low Cost" airlines, there's a few others that had similar bad ratings:

      http://www.jdpower.com/travel/ratings/airline-ratings/low-cost/ [jdpower.com]

      • by nwf (25607)

        And it would match Consumer Reports who rated usair the worst of any domestic carrier. I'd link, but they keep almost everything behind a paywall. Usair used to be really good, until they merges with America West. United is second to last and that's who I'm waiting for now at the airport. Only 75 min late currently.

        Since united and continental are merging, I'll bet the combined airline will be far worse than either alone.

    • by Drathos (1092)

      I've never had any major problems with USAirways (a few delays here and there, but only one more than 30 mins). I actually flew through CLT yesterday, but I guess I got out before the problems started.

      Delta, on the other hand, is a constant problem. Left stranded in ATL twice, many delays, and terrible customer serice. Maybe if they didn't overload ATL, it wouldn't be so bad (that doesn't help the customer service issues, but it might with the others). I avoid them whenever possible.

  • Why isn't there a backup available in case a glitch occurs?
    • Because they've calculated the customer apathy and money lost is less than implementing backup procedures. Remember it's all about $.
      • by linest (157204)

        Because they've calculated the customer apathy and money lost is less than implementing backup procedures. Remember it's all about $.

        I'm not crazy about the way that's phrased, but you are essentially correct. Establishing backup data centers, populating them with hardware, purchasing additional software licenses, establishing, testing and maintaining fail over procedures is nontrivial. When you consider the overall health of the airline industry, it's not surprising that the extra tens of millions of dollars were not spent.

        It'd be interesting to know how many millions of dollars this will end up costing US Airways. I'll bet accepting th

        • by jd (1658)

          That depends on how you define cost. The instantaneous cost is one part, but only a part. Even so, that cost isn't just frustrated customers. It's parking costs for the aircraft, staff wages, any technician overtime needed, costs due to food spoilage, loss of in-flight sales, etc. Delayed costs also matter. There's any loss of future custom to consider, since that is also a cost to the company. There's any increase in insurance costs for them as a result of any successful claims. It may well impact the airl

          • by digitig (1056110)

            The problem is that many aren't quantifiable - too many unknowns - so an airline is incapable of knowing if a backup system is cheaper or not.

            But the airline still has to make the call. Pay for a backup for system x or take the hit if it fails. And they can only sensibly make that call if they can make an estimate of how much the hit will cost and how often it's going to happen. And don't forget: multiply redundant systems can fail, too. I've seen a power system based on main and standby UPSs fed by different power supply companies, backed up by main and standby generator sets, each of which comprising main and two standby generators, go down com

          • by linest (157204)

            The problem is that many aren't quantifiable - too many unknowns - so an airline is incapable of knowing if a backup system is cheaper or not.

            I like that observation a lot. It could be carried further. You're talking about actual risks. Real decisions are made based upon perceived risks that sometimes consist of little more than assumptions. Especially once you get out of the IT realm and need something paid for. Since these are IT risks and they need to be communicated to non-IT people, there is a challenge there. It's not easy.

            Your comment about software licenses being free for disaster recovery, on the other hand, seemed a bit too off hand. I believe the issues are the databases for crew scheduling and aircraft maintenance. That's the stuff that'll keep you on the ground. To my knowledge, there is no dominant application for aircraft maintenance packages. If that data is stored in an Oracle database, you're going to pay big bucks for DR licenses. On the other hand, if we assume that US airways is using a Jeppesen package for crew scheduling (I'd bet a small amount of money on this), then it relies on a 10 year old version of Informix. Disaster recovery for the database server software would (in my experience) be free. IBM's OK that way.

    • Why isn't there a backup available in case a glitch occurs?

      It's called "risk management." Let's say a backup system would cost $150M over 20 years, and the current system is calculated to fail every seven years, at a cost per failure of $30M (cost in terms of lost business / OT / brand damage etc.). Running without the backup system you're many tens-of-millions of dollars ahead in the game. These figures are just made up, but these sorts of calculations go on all the time in many different industries.

    • There was a backup plan in the suit pocket of the CIO, but it was casual Friday, so everyone was, you know, wearing a Hawaiian shirt.
    • Utrecht is a central city in Holland where pretty much all the railway lines intersect for no smart reason.

      So, if there is an issue at Utrecht like a fire alarm at the control center, there better be a backup or all train travel in Holland is seriously affected.

      Luckily the backup control center is there.... right there... in the same building... small building... affected by the same fire alarm...

      But hey, lets not immidiately order a load of busses to deal with stranded passengers, people have become so use

  • I dont know what the "outage" was, but it seems redundancy is an afterthought with US Airways.
    So, I can assume the following
    There is no redundancy(zing)
    There is no Recovery plan
    There is no DR plan
    There is no SoP on releases
    There is no SoP on testing
    There is no SoP on handling outages
    Bravo! Now, did those new TSA rules go into effect yet? Can US Airways be fined into oblivion because of this?

    Also, what are the extra fees US Airways charged its passengers for having to handle their complaints and a
  • I know IT fully embraced Patch Tuesday leaving us with up to a month's worth of accumulated crud, but now they've gone too far!

    • I read it exactly the same way. I guess using "A" and "on" was too much extra typing. :D
    • by jd (1658)

      Maybe the computer glitch is an artificial intelligence called Friday. It then called up the airports and ordered the planes not to take off.

  • by Anonymous Coward

    FTFA: "The Tempe, Ariz-based carrier cited a power outage near one of the airline's data centers in Phoenix as a possible cause."

    A POWER OUTAGE?! So, no UPSes, no generators, and no multiple utilities at a main data center for a major company? Come on now...

  • What?! (Score:1, Insightful)

    by Anonymous Coward

    I love it when people melt down and scream and yell. Fucking christ. Shit happens. Shut your mouth, go over to the airport Starbucks and buy yourself an overpriced airport coffee and calm down. You're not helping. You're not the center of the universe, you twit; in fact, they're probably not even going to even up the scales.

    (I'm one of those in the calm catagory when it comes to Emergencies, or (more likely) "emergencies.")

    • by egburr (141740)

      A cup full of caffeine and sugar is supposed to help you calm down?

      I used to carry a couple extra books in my carry-on bag, as I have seldom had a flight even come close to being on time (except for connecting flights which are almost always on time no matter how late my initial flight is).More recently, I just make sure my phone charger is with me, so I can keep my battery topped off as I read ebooks or play games on my phone instead. My only real complaints are that I have yet to find an airport with even

      • Yeah, I also thought that Starbucks wasn't the best example.
        Last I I flew, it was for once it was good that I tend to overpack, as that in part included extra reading material that I hadn't gotten to the rest of the trip.

      • by Abstrackt (609015)

        My only real complaints are that I have yet to find an airport with even slightly comfortable seating and that there is never any place at all to get away from all the noise of the multiple TVs tuned to multiple stations all at full volume and the incessant security announcements just in case there is someone in the airport who hasn't flown in the past 20 years.

        HHGTTG actually got this one right: bring a towel. You can use it as a cushion on an uncomfortable seat or even as a pillow if you want to nap on the floor. Just make sure it's a thinner towel, ideally a travel one, so it doesn't take up too much space in your bag/suitcase. As for the noise, IEMs are a godsend. Once my music or ocean sounds are on I don't hear anyone or anything else, perfect for those six-plus hour stopovers.

  • I specifically chose United over USAir for travel yesterday as I've had the most trouble with them. However United was also in poor shape yesterday. It was termed 'operational delays', with two hour delays across the board. Calls into United faced 25-30min wait times. And many overbooked flights.

    Seems the whole industry is going down the tubes... and decreased competition from these mega-mergers are not helping.

  • by Cutriss (262920) on Saturday June 11, 2011 @12:18PM (#36412002) Homepage
    As I am now located in proximity to an airport with a US Airways "service focus" and have had the "pleasure" of flying with them several times, I have to ask - how would you be able to tell the difference? Every time I've been in a US Airways terminal, there's always a significant number of non-weather-related delays and cancellations (compared to the other airlines' monitors). My wife and I have independently had three separate incidents this year where we were 4th and inches from having to stay overnight at an airport due to cancellations/late planes/overbooked crew/etc. In two of those cases, I had flights where we took off at the 2'55" mark, just shy of the three hour requirement to return to gate and let everyone off. The cynic in me suspects that US Airways is actually using that three hour window to plan its flights.

    It's an abhorrent mess, and when I see the US Airways CEO defending against his last place customer service ranking, I have to wonder just how much denial one management team can stand.
    • by jittles (1613415)
      See I have the exact opposite experience with US Air. Free upgrades to first class, and have not experienced any more delays than any other airlines. My worst experiences have been with Delta and AA.
  • by Anonymous Coward

    ...before computers came along and made our lives so much -easier-?

    • Re: (Score:3, Insightful)

      by 246o1 (914193)

      ...before computers came along and made our lives so much -easier-?

      Not fly anywhere, because it was too expensive.

    • by blair1q (305137)

      they weren't supposed to make your life easier.

      they were supposed to make your ticket price leave more profit in the airline's pocket.

      all those people who couldn't get anywhere? just how many got their money back?

      the system is working exactly as designed, even if the passengers don't get to feel all special about it.

    • Back in the 60's when airlines started computerizing, air traffic volume was a fraction of what it is today. You'd have to be nearly fifty to have even been alive at a time when computers weren't starting to make our lives easier in a variety of ways. ("Computers" !=" PC's".)
       
      I remember trying to make airline reservations back before the web. You couldn't pay me enough to go back to those days. (Unless you could also give me my twenty year old body as well.)

  • by xkr (786629) on Saturday June 11, 2011 @01:10PM (#36412342)

    I fly US Airways regularly. Last flight out was late taking off for no apparent reason. Our luggage did not make the connection in their own Hub. Neither did anybody else's. It took over an hour for the luggage clerk to process the long line. I counted over 500 keystrokes required per person. Staff didn't care at either airport. They would not put out luggage on the next plane in (another airlines, and they would have to pay a fee to that airline) so it was over a day to get out luggage. Two days, or three, unless we came back to the airport to pick it up. On the way home to SFO, it took over an hour for them to get out luggage onto the carousel. They had the nerve, over the PA system, to blame the passengers for having, "too much luggage," for the delay.

    Consumer Reports rated US Airways at the bottom of customer satisfaction.

    Planes fly. Southwest regularly makes last second changes, including flag stops (unscheduled) and re-using planes for "second runs."

    There was LOTS that US Airways could have done. First, they could have flown the planes if they wanted too. They planes had already been scheduled, so there were no questions of maintenance or fuel, or flight plans. Second, they could reimburse passengers for the delays. Third, they could have rescheduled some passenger.

    Then, of course, as said, there is simply no excuse for the IT to be down for that long, if at all. They had no (working) backup systems, either computers, paper, or people. That is the very definition of incompetent.

    I work in IT. As a guy said in my last meeting, “Anybody who designs in RAID 5 should be shot.” Duh.

    The fact is that the airlines management is incompetent. This is not an opinion. Simply too many facts. The board should completely clean house. When the questions comes up in the next board meeting of, “What to do?” the answer is, “Duh.”

    • by T-Bucket (823202)

      You are obviously clueless as to how an airline is run.

      They could not have just "flown the planes if they wanted to". Despite scheduling the flights, EVERYTHING is handled by computers now. Those flight plans you mentioned? Yep, filed by a dispatcher USING A COMPUTER. The performance calculations that determine how much fuel that flight plan will require? Yep, computer.

      Add to this the fact that the computers also control gate assignment, weight and balance, baggage routing, etc etc. There is NO WAY a m

    • by Xacid (560407)

      “Anybody who designs in RAID 5 should be shot."

      This runs counter to everything I've been taught - so perhaps you have some real world advice to support this? Merely curious.

      • by IQgryn (1081397)
        RAID 5 is good at losing a second disk soon after the first, even with a hot spare (which means you lose the array). I think the GP meant something like "RAID 6 should be the bare minimum", or perhaps "any level of RAID is not as good as having an independent backup system".

        But I'd like to know what they actually meant, too.
        • by lanner (107308)

          Yea, that's pretty much what he meant.

          The reality is that an entire flight scheduling system like the one that US Airways uses could probably be replaced by $50K worth of junk off of dell.com. The software has to be written custom, but this isn't computational proteomics here. A couple of SF bay goons could do this in six months.

          For this kind of a small-scale implementation, you should have at least three separate data centers across the world/continent, which duplicate the information with automatic term

          • by adri (173121)

            Wow, and that likely completely underestimates the scale of this kind of project.

            Chances are there's lots of systems all tied together, feeding data in and out. There may not be a "small scale implementation". This may be a pain in the ass to integrate into lots of these inter-operating system.

            So maybe the system-as-a-whole is flawed, sure, but redesigning that to be less of a clusterfuck is likely not within the scope of your "$50k and 6 months."

            Then there's keeping everything in sync. You end up having lo

    • by Idbar (1034346)
      Even worse, they have managed now to blame anything and everything on weather so they don't have to reimburse people. Such that when they should have a properly deicer working (but they don't), they'd blame it on the cold weather.
    • There was LOTS that US Airways could have done. First, they could have flown the planes if they wanted too. They planes had already been scheduled, so there were no questions of maintenance or fuel, or flight plans.

      If it were a static problem, you'd have a point. But it's a dynamic problem involving dozens of airports, hundreds of aircraft, and tens of thousands of passengers and their luggage. Not to mention that flight plans are filed immediately before departure... so, no system, no flight plan. Not t

  • by Anonymous Coward

    Given a choice between two competing airlines flying to the same place, the vast majority of passengers will book based solely on published cost.

    Leaving aside for the moment the question of hidden fees, that means that the airlines have no choice but to trim every possible cost to be competitive. I'm not in the airline industry, so I'm guessing here. I suspect that means no extra flight crews on standby for unexpected events, no extra gate crew coverage, the absolute minimum of phone lines to handle probl

  • by Etraud (2256864)
    Closer to Caos... lol
  • I love the weather comments. They show that people just don't think on a large scale. I work at Dallas Love Field and I went home an hour late last night because of bad weather in the morning on the other side of the country. Most people would not realize that.

  • The word "backup" is often confused with "practice" ... backups who needs backups?

  • Please, how about printing all the important information ad a 2D barcode on the tickets (cryptographically signed. of course), which can be read without a connection to the central system and having for the really important stuff = scheduling of planes a second independent system. I always wonder that the cost of these sw bugs could easily go into the tens of millions of $ , so it should be possible to take measures not to be completely dependent on a single point of failure.

Did you know that for the price of a 280-Z you can buy two Z-80's? -- P.J. Plauger

Working...