FAA Computer Glitch Causes Widespread Airline Delays 133
seven of five writes with this excerpt from an Associated Press report: "A problem with the FAA system that collects airlines' flight plans caused widespread flight cancellations and delays nationwide Thursday. It was the second time in 15 months that a glitch in the flight plan system caused delays. The FAA said in a statement that it is having a problem processing flight plan information. 'We are investigating the cause of the problem,' the agency said. 'We are processing flight plans manually and expect some delays. We have radar coverage and communications with planes.'"
Here I sit... (Score:3, Interesting)
...stuck in Atlanta...
Re:Here I sit... (Score:4, Funny)
It could be worse. You could be stuck in Lodi (again).
Re: (Score:3, Funny)
Sorry to hear that. But, as the old saying goes, "it's better to be DOWN HERE desperately wishing you were UP THERE, than UP THERE desperately wishing you were DOWN HERE."
Re: (Score:1)
No it is supposed to go:
"Here I sit, broken hearted, came to s**t and only farted."
Re: (Score:1)
Some come here to shit and stink
and pick crabs off their balls;
I come here to sit and think
and write stuff on the walls
re problem (Score:1)
Re:Hmm could it be a windows problem? - of course! (Score:1, Informative)
Link to deployment announcement about "modernization" of airspace network.
http://www.stratus.com/news/2005/20050314a.htm [stratus.com]
Re: (Score:2)
apparently people didn't understand the difference between the question mark at the end of my sentence and a statement, since people decided to mark it a troll. Do people not know what a question mark means? I was hoping someone would reply with actual informative info.
Re: (Score:1)
Do you know what 'troll' means? I await you informing me of your informative information, informant.
Info.
Re: (Score:2)
maybe you should do a little research, because I did.
http://www.stratus.com/news/2005/20050314a.htm [stratus.com] Do you notice something? They installed windows due to defining it as an "open platform". They were deceived. This is the result.
Re:Hmm could it be a windows problem? - of course! (Score:4, Informative)
Well, I work w/ the FAA right now, and they're becoming relatively platform agnostic actually. ERAM [wikipedia.org], for instance, was written by Lockheed Martin to run on top of some flavor of IBM UNIX or Linux (forget which). In the old days, everything ran on custom, purpose-built hardware and OSes, but that really turned out to be a maintenance nightmare. Using COTS Hardware/Software means upgrading systems or providing new capabilities can be pretty easy.
I think ASDE-X [wikipedia.org] runs on top of some sort of POSIX type OS as well... I know its data stream is standard UDP over Ethernet type stuff.
Re: (Score:2, Insightful)
problems? (Score:1)
Re: (Score:3, Interesting)
By "glitch" they mean "totally offline delaying and canceling flights".
I'm pretty sure they had lots of other bugs and "regular glitches" in this time.
On the other hand, I'm also pretty sure that this kind of software does indeed go through a much better development and verification process than most commercial software around.
Re: (Score:2)
No, by glitch they mean glitch. "totally offline delaying and canceling flights" is the result the glitch.
"On the other hand, I'm also pretty sure that this kind of software does indeed go through a much better development and verification process than most commercial software around."
the scheduling software? probably not. It was done out of house for contract by people who don't give a damn the moment the software releases.
Re: (Score:3, Insightful)
Re: (Score:2)
I'm not sure about where the FAA's flight plan software falls in, but I'm guessing that since it's not safety critical, and only an operational risk...
Considering that flight plans exist to prevent planes from crashing into one another, I'd lump them into your category of "safety critical."
Re:problems? (Score:5, Informative)
Considering that this did not affect the development of flight plans, only delayed them, I'd rip this back out of the category of "safety critical".
Online development of flight plans is much faster than offline. But that's precisely what makes the offline ones safe - as you delay the development of flight plans you also reduce the number of planes being cleared for takeoff at the same time. Once the flight planning software goes back online, you can start granting flight plans quickly again and get traffic volumes back to normal.
And it's a bit of a stretch to assume a risk of a collision even in the complete absence of flight planning. ATC, Radar, airplane internal collision avoidance, and pilot eyeballs are all unaffected by this. The only thing this could possibly do is put two planes in the vicinity of each other, but even that is unlikely as the sky isn't "the wild west" - it's carved up into clearly-defined highways with speed limits and defined routes which are all used in flight planning. So if you have two planes in the vicinity of each other, they are going in the same direction at roughly the same speed, so there's TONS of time to react.
And the first precaution, as we've seen here, is to reduce the number of clearances. Arguably, this makes the actual flying safer, since there are fewer planes up there for ATC to have to track and communicate with.
Re: (Score:2, Informative)
Re: (Score:2)
And some of which still runs on cryo-cooled Univac machines...
Re: (Score:2)
I'm not sure about where the FAA's flight plan software falls in, but I'm guessing that since it's not safety critical, and only an operational risk...
Considering that flight plans exist to prevent planes from crashing into one another, I'd lump them into your category of "safety critical."
Last I looked, this was mainly an issue near airports - airspace is really huge, and I don't see flight corridors as hugely risky.
Re:problems? (Score:5, Informative)
Sorry, this is flight planning/scheduling software. This is the system by which airliners are told what route they are to take when they finally get their flight clearance.
The primary reason this has not received "oh my god the sky is falling" priority is because, well, it isn't. Nor are shiny metal tubes blasting through the sky at 500 miles an hour going to smash into each other because of this. People will be inconvenienced and that's regrettable and needs to be fixed, but this is not a safety issue.
- Radar control systems: Unaffected.
- Air Traffic control: Unaffected.
- Communications: Unaffected.
- Landings: Unaffected.
- Any flights already in progress: Unaffected.
- Ground control: Unaffected.
- Passenger Safety: Unaffected.
This only delays the granting of flight clearance. The planes that are inconvenienced by this are safely on the ground, and the effect is that they stay on the ground longer than they should. Again, this is regrettable, but not fatal. There is no plausible scenario that would lead from this to a failure of traffic control. In fact, with fewer planes in the sky, you could argue that flying is actually safer (for those people who are lucky enough to have gotten clearance, of course!).
In the medical field, this would be a failure of the system that the receptionist uses to schedule your next appointment. In the automotive analogy, this would be a failure of your garage door opener. In the heart monitor analogy, there isn't even an analogy because heart monitors don't have (as far as I know) any non-critical systems.
Re: (Score:1)
Not receiving clearance means flight delays and flight cancellations. That translates to lost revenue to the airlines; inconvenience and possibly lost wages to the traveller; and inconvenience and possible lost revenue to to the travellers' employers. It is not a trivial matter as the losses could add up to millions of dollars per hour. Suer, no-one will die beczause of it but many will lose money.
To use one of your analogies; your garage door does not open, you are late for a sales meeting and lose a $1M c
Re: (Score:2)
1) Money
2) Management
If nobody see the need for spending a certain amount of money for a system, then you may get a system which fails on occasions.
If once every 15 months is acceptable, then no need for more money.
But if you want a more robust system, then more money may need to be spent.
Which would mean that companies would need to be giving the FAA more money. So suing the FAA would defeat the purpose of what they are trying to achie
Re: (Score:2)
I was responding to a post that implied that this failure could cause a midair collision. It could not. I didn't address the possible economic impact of the failure because that wasn't the topic I was responding to.
But, since you brought it up, what would be the economic impact of fixing it so it could never, ever fail under any circumstances? Moreover, is that even possible?
Keep in mind that flight safety stuff is standalone. Each zone has their own RADAR, their own control systems, and is a self-conta
Re: (Score:2)
Re: (Score:2)
LOL, good point.
Let me refine that: Not an FAA safety issue.
FTFM (Fixed That For Myself).
The TSA, on the other hand, might consider it a wee bit of a safety issue.
Re: (Score:1)
I ran an airline for a time, and I see where you left off a few critical sub-systems.
- Dispatch Planning (what cooks up the flight plans in question for submission, specifying the fuel burn, max takeoff weight, planned landing weight, operating mach speed, altitude, etc.)
- Operations Control (where every plane and crew is, in real time.). Think of a Gantt chart that would make your jaw drop, it's kind of like that.
These are the things that make an airline run. The FAA only has one responsibility -- keep a
Re: (Score:2)
>This is the system by which airliners are told what route they are to take when they finally get >their flight clearance.
>The primary reason this has not received "oh my god the sky is falling" priority is because, well, >it isn't
You lack of imagination leaves to me to think you would be a bad programmer.
See I could tell you to take off runway 7, which in itself would be harmless, however , the glitch comes from the same software that told the other air traffic controller that it was ok to use
Re: (Score:2)
The system works like this:
1. Pilot works with Ground or calls Flight Services and files a Flight Plan. Flight Plan includes:
- Airport you are taking off from (A).
- Airport you intend to land at (B).
- Airports you will use as a "contingency" if things go badly (C,D,etc).
- What routes you intend to use to get from A to B.
This allows Flight Services to figure out how many airplanes will be at altitide in different areas at any given time so they can control congestion enroute
Re: (Score:2)
> If Flight Planning is down, Flight Plans cannot be filed.
Already you are wrong in your assumption, as per the article states,
they were doing this manually.
This error on your part would cause all subsequent points to be null and void.
You would be at home with those developers responsible for the glitch!
Re: (Score:2)
No, if you read some of my other posts I already stated that this outage did NOT mean Flight Planning was down. It meant that flight plans could not be submitted automatically.
Flight Planning was not, in fact, down during this incident. It was severely delayed because the Flight Plans could not be submitted automatically and the manual process takes longer. However, I was making the point that Flight Planning is NOT involved in Flight Operations (Air Traffic Control, and Flight Services) and a theoretic
Re: (Score:2)
To clarify, I was responding to this point of yours:
"See I could tell you to take off runway 7, which in itself would be harmless, however , the glitch comes from the same software that told the other air traffic controller that it was ok to use runway 7 to land"
If you told me to take off on RWY7, you would be an Air Traffic Controller. Your software (if you used any) would be contained within the tower. It is not Flight Planning software, it is Flight Operations software.
The article make absolutely no me
Re: (Score:2)
Sounds like you have experience in the field, I like the fact they have backup plans
for these type of things...I just hope that most of what you say takes into consideration the last time they were stuck using this was way back when we did not have as much air traffic as today...
I remember hearing someone telling me all about their backups, and the technology used, and everyday they made backups...until the day they had to restore, that's when they found out the backups were no good....could have been avoid
Re: (Score:2)
PS: You are confusing "Air Traffic Control" and "Flight Services" with "Flight Planning". They are three separate entities.
An "Air Traffic Controller" is a person (or multiple people) sitting in a tower or other control area. They have one primary job - to make sure planes don't smack into each other while in their domain. They sequence takeoffs and landings, taxiing, etc.
ATC has absolute and utter control within their domain. If a Controller tells you to jump, you start jumping then ask "how high?" on
Damn Excel (Score:4, Funny)
Some one messed up their $2,000,000 excel macro that list morning.
Honest! (Score:5, Funny)
I only changed one little line of code! It wasn't even important enough to test!
Re: (Score:3, Interesting)
HAHAH, around my work you could add "or Document" to the end of that statement. I cant count the number of times people have changed things with no documentation and they did not consider it important.
Re: (Score:2)
Document? I don't understand. What is "Document"?
Re: (Score:1)
Re: (Score:2)
I think it's a Microsoft product. That's why not many around here use it.
Re: (Score:2)
No you see the problem was that someone tried to comment out a comment. However, their skewed vision on how comments work, and by some MIRACLE of programming and compilation, All the code ended up commented and the comment was compiled instead.
what do you expect from 70's technology ?? (Score:2, Informative)
Re:what do you expect from 70's technology ?? (Score:5, Insightful)
So when will they upgrade? (Score:1)
We've heard about the antiquated ATC system for over a decade. Does anybody on the inside know if, and when they'll upgrade?
Re: (Score:2)
They are also phasing in more GPS approaches (nav systems specifically designed for landing) and phasing out the NDBs (non-directional beacon) and taking failing VORs out of service that are often used at fields which have (ILS) instrument approach runways.
That said, it is a very slow process. If your car's speedo goes out, who cares. If the
Re: (Score:2)
Assuming that most planes are flying where
GPS speed - headwind > stall speed
unless they're on take-off or approach, wouldn't a GPS plus a laser wind-speed detection system at airports solve that problem?
Re: (Score:2)
Re: (Score:2)
It's an emergency when your airspeed indicator is malfunctioning, and it is the most direct indication of the amount of energy relative to the airstream your aircraft has. It's not something you want to muck with by putting multiple layers of indirection/abstraction (and thus additional things that can fail) between the sensors and the indicators.
The GP was mentioning failures too - why not have a backup system available if this is something which happens enough to be concerned about?
Re: (Score:2)
a) It assumes the wind at the ground station is near the wind speed and direction of the aircraft. This is almost always a wrong assumption, as the wind speed rapidly increases with altitude, and tends to cant in quite a different direction as well. For a sensor that has to be accurate within a knot or two, this is completely in
Re: (Score:2)
a) It assumes the wind at the ground station is near the wind speed and direction of the aircraft. This is almost always a wrong assumption, as the wind speed rapidly increases with altitude, and tends to cant in quite a different direction as well. For a sensor that has to be accurate within a knot or two, this is completely inadequate.
That's why I suggested the laser systems above - I don't understand exactly how they work, but by shining lasers through air masses they can determine wind speed, density, e
Re: (Score:2)
This brings up the greater question of how old must a technology be before it is considered mature enough for purposes as mission-critical as air planes. 5 years? 10 years? 20?
I think that reasonably, you ought to be able to trust technology that's been around for a decade.
Re: (Score:1)
Not necessarily, if the technology was speced to handle 100,00 flights per day and they are trying to pump 200,00 flights per day through it, the software may not be able to handle it no matter how old it is.
Re: (Score:2)
C++ stripped to essentially C may have hidden problems
That sounds about right, yeah.
Re: (Score:2, Insightful)
They are comfortable running systems on ancient mainframes running ULTRA.
I would be comfortable doing the same thing.
Why do people shit on mainframes? They may cost a lot of money and time, but they're orders of magnitude more dependable than any other server solution that exists.
Yes, that includes the "cloud".
Re: (Score:2)
Re: (Score:1)
Uh, that's a very small part of the reliability.
The hardware reliability comes from having fault-tolerant, hot-swappable, redundant, well-tested hardware. The "cloud" hopes to achieve this, yet doesn't.
The software reliability comes from having fault-tolerant, well-designed, well-tested software. This takes time, sure, but it can just as easily be done with forward-thinking to allow for growth and change.
The problem is not mainframes, mainframe programmers, or PHBs resistant to change. The problem is peo
Re: (Score:2)
Re: (Score:1)
Terrible analogy.
Re: (Score:2)
Resolved... (Score:3, Informative)
...according to the Wall Street Journal. [wsj.com] Wonder if they'll give me a lift home?
Re: (Score:2)
All true, but note line 1 of TFA:
Re: (Score:3, Insightful)
Even if it's fixed, there will still be delays for a while. There are probably tons of flights that are still awaiting clearance and the requests are being dealt with as quickly as safely possible. Meanwhile, a normal load of requests is still coming in, and those go into queue as well.
Trouble with things like flight planning is that they are, well, planning. If they go down, the manual process can't keep up with the load, and a queue develops. Once the system comes back up, it's gotta munge through the
Re: (Score:1)
I bet their landlines were still working.
obviously (Score:1)
Re: (Score:2)
Re: (Score:2)
Re: (Score:1)
Maybe it's like the internet... (Score:2)
Already fixed before it even got posted on /. (Score:3, Informative)
http://online.wsj.com/article/SB125863837097855555.html?mod=WSJ_hpp_LEFTTopStories [wsj.com]
To quote some technology middle manager (Score:4, Funny)
"That's it, you're all grounded!"
Glitch is now fixed (Score:3, Funny)
Apparently they fixed the glitch so the problem worked itself out naturally.
Re: (Score:2)
Foot TCP (Score:2, Insightful)
We all know how large out of touch behemoths sometimes structure their IT. By 'packet switch' they mean 'guy who couriers hardcopy flight plans' and by 'database mistmatch' they mean their dewey-decimal-system was mixed-up by some jokesters.
Honest... (Score:5, Funny)
**ducks**
Just another great goverment run program... (Score:3, Interesting)
Re:Just another great goverment run program... (Score:4, Insightful)
Under their control, 87,000 flights a day cross the skies of the US. Despite incredible crowding at and in the airspace surrounding a large number of airports, collisions are one-in-billions events. The vast majority depart and arrive without undue delay. (And anyone expecting no delay in such a dynamic system with so many variable is smoking some good stuff.)
In this incident, a problem was detected, backup procedures implemented, the problem was fixed, and full functionality was restored - all in a matter of hours without halting the system.
I'd be the first to admit that there are a lot of badly broken government programs - but in this case, you're just blowing smoke.
Re: (Score:1)
Re: (Score:2)
Repeating the usual party line like a parrot, all the while ignoring the points I made, very impressive.
Not.
Re: (Score:3, Interesting)
That's true. The U.S. GOVERNMENT built the best air traffic control system in the world back in the 50's, one so robust, well designed and well managed that it has been the safety and operational count leader world-wide
Re: (Score:2)
The radar technology used by air traffic controls is from the 50's.
If by this you mean "the basic technology was invented in the 1950s," then sure. If you know of a better general-purpose way to find out where things are in the sky than to bounce radio waves off of them, I'd like to hear it. That said, the systems which process and display radar data however have received several updates since the 50s.
No GPS...
Are you referring to ADS-B? I
Re: (Score:1)
The major concern of ADS-B is that is is susceptible to hacking (you can broadcast your own "phantom" aircraft if you know the standard) and reduces the anonymity of private and business aircraft users (every ADS-B transponder broadcasts a code that uniquely identifies an individual aircraft).
Aren't the pilots already doing this with their radio (the code being the flight/tail number)?
Re: (Score:2)
Re:Just another great goverment run program... (Score:5, Insightful)
Yea, these are the guys (the feds) I want building cars, taking over health care...thanks god they are not building the planes. I'm just trying to think one government run organization that works as well as any private one. Any idea?
This morning I was woken by my alarm clock powered by electricity generated by the public power monopoly regulated by the US Dept of Energy. I then took a shower in the clean water provided by the municipal water utility. After that, I turned on the TV to one of the FCC regulated channels to see what the national weather service of the national oceanographic and atmospheric administration determined the weather was going to be like using satellites designed, built and launched by the national aeronautics and space administration. I watched this while eating my breakfast of US Dept. of Agriculture inspected food and taking the drugs which have been determined as safe by the food and drug administration.
At the appropriate time as regulated by the US congress and kept accurate by the national institute of standards and technology and the US naval observatory, I get into my national highway traffic safety administration approved automobile and set out to work on roads built by the local, state and federal departments of transportation, possibly stopping to purchase additional fuel of a quality determined by the environmental protection agency, using legal tender issued by the federal reserve bank. On the way out the door, I deposit any mail I have to be sent out via the US postal service and drop the kids off at the public school.
After work, I drive my NHTSA car back home on the DOT roads, to a house which has not burned down in my absence because of the state and local building codes and the fire marshal's inspection, and which has not been plundered of all it's valuables thanks to the local police department.
I then log onto the internet which was developed by the defense advanced research projects administration and post on freerepublic.com and fox news forums about how SOCIALISM in medicine is BAD because the government can't do anything right.
Re: (Score:2)
Bravo, bravo! A few minor nits to pick and some stylistic points, but against such a performance I'll let them pass. Again, bravo!
Re: (Score:2)
I have a lot of problems with government and many departments within that bureaucracy, but the FAA is not one of them. It's pretty amazing what they do and there probably aren't many other entities out there that could do what they do.
Re: (Score:2)
A government-run organization that works as well as a private one: US Postal Service.
Seriously. Those guys move tons of stuff around the US every day, and significant portions of daily business would grind to a halt if they stopped. Think about it: for around 50 cents, you can send something from Maine to California and expect it to arrive within a few days. They make extensive use of OCR, have an extremely sophisticated sorting and routing system, and are in many ways more efficient than FedEx or UPS (for
Re: (Score:1)
Considering how many companies have failed in the past 200 years, how the financial industry was central to our current economic crisis, and how we're still living under the same government we had 200 years ago, I'd say your faith in the private sector is severely misplaced.
But don't let that stop you.
I had a longer comment prepared, but I am going to shorten it. The government's failures would be as easily producible as failures of industry. Even worse is that government departments are usually not created until a real need is noticed - which should increase their chance of success. Also, your example of the financial industry is a joke, and not relevant to any discussion of a free market. Lastly the proposition that we are under the same government we were under 200 years ago is only acceptable upon a
I'm this many. (Score:2)
It was the second time in 15 months that a glitch in the flight plan system caused delays.
Thanks for the arbitrary use of months. Is it a baby, or were you working from fortnights and thought you might as well round to the nearest lunar cycle and then convert to Gregorian? Do you also append "and a half" to your age, as appropriate?
Yeah, right. A cynic's view. (Score:3, Interesting)
Re: (Score:2)
Re:Yep... (Score:4, Informative)
I just read a post on Facebook by an Air traffic controller I know. They had to e-mail or fax all icao [wikipedia.org] flight plans to the FAA [wikipedia.org]. The FAA manually typed in every flight plan for every flight in the country.
Re: (Score:2)
You and dtmos (see post further down in this thread) should get together for a beer. You're both stuck in Atlanta, and it's after noon. :)
Slashdotters meeting in person? Whoda thunk it. LOL.