Boeing Identifies New Software Problem On Grounded Boeing 737 Max (bloomberg.com) 77
An anonymous reader quotes a report from Bloomberg: Boeing Co. engineers have discovered a new software problem on the grounded 737 Max that must be patched before the plane can return to service, Federal Aviation Administration chief Steve Dickson said. A light indicating that the stabilizer trim system wasn't working properly "had been staying on for longer than a desired period," Dickson said, without providing more detail. The issue involves an alert designed to warn when the so-called trim system, which helps raise and lower the plane's nose, isn't working properly, according to two people familiar with the issue who weren't authorized to comment on it.
It's not clear how much of a delay, if any, the issue will create as Boeing finalizes numerous fixes required to get its best-selling plane back into service. One of the people familiar with the trim alert problem said it's not likely to change Boeing's projection of returning the plane to service by mid-2020 because the company had built padding into its schedule. The trim alert issue resulted from Boeing's redesign of the two flight computers that control the 737 Max to make them more resilient to failure, the two people said.
It's not clear how much of a delay, if any, the issue will create as Boeing finalizes numerous fixes required to get its best-selling plane back into service. One of the people familiar with the trim alert problem said it's not likely to change Boeing's projection of returning the plane to service by mid-2020 because the company had built padding into its schedule. The trim alert issue resulted from Boeing's redesign of the two flight computers that control the 737 Max to make them more resilient to failure, the two people said.
Two Flight Computers? (Score:3)
Donâ(TM)t you ideally need 3 redundant systems to reliably âoevoteâ on which system is correct?
Boeing needs a FAR better CEO! (Score:2)
Boeing needs a FAR better CEO, someone who is socially sophisticated, extremely technically knowledgeable, and who enjoys technical detail. Boeing managers need to "manage by walking around". That means they must be located where the airplanes are being manufactured, not in Chicago.
Boeing Employees Mocked F.A.A. and 'Clowns' Who Designed 737 Max [nytimes.com] (Jan. 9, 2020)
You can download "Internal Boeing communications about the 737 Max [nyt.com]". (PDF
Reliability: 3 systems are needed. (Score:2)
Re: (Score:3)
The problem with digital systems and software, you can have hundreds of system that all report wrongly due to poorly written software and numerous points of digital failure. This does not even touch software security, exactly how hackable are modern airliners and how readily can they be crashed by false data.
How much software should be allowed in a modern jet, should it all be required to be done in hardware, permanently written in, the only way to change it, swap out the chip. Doing it is software just bec
Re:Reliability: 3 systems are needed. (Score:4, Informative)
Doing it is software just because it is cheaper, is insane, it should all be done in hardware, permanently written in and chips swapped out.
How do you know this isn't how software changes are done already?
Software that is burned to a ROM versus an FPGA that has the logic burned in sounds like a distinction without a difference. In either case this can mean a system that cannot have the instructions it runs modified in flight, or even on the ground without prying out a chip and swapping in a different one. It's been a few years since I worked on aircraft hardware, and none of them involved the systems that controlled the attitude control surfaces, but all the software was burned into ROMs. Even so the very nature of a flight computer does not allow for easy modification of the software. It's not like there's a USB port on the flight deck to plug in a flash drive, or an Ethernet port to plug in a laptop. What you get is a circuit breaker to turn it on and a keypad that's somewhat similar to those found on an old flip-phone. There's buttons on the sides of the displays to hit menus, there's a numeric keypad, arrow keys, and maybe a few other buttons. There isn't a full qwerty keyboard, and nothing like a file system to store arbitrary files.
There's no doubt that these flight computers are complex. There's no doubt these computers run on many many lines of code. What they don't have is any kind of "hackable" interface to modify critical functionality.
I assume every student of computer science and/or engineering will learn that there were two architectures of early computers. There was the Von Neuman (or Princeton) model and the Harvard model. The Von Neuman model of computer architecture is how most systems are built today, this is a model with a shared instruction and data memory. The Harvard model, which is quite rare today on consumer electronics, has separate memories for instructions and data. It's the Harvard model that is used on systems like flight computers because of the inherent security this provides. No matter what the user might enter for data, or how furiously they type at the interface, there is no means by which the data entered could corrupt the instructions. They are in separate memories which will render so many "hacks" used on consumer electronics impossible to do anything to the instructions that the computer runs.
Then there's the matter of the data these systems get. What contributed to the 737 MAX crashes were failed sensors and limited means to detect this failure. From what I understand the MCAS was able to detect a failure of the single sensor it relied upon for input. The pilots were not necessarily given instruction on just how bad this kind of failure could be for them, as even the people that built MCAS didn't know. A single AoA sensor failure would be indicated with a warning light, and based on that the pilots had a procedure to deal with it. A failure of both AoA sensors might mean this was not detected, and no warning lights. At least no warning lights until this failure became more apparent with some other problems. My point is that even if the software is "perfect" the system can fail because of bad data from another system. There's only so much that the computer and operator can do to make sure the system is getting good data.
Separating the instructions from the data removes many security issues. Having redundant sensors removes many problems of getting bad data. What seems to be at issue here is a problem of bad data not being detected combined with it being acted on in a way that can lead to a cascade of events that is difficult or impossible to recover from. It sounds like this latest issue is a matter of a false positive warning, which can create it's own cascade of events that might be unsafe and the crew would have to deal with this.
Re: (Score:2)
There's no inherent reason why doing something mechanically should be better than doing it in software. Mechanical engineers can and do screw up too.
What you need is a proper process. Have multiple software systems written in different languages by different teams and compare the results. NASA did that on the Space Shuttle I believe. And then properly test it.
Re: (Score:2)
Re: (Score:2)
Yes, "3 redundant systems" for reliability.
It is unnecessary to have 3 deep redundant computers IF you have redundancy through other means.
In this case, they simply have an indicator that says (Hey! The two computers disagree!) and then the pilots decide which one is right. So, you get a "master caution" alarm when the two computers are in disagreement and then you hand the pilots a check list to run when the alarm sounds.
So yes, multiple redundant systems but NO, two computers are fine and meet the requirement when the pilots can intervene.
Re: (Score:2)
I dont really want the issue of two systems being in an arbitrary state of disagreement.
It's like having two watches. Without a third source, like the Sun, you don't know what time it is.
Re: (Score:2)
MCAS is not a critical system at least in the sense that not having it is not a big deal. It is crtical in the sense that it can crash the plane. Not knowing the correct time isn't going to crash the plane though and neither is not knowing the correct AoA.
The real reason they don't need 3 AoA sensors is that the new software shuts down MCAS entirely whenever they disagree. Without the autocrashing MCAS engaged the pilots can just fly the plane as normal and hopefully the bad AoA sensor will get replaced eve
Re: (Score:1)
Re: (Score:2)
I disagree. The MCAS is not a flight critical system, in that pilots can fly just fine without it.
At issue is the avoidance of low speed stalls, which pilots are trained to recognize and avoid. The MCAS system was intended to help them recognize as they were approaching a stall and help them avoid it. I believe the 737 MAX has multiple systems designed to avoid stalls. There is the AOA sensor indicators, a stick shaker and the stall warning horn, all which indicate you are approaching a stall. The MCA
Re: Boeing needs a FAR better CEO! (Score:1)
Re: (Score:2)
That's laughable. If pilots could "fly just fine without it" it would not be there and Boeing could have avoided trying to hide it. I've yet to find a company that adds complexity and cost to their systems for shits and giggles.
You assume they tried to hide it. They could have simply overlooked the impact of this system like they claim.
I think a more likely scenario is that the MCAS design was done in stages to address specific problems discovered during flight testing. The system was initially seen as something of an "autopilot" system, similar to all sorts of things that automatically adjust stab trim. Put down the gear? Stab trim is adjusted. Add some flaps? More automatic trim adjustments. It's common, even the autopilo
Re: (Score:2)
Re: (Score:2)
But it's not arbitrary. You have a general clue as to what is going on.
If the two sources are wildly off, you generally can figure out which clock is saying the correct time based on the state of the world and other things - the two don't just produce two results that you don't have a clue based on historical data.
So if one
Re: (Score:2)
Um... If computers are flying the aircraft and the pilots cannot intervene then yea, you need more than single fault tolerance.
In this case, we have two pilots and two computers. If the computers disagree, a warning light is illuminated and the pilots sort out which one is correct. In the 737 they are actually flying the thing and have controls which are directly connected to the control services (with hydraulic assistance of course). When the computer's are not in agreement, you just turn them off and t
Re: (Score:2)
Or you can just turn off a noncritical system like mcas when there is any sign off incorrect inputs such as a large enough disagreement between the two sensors. The pilots just have to see the mcas disable warning light and assume that the computer has noted some erratic inputs probably from an AoA sensor. I guess if autopilot depends on AoA readings it will also have to be disabled though which means the pilots will have to be able to fly the aircraft manually.
Re: (Score:2)
Yup.. And THAT is the real issue here, the pilots didn't know what the MCAS system was doing or that it was off in the weeds in some fault mode as it was doing it. Had they known what was happening, the fix is easy.
Re: Two Flight Computers? (Score:1)
Re: (Score:2)
It doesn't matter how many redundant computers you have if the software they're running is incorrect.
A friend of mine's father worked at Draper Labs on the Space Shuttle's fight control computers. The shuttle had, if I recall *five* redundant computers that checked each other's results for errors. He thought this was a terrible idea, that the complexity of the solution introduced more uncertainty into the system than simply trying to make one or two computers more reliable. He was proved right on the very
Re: (Score:2)
I'm just saying complexity imposes certain limits on how certain you can be about a system. Sure two computers is clearly better than one. Maybe three is better than two. But is five better than three? At some point the statistical gains in reliability you get on paper are questionable; a sufficiently complex system exhibits behaviors that aren't possible to anticipate or find in simulations.
As for the cost, it only delayed the launch by two days; I assume when they discovered the fault was simple enough
Re: (Score:1)
Not a pretty picture, but plausible I think.
No, that is not plausible.
Re: (Score:2)
Only if you have no diagnostic capability. For completely dumb systems (think AOA sensors) you need 3 sources of information. For intelligent devices a large number of failure modes can be covered with only a redundant pair.
The number of possible failures modes covered is a matter of careful design (so let's not pretend that Boeing did a good job there).
Re: (Score:2)
You got a ton of replies to this question, but none seem to have the correct answer:
Yes, you need 3 systems to reliably vote. The flight computers were designed in the 70's where 2 computers were considered sufficient for the purpose of the 737. Changing to 3 computers would require reprogramming the entire flight software which would also mean recertifying that part of the plane (and likely recertifying the pilots as well, as it would almost certainly change how the software reacts to anomalies). This was
Mod parent up (Score:2)
Yes you do. When you only have two systems, you know that one's right and one's wrong, but no idea which one is which. In a system with 3 computers, a simple voting protocol (the two systems that agree are right) is absolutely critical to safety. And if you're going to Mars [scientificamerican.com], it's even harder as cosmic rays can cause memory to change.
"We had to actually HIL test our components." (Score:2)
And we found severe deficiencies in what we said we did the first time with the MAX8, but we really didn't because it was "already certified" and we only changed a little bit of the code.
Meanwhile GE and all the other suppliers with equally poor quality control and 'oh just throw that in it's already certified' mentality are breathing a sigh of relief that they're not the ones at fault, this time.
I would prefer to never fly on this plane. (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
A professional racing driver can probably stop a car without brakes too. It was sometimes possible to fight MCAS and win with the old software, but it was very tricky and depended on certain conditions like air speed being low enough. It was most definitely not in any way 'safe'. If grounding was not necessary it would only be because AoA sensor failures are rare and not because it is easy for any decent pilot to fight MCAS and win.
Re: (Score:1)
Re: (Score:1)
Re: (Score:2)
It will be thoroughly tested and audited, but the fundamental flaws of the design will not have been fixed.
That would require a complete redesign of the flight computers to a modern standard plus retrofitting longer landing gear and moving the engines. It is not feasible to do those things to already-built planes.
I thought for the longest time that I would be happy going on the MAX once it was recertified. I have changed my mind since. I will never be flying on a 737 MAX, and I will carefully consider wheth
Re: (Score:2)
It is not feasible to do it to the 737 at all. Longer landing gear cannot be retrofitted because it has no overwing emergency slides. Adding them would result in so large changes of the airframe that it will lose its grandfathered certification and it can't be recertified because it is built to old standards.
Re: I would prefer to never fly on this plane. (Score:1)
The 747 was a magnificent plane. I still see a few of those majestic birds in service internationally - but alas, they are mostly gone.
For newer Boeing planes I agree with your sentiment.
What troubles me is that they blamed the pilots... (Score:5, Insightful)
...The issue involves an alert designed to warn when the so-called trim system, which helps raise and lower the plane's nose, isn't working properly...
They blamed the pilots, their training or lack thereof, and stuck to this mantra for months. Talk show hosts made things worse by toeing the line [of Boeing].
The fact that the crashed planes were from the so called "third world", did not help quell the story line.
In their face, Boeing, with so much experience building planes, could not go wrong. "It was their fault" they kept saying - those 3rd world fellas.
We learn later on, that their own "engineers" had pointed out [safety] issues.
I am saddened about all this, especially talk show hosts who never really investigate stories before airing them. It's sad.
Imagine the terror when the planes were going straight down with nothing to do about it in time. I am afraid of that Boeing MAX plane.
Someone needs to go to jail.
What troubles me is that they blamed the FAA (Score:5, Interesting)
The blame shifting that Boeing is doing makes me wonder why some of these execs aren't in jail. Jail is a valid option for punishing people who caused 300 people to die.
Re: (Score:2)
If corporate executives went to jail for the deaths they were responsible for then tobacco and asbestos company executives would be the first in line, having covered up the evidence that their products caused cancer for 45 years.
Re: (Score:2)
If corporate executives went to jail for the deaths they were responsible for then tobacco and asbestos company executives would be the first in line, having covered up the evidence that their products caused cancer for 45 years.
There it is right there!
Re: (Score:2)
Commercial passenger airline pilot training (and certification) is horrible at some airlines in some countries. In particular, I urge you to read this article [nytimes.com] that discusses the disaster that is pilot training and certification in some countries and on some carriers. This article, for example, exposes that in spite of normal simulator training being done with three people - the two pilots and an examiner/instructor, some Indonesian airlines would have seven in the simulator - two pilots flying, one instruct
Re: (Score:2)
Commercial passenger airline pilot training (and certification) is horrible at some airlines in some countries. In particular, I urge you to read this article [nytimes.com] that discusses the disaster that is pilot training and certification in some countries and on some carriers.
It's funny that that article specifically mentions Navy fighter pilots. A lot of former military fighter pilots, especially single seat pilots, have difficulty transitioning to civilian, commercial flying. The handling is different, the controls can often be different, and most of all they are very uncomfortable with CRM. They aren't used to flying as a crew so their communication and coordination can be lacking, which is very important if there is an issue inflight. The bust rate is actually significan
Re: (Score:1)
Re: (Score:2)
McDonnell Douglas is after all the company behind the DC-10 cargo door debacle/disaster. They identified an issue with the cargo doors not latching properly, but still giving the impression of being properly latched in their own pre-production tes
Software bugs vs design flaws (Score:5, Insightful)
The complete design is flawed. The concept of using a computer to stabilize a plane because its wing and engine design makes it unstable should be completely eradicated. Now, they have detected a software bug, but they simply cannot solve the fundamental problem of the plane because it is inherent to the design.
Boeing should return to the old 737 (not MAX) while they design a proper successor from the ground, with proper and specific pilot training. I will never fly in a MAX.
Re: (Score:1, Insightful)
Re:Software bugs vs design flaws (Score:4, Informative)
As repeated in every other MAX story - there can be no return to the 737NG. Right now, it would take a year to ramp up the NG production line again, minimum - suppliers have ramped down production (including engine suppliers) and switched over to the newer toolings required for MAX production. Switching back would require suppliers to switch back to the old tooling (if it even exists) and that will take significant amount of time.
Aside from that, switching back to the NG would be an economic disaster for Boeing, even more so than this current production halt is - yes, they could produce NGs, and yes airlines would take them, but at a *huge* discount along with significant ongoing fuel burn penalties being paid by Boeing. And airlines would dump them very quickly when better aircraft came along, with Boeing having to guarantee resale values as well.
The MAX, in some form or another, is the way forward for Boeing - anything else will see Boeing exiting the narrow body market, probably permanently.
Re: (Score:1)
Re: (Score:2)
As repeated in every other MAX story - there can be no return to the 737NG. Right now, it would take a year to ramp up the NG production line again, minimum - suppliers have ramped down production (including engine suppliers) and switched over to the newer toolings required for MAX production. Switching back would require suppliers to switch back to the old tooling (if it even exists) and that will take significant amount of time.
Boeing has over 400 MAXs sitting, assembled, ready to be delivered. That is in addition to all the other parts they have already received from suppliers before assembly shut down. It will take a year or 2 at least just to deliver all of those aircraft that have already been completed (and that doesn't include the installation of any fixed that have been found since the groundings, possible additional inspections/testing of the aircraft themselves since they have been sitting for months, etc). Airline acc
Re: (Score:2)
Even if you ignore the timescales (I was being conservative), you are ignoring the fact that the NG is a second rate product that no one wants - airlines bought the MAX for a reason, that reason doesn’t go away just because the MAX might. Airlines will drop their orders without penalty and go elsewhere, and Boeing will be left with the dregs who couldn’t get slots at Airbus.
Make no mistake, while Airbus is currently production line slot restricted, there are ways for it to increase production t
Re: Software bugs vs design flaws (Score:2)
With Airbus, dont forget that they will soon have a production facility in Mobile making the 220-100 and -300
Re: (Score:2)
For Pete's sake, handling characteristics of all aircraft have issues, especially at the edges of the flight envelope. The 737 is not unique in this, nor is this the first instance of handling problems with the 737's design. Past models have had issues too and all sorts of aerodynamic tricks and adjustments have been made over the years. I see all sorts of vortex generators, strakes and appendages on all sorts of aircraft, designed to "fix" issues at the edges of the flight envelopes.
The issue is not uni
Re: (Score:2)
Re: (Score:2)
The "real" solution is a clean sheet design to replace the 737. However, that's a decade and a few billion dollars away from flying.
The 737 MAX is not a "really different animal" except at the edges of the flight envelope where subtle differences in aerodynamics are amplified. The MCAS was designed to help one of these differences not "feel" as different, and except for the unfortunate failure mode, does just that.
The correct solution depends on the problem... In this case the problem is really a cascade
Re: (Score:2)
Stall avoidance should be a primary skill of all pilots. They should NEVER approach the stalling AOA when maneuvering, doing so is a stick and rudder skill problem. In fact, you should NEVER approach the edges of the flight envelope without extreme caution and only with good reason. The MCAS doesn't get involved until the air speed is low enough and the AOA high enough that you are dangerously near the edges of the flight envelope. Edges that "felt" different on other versions of the 737. While that does explain how 737 Max pilots could find themselves on the edges, it doesn't really forgive them for pushing things that far.
In both cases, the pilots were not at the edges of the flight envelope. MCAS got involved because of faulty, non-redundant AOA sensors.
Here's the flight data from ET302: https://leehamnews.com/wp-cont... [leehamnews.com] . Take a look at AOA-R (blue), and AOA-L (red). The red line pegs when the AOA-L sensor fails (it's the one that MCAS uses). You can see from AOA-R and airspeed that they were aviating fine (200+ kts, low AOA) until MCAS stepped in.
You can see a similar story in the Lion Air flight data: https://stati [seattletimes.com]
Re: (Score:2)
Have you asked yourself why the 737 is several tons lighter than an A320 despite the latter being a far more modern design using newer materials?
The answer is that the 737 is built to 1950s design specifications. It is lighter because
1) it is flimsy. This is why it tends to break in several parts after a runway overrun, like the one happened a couple of days ago.
2) it has only two hydraulic lines instead of three.
3) it has no overwing slides. In case of an emergency the passengers are supposed to slide over
Re: (Score:2)
3) it has no overwing slides. In case of an emergency the passengers are supposed to slide over the flaps.
That's ok, you just keep the wings low. Passengers already regularly break their ankles sliding down the slides anyway. Of course, if the wings are low and you want to use bigger engines, you just have to change the mounting and mount them more forward on the wings. This can change flight/handling characteristics, but that can be easily overcome with software.
Re: (Score:2)
Have you asked yourself why the 737 is several tons lighter than an A320 despite the latter being a far more modern design using newer materials?
The answer is that the 737 is built to 1950s design specifications. It is lighter because
1) it is flimsy. This is why it tends to break in several parts after a runway overrun, like the one happened a couple of days ago.
So, you are upset because an aircraft comes apart during an accident? That's stupid. There are all sorts of examples of structural problem in aircraft, but the safety factors in currently manufactured aircraft are quite safe up to their rated loads. Accidents usually exceed these rated loads, even the "gentle" ones like you describe. This isn't a design flaw, nor is it a safety issue. Unless you are trying to claim aircraft need to be designed to survive accidents.. To which I respond "What kinds of ac
Re: (Score:2)
This is exactly my point. The 737 is rated to loads that were good enough more than half a century ago when deaths in an airplane crash were considered an acceptable outcome. The standards today are far more demanding, but the 737 is grandfathered. It is not safe at all in comparison to modern jets. Its floors are too weak to support the modern 16G seats, which makes the aircraft even more unsafe in the case of
Re: (Score:1)
also, too. money
boeing is trying desperately to spend as little money as they can to make as much money as they can. that's their goal, and there thinking is oriented around that.
if they actually made designing quality software, or a quality airplane the goal, then that's what they would get.
meanwhile they try to do the minimum they can and then attempt to deceive people that they got it right.
and yes, they are absolutely going to do the math as to how many planes they can allow to crash to still make money
Re: (Score:1)
Re: (Score:1)
Re: (Score:2)
The complete design is flawed.
The concept is no such thing and every fly by wire type aircraft has some form of computer adjustment to its flight characteristics. Boeing's implementation is the only thing here that is flawed, and very much so.
Re: (Score:2)
Why are software bugs somehow worse or more serious than any kind of design flaws?
The fact is, they aren't. Software is just more abstract, so we think of it as somehow less dependable.
Any kind of engineering, software or hardware, depends on attention to detail and making reasonable trade-offs. Every building, every road, every vehicle, everything constructed by mankind has design flaws and compromises. Software is not uniquely un-trustable.
If this keeps up (Score:2)
Not sure where we go as a modern tech culture. Drip, Drip, Drip, Leak, Leak, Leak!
Just my 2 cents
Re: (Score:2)
Re: (Score:2)
It may not meet spec, and as such it needs to be addressed for certification purposes,
Na Bro, we self certify. We only need to worry about it after the crash.
It never ends (Score:2)
Re: (Score:1)
This is actually good (Score:2)
The way I see it, the plane isn't allowed to fly anyway. So there are massive reviews going on. I'd rather them catch the errors now vs after the plane is back in production.
Bug or problem triggered by previous bug fix (Score:2)
Well, to be fair, this is common to software development. If you change something then you may have problems down the line. It is not clear if this was a dormant bug or that it was just the system misbehaving because the parameters were changed in a way incompatible with the system. And yes, they caught it by testing.
If it was a dormant bug it would be somewhat of a problem as it indicates that everything may not have been tested as well as should be expected. So in that case it could be an indication of th
Forget 737 Max (Score:2)