Slashdot is powered by your submissions, so send in your scoop

 


Forgot your password?
Close
typodupeerror
×
Transportation Businesses

Delays in Boeing Max Return Began With Near-Crash in Simulator (bloomberg.com) 95

Posted by msmash from the can't-catch-a-break dept.
Boeing engineers were nearly done redesigning software on the grounded 737 Max in June when some pilots hopped into a simulator to test a few things. It didn't go well. From a report: A simulated computer glitch caused it to to dive aggressively in a way that resembled the problem that had caused deadly crashes off Indonesia and in Ethiopia months earlier. That led to an extensive redesign of the plane's flight computers that has dragged on for months and repeatedly pushed back the date of its return to service, according to people briefed on the work. The company -- which initially expressed confidence it could complete its application to recertify the plane with the Federal Aviation Administration within months -- now says it hopes to do that before the end of the year.

Changing the architecture of the jet's twin flight computers, which drive autopilots and critical instruments, has proven far more laborious than patching the system directly involved in 737 Max crashes, said these people, who asked not to be named speaking about the issue. The redesign has also sparked tensions between aviation regulators and the company. As recently as this week, the FAA and European Aviation Safety Agency asked for more documentation of the changes to the computers, said one of the people, potentially delaying the certification further. Developing and testing software on airliners is an exacting process. Manufacturers may have to demonstrate with extensive testing that a software failure leading to a crash would be as rare as one in a billion. Further reading: Boeing Has So Many Grounded 737 Max Planes Waiting To Be Fixed They're Parking Them in the Employee Parking Lot.
This discussion has been archived. No new comments can be posted.

Delays in Boeing Max Return Began With Near-Crash in Simulator More

Delays in Boeing Max Return Began With Near-Crash in Simulator

Comments Filter:

  • Boeing’s 737 Max Software Outsourced to $9-a (Score:5, Insightful)

    by sbrown123 ( 229895 ) on Friday November 08, 2019 @10:10AM (#59393774) Homepage

    Boeing’s 737 Max Software Outsourced to $9-an-Hour Engineers

    https://www.bloomberg.com/news... [bloomberg.com]

    "Increasingly, the iconic American planemaker and its subcontractors have relied on temporary workers making as little as $9 an hour to develop and test software, often from countries lacking a deep background in aerospace -- notably India."

    Think of all the money they saved by outsourcing.

    • Think of all the money they saved by outsourcing.

      Money well spent. They got the software they wanted and a scapegoat as a bonus.

    • Think of all the money they saved by outsourcing.

      If you go by the market cap lost as a result of the stock dip from all of these problems, they must have had several tens of millions of hours worth of software development they needed done to have made any money. I'm not sure how to factor in the added costs of the government crawling up their asses or anything like that, but it's probably only in the hundreds of thousands of hours before the savings paid off.

    • Boeing’s 737 Max Software Outsourced to $9-an-Hour Engineers

      And that is relevant how? The money being spent on engineers does not guarantee quality. You can pay someone $200/h from the old school with a long grey beard and they can still produce unreadable non-working and buggy code if your specifications and quality systems are not in place (and at Boeing they aren't).

      Contractor oversight at Boeing is horribly fucked, and you can't fix that by paying more money to the contractor.

    • You can get a full engineering team in India for the price of a single engineer in the USA. You do have to take care as those "Third World" countries tend to be a bit sloppy about qualifications though. The next step is outsourcing management. Expect massive savings from your minimum wage boss, with the added benefit that you can really punish them when they make a "mistake" that costs lives.
    • Hey Boeing,

      How did those $9/hr coders work out? Save any money?

      Note: Next time don't high a bunch of $9/hr engineers to build your next plane.

  • Who's surprised? (Score:5, Insightful)

    by BytePusher ( 209961 ) on Friday November 08, 2019 @10:18AM (#59393786) Homepage
    While intuitively one might think a company like Boeing would learn from their mistakes, we've seen over and over again that bad leadership almost always doubles down on their toxic habits. This is the curse of privilege; they're thrust into a position of influence and power, showered with praises, middle management biases the feedback they receive from below, until ultimately they live alternate universe with alternative facts. Their worst habits aren't accidental, they are carefully thought out and decided upon, influenced by their core philosophy. They often believe that the problem was not their core philosophy, but that they didn't push their core philosophy on everyone enough. So instead of sitting down and questioning themselves, they turn the blame to everyone else. CEO failure of this kind is almost always followed by more failures, not learning, despite all the positive bullshit they spout off about learning from failures. That's just gaslight to allow them to keep playing the game they suck at.
    • Spot on. It's these bean-counting narcissists who have taken down what WAS one of the most successful, industry-leading manufacturers (they built the B-52s that have been in the air for over 50 years and the B-17s & B-29s excellence changed the world) tossing a decades long reputation into the garbage. You won't get me on a Boeing plane again (no great loss). Such good management.

    • Re:Who's surprised? (Score:5, Insightful)

      by Rosco P. Coltrane ( 209368 ) on Friday November 08, 2019 @10:40AM (#59393880)

      Nobody's surprised that Boeing cuts corners, won't learn a thing from their mistake and will do it again. They're a private corporation hell-bent on making a profit - like all corporations.

      What surprises me is that the FAA doesn't seem to have anything coming their way, when they clearly haven't done their job. I've worked in the aero industry as an ARP-4754 / DO-178 quality engineer, and I know for a fact that it takes extraordinary, almost conspiratory efforts to certify a design for a DAL-A feature such as the MCAS based on a single sensor. It's such a biggie even a low-level QA monkey such as me with zero access to the design file or the certification file can tell with 100% certainty that someone (several someones probably) at the FAA had to take deliberate steps to cover this up and turn a blind eye, and that those people up for jail time for a couple decades.

      Knowing that, I'm staggered to see all the blame put on Boeing, and no inquiry or indictment coming the FAA's way. I just can't figure out how that's possible...

      • Does the FAA have a revolving door to industry the same as most other federal agencies do? They probably have it built in to protect anyone that was abusing their positions at the FAA.

      • The FAA can say that the MCAS system would never have been approved if it were accurately described to them by Boeing in the first place. To me, as an outsider, that is a plausible explanation, even if you might have good reason to be dissatisfied based on personal experience.

        A single point of failure system that will repeatedly attempt to crash a plane without intervention violates so many basic design principles for aircraft safety that there is no way Boeing is not swimming in evidence that the problem

      • count me surprised. I worked closely with part of their teams (non avionics) team a little under two decades ago (pre 9/11). I found them to have a culture of rigorous engineering, and if anything, slow moving and unwilling to compromise on safety. That said, looks like a lot has changed. It's amazing that they're outsourcing avionics testing. They're talking about humans lives. Sad.

      • Re: (Score:3)

        by cusco ( 717999 )

        Once upon a time the FAA (and most of the other regulatory agencies) had staff on board that could examine and test manufacturer's claims. Then Newt Gingrich realized that he didn't have to eliminate regulatory agencies, just squeeze their enforcement budgets to almost nothing and the result is the same. Now the FAA has to blindly accept whatever Boeing (or any other manufacturer) says because the Magical Mystical Free Market Fairy says they'd never lie to regulators. The FDA, EPA, etc. are very much in

      • Since big business bought off the Democrats and Republicans America has been sinking quickly into "Third World Bananna Republic" status. Deregulation of the FAA, deregulation of the banking industry... this road doesn't go anywhere you want to be.
      • Agreed. No one should be surprised that the design problems that killed two airplanes full of people would show themselves in a simulator. What we should be surprised at is that our systems for keeping people safe, ignored the same problems months earlier.

    • cost cutting over safety! we need software PE's with licenses on the line for some things.

    • Re: (Score:2, Funny)

      by Ryzilynt ( 3492885 )

      While intuitively one might think a company like Boeing would learn from their mistakes, we've seen over and over again that bad leadership almost always doubles down on their toxic habits. This is the curse of privilege; they're thrust into a position of influence and power, showered with praises, middle management biases the feedback they receive from below, until ultimately they live alternate universe with alternative facts. Their worst habits aren't accidental, they are carefully thought out and decided upon, influenced by their core philosophy. They often believe that the problem was not their core philosophy, but that they didn't push their core philosophy on everyone enough. So instead of sitting down and questioning themselves, they turn the blame to everyone else. CEO failure of this kind is almost always followed by more failures, not learning, despite all the positive bullshit they spout off about learning from failures. That's just gaslight to allow them to keep playing the game they suck at.

      Offtopic - This article is about the Boeing 737 Max - It has absolutely nothing to do with the trump administration.

    • EO failure of this kind is almost always followed by more failures, not learning, despite all the positive bullshit they spout off about learning from failures.

      Followed by a bonus, of course.

  • Turns out cutting corners can be expensive.

    • When PE does it they can face hardtime but for CEO they get fired with an big cash out and do no time.

    • Re: (Score:2)

      by sycodon ( 149926 )

      Turns out giving some computer system primary access to control surfaces based on the inputs on only one sensor can be deadly.

      Seriously, that is apparently the primary cause of the accidents, faulty reading from a single sensor.

      Any system that can move the control surfaces should be triple redundant in all respects.

      • Any time the computer can override the pilot it's a bad situation. If the computer says trim down and the pilot says trim up (he's got a button for that) the plane should obey the pilot. If the pilot is wrong you've got a scapegoat, if the computer is wrong you've got a lawsuite.

  • Hot standby (Score:4, Interesting)

    by amorsen ( 7485 ) <benny+slashdot@amorsen.dk> on Friday November 08, 2019 @10:44AM (#59393900)

    Most modern, computerized aircraft -- such as more recent Boeing models and Airbus SE’s jets -- use three computer systems to monitor each other, Hansman and Lemme said.

    By contrast, the 737 Max had two separate computers. One operated the flight systems and another was available if the first one failed, with the roles switching on each flight. But they interacted only minimally.

    Boeing decided to make the two systems monitor each other so that each computer can halt an erroneous action by the other. This change is an important modernization that brings the plane more in line with the latest safety technology but raised highly complex software and hardware issues.

    I am not primarily a software person, but changing from a pure hot standby solution to half-way active-active seems like a truly daunting task. Especially when it is safety critical. Can that really be achieved in six months or a year?

    • No, it can't

    • Re:Hot standby (Score:5, Interesting)

      by Zocalo ( 252965 ) on Friday November 08, 2019 @11:18AM (#59394064) Homepage
      In short, almost certainly not, especially on something as complex as a flight control system and with the entire aviation world, especially the certification bodies like the FAA and EASA, watching.

      Master-Master systems are notoriously complex to get right even when designing them that way from scratch. Trying to turn a Master-Slave setup into a Master-Master "just by adding a wire" and "in a couple of months" tells you all you need to know about the sheer arrogance and lack of technical acumen in the higher levels of Boeing right now. Also, keep in mind that most truly modern aircraft systems have not two, but *three*, control systems to have a deciding "vote" in the event of conflict and avoid a scenario called split brain, which would be a scenario where, for whatever reason (perhaps a fault, or maybe even the cosmic rays mentioned in TFS), one of the aircraft's flight systems is indicating "pull up" and the other "nose down".

      • keep in mind that most truly modern aircraft systems have not two, but *three*, control systems to have a deciding "vote"

        My favourite quote to explain to non-safety systems engineers why two devices is a bad idea:
        A man with a watch thinks he knows the time.
        A man with two watches is never sure.
        A man with more watches can figure it out.

      • In most applications of Master-Master the I'm familiar with ie: Server node clustering, having at least three nodes and always an odd number is usually a requirement. I don't know how you can prevent split brain with two nodes in a closed system.

        • Re: (Score:3)

          by cusco ( 717999 )

          You can't, but Boeing desperately wants to avoid the extra cost of a third computer or recabling to allow each computer access to all of the sensor inputs. As someone above noted, this is a typical reaction from executives who have never done anything but manage. Rather than learn from their mistakes and change they will double-down on their prior behavior, assuming that (since of course they are perfect in every way) the initial problem was caused by underlings' incompetence rather than their own stupidi

    • Re: (Score:1)

      by Anonymous Coward

      No :-(

      The three-way 'vote to win' solution was always expensive, but was always intuitively and empirically the safest way to operate. You can even go as far as to have three entirely different software versions running and still operate safely.

      The dual master solution is incredibly difficult to get right in every circumstance. Essentially, you have the primary making the decision and the secondary hopefully making the same decision. If the secondary disagrees though, there's no clear way to determine who w

      • Re: (Score:2)

        by amorsen ( 7485 )

        That third computer must have been very, very expensive for anyone to think dropping to two would save money.

        I think it is more likely that the hot-standby design goes back to the 70's or 80's. Especially in the 70's constraints on size, weight, and power consumption would have made it attractive to use fewer computers, and software development has come a long way since then.

        • That third computer must have been very, very expensive for anyone to think dropping to two would save money.

          I think it is more likely that the hot-standby design goes back to the 70's or 80's. Especially in the 70's constraints on size, weight, and power consumption would have made it attractive to use fewer computers, and software development has come a long way since then.

          Or 60's. The 737 base design is that old.

          Boeing wants to have their cake and eat it too. Produce a brand new modern plane, but have it grandfathered in to safety standards riding the coattails of the previous models. This cost-cutting mentality has killed people, and it will kill more people in the future. I predict nothing will change, except perhaps a government bailout once the stock prices get too low.

      • If the secondary disagrees though, there's no clear way to determine who was correct - it's worked out by a complex algorithmic decision, based on previous actions, possible futures, and known data about the present.

        But in this case, if the two sensors disagree, just turn off the MCAS and tell the pilot that he is on his own. The pilot can then fly the airplane using the other instruments (including looking outside if the weather conditions allow).

        • Re: (Score:2)

          by Nidi62 ( 1525137 )

          If the secondary disagrees though, there's no clear way to determine who was correct - it's worked out by a complex algorithmic decision, based on previous actions, possible futures, and known data about the present.

          But in this case, if the two sensors disagree, just turn off the MCAS and tell the pilot that he is on his own..

          Wasn't part of the problem to begin with that the MCAS only ran off one sensor and that the AOA Disagree light was only enabled when purchasing an upgraded package? One would think that anytime an aircraft had multiple sensors that agree/disagree detection would be standard.

          • Re: (Score:2)

            by cusco ( 717999 )

            That, and Boeing didn't even tell the airlines about the MCAS nor how to disable it, much less the pilots.

            • If the new model is just a slight variation of an existing design you don't need to retrain the pilots. Turns out this was more than a slight variation.

      • Re: (Score:2)

        by cusco ( 717999 )

        Not only is the addition of a third computer expensive, but to do this right each of the computers has to have access to all of the sensors and rewiring a previously-built aircraft is horribly expensive. Even for something like a Cessna the installation of a new wiring harness can cost 10% or more of the value of the airplane.

    • Re:Hot standby (Score:5, Informative)

      by Distan ( 122159 ) on Friday November 08, 2019 @01:40PM (#59394712)

      > By contrast, the 737 Max had two separate computers.

      The 737 Flight Control Computer (FCC) system has dual-dual redundancy. The 737 contains two FCCs that each contains two processors running independently. In the event the two processors in one FCC ever disagree that FCC shuts down. If the FCC that shuts down was the active one then the secondary FCC takes over. This system provides at least as much hardware redundancy as a three-way voting system.

      A key problem with the initial MCAS implementation wasn't the lack of CPU redundancy but the lack of sensor redundancy. Each FCC was programmed to accept only one AOA sensor's input. In the event of erroneous AOA input to the active FCC there was no way for the system to detect the error.

      MCAS as initially implemented also represented a very significant shift in the role of the pilot vs the computer. The 737 is an old air-frame that is fundamentally controlled by steel cables and pulleys. The Pilot has full authority over the aircraft and the computers are best seen as "pilots aids". MCAS inverted this expectation by inserting a new software component that had nearly unlimited authority over the pilot.

      Contrast this with Airbus where the computer is responsible for flying the plane under the authority of the pilot. Under "normal law" on an Airbus the pilot has no direct control over any flight surfaces. The pilots commands are simply one input to the computer, but the computer ultimately decides what to do.

      From the traditional Boeing 737 engineering point of view, having full redundancy on all systems wasn't necessary because the pilot is the one flying the plane and if one of the systems misbehaves the pilot could just turn it off. From an Airbus engineering point of view, having full redundancy on everything is critical because the computer is flying the plane.

      • Additionally MCAS final implementation gave it much more control over the horizontal stabilizer than the initial version submitted to the FAA. The FAA sent them back with permission to tweak the system. That 'tweak' essentially gave the MCAS full control of the horizontal stabilizer over time.

  • Test Imperatives (Score:5, Insightful)

    by Shotgun ( 30919 ) on Friday November 08, 2019 @10:57AM (#59393962)

    (The customer would never do that!) I don't test what the customer would do. I test what the customer COULD do, 'cause if they can do it, they will.
    (It was an edge case.) "Edge" case is not a thing. It is just a case.
    (It was glitch.) "Glitches" are just a different set inputs that the software allowed.
    In every case, if you want to know of the software will process a set of inputs correctly, test it.

  • Why not put in a test to limit the amount of adjustment to a gentle descent while notifying the pilots of the situation thus allowing them time to override or disable the function ? Seems like a simple fix if the system works correctly most of the time. Unless the hardware design is so broken that the plane would crash without allowing the software full reign. Perhaps that's the real problem.

    • Based on some comments from people who speak with industry experience, simply ripping out MCAS is not necessarily wrong from engineering perspective. The plane is probably safe enough with the correct pilot training.

      But new pilot training is expensive, makes customers mad, and the necessity of it for this aircraft may trigger the FAA to make a deeper and more detailed assessment of the entire plane, which could cost a lot of time and money.

      The problem spiraled into a horror show because Boeing desperately

  • Recycle Time (Score:5, Insightful)

    by HangingChad ( 677530 ) on Friday November 08, 2019 @11:44AM (#59394194) Homepage

    It's time to admit the 737 Max is never going to fly right and consign it to the scrap heap of aviation history.

    • I wish that was true. Then it would stay as the example in history where "cutting corners to accelerate time to market can make a plane a total failure and even endanger the largest companies". Everyone would win from that.

    • Nah man, we'll just patch out the bugs mid flight while the customers are beta testing.

    • The actual solution would be scrap the MCAS system entirely, accept that the plane has different dynamics than previous 737's, and retrain the pilots. I'm not even sure why they are putting all this effort into "fixing" it - the original point of the system is so that pilots that were familiar with previous iterations of the 737 could (supposedly) fly the MAX without retraining. Do you really think at this point, no matter what Boeing changes, is anyone going to let a pilot fly a 737 MAX without a bunch o

  • successful test (Score:3)

    by bugs2squash ( 1132591 ) on Friday November 08, 2019 @11:53AM (#59394222)
    A successful test is one that finds a problem. I'm far happier that they found it in a simulator than on a beat-up flight recorder extracted from wreckage.

    • THIS! It's amazing how many people think finding problems during testing is a bad thing. I've had to explain this to my boss as well who also questioned how we could possibly find a fault. The answer was: If we were certain there was no fault, then we wouldn't wasted time testing.

    • Re: (Score:2)

      by labnet ( 457441 )

      I know an ex head engineer from Boeing. They actually plot bugs found on a graph, and they are not finding enough bugs, they assume they are not testing well enough.

  • The right thing to do would be to buy back and scrap all the max's already delivered and design a full replacement for the 737 while continuing to sell the last good version. I assume the cost of this would kill Boeing so they are half-assing the response.
  • One in a billion flights? One in a billion flight hours? One in a billion CPU clock cycles?

  • Remove the code that commands the elevators and replace it with audible warnings when sensors detect pitch exceeding reasonable bounds and a manual elevator trim that is easy to use and near-at-hand.

    Problem solved. Send my check to my usual address.

    • Re:A modest proposal (Score:4, Interesting)

      by bsolar ( 1176767 ) on Friday November 08, 2019 @01:11PM (#59394584)
      As far as I understand that would require all the pilots to be re-trained, which is the reason the correction was designed to be "hidden" through an automated system in the first place.
      • Correct. The new engines gave similar flight characteristics, but in some situations/angles they gave the aircraft very different flight characteristics. The hope appeared to be that the designers could use software to prevent the pilot from putting the plane in that situation. The software would sense the undesired state, and correct the flight of the aircraft to avoid it. Sadly for all the lives lost, there were both design and implementation flaws.

    • Doesn't solve the fact that the plane doesn't handle properly with the the larger engines.

      It was a band-aid, when it needed surgery.

  • Trying to square what I know about flight computers (minimal) to what's in the article, and I'd love input from someone who actually knows what's going on. I worked, very briefly, on a specific flight computer piece hardware. (No, I don't know what plane it went into. I BELIEVE it was military, based on the customer. (Rhymes with bunny-well)) But this particular design was technically 2 computers in one. It had one PCB, but it literally was a cut-paste copy of two distinct hardware paths. Each side r

    • Comment removed based on user account deletion
    • The 737 is a mess of dissimilar components semi-networked together from various manufacturers. For example, the Angle of Attack sensor is its own PCB, probably just receives AC power (which internally is converted to DC), has some small embedded microcontroller from the 1980s, runs simple assembly language or C developed code, or is simply a bunch of logic gates with Analog to Digital converters. It may interface with an AIRINC429 data bus or use some other lesser serial bus. This system probably triggere
      • But even Airbus can have a 'corner case' which dumps you in the Atlantic while pilots ask "What's it doing now?".

        • AF447 did exactly what it was designed to do - give control to the pilots when the flight control systems couldn’t trust the data it was being fed. The pilots then fucked up. No edge case there.

  • “Developing and testing software on airliners is an exacting process.”

    Boeing’s 737 Max Software Outsourced to $9-an-Hour Engineers [bloomberg.com]

    • Developing and testing software on airliners is an exacting process.

      They wouldn't be finding themselves in this situation if that were actually the case.

  • Makes you wonder how much pork-barrel spending is going to be be wasted on safety testing SLS before it ever flies.
    • The purpose of SLS is to spread the pork, not fly. If NASA wanted to get to Mars in five years they could, but it would require cutting all the pork to have sufficient money to pay for it. An upscale Apollo mission would work, just everything bigger, and a nuclear thermal rocket for the command module to reduce trip time.
  • My ex is responsible for the OS used a large portion of the western world's aircraft. They misinterpreted a processor errata on cache misses. There was no one in the certification process that actually had the ability to understand the errata. Her company screwed up but the point of certification is to catch these mistakes. The certification process has lots of checks, takes a long time and is very expensive but for the most part it lacks any competence. Something complicated like the OS task swapping s
  • How do airlines company handle the cost of all that grounded planes? Parking costs money, and grounded planes mean missed income.

  • Everybody is clear on this, right? The root engineering problem is, the airframe is junk. Stretched too many times, the wheels are now too stubby for the engines, which were moved to a new position that creates aerodynamic instability under certain common flight conditions. Trying to paper this over with software is putting lipstick on a pig. The airframe itself is inherently dangerous. Just junk it and move on.

Slashdot Top Deals

A morsel of genuine history is a thing so rare as to be always valuable. -- Thomas Jefferson

Close