Inside Boeing's Factory Lapses That Led To the Alaska Air Blowout (seattletimes.com) 52
Remember when a door-sized panel blew off a Boeing aircraft back in January? The Seattle Times reports that the "door plug" incident "was caused by two distinct manufacturing errors by different crews" in a Boeing assembly plant in Renton, Washington last fall. (And that Boeing's quality control system "failed to catch the faulty work.")
But the details tell a larger story. The newspaper bases their conclusion on "transcripts of federal investigators' interviews of a dozen Boeing workers, synchronized with an internal Boeing document obtained by The Seattle Times," tracing the whole history of that panel's production. Within a day of its fuselage arriving at the factory, "a small defect was discovered: Five rivets installed by Spirit on the door frame next to the door plug were damaged." That day, the Friday before the Labor Day weekend, repair of those rivets was handed to Spirit, which has contract mechanics on-site in Renton to do any rework on its fuselage. In the meantime, inspectors gave mechanics the OK to install insulation blankets, which covered the door plug. By the following Thursday, a Spirit mechanic had logged an entry in the official Federal Aviation Administration-required record of this aircraft's assembly — the Common Manufacturing Execution System or CMES, pronounced "sea-mass" by the mechanics — that the rivet repair was complete: "removed and replaced rivets." But that day, a Boeing inspector responded with a scathing rebuttal, stating that the rivets had not been replaced but just painted over. "Not acceptable," read the work order. On Sept. 10, records show Spirit was ordered a second time to remove and replace the rivets...
["Shipside Action Tracker"] entries show that after several days, the still-unfinished work order was elevated to higher-level Boeing managers. On Sept. 15, Boeing cabin interiors manager Phally Meas, who needed the work finished so he could get his crew to install cabin walls and seats, texted on-site Spirit manager Tran Nguyen to ask why the rivet work hadn't been done, NTSB interview transcripts show. Spirit mechanics couldn't get to the rivets unless the plug door was opened, Nguyen responded. He sent Meas a photo from his phone showing it was closed, according to the transcripts. It wasn't Spirit's job to open the sealed door plug. Boeing's door team would have to do that, the records show. "He kept asking me how come there wasn't work yet," Nguyen told the NTSB. "The door was not open. That's why there wasn't work yet."
By Sept. 17, the door was still closed, the rivets still unrepaired. The job was elevated again, to the next level of managers. On that day, according to the SAT record, senior managers worked with Ken McElhaney, the door crew manager in Renton, "to determine if the door can just merely be opened or if it needs removal...." [On September 18] at 6:48 a.m., a Boeing mechanic identified as a Door Master Lead texted a young Trainee mechanic on his team to come to the Alaska jet and open the door. The NTSB interviewed but did not name the Trainee or the Door Master Lead, who had almost 16 years at Boeing.
Filling in for the veteran mechanic on vacation, the Trainee was perhaps the least equipped to do this atypical job. He'd been at Boeing for about 17 months, his only previous jobs being at KFC and Taco Bell. "He's just a young kid," the Door Master Lead said...
More key quotes from the article:
But the details tell a larger story. The newspaper bases their conclusion on "transcripts of federal investigators' interviews of a dozen Boeing workers, synchronized with an internal Boeing document obtained by The Seattle Times," tracing the whole history of that panel's production. Within a day of its fuselage arriving at the factory, "a small defect was discovered: Five rivets installed by Spirit on the door frame next to the door plug were damaged." That day, the Friday before the Labor Day weekend, repair of those rivets was handed to Spirit, which has contract mechanics on-site in Renton to do any rework on its fuselage. In the meantime, inspectors gave mechanics the OK to install insulation blankets, which covered the door plug. By the following Thursday, a Spirit mechanic had logged an entry in the official Federal Aviation Administration-required record of this aircraft's assembly — the Common Manufacturing Execution System or CMES, pronounced "sea-mass" by the mechanics — that the rivet repair was complete: "removed and replaced rivets." But that day, a Boeing inspector responded with a scathing rebuttal, stating that the rivets had not been replaced but just painted over. "Not acceptable," read the work order. On Sept. 10, records show Spirit was ordered a second time to remove and replace the rivets...
["Shipside Action Tracker"] entries show that after several days, the still-unfinished work order was elevated to higher-level Boeing managers. On Sept. 15, Boeing cabin interiors manager Phally Meas, who needed the work finished so he could get his crew to install cabin walls and seats, texted on-site Spirit manager Tran Nguyen to ask why the rivet work hadn't been done, NTSB interview transcripts show. Spirit mechanics couldn't get to the rivets unless the plug door was opened, Nguyen responded. He sent Meas a photo from his phone showing it was closed, according to the transcripts. It wasn't Spirit's job to open the sealed door plug. Boeing's door team would have to do that, the records show. "He kept asking me how come there wasn't work yet," Nguyen told the NTSB. "The door was not open. That's why there wasn't work yet."
By Sept. 17, the door was still closed, the rivets still unrepaired. The job was elevated again, to the next level of managers. On that day, according to the SAT record, senior managers worked with Ken McElhaney, the door crew manager in Renton, "to determine if the door can just merely be opened or if it needs removal...." [On September 18] at 6:48 a.m., a Boeing mechanic identified as a Door Master Lead texted a young Trainee mechanic on his team to come to the Alaska jet and open the door. The NTSB interviewed but did not name the Trainee or the Door Master Lead, who had almost 16 years at Boeing.
Filling in for the veteran mechanic on vacation, the Trainee was perhaps the least equipped to do this atypical job. He'd been at Boeing for about 17 months, his only previous jobs being at KFC and Taco Bell. "He's just a young kid," the Door Master Lead said...
More key quotes from the article:
- Boeing put both employees on paid administrative leave.
- "A company investigator accused one of them of lying. That employee told the NTSB that Boeing has set the pair up as scapegoats."
- "A 35-year veteran on the door team told NTSB investigators that he is 'the only one that can work on all the doors' and he was typically the only mechanic who would work on door plugs. That mechanic was on vacation on the two critical days, September 18 and 19 last year, when the door plug on the Alaska MAX 9 had to be opened and closed..."
- "No quality inspection of the door plug was conducted, since no record of its opening and closing was ever entered in the system, documents show."
- The FBI ís investigating Boeing "for potential criminal negligence," according to the article, "and has issued subpoenas using a Seattle grand jury."
Thanks to long-time Slashdot reader schwit1 for sharing the news.
So a cascade of failures... (Score:4, Insightful)
Where have I heard that recently as well?
Ah, yes: https://www.cisa.gov/sites/def... [cisa.gov]
Re: So a cascade of failures... (Score:2)
What an amazing degree of detail (Score:4, Insightful)
I find it absolutely amazing how FAA investigations can uncover this degree of detail when investigating something as small-detailed as "who didn't install this bolt on this aircraft." It leads me to two conclusions:
1) Given the degree of auditing that goes into the construction of an aircraft, Boeing ex-CEO Dave Calhoun must be a really special degree of stupid get for thinking that some PR swipe-it-under-the-rug BS like "[We're] unable to provide that information...Boeing has no records of the work being performed" [go.com] would be accepted at face value.
2) I'm proud of my taxpayer dollars doing to the FAA. This is some truly high-quality investigatory work they've performed.
Re:What an amazing degree of detail (Score:5, Interesting)
I find it absolutely amazing how FAA investigations can uncover this degree of detail when investigating something as small-detailed as "who didn't install this bolt on this aircraft." It leads me to two conclusions:
1) Given the degree of auditing that goes into the construction of an aircraft, Boeing ex-CEO Dave Calhoun must be a really special degree of stupid get for thinking that some PR swipe-it-under-the-rug BS like "[We're] unable to provide that information...Boeing has no records of the work being performed" [go.com] would be accepted at face value.
2) I'm proud of my taxpayer dollars doing to the FAA. This is some truly high-quality investigatory work they've performed.
I guess it's an indication of how big the paper trail is, all the FAA probably had to do was download the list of tickets. I wouldn't credit the FAA for the investigation as much as setting up the auditing system that made the investigation possible.
The other interesting part is that this issue was clearly known by Spirit and Boeing, and both were trying to get it fixed, but the ball was clearly dropped. My (naive) takeaway is that Boeing is a bit too vertical. It makes sense to elevate one level of management since the first level screwed up, but that second level should have had the authority to resolve it, instead it got elevated to a 3rd level that was too far removed and too unfamiliar and that's where the real screw-up happened.
Re: (Score:2)
Re: What an amazing degree of detail (Score:2)
My takeaway is that there are too many chefs in the kitchen. If you have that many leads and managers, you can make one of them (and a deputy) own the whole plane, not one owns the door, one owns the interior, one owns x, y, etc and they all talk amongst themselves.
Re:What an amazing degree of detail (Score:5, Insightful)
Re: (Score:2)
Interesting. I missed that. Unlike the FAA, the NTSB is not (yet) tainted by too much political influence.
Re: (Score:2)
I'm just waiting for some judge in Texas to rule that under Loper Bright, the NTSB has no authority to do anything.
Re: (Score:2)
Your understanding of Loper Bright is deficient if you believe it is applicable in any way to the facts of this investigation.
On the other hand, the judiciary in the last couple of decades have stretched laws and precedents beyond anything recognizable in order to benefit corporations, and the Supreme Court has gone happily along with it.
Re: What an amazing degree of detail (Score:1)
Re: (Score:2)
NTSB doesn't have the dual charter of safety plus promoting the health of the industry.
Re: (Score:2)
That makes a ton of sense. Perverted incentives (as the FAA seems to have) never work out.
Re: What an amazing degree of detail (Score:2)
Re: (Score:2)
As long as the FAA is does not get hobbled by politics, it does excellent work. Calhoun should go to prison for his statements though.
Re: (Score:3)
Aviation
Haven't we all seen this? (Score:2)
Re:Haven't we all seen this? (Score:4, Funny)
So, you work for Crowdstrike?
Overall, seems like QA is failing at their job more and more. When the bean counters start downsizing, QA is often the first to go.
Re: (Score:3)
Bean counters kill businesses. It just takes a while.
Re: (Score:2)
Re: (Score:2)
You are spot on. As a rookie with just 25 years of experience your narrative hit hard. Well ish. Most of the oversights that got my blood boiling never started at the engineering level, they came from up top
No way Boeing can recover (Score:3, Insightful)
Re:No way Boeing can recover (Score:5, Insightful)
Indeed. For something critical to go _this_ badly sideways, the leadership has to be rotten. (Same, incidentally, for Crowdstrike, Microsoft, etc.) For example, one expert that can do the doors? WTF? Any halfway competent risk-analysis should have shown that this needs 3 people that can stand in for each other. That this risk analysis was not done or ignored is a pure leadership failure.
Re: No way Boeing can recover (Score:2)
I've seen it's equivalent play out a thousand times.
Re: (Score:3)
Well, yes. But these people are building airplanes. Apparently they have forgotten what that means.
Re: (Score:1)
So short-term economic efficiency trumped engineering efficiency, and why do economists get to claim that capitalism is efficient again?
Re: No way Boeing can recover (Score:1)
Re: (Score:1)
What if AI optimizes each person's virtual environment for them personally, simultaneously for everyone? Is that the win-win-win utopia we've all been dreaming?
Re: (Score:2)
For example, one expert that can do the doors?
We have found the "Bus Guy" at Boeing. As in, what happens when this guy gets hit by a bus?
Re: (Score:3)
A 35-year veteran on the door team told NTSB investigators that he is 'the only one that can work on all the doors' and he was typically the only mechanic who would work on door plugs.
The mechanic who said he is the only one that can work on ALL the doors might mean he is the only person certified to repair emergency exit AND pilot doors AND food service doors AND cargo hold doors AND equipment bay doors. Others might be able to work on four of the five types, but he is the only mechanic that has certifications for every door type. A bit like an IT person claiming they are the only one that has worked on EVERY type of networking technology - from Token Ring to Fiber.
As for the claim th
Re: (Score:2)
Exactly.
This is what happens when (Score:5, Interesting)
Some things need redundancy. SOMETIMES, THE COST IS WORTH PAYING.
Re: (Score:3)
Indeed. And sometimes, deciding to not have that redundancy should land you personally behind bars and if people dies or could easily have died, then for an extended period of time. Incidentally, all IT security is redundancy in a sense. The current extreme mess there is due to essentially the same problem.
Re: (Score:2)
Re: This is what happens when (Score:1)
Here's a proposal (Score:2)
Re: (Score:2)
Re: (Score:2)
The black hole . . .
Bureaucracy at its finest. (Score:3)
Between being an old company fat with middle management, outsourcing most their work to third parties and still having to deal with union regulations it's surprising Boeing gets anything done at all.
Who's going to step up and replace Boeing?
Re: (Score:2)
Irkut in Russia is getting its new MC-21 certified now, which is supposed to be a replacement for the 737. Of course there's those annoying sanctions in the way, but 2/3 of the world's countries are ignoring them so who knows?
Re: (Score:2)
They have been "getting its new MC-21 certified now" since 2016. By the looks of it, even the 777X will be certified before it.
Re: (Score:2)
The elephant in the room is Airbus. Between the A320/321/Neo/etc and their venture with Bombardier to make the A220, they're well situated to take as much marketshare as their build rate allows. One of the main reasons more 737 orders haven't been cancelled is that both companies have a 10+ year backlog. Comac is working hard to join them but it seems unlikely to me that we'll see them in the US. Maybe another regional jet manufacturer will try to step up like Bombardier did. Wouldn't shock me if a com
Imagine if they just did the work (Score:3)
Just crazy to me. Got adversarial over doing the work and delayed it until it became an emergency in enough eyes to just cut the corners to this
Boeing inspects, says to fix the rivets - IMO this should be the end of it - Spirit do what your customer said - but instead.....
Spirit two days later says nah - we think they're good - not gonna do it.
Boeing three days later says - go fix them like we said you !@#%wads.
(We've blown a week arguing over a fix that the inspector identified now basically)
Another four days of elevation to higher and higher managers eventually Spirit says fine we'll fix it, now you have to open the door for us.
Three days later the door is still not open (why the hell did that not happen like same or next day?), and someone - not yet know apparently - opened it in a rush and didn't document it - during a two day window where the one guy who knows door plugs in Renton is on vacation.
The people who don't want any more delays pushed folks to hurry up - it gets closed wrongly without the bolts. Since it wasn't documented no flag for the inspections that should have happened.
What a shit show. At least someone made a few bucks by having sold off what became Spirit so a simple internal re-work order turned into this steaming pile of dung.
Re: (Score:3)
Re: (Score:2)
Just crazy to me. Got adversarial over doing the work and delayed it until it became an emergency in enough eyes to just cut the corners to this
Boeing inspects, says to fix the rivets - IMO this should be the end of it - Spirit do what your customer said - but instead.....
Spirit two days later says nah - we think they're good - not gonna do it.
Boeing three days later says - go fix them like we said you !@#%wads.
(We've blown a week arguing over a fix that the inspector identified now basically)
Another four days of elevation to higher and higher managers eventually Spirit says fine we'll fix it, now you have to open the door for us.
Three days later the door is still not open (why the hell did that not happen like same or next day?), and someone - not yet know apparently - opened it in a rush and didn't document it - during a two day window where the one guy who knows door plugs in Renton is on vacation.
The people who don't want any more delays pushed folks to hurry up - it gets closed wrongly without the bolts. Since it wasn't documented no flag for the inspections that should have happened.
What a shit show. At least someone made a few bucks by having sold off what became Spirit so a simple internal re-work order turned into this steaming pile of dung.
Boeing only had one guy handling the door plugs, there's no way Spirit should have tried to open it on their own.
The root problem was turning the assembly of the plane from a one company job (Boeing) into a two company job (Boeing + Spirit). I didn't see any adversity in the interactions, but the moment there's two orgs involved communication becomes a major issue. If you talk directly to workers in the other org it becomes a gong show because management doesn't know whats going on. And if you go only throu
Troubling signs (Score:4, Insightful)
As someone who has had a career in process and quality management for over 15 years and has spent a decade as Quality Manager over an AS9100 certified operation, seeing the quoted phrase "failed to catch the faulty work" tells me they are still failing. Until their perspective matures to "Our management system failed to prevent the faulty work" they will continue to experience critical failures. Inspections don't create quality. Quality is either there or not there at the time of manufacturing. If they continue to approach their problems retroactively as that statement shows they are, it is only a matter of time before more bad work slips through. It's not the way to run a business on which lives depend.
Re: (Score:1)
Agree. "You can't inspect-in quality." Quality comes from design of the processes and design of the product. The door was designed such that it could be snapped in and look OK without the bolts (and work OK for many flights.)
What about springs that would prevent the door from seating without the bolts?
At least there should be warning labels on the inside of the door near each bolt:
"Bolt #1 of 4 [bolt dimensions] [bolt part number]"
Yes, there are process problems: The workers removed and apparently lost the
Rights without responsibility (Score:2)
How can the board and the executives be allowed so much profit without being required to do due diligence to prevent these failures, and not sharing in the responsibility for them?
It's past time to institute a system in which the execs are held responsible for the failures of the company. When they permit expensive problems with negligence (or worse, cause them by creating a toxic culture of corner-cutting) take the money out of their compensation.
Stop work order (Score:2)
I'm actually impressed by the degree of documentation here, and this seems to me to be precisely a lesson in why documenting actions is crucial.
One thing I wonder about here is whether they have a capability in place similar to what is common in the oil and gas industry (with which I've worked over the years). At many O&G sites, like compressors and processing plants, if any worker sees something alarming that needs to be immediately resolved, they have the ability to issue a stop work order. This means