Forgot your password?
typodupeerror
The Military Technology

Mechanic's Mistake Trashes $244 Million Aircraft 428

Posted by Unknown Lamer
from the total-tool-awareness dept.
Hugh Pickens writes "An accident report is finally out for the Air Force E-8C Joint Surveillance Targeting and Attack Radar System that had started refueling with a KC-135 on on March 13, 2009 when the crew heard a 'loud bang throughout the midsection of the aircraft.' Vapor and fuel started pouring out of the JSTARS from 'at least two holes in the left wing just inboard of the number two engine.' The pilot immediately brought the jet back to its base in Qatar where mechanics found the number two main fuel tank had been ruptured, 'causing extensive damage to the wing of the aircraft.' How extensive? 25 million dollars worth of extensive. What caused this potentially fatal and incredibly expensive accident to one of the United States' biggest spy planes? According to the USAF accident report, a contractor accidentally left a plug in one of the fuel tank's relief vents (PDF) during routine maintenance. 'The PDM subcontractor employed ineffective tool control measures,' reads the report. Tool control measures? 'You know, the absolutely basic practice of accounting for the exact location of every tool that is used to work on an airplane once that work is finished.' Air Force Chief of Staff Gen. Norton Schwartz just told Congress, 'there is a JSTARS platform that was damaged beyond economical repair that we will not repair.' So, if this is the one Schwartz is talking about, then one mechanic's mistake has damaged a $244 million aircraft beyond repair."
This discussion has been archived. No new comments can be posted.

Mechanic's Mistake Trashes $244 Million Aircraft

Comments Filter:
  • Shit Happens (Score:5, Insightful)

    by rotorbudd (1242864) on Tuesday January 31, 2012 @09:11AM (#38876097)

    I've been an A&P for over 35 years and I've seen worse.
    (by pilots and mechanics)

    • yes - they are lucky no-one died. I've seen tool control related accidents (fod) and other problems due to maintenance issues go a lot further south than this - though the dollar total is impressive.

      • oh - reading the executive summary (3rd link) it says damage was 25 million.

    • Re:Shit Happens (Score:5, Interesting)

      by Suki I (1546431) on Tuesday January 31, 2012 @09:51AM (#38876441) Homepage Journal

      I've been an A&P for over 35 years and I've seen worse.
      (by pilots and mechanics)

      In Chuck Yeager's biography he talked about an assembly mechanic who was installing a bolt the wrong way, even though his instructions said the right way to do it. Resulted in numerous fighter plane crashes and almost killed Yeager when he was test flying one of the planes to see what was causing the crashes.

      • Re:Shit Happens (Score:5, Informative)

        by Garybaldy (1233166) on Tuesday January 31, 2012 @10:00AM (#38876515)

        I remember reading that as i have repeated the story many times. The women on the assembly line could not grasp why you would stick a bolt in upside down. Always being taught to put it in facing down. So if the nut ever came loose the bolt would not come out. Even though as you said the instructions said to put it in upside down.

        The reason being the head of the bolt was shorter and would not interfere with a control cable.

        • Re:Shit Happens (Score:5, Informative)

          by vlm (69642) on Tuesday January 31, 2012 @10:11AM (#38876621)

          She knows clearance issues are why you install a shorter bolt Again, engineering design failed, miserably, so a way to blame the peon.

          If you insist on putting the brake pedal on the right foot and accelerator on the left, it doesn't matter how loudly you blame the driver, its still a design failure.

          This specific incident was hashed out in one of those freshman "intro to engineering ethics" classes I had to take a long time ago. Still remember it. It was a huge design failure, although you could claim it was also a huge management and PR success to put all the blame on some poor chick. Was used as an object lesson for how management picks the winner and loser, sometimes engineering gets it, sometimes operations/factory floor gets it, and part of being an engineer is "toughening up" that you're going to be involved in corporate BS like that, so get used to thinking about it.

          • by MightyYar (622222)

            I'd say she shares some blame, by making a design decision at assembly time rather than bring the matter to the attention of her supervisor.

            But yeah, bad design. If bolt orientation is so critical, you need to make the design idiot (or self-declared assembly expert) proof. I suspect "shorter bolt" wouldn't have worked - and in any event, having one bolt shorter than the others might be asking for trouble as well. But even something as simple as stamping or stenciling "bolt head down!" might have averted the

            • Re:Shit Happens (Score:5, Interesting)

              by vlm (69642) on Tuesday January 31, 2012 @10:33AM (#38876839)

              Or if you're going to intentionally violate international standards of assembly, management needs to hire a QA/QC guy who's sole job is to make sure things are put together the wrong way. Unless of course he failed in this case, but he was someone important's son, so he can't be blamed...

              There's always a way to design something the "right" way. If clearances are that tight, g-loading of the frame would have screwed it up eventually, or a tiny piece of shrapnel could take down a plane... A "combat" style repair during an emergency on a distant island could cause the loss of a plane, this isn't just a manufacturing problem.

              This incident was an hour long seminar in class and at the end of class, there's no way around it, it was an engineering failure but some lowly peon took the hit, with a sub-text esoteric or whatever meaning that even when engineering "wins" in a corporate BS scenario, everyone else really "loses".

              We came up with all manner of solutions like "shorter bolts everywhere not just one shorter bolt", "rivets not bolts", "reroute the cable". One unpopular one was "well, in wartime, you're gonna take losses, just deal with it".

              The funniest, yet best human factors solution, which won the award for the best solution, was to work with human nature, not against it, and make the build fixture upside down. So the plant workers install the bolts right side up, from their perspective. Don't even tell the bolt installer plant workers that they're working upside down. I wish I could say that was my bright idea, but mine was a crappy solution involving spray painting bolt heads and spray painting the holes on the bolt side using a fixture, which got shot down, something about F-ing up corrosion control chromate primer or whatever.

              • by stevew (4845)

                Me thinks your professor was an idiot.

                The engineering design failed in the opinion of the professor. Yet, there was documentation saying HOW to install something that wasn't followed?

                Further - somewhere someone had figured out there was a problem in this area and had written corrective procedures to avoid the problem. That of and by itself can be considered an appropriate engineering response to a problem! Don't forget - engineering is the application of science to real world problems while optimizing the

                • Re:Shit Happens (Score:5, Insightful)

                  by LWATCDR (28044) on Tuesday January 31, 2012 @12:03PM (#38877921) Homepage Journal

                  No the professor was right.
                  There is a benefit to putting in the bolts the way the worker was taught to do it. It is also the standard way.
                  And the class came up with a number of solutions that would have been better than the upside down bolt.

                  You should always make assembly errors as unlikely as possible. Having a design that will fail if a single bolt is installed in that standard way vs a special procedure is just asking for trouble. Doing when other solutions are available is a fail.

          • by Sponge Bath (413667) on Tuesday January 31, 2012 @10:41AM (#38876935)

            If you insist on putting the brake pedal on the right foot and accelerator on the left, it doesn't matter how loudly you blame the driver, its still a design failure.

            They should take the Apple route and put both functions in one pedal. Simply Genius! (tm)

    • by mcgrew (92797) *

      Indeed, when I was in the USAF I spent the 1st 3 years on the flightline, and there was a lot of accidental damage. One poor fellow backed a C5-A into a hangar and did $50 million in damage. He was sweating bullets for a week until the wing walker got the blame. They grounded the fleet when one of the giant buckets they serviced the tails fell over in another base and killed two mechanics. I saw quite a few land without landing gear on a foam runway, and at least one had an engine fall off. I also saw a C-1

  • by Anonymous Coward on Tuesday January 31, 2012 @09:13AM (#38876109)

    Government contractors. Saving you money like they have never saved it before.

    • by alcmaeon (684971)

      In the real world, a contractor damages $244,000,000.00 of someone's shit, the contractor is paying $244,000,000.00 plus loss of use costs until replacement.

      In the government run world, everyone will have a laugh and the taxpayers will pick up the tab.

      • by Geraden (15689) on Tuesday January 31, 2012 @09:56AM (#38876483) Homepage

        In the real world, faced with $244,000,000 in lawsuits, the contractor folds up and declares bankruptcy.

        Then everyone will have a laugh and the taxpayers will pick up the tab.

        • by s-whs (959229) on Tuesday January 31, 2012 @10:21AM (#38876735)

          In the real world, a contractor damages $244,000,000.00 of someone's shit, the contractor is paying $244,000,000.00 plus loss of use costs until replacement. In the government run world, everyone will have a laugh and the taxpayers will pick up the tab.

          In the real world, faced with $244,000,000 in lawsuits, the contractor folds up and declares bankruptcy. Then everyone will have a laugh and the taxpayers will pick up the tab.

          In the real world, whatever happens, everyone will pay for this. What do you think happens if that firm is properly insured? The insurance company pays and will increase rates for everyone, not just that firm that made the mistake (you can't do stats on a single mistake anyway, and the insurance firm needs to get that money from somewhere if they are to remain as profitable).

          So everyone pays more insurance, this means the companies who pay more insurance have more costs and increase their rates etc. This is not something insulated. Ditto for bankruptcies, not everyone pays as much everyone pays for it in the end.

  • by Anonymous Coward on Tuesday January 31, 2012 @09:16AM (#38876137)

    The most I ever cost my employer for a screw up is about $1.1 million.

  • RFID (Score:5, Interesting)

    by Dr_Barnowl (709838) on Tuesday January 31, 2012 @09:16AM (#38876143)

    Sounds like a great case for RFID inventory control ; tag every tool, log them out of the toolbox with a loop mounted on the side, log them back in again when you return them.

    The article linked mentions this on the second page ; I don't see why you should be limited to the 3M solution though (except maybe they'll bribe someone to make it a regulatory necessity). You can get nearly 2,000 tags for about $100, so it's not like it would be expensive.

    • Re:RFID (Score:4, Insightful)

      by phonewebcam (446772) on Tuesday January 31, 2012 @09:21AM (#38876185) Homepage

      "You can get nearly 2,000 tags for about $100"
      You or I could, but the essential middlemen selling the same stuff to the government would add at least three zeros to the end of that figure

      • Re:RFID (Score:5, Insightful)

        by O('_')O_Bush (1162487) on Tuesday January 31, 2012 @10:02AM (#38876535)
        There are a lot of misconceptions about how contractors work, because typically, their profit margins are no higher than in other lines of business.

        The government is big on COTS hardware/software, and only turn to contractors for specialized circumstances. Those extra zeros come from the unusual design requirements and low volume orders.

        Take the x thousand dollar hammer example. On the surface, that seems absurd, since one can buy a hammer for less than 10$. But when the hammer is going into space and is made of a difficult to machine titanium alloy (tool steel shatters at cold temperatures), is egonomic even through spacesuit gloves, is lightened without reducing mechanical efficiency (makes sense at an estimated 1000$/pound/launch), and only 10 are made (despite flat machining costs), that X or XX thousand dollar price tag seems very affordable.

        The same thing happens in other areas. I work on submarines and some components use joysticks. Sure, commercial joysticks can be obtained for under 100$, but a waterproofed, pov only motion, high durability (sailors treat equipment like crap, and failure is not an option) piece of clockwork machinery that maybe 50 will be made, you are looking at just shy of XX thousand per.
    • by ByOhTek (1181381)

      Seems overly complex. Why not just have the toolbox be able to detect what tools are contained within? Not even bother with the side loop. It could then have a nice little display of how many (and even what) tools are not inside.

      • Simply weight the toolbox on the way out and again on the way back in.

        • by T-Bone-T (1048702)

          That doesn't tell you what tool(s) is/are missing, only that the set is incomplete.

        • Re: (Score:3, Informative)

          by Anonymous Coward
          Simply weight the toolbox on the way out and again on the way back in.

          Wouldn't work. Consumables. Safety wire, cotter pins, packing material. Even small, any of those is enough to cause a major problem. And far too small to be noticed when weighing a 75lb toolbox.
          The way it is normally done is by foam cutout for each tool. A quick look can tell you if something is not in place. Of course, you have to have the brainpower to actually look when you are leaving the area.

          (anon to not screw up previous mods)
          • by ByOhTek (1181381)

            Put the consumables in a separate box.

            However, the issue of WHAT is missing, as mentioned in the other thread, might be critical.

            Also, grease will get on/off tools, and I think that could make enough of a difference if there are any particularly light tools.

      • by T-Bone-T (1048702)

        That is one method of detecting what tools are contained within. The toolbox has to have some method of determining what tools it contains and a tag on each tool and a single reader on the toolbox is about as simple as it can get.

        • by ByOhTek (1181381)

          I was suggesting the whole box be a reader, as having the extra activity of swiping the tool through a loop may be omitted in a rush.

          It would be best if no extra activity were needed to detect which tools are in the box - which means, there is either (a) no way to access the tools except through the "loop", or (b) the toolbox can do a live inventory of it's contents at any time.

      • Re:RFID (Score:4, Informative)

        by rickb928 (945187) on Tuesday January 31, 2012 @11:35AM (#38877511) Homepage Journal

        In the 70s, our tool kit was a canvas bag. We had to check it before we went to the flight line and both of us signed off that it was complete. Then before we LEFT the flight line, we counted again and signed off that it was complete. If the bird was scheduled to fly before we could get back to the tool crib, the crew chief also counted and signed off. Then we returned to the shop, checked the bag in and it was counted again before we could sign off on the work.

        If the tool crib did not get all the tools back, the bird would held until we found the tool or the bird was inspected inside and out. For 2 years I was there, we never lost a tool, and I never heard of anyone losing a tool from any other shop. In fact, my usual task was to lock a fixture, and I had the speed wrench on a wrist leash. Fortunately I never worked on a bird with engines running, which was a whole different protocol.

        It is not that hard to count. From the description of this process, I'm disappointed that the shop didn't have a tool board that would show an empty spot, nor any process to question a missing tool. In our shop back then, a missing tool for ANY reason would have been grounds for a complete inspection, evaluation, and questioning. I wasn't allowed to carry tools into the shop, even that teeny screwdriver we used for rotary switches. Absolute control within the shop system.

        Leaving something on equipment was just inexcusable. Shocking really.

    • by webnut77 (1326189)

      so it's not like it would be expensive

      This is the government. It WILL be expensive.

    • by vlm (69642)

      Sounds like a great case for RFID inventory control ; tag every tool, log them out of the toolbox with a loop mounted on the side, log them back in again when you return them.

      Who fills out the ISO9000 report paperwork documenting the RFID hasn't fallen off the tool and remains in the bottom of the toolbox? You could generate an exception report of tools that were supposed to be used but haven't been used in "X" months, but then someone needs to review that and follow up and most importantly, document it and get a sign off from their boss.

      If the RFID falls off a $125000 radar spectrum analyzer, does that make it non-compliant and eligible to be sold to techs buddy for $50 govt s

  • It's all ball bearings these days.
  • Only 244 million? (Score:2, Insightful)

    by Anonymous Coward

    244 million? Isn't that minuscule? CEOs regularly crash the stock market. But at least they take responsibility! Like... becoming CEO somewhere else?

  • by argStyopa (232550) on Tuesday January 31, 2012 @09:24AM (#38876203) Journal

    A few points occur to me:

    1_ ...how $25 million in repairs is "beyond economical repair" on a $240 million plane? If I have a $20,000 car that's in an accident, it's not uncommon to have $2000 in repairs...that's hardly "totalled".

    2. Now, looking at the pictures, that's pretty serious...but then it's more than $25 million in damage.

    3. the E8 is a converted 707...didn't they stop building those in the 1970s? If this is a 30 year old airframe (at best) then either that damage is $25 million or the plane is worth less than $240 million today.

    4. Finally, as I understand it this damage was done by a subcontractor. When I use subcontractors, they have liability insurance to cover the systems they're working on, plus potential liabilities. Doesn't the US government require AT LEAST such protections when farming out work to contractors?

    By the way, I'd like to further remind the Air Force that this is a COMBAT aircraft. Granted, it's not supposed to be in dogfights or shot at, but this is a piece of military equipment, maintained in difficult conditions/circumstances by relatively inexperienced crew (for example an aircraft carrier's crew largely is swapped out about every 18-36 months). That seems incompatible with its evident fragility.

    • by confused one (671304) on Tuesday January 31, 2012 @09:30AM (#38876265)
      You've hit the nail on the head with #1 - #3. They totalled a 707 airframe, which is not a $244 million dollar plane. Most of that $244 million cost is what makes a 707 a JSTARS -- the payload. And the payload will probably be salvaged and re-used either to build another JSTARS or as spares to support the existing JSTARS platforms. This is being way over-hyped. Big oops for the contractor -- I wouldn't renew the contract; but, I'm not government.
      • by Svartalf (2997)

        It probably basically totalled the airframe. At that point it's cheaper to take the payload out- but don't think pulling the payload and putting it into a new 707 is going to be cheap. It's probably going to cost something on the order of a third to half of the 25 mil at least to do it and then recertify the new plane for service.

    • by Kjella (173770)

      3. the E8 is a converted 707...didn't they stop building those in the 1970s? If this is a 30 year old airframe (at best) then either that damage is $25 million or the plane is worth less than $240 million today.

      They're probably comparing apples and oranges here, the new cost was $244 million but the planes have been in service of some form since 1991, the accident was in 2009. Secondly, that probably includes a lot of R&D costs so a $25 million dollar could be a much larger part of the production cost. Third, maybe the military's needs have changed or other types of craft do better, it might make sense to operate but not necessarily to spend that much to keep it in service.

      Finally, as I understand it this damage was done by a subcontractor. When I use subcontractors, they have liability insurance to cover the systems they're working on, plus potential liabilities. Doesn't the US government require AT LEAST such protections when farming out work to contractors?

      They certainly could, but nothing com

    • by tlambert (566799) on Tuesday January 31, 2012 @10:41AM (#38876919)

      If you read the actual congressional testimony, you would have seen that Schwartz didn't say that it wasn't repairable for ~$25M, which is 10% of the cost of the whole system, he bemoaned his budget constraints, and said they wouldn't repair it as an example answer to the question "Is there any sacrifice you're seeing in ISR...?". Also note that they're only not repairing *the platform*.

      The title of the press release from the Public Affairs office more or less says it all: "Air Force Strategic Choices and Budget Priorities Brief at the Pentagon".

      -- Terry

  • by crimguy (563504) on Tuesday January 31, 2012 @09:25AM (#38876211) Homepage
    $25 million? It's not as if they had to repair the toilets or anything . . .
  • So this is a plane that might get, you know, shot at? In a war or something? And it can't handle two little holes, or be repaired? Sounds like a design flaw to me.

    • Just because the holes were visible from the outside doesn't mean that it is the only damage the aircraft suffered.
    • by cptdondo (59460) on Tuesday January 31, 2012 @09:40AM (#38876349) Journal

      None of the AWACS/JSTARS/etc planes are "made to be shot at". They're civilian airframes stuffed to the gills with super-secret electronics. They rely on fighters and ECM to stay up; they don't do any fighting themselves. Heck, they're unarmed.

    • the sealed tank overpressurized and started inflating like a balloon, inside the airframe. There's all sorts of structural damage where it literally ripped apart the structure from the inside.
    • by felipekk (1007591)

      Yes, and that's probably why the airplane was able to land after the damage suffered.

      It doesn't mean that after getting shot you don't have to repair the equipment.

  • typically misleading (Score:5, Informative)

    by confused one (671304) on Tuesday January 31, 2012 @09:26AM (#38876225)
    You lost an airframe. A significant fraction of that $244 million is payload and equipment that will be recovered and used as "spare parts" to maintain other JSTARS aircraft. The airframe is all that was lost. The airframe is a commercial 707 derivative. It's not an $244 million aircraft, it's a tricked out $5 million dollar aircraft. The issue, now, is replacing the system -- which means assembling another JSTARS. Given typical government contracting practices that will cost another $325 million (inflation adjusted from initial cost of $244 million in 1998).
  • At least he chose to study engineering and not medicine.
  • Oh well (Score:5, Funny)

    by obarthelemy (160321) on Tuesday January 31, 2012 @09:32AM (#38876277)

    If he were a banker he'd get a bonus ?

  • This is for what the 'epic' in 'epic fail' was invented.
  • Acronym brought to life.
  • Well, there's your problem.

  • by 140Mandak262Jamuna (970587) on Tuesday January 31, 2012 @10:15AM (#38876661) Journal
    If I remember Srimati teacher correctly, apparently a whole kingdom was lost because some cobbler missed nailing one nail in the shoe of one horse. I, along with rest of the class, had actually memorized the entire report of the investigation committee. We delivered the report to an assembly of interested parties ( Mohan master accompanied us on the xylaphone) on the annual day of the Mahatma Gandhi Elementary. From the standing ovation we got, I assume our report was spot on and was accepted with great appreciation.
  • by tekrat (242117) on Tuesday January 31, 2012 @10:19AM (#38876701) Homepage Journal

    World War II, if you watch enough of the History Channel, boiled down to quantity winning over quality. Our Sherman Tanks, for example, were utter crap compared to the Panzer and Tiger tanks. But, the USA was able to build a lot of them and they were simple and cheap. The Panzer and Tiger, however, were built in small numbers because they were complex machines.

    Germany was 10 years ahead of the USA technologically. But, Germany wasn't able to build to the quantity needed to fight an industrial giant like the USA, especially while we were bombing their industrial capacity to zero (and losing 60% of our aircraft to do it).

    It is sad that USA is now following Germany's example. We are building overly complex, hugely expensive equipment that cannot be easily field serviced, and building them in limited numbers because we cannot afford them in great quantity.

    Eventually, even though we are 10 years ahead of every opponent technologically, someone will be able to over-run us in a drawn out war simply by having great numbers of simpler, cheaper equipment, and a lot of it.

    And I think we all know who's the industrial giant now, that can produce great quantities of material quickly and cheaply.....

    • by vlm (69642) on Tuesday January 31, 2012 @10:48AM (#38877049)

      A often over looked factor is attrition in WWII. Made up numbers:

      Lets say the US had zero elite level tankers but millions of noobs and we didn't start the land war until, well, frankly pretty much d-day 1944. Solution, make millions of noob-tanks. We didn't have any elite combat veteran tankers anyway to make use of elite level tanks.

      Lets say the Germans had a hundred thousand elite combat vet tankers, but a quarter of them die in combat every year starting in 1939, so by 1945 you've got 12 year olds with hunting rifles "defending" Berlin at the last stand. Solution, make tens of thousands of elite-tanks and hope each elite-tank blows up more than 10 noob-tanks. Eventually you end up with dudes from the assembly line trying to be tankers, that didn't work out so well.

      They darn near won, despite the attrition, so I wouldn't harsh their strategy too much.

  • by automandc (196618) on Tuesday January 31, 2012 @12:24PM (#38878211)

    Forgive the rant, but:

    It is not a "spy" plane, it is a "surveillance" plane. Ever since the 2001 Hainan Island incident this mistake has really irked me. The Chinese used it as a rhetorical club to beat us with when GWB chickened out and let them chop up our plane and imprison our crew.

    A "spy" plane would be one that is designed/intended to escape detection and/or interception while conducting surveillance in places it has no right to be (such as the U2 and SR-71 or the Global Hawk). During the cold war, the Soviet Union consistently protested our overflights of their territory with the U2 and SR71, and sought (and once succeeded) to shoot them down, as was their right. Those were "spy" planes, and Francis Gary Powers was, technically, a "spy."

    The JSTARS E-8 and the Hainan EP-3E are both military versions of the Boeing 707 -- they aren't designed to hide from or evade anyone trying to see and/or catch them. They are big obvious platforms that fly in neutral territory (or over an actively declared battle zone when we have air dominance) and provide surveillance and other capability. They aren't hiding or trying to deceive anyone.

  • by gerardrj (207690) on Tuesday January 31, 2012 @12:35PM (#38878363) Journal

    Why would a plane with so much advanced electronics on board not have a check system or pre-flight checklist item to look for such an installed plug. Supposed a swarm of bees had built a nest in there and blocked it instead of the mechanic's error?

    If something as simple as a plugged vent can cause complete and catastrophic damage to the craft then there needs to be pre/in flight monitoring of that system. Seems a simple pressure gauge in the tank would have prevented this situation from becoming life threatening.

When in doubt, mumble; when in trouble, delegate; when in charge, ponder. -- James H. Boren

Working...