Follow Slashdot blog updates by subscribing to our blog RSS feed

 



Forgot your password?
typodupeerror
Software Transportation IT

Missing Files Blamed For Deadly A400M Crash 253

An anonymous reader writes: Think you had a bad day when your software drivers go missing? Rejoice, you get to live! A fatal A400M crash was linked to data-wipe mistake during an engine software update. A military plane crash in Spain was probably caused by computer files being accidentally wiped from three of its engines, according to investigators. Plane-maker Airbus discovered anomalies in the A400M's data logs after the crash, suggesting a software fault. And it has now emerged that Spanish investigators suspect files needed to interpret its engine readings had been deleted by mistake.This would have caused the affected propellers to spin too slowly causing loss of power and eventually, a crash.
This discussion has been archived. No new comments can be posted.

Missing Files Blamed For Deadly A400M Crash

Comments Filter:
  • Good god. (Score:5, Insightful)

    by Anonymous Coward on Thursday June 11, 2015 @03:28PM (#49893713)

    Is it so hard to have a integrity check and diagnostic set run as part of the preflight checks? If you can place hundreds of miles of wire and know what's what, surely they have computer engineers competent enough to make something like this to catch such glaring errors.

    • Re:Good god. (Score:5, Insightful)

      by Anonymous Coward on Thursday June 11, 2015 @03:33PM (#49893755)
      We've lost that kind of 'slow down and make sure it's right' attitude that engineers really need to have. In this fast-paced road of cutting costs and letting the marketing group run the show, the pressure to get product out the door as quickly as possible no matter what is unstoppable for software in particular, but really almost anything that is able to be 'patched' later. Making consumers into your beta testers is douche-y enough, but doing it when lives are at stake should be punished as criminal and in an extremely harsh and public way.
      • Re:Good god. (Score:5, Insightful)

        by TWX ( 665546 ) on Thursday June 11, 2015 @04:06PM (#49894013)
        Case in point, the Toyota vehicle acceleration problem.
      • Re:Good god. (Score:5, Insightful)

        by joh ( 27088 ) on Thursday June 11, 2015 @04:37PM (#49894223)

        We've lost that kind of 'slow down and make sure it's right' attitude that engineers really need to have. In this fast-paced road of cutting costs and letting the marketing group run the show, the pressure to get product out the door as quickly as possible no matter what is unstoppable for software in particular, but really almost anything that is able to be 'patched' later. Making consumers into your beta testers is douche-y enough, but doing it when lives are at stake should be punished as criminal and in an extremely harsh and public way.

        As far as I know aerospace software is far away from what you describe. Of course you're right if you say that these things are a reason for problems, but THIS is very well understood and usually software for planes is nothing like a consumer product.

        They screwed up, yes, but if they would be "punished as criminal and in an extremely harsh and public way" nobody would ever do anything useful anymore. The problems leading to this crash have to be analyzed and understood and then they have to make sure that the same thing can't happen again.

        But of course: If this was due to someone not following procedures or messing around with maintenance this can (and will) have consequences. I'm also pretty sure that one or more people will lose their job over that.

        But if you really think you can make shit never happen and things working 100% all the time by "hard punishment" you're just wrong.

        • Re:Good god. (Score:5, Insightful)

          by tlhIngan ( 30335 ) <slashdot@w o r f . n et> on Thursday June 11, 2015 @06:05PM (#49894759)

          They screwed up, yes, but if they would be "punished as criminal and in an extremely harsh and public way" nobody would ever do anything useful anymore. The problems leading to this crash have to be analyzed and understood and then they have to make sure that the same thing can't happen again.

          You know, when an accident happens, the safety board (NTSB, TSB, BEA, etc) interviews are actually privileged information. As in, if you're being interviewed by the safety board, anything you say cannot be used as evidence against you.

          It's a privilege that the safety boards all fight for.

          The reason for this is the safety board's goal is to not find fault, but to find solutions to preventing it from happening again. Doesn't matter if someone hit a button that said "Crash this plane" and pushed it on purpose. They know that if the interviews were not privileged communications, no one would speak to them for fear of self-incrimination. And when that happens, everyone clams up, and you can't figure out why an accident happened or make recommendations to prevent the issue the next time it happens.

          This is especially more so when most complex accidents are a chain of events - this happened, then that happened, then this next thing, plus X, Y and Z and if any of them didn't occur, the accident wouldn't have happened. Almost never is it the result of one definitive action.

        • by mjwx ( 966435 )

          But of course: If this was due to someone not following procedures or messing around with maintenance this can (and will) have consequences. I'm also pretty sure that one or more people will lose their job over that.

          If someone was deliberately messing around with the files when they weren't meant to, that would be grounds for criminal negligence. They could end up in front of a court, not just out of work. But I agree with your point, harsh punishments dont work as much as people pretend they do. But thi

      • We've lost that kind of 'slow down and make sure it's right' attitude that engineers really need to have.

        Oh, they slowed down alright, but the attitude was not right.

        this would have caused the affected propellers to spin too slowly causing loss of power and eventually, a crash.

        • A "landing" is where you run out of altitude, airspeed and ideas at exactly the same time you are over the touch down zone of the runway. They ran out of altitude before they ran out of airspeed and ideas, and they did it at the wrong location.
    • Re:Good god. (Score:5, Insightful)

      by fuzzyfuzzyfungus ( 1223518 ) on Thursday June 11, 2015 @03:37PM (#49893785) Journal
      What's surprising to me is not merely that; but if the calibration data are so important that the engine shuts down without them, how did the aircraft take off?

      If the calibration data are nice, good for fuel economy, improve reliability, etc. you'd expect things to continue working without them, albeit possibly not as well as the manual specifies.

      If the calibration data are Absolutely Vital Lest The Engine Throw A Propellor Right Through The Cockpit, or something of that nature, how did the aircraft allow you to take off with 75% of the data missing? An actual error handling arrrangement would, of course, be in good taste; but even without one I would have (naively, apparently) expected the situation to take one of two courses: if the data are semi-optional, things would work, if perhaps not well. If they are Vital, attempting to get off the ground would have failed. Successful takeoff, followed by shutdown and fiery death, though, seems weird.
      • Re:Good god. (Score:4, Insightful)

        by ChumpusRex2003 ( 726306 ) on Thursday June 11, 2015 @03:52PM (#49893907)
        My guess is that rather than "files" per se, these are look-up tables which were statically linked into the binary.

        On this type of safety critical application, it's a key design aim to avoid code which might fail or throw an exception at runtime. So, rather than load data from a file, which could fail due to a memory allocation failure, a file system failure, etc. the relevant data is static linked, so if the executable successfully launches, it cannot fail to have the data available.

        I don't know what these tables might have been mapping, but conceivably if they torque tuning parameters, the engine might still have run if the data was all NULLs, but delivered the incorrect torque in response to control inputs. Of course, if the missing data was things like fueling data, then the engine may have failed to start.
      • Re:Good god. (Score:5, Informative)

        by Spy Handler ( 822350 ) on Thursday June 11, 2015 @04:01PM (#49893979) Homepage Journal

        if the calibration data are so important that the engine shuts down without them, how did the aircraft take off?

        One engine delivering full power and 3 engines running at low RPM would be enough to take off, since the plane was empty and probably had a small fuel load as well.

        Wiki has an article on the crash: http://en.wikipedia.org/wiki/2... [wikipedia.org]

        Looks like they took off, but noticed a problem with the engines, turned around to do an emergency landing, but hit an electrical pylon and crashed. So it's not like they lost all power and fell out of the sky, they had some power and were doing an emergency landing when they hit an object on the ground just before touchdown. 2 of the 6 people on the plane survived.

        • Re:Good god. (Score:5, Informative)

          by tomhath ( 637240 ) on Thursday June 11, 2015 @04:07PM (#49894017)
          As I read it, the files weren't used until the plane was 400 feet off the ground. So takeoff wasn't a problem.
        • This is pretty bad bit of piloting then. There are at least two ways they should have known something was wrong if the engines where not producing enough power.

          1. The engine gauges should be abundantly clear between the RPM and pressure ratio. Both are an excellent indicator of how much power you are getting and if either was incorrect or unsteady after throttle up from flight idle, in ANY of the engines you DON'T proceed but abort your takeoff. You NEVER take an airplane airborne with an engine problem

          • by serbanp ( 139486 )

            It is possible though that the engine uses two sets of tweak parameters, a "default" one during take-off and the "optimized" one when altitude hits 400ft. Everything went downhill (bad pun) when the empty LUT was switched in (at 400ft) and only then the 3 engines lost their thrust.

            • If that's true, this this is an incredibly bad design flaw by Airbus, something on par with forgetting to connect the emergency break handle to the devices designed to stop the wheels from turning.

          • This is pretty bad bit of piloting then...

            Do us all a favor. When the full accident report comes out and you find out you had your head in your ass, please try to learn from it. Then apologize to the families of those you've slandered.

            I wouldn't fault your ignorance of how transport category aircraft are operated if you didn't try to pass judgement on men who are/were far more accomplished than you will ever be.

      • My 2001 Jeep has a base "go home mode" if there is a problem to where the normal engine parameter look up tables can't be used. This way you can still drive the thing to a shop even if some sensors, etc are shot. I know this is a pretty primitive comparison but, at the very least, you'd think the A400M engine software would have a *baked in* "go home without crashing" dataset.

        • by TWX ( 665546 )
          It's referred to as "limp mode" and it limits you to 2nd gear for forward and reverse. It's a transmission control feature, not an engine feature. It's designed to keep a transmission fault from stranding you but to also attempt to mitigate damage; by limiting to second gear the hope is that most owners will seek to get it fixed rather than continuing to drive it that way.
          • Re: (Score:2, Interesting)

            by Anonymous Coward

            limp mode also governs engine RPM to a rather low threshold (sometimes it will simply force the vehicle to a high idle and ignore the throttle entirely if it's drive-by-wire). It is activated if the ECU detects significant engine issues, most especially extreme knocking. It is not limited to the transmission. I've had that mode happen to me on the highway when I only half-way plugged in a MAF sensor and the ECU received significantly faulty data causing wildly incorrect fuel-air mixture ratios. Rather f

          • Re: (Score:3, Informative)

            by Casper0082 ( 2274124 )
            As others have mentioned, limp mode is not just a transmission control feature. It is a fail safe built into the Engine Control Unit. When all sensors are operating correctly, the car has a map to determine the appropriate air/fuel mixture taking into consideration temperature, pressure, exhaust etc to ensure your car runs optimally. When in limp mode, the ECU cannot trust the sensors to determine the optimal air/fuel ratio. There is a base map that will allow your car to run with a rich fuel mixture (s
        • Re:Good god. (Score:5, Informative)

          by Thelasko ( 1196535 ) on Thursday June 11, 2015 @05:13PM (#49894459) Journal

          ... you'd think the A400M engine software would have a *baked in* "go home without crashing" dataset.

          From how I read the article, it does have a default dataset that it switches to when it detects a problem. From TFA:

          The automatic response is to hunker down and prevent what would usually be a single engine problem causing more damage.

          Limiting the speed of a ground vehicle is safe. However, limiting the speed of an aircraft causes a crash. It sounds like they need to reevaluate their "limp home" calibration, as we call it in the industry.

          • Except in an aircraft you have multiple redundant engines, so the idea of dropping a faulty engine to idle is not that unreasonable.

            The problem here was 3 faulty engines, for which there was insufficient redundancy - in this case a "common cause" failure that's a much more difficult problem to deal with.
      • Re:Good god. (Score:5, Insightful)

        by Anonymous Coward on Thursday June 11, 2015 @04:09PM (#49894037)

        Read the article... the warning was not designed to kick in until the aircraft was at an altitude of 400ft (120m).

        Not only do you not know you have a problem until are in the air. You don't know you have a catastrophic problem until you are at an unsurvivable altitude. Too low to effectively use a parachute. Too high to just 'jump out' or belly-land it.

        The worst thing is... a committee signed off that this was an 'acceptable risk'. Members of that committee should be brought of up on criminal negligence and manslaughter charges.

        Not a Luddite, but give me my bicycle back...

        • Absolutely this. Any kind of software on any airplane has to go through layers upon layers of specification, design, validation and acceptance before it ships. This was a deliberate choice by bureaucrats who have no idea what they are doing as much as it's a failure by the software engineers to ensure integrity of the system at all times. They should all be brought up on charges.
        • I'm thinking that if they build these decisions into the software then they should also build a couple of test scenarios into the flight simulator and have a pilot fly them a bunch of times to see what happens. Especially the ones where you decide to have them kick in at 120m.

        • "Jump out"? This thing has a rotation speed around 120 knots... good luck surviving a jump like that even if you managed to get the rear cargo door open in time. What kind of fixed-wing aircraft other than a glider has a stall speed low enough for you to survive jumping?
    • by presidenteloco ( 659168 ) on Thursday June 11, 2015 @03:40PM (#49893807)

      My printer at home does it every time it starts up.

      Too bad the airplane doesn't.

      I guess production delays are more expensive than debugging-by-crash. Sad.

    • by khb ( 266593 )

      #!/bin/ksh

      if [[ ! -r $config_file ]] ; then
            abort "Cannot find engine configuration files"
      fi

  • Tuesday is crash-day, oops I meant patch-day.

  • The engineers probably did one but obviously it wasn't good enough. Software fails, but you can design it to fail in safe ways.
    • by tomhath ( 637240 )
      That was kind of the problem. They designed it to shutdown the engine if something appeared drastically wrong. Normally a single engine shutdown wouldn't be a problem, but three engines all detected the same problem at the same instant and all shut themselves off.
      • The article said 3 of the engines "hunkered down" not shut themselves off. They stopped responding to input and stayed at their current power.
        They did turn around and attempt to land the plane, but hit a pylon in the process.

    • The FMEA's I was party to were basically to give cover. "We had an FMEA, and it still managed to fail." When usually it was actually managed into failure despite engineers asking for more time and less feature creep, and specification uncertainty.

  • So, how did ... (Score:5, Interesting)

    by PPH ( 736903 ) on Thursday June 11, 2015 @03:29PM (#49893731)

    ... the engines even start. Or throttle up to take-off power?

    Come on, folks. Turn the power on to the engine controllers at the flight line and the status display should have been flashing warnings. Nobody should have even started this thing.

    • Re:So, how did ... (Score:5, Informative)

      by Anonymous Coward on Thursday June 11, 2015 @04:01PM (#49893973)

      The story seems to massively simplify how the ECUs work. Each engine needs to be calibrated after production so that the sensor data it hands to each ECU is actually meaningful due to the way it's actually acquired in the engine. The parameter set isn't stored in the engine, but in the associated ECU. To prevent them from getting out of sync, the engine itself contains a little register with the checksum of the parameter set. If that checksum doesn't match, the ECU shouldn't power up the engine. However, the register and the ECU are initially loaded with a default parameter set used in testing scenarios. Looks like that one might have been untouched for the engines on that flight. Now, this is bad because the ECU now misreads the true engine status in various ways and can even think that an engine which is otherwise running fine is seemingly in some critical condition - e.g. power output too high, which causes an immediate shutdown to prevent engine damage. A jet engine that fails by disintegration has a high chance of slicing other airplane parts with ripped off fan blades. This is why hard engine shutdowns do make sense. But when putting the pieces of this puzzle together, this is starting to look similar to how Murphy's law came to be: an exceptionally unlikely chain of human errors ruining everyone's day.

      • You forget where you are posting..

        Every accident in the private market is always the fault of the free market and money hungry CEOs.

      • Re:So, how did ... (Score:4, Interesting)

        by TubeSteak ( 669689 ) on Thursday June 11, 2015 @07:56PM (#49895255) Journal

        A jet engine that fails by disintegration has a high chance of slicing other airplane parts with ripped off fan blades.

        It's actually exceedingly rare for there to be an uncontained failure.

        That engine shroud is intended to handle catastrophic failures at full throttle.
        This video is a test of the Rolls-Royce Trent 900 engine that went into the Airbus A380. The test starts ~3:25 in.
        https://www.youtube.com/watch?v=j973645y5AA [youtube.com]

        Then again, this is the same engine after an oil leak led to an internal engine fire
        https://www.atsb.gov.au/media/2891294/vh-oqa-fig7.jpg [atsb.gov.au]
        https://www.atsb.gov.au/media/4173628/ao-2010-089_vh-oqa.jpg [atsb.gov.au]

        The Australian Transport Safety Bureau (ATSB) found that a number of oil feed stub pipes within the High Pressure / Intermediate pressure (HP/IP) hub assembly were manufactured with thin wall sections that did not conform to the design specifications. These non-conforming pipes were fitted to Trent 900 engines, including the No. 2 engine on VH-OQA. The thin wall section significantly reduced the life of the oil feed stub pipe on the No. 2 engine so that a fatigue crack developed, ultimately releasing oil during the flight that resulted in an internal oil fire. That fire led to the separation of the intermediate pressure turbine disc from the drive shaft. The disc accelerated and burst with sufficient force that the engine structure could not contain it, releasing high-energy debris.

        Most of the shroud's strength is focused around the main fan blades instead of the turbine blades that are much deeper in the engine.

    • by Rinikusu ( 28164 )

      I dunno, the asshole in me thinks this sounds like one of those Microsoft Validation things.. Free Engine Software for 500 miles, then activation required!

  • You'd think there would be some kind of checks in place that wouldn't allow the plane to operate when critical files are missing. Or that the files couldn't be deleted.

    Stories like these are the reason I can't believe auto manufacturers are even considering being able to push updates to cars. The checks in place for aircraft hardware is extremely rigorous. Pretty much every nut and bolt has a complete history log. If this kind of thing can happen on an aircraft, what happens when some weird conditions occu

    • The checks in place for aircraft hardware is extremely rigorous.

      Yes, but how many of those regulations and checks trace back to accidents versus an engineer's foresight? I'd expect that most items in a pilot's pre-flight checklist do trace back to accidents. And it seems the computer's pre-flight checklist will too.

      I once heard that the expression "Navy regulations are written in blood" was used to explain to new sailors why so many tasks are to be performed exactly the way the regs say and in no other manner. The phrase was then elaborated on explaining to the sailo

  • Separated by cause: Software bug vs Hardware bug.

    • That would depend who you ask. If you ask the hardware guys, it's always a software problem and vice versa. At least that's the way it is with the hardware guys I work with ;)

  • by Frosty Piss ( 770223 ) * on Thursday June 11, 2015 @03:57PM (#49893945)

    Just my take as a software engineer and current DoD employee that works with C17...

    There should have been some process on firing up the jet / avionics / computers that ran checks to see that even if software was not latest, was it CONSISTENT?

    Big fail from the software engineering standpoint.

  • Return codes? (Score:5, Insightful)

    by cfalcon ( 779563 ) on Thursday June 11, 2015 @03:57PM (#49893947)

    This is a tragedy, but since we're on a tech site, lets talk tech.

    Return values are handled oddly in pretty much every major language. Many API calls want to return something simple- int or bool- and if anything is more complex than that, generally require an actual data structure to be returned, often as a reference. This means that the "I didn't do this" action has a variety of ways to be be passed back- none of them even close to standard.

    If something returns a distance, magnitude, or size, "0" normally means "Error, nothing happened" which is often the same as "Sure, I wrote 0 bytes. Really."
    If something needs to distinguish between success ("I did the thing 0 times as requested" and failure "I couldn't do the thing because of an error condition"), then sometimes a -1 is returned, or an exception thrown, or something else.

    In this plane, something was, at some point, responsible for getting data about the engines. Likely, this happened in layers, each one having access to the results of the lower pieces. One of those pieces had the task of parsing those files.

    So EITHER someone (process, program, whatever) meant to say "This is a problem" and instead said "Here's some default data", OR someone ELSE in that chain of commands (process, program, whatever) has a default for a "This is a problem" result to use as a failsafe, and it was never tested or never communicated up.

    We probably won't get the technical details that go from "files missing" to "engines don't work". Certainly, several level of software or hardware could allow for any number of workarounds in this case, and I'm sure they have a complex system and this was some eventuality that was hard to test for.

    Still, interesting to think about the error return methodology, and how it's so different everywhere in CS.

  • The summary, as usual, is terrible. The missing files were calibration data for the engine controllers, not executables of any kind.

    However, the article says some astonishingly stupid things, like: "'Nobody imagined a problem like this could happen to three engines,' a person familiar with the 12-year-old project said."

    Well, duh.

    Since the human imagination is known to be almost completely useless as a tool for understanding reality or predicting the future, this has to be the most obvious observation since

    • However, the article says some astonishingly stupid things, like: "'Nobody imagined a problem like this could happen to three engines,' a person familiar with the 12-year-old project said."

      If it was a mechanical issue, then yes, I would believe it would be a billion to one chance for something to happen to all three engines. On the other hand, since it is software, I would say that it is a billion to one chance that it would NOT happen to all three engines.

  • by dav1dc ( 2662425 ) on Thursday June 11, 2015 @04:07PM (#49894015)

    + http://en.wikipedia.org/wiki/T... [wikipedia.org]

    The first computer controlled X-ray machine.... which accidentally irradiated some people to death...
    due to *gasp* software faults! (say it ain't so!)

    I first heard about the Therac-25 during my "Ethics in Computer Science" class many years ago - it made an excellent case study... about problems just like this one.
    Once the textbooks get updated, Therac-25 will be replaced with a case study about the a400m roll out. ^_^

    • Will the case study say when a risk is identified and a manual procedure put in place to mitigate the risk is specified, don't be surprised when the procedure fails?

  • by Gordo_1 ( 256312 ) on Thursday June 11, 2015 @04:09PM (#49894039)

    FTFA: "...Without the vital data parameters, information from the engines is effectively meaningless to the computers controlling them. The automatic response is to hunker down and prevent what would usually be a single engine problem causing more damage. This is what the computers apparently did on the doomed flight, just as they were designed to do."

    So, in other words, each engine did exactly what is was designed to do, which is to act independently and shut itself down. There's no executive override function that says "hmmm, maybe we shouldn't shut down 3 engines at the same time!" The crew had no chance against an obviously buggy software implementation. Pilots need more control to override complex software like this in emergencies.

  • They were spinning too slowly? Isn't this why the pilot has a throttle? And if they are supposed to 'correct' and 'adjust' the input from the pilot, as one article explains, then how did it ever take off in the first place? Shouldn't there be a basic check like 'if altitude != 0 { allow_engine_off("NO!") } I'm sure there are all sorts of reasons why it's better this way, but it seems like when the plane is able to just ignore the pilot, then you are simply waiting for a catastrophe to occur.
    • A throttle is of no use when the engines can't respond to it. The engines (3 out of 4) initially did not respond to the pull-back from takeoff power at around 1500 feet. The pilot then pulled back to flight idle, and the engines dropped to flight idle. Unfortunately they then did not respond to the throttle back up, and remained stuck at flight idle. You can't keep a plane in the air with 3 out of 4 engines at idle for very long at 1500 feet. (You also can't fly a plane very well with the engines stuck

    • Apparently the engines did not turn off. They just didn't respond to an increase in power.

      When the airflow meter in a car spits out invalid date, the engine ECU doesn't shut the engine off. It logs a check-engine code and ignores data from that sensor and runs in a safe mode, using parameters it knows will not result in catastrophic failure.

      Same thing happens in automatic transmissions, they'll stick themselves in 2nd or 3rd gear and refuse to shift to any others except N or R (Park is still a mechanical se

  • The last word... (Score:4, Insightful)

    by Unknown74 ( 3041957 ) on Thursday June 11, 2015 @04:14PM (#49894073)
    " The more they overthink the plumbing, the easier it is to stop up the drain. " - Montgomery Scott, Star Trek III
  • I highly doubt the maintenance software is used to operate the engines during flight. If it is that is a VERY big fuck up on the part of the manufacture. None of the planes I've worked on used the maintenance software for flight ops. Sure it records during flight, but that is separate from the instruments and the throttles.

    The high tech word of aviation is at least 30 years old. There is a reason for that, it works and it rarely fails. All the fancy stuff is bolted on top of the bombproof legacy gea

    • The maintenance software is probably used to load the software on the ECU.
      It was ECU software configuration that caused the failure.
      Perhaps that's why they're telling their customers not to use the maintenance software.

  • by toygeek ( 473120 ) on Thursday June 11, 2015 @04:51PM (#49894313) Homepage Journal

    Engineer 1: "Hey, I know, I'll build in a function that wipes the entire control system when it starts a firmware update so that no old software gets left behind after the update."
    Engineer 2: "It'll save a ton of time on this firmware update if I leave out the engine control functions, since those aren't being updated. My bosses will love me!"

  • I like reminiscing about the rope-and-pulley days but i've been stranded with a broken clutch steel-rope cable, I've had another one snap on a bike, and points-and-condenser ignitions are inhumane and intolerant of lapses in maintenance. That peculiar smell that old cars and old planes had? incomplete combustion.

    I like this computer-controlled world. Things work much better.

    The rope-and-pulley analog here would be "Hey Bertie, did you put the cotter pin on that rod?" "Ya ya, sure sure!"

    Meanwhile, as the

The party adjourned to a hot tub, yes. Fully clothed, I might add. -- IBM employee, testifying in California State Supreme Court

Working...