Toyota's Engineering Process and the General Public 345
Doofus writes "The Washington Post has published in today's paper an article titled 'Why it's so hard for Toyota to find out what's wrong' by Frank Ahrens on the Toyota situation and the difficulties of adequately conveying to Senators and Representatives — most of whom are non-technical — the debugging process. Ahrens interviews Giorgio Rizzoni, an 'expert in failure analysis' at Ohio State, who describes the iterations of testing that NHTSA will likely inflict on the Toyota sample cars they have purchased, and then moves into the realm of software and systems verification: 'He explained that each vehicle contains "layers of computer code that may be added from one model year to next" that control nearly every system, from acceleration to braking to stability. Rizzoni said this software is rigorously tested, but he added: "It is well-known in our community that there is no scientific, firm way of actually completely verifying and validating software."' Ahrens ends the piece with a quote from a 2009 LA Times interview with former UCLA psychology professor Richard Schmidt about how user reports are often unreliable: 'When the driver says they have their foot on the brake, they are just plain wrong. The human motor system is not perfect, and it doesn't always do what it is told.'"
Toyota is currently planning an event to challenge evidence presented by professor David W. Gilbert that called into question Toyota's electronic throttle system.
Re:What? (Score:5, Informative)
Don't get me wrong, it would be incredibly cost, labor and time expensive, and require real computer scientists, but it is certainly possible.
Speaking as a "real" computer scientist, I think you might have underestimated the time requirement. Most problems in automatic verification are either undecidable, or intractable.
falsely blaming the user (Score:5, Informative)
When the driver says they have their foot on the brake, they are just plain wrong. The human motor system is not perfect, and it doesn't always do what it is told.'
This was true with Audi in the 80's, when 60 Minutes did a report where, among other things, they faked a car accelerating out of control (the car was modified extensively.) And yes, a large number of drivers, particularly the elderly, hit the wrong pedal all the time.
However, there are cases where driver reports are plenty accurate. A great example of this would be the problems Volvo V70R and S60R owners have with brake failure while going up hills [google.com].
I've experienced it three times in the 6 months or so that I've owned my car. Each time, I was headed up a hill towards a stop sign, put my foot on the brake, and there was nothing there- I had to push so hard I was pulling against the steering wheel for leverage. This is a car with big, high-performance brakes that can stop on a dime.
Volvo claims there's no problem, despite numerous reports on the V70R.com and Swedespeed forums. No other models demonstrate the behavior.
Re:What? (Score:5, Informative)
There's even hardware to do it. dSpace [dspaceinc.com] sells some very nice (and very expensive) hardware to do testing. You can setup scripts to test almost any scenario. It'll fake out all the basic sensors and then you can test to see what happens when you hit the brake at 10 mph, 20 mph, 30 mph. You can do burn in tests. Software is very very repeatable. You can often trace right through the Simulink model and find out what is going on.
In the latest versions of CANape you can even view your Simulink Model EXACTLY how you built them and add all of your signal channels to it [vector.com]. If there is a bug or people are experiencing problems, it takes all of an hour at most to figure out what is going on and what is causing it.
And given the short cycle time, you don't have time to rewrite everything. Every company that uses Simulink for models even has verified and validated library blocks. We have a "C to K" block (because one isn't built in). That automatically matches In & Out data types, etc. We have low pass filters that are designed to our companies standards....
And we have engine control models that have been ported from Assembly that have been used for 30 years that 'work'. We're not going to throw that all out the window every development cycle.
Previous comments on how Simulink [slashdot.org] is used to write code in companies that use it.
SAE Paper on how Caterpillar [mathworks.co.kr] uses auto coding generation to write their stuff.
Anyone else think it odd? (Score:4, Informative)
Why does the vehicle ABS (from what I know from the news) get tripped up on instant breaking? Really? ABS... the thing that is supposed to pump the break to allow for cleaner stops triggers breaking problems and increased acceleration?
I just think bad coding in general here. Regardless of "testing"
Re:What? (Score:4, Informative)
Some of these same problems are impossible for humans to verify simply because "solution space" is outside the combined lifetime of every human on the planet. That's why "automatic verification" and why even automatic (or more properly, automated) verification, becomes an intractable problem - simply not enough TIME.
If it will take 100 years to verify every possible code path and input, and the system is needed sometime in the next 50 years, forget it.
Re:What? (Score:3, Informative)
The temperature absolute zero [wikipedia.org] is a temperature we can never reach.
You can actually prove that some small snippets of code are really and truly bug-free, however. You can prove many algorithms correct, and prove that a block of code correctly implements the algorithm.
Re:What? (Score:3, Informative)
Sorry, but you are not correct in the general case. Within a very constrained problem space, you can have formal, verifiable proofs that are turned into programs, yes. But in the broader context of Turing-complete programming languages, you deal with the halting problem. As soon as you add unlimited recursion into the mix, you throw out complete verification.
Which of these paradigms is more appropriate really depends on the scale of the input space and the complexity of the problem you are trying to solve, and how well you can express the requirements formally.
Re:"An event to challenge Evidence" (Score:2, Informative)
Re:What? (Score:2, Informative)
When I was getting my CS degree I took classes on formal methods for proving that your software is correct. It's not a clear-cut thing. You have to design your language to be verifiable, you have to restrict things like branching and loops to conform to loop controls that preserve base assumptions, and you essentially have to write your code to be verifiable. One thing that I can remember off the top of my head that can impact your ability to formally prove anything about your code are side effects - you might be able to prove that when your loop terminates your loop control variable will be equal to zero, but if your language supports side effects you might not be able to formally prove that variables that the proof methodology suggests should be untouched actually have the same values coming out of the loop that they had going in. You can generate examples on a case-by-case basis, but you can't prove it in the general case because side effects are outside the typical mathematical framework used to do proofs.
Assuming their software is written in bog-standard C and they didn't use these kinds of methods when designing it (which is a reasonable assumption - few areas actually spend the huge amounts of time and money to code this way) then I doubt they could possibly retrofit a proof methodology back onto the system they've built. There's an argument to be made that they should have designed it that way in the first place, but that would have cost money. There's also an argument that they should be using the very expensive redundancy methods that are used to make the code and devices that run airplanes with high safety-critical needs. But, of course, that would also cost money. The market ensures that you're going to get the code that is "good enough" to run the car without killing people rather than the code that you might like to have in the car. External pressure is probably going to end up forcing the auto companies to increase their expectations in what the phrase "good enough" means, but it also will likely mean more expensive testing and coding processes which will mean larger price tags on the cars in the future.
followup comments (Score:5, Informative)
If neutral won't work- you can also turn off the ignition, but don't turn the key completely off, or you'll engage the steering lock(ie, go to the 'accessory' position.) You will not "lose steering"; at any speed over about 2-3MPH, steering assist becomes less and less necessary, particularly if you don't have very wide tires.)
If you "ride" the brakes, the pad and rotor will heat up and "cook"; consumer, mass-market pads are designed to have good "cold" (ie instant) grab, be easily modulated, quiet, not cause excessive wear on the rotor, and not generate brake dust that is impossible to remove from the wheels. Racing pads are designed for higher temperatures (where among other things, you get much more heat transfer from the rotor to the air blowing past/through it), but they have very lousy "cold" bite. Also, heat up the calipers enough, and you will cause the moisture in the brake fluid to boil (your brake fluid should be changed at a MINIMUM every 2 years, because it is hygroscopic), and that boiling will result in "vapor lock"- no brakes. The brakes MUST be bled after such an incident.
Audi successfully defended itself from several lawsuits and even won a countersuit in a case where a mother crushed her boy against their garage wall (after going through the garage door!). Interviewed by an officer afterwards, she repeatedly said she'd hit the wrong pedal. They sued a few months later claiming the car had "gone out of control". As someone who knows Audis well, particularly the mid-80's 5000 turbo series- the idle stabilization valve (the only way the car computer can increase engine speed) simply cannot allow enough air to bypass the throttle enough to cause the car to lay down burnt rubber, crash through a garage door, and embed itself in a house wall.
The problems with the Volvo "R" models have been reported in a number of other european cars; you'll also see the words "ice mode" thrown around occasionally. Many ABS controllers since 1990 or so have an accelerometer to detect when all the wheels stop simultaneously but there is no corresponding negative acceleration. "Ice mode" is supposedly some sort of variant of this, and there has been great debate as to whether this "mode" is internet folklore, but you'll find many, many posts on all sorts of varying car enthusiast forums.
Yes, interesting. (Score:5, Informative)
"... it was determined that [Toyota] Electronic Control Module (ECM) malfunction detection strategies were not sufficient to identify all types of fundamental APP sensor and/or circuit malfunctions. Some types of Electronic Throttle Control (ECT) circuit malfunctions were detectable by the ECM, and some were not. Most importantly, the Toyota detection strategies were unable to identify malfunctions of the APP sensor signal inputs to the ECM. APP sensor signal circuits must be undeniably correct to electrically convey the appropriate driver commands to the ECM."
Next paragraph:
"With the two APP sensor signals shorted together through a varying range of resistances, all four Toyota vehicles tested thus far reacted similarly and were unable to detect the purposely induced abnormality. The types of signal faults introduced into the APP circuit should have triggered the vehicles' ECM to illuminate a warning lamp within seconds."
Bottom of page 4:
"In addition, the shorted APP signal circuits were connected momentarily to the sensor's five-volt supply circuit with the vehicle in drive. In all test vehicles, the ECM did not set a DTC and the engine speed increased rapidly to full throttle. This result shows that unusual or sudden unintended acceleration of the vehicle was possible in the ETC test vehicles."
Re:Good time to buy a Toyota (Score:1, Informative)
Over 40% of sudden acceleration are from Toyota drivers http://blogs.consumerreports.org/cars/2009/12/sudden-unintended-acceleration-sua-analysis-2008-toyota-lexus-ford-gm.html [consumerreports.org]. Which is far higher then Toyota's share vehicle parc (number of vehicles in use). This is an indication that there may something other then human error. (Ford is also higher then it should be, with most of its complaints coming from the F-series, the common explanation for this is the shape of the transmission tunnel in certain bodystyle causes the driver to place his or her right foot in an unusual manner causing the driver to hit the wrong peddle)
Re:What? (Score:5, Informative)
Oh, dear, dear, dear. Have you evern _looked_ at the details of the TCP protocol, or how and why RAID works? It's only in a non-existent universe with point sources, frictionless bearings, and perfectly spherical fields that such mathematical precision is completely reliable. Even then, the 3-body problem has _not been solved_, nor is the Schrodinger equation easily solved for even the smallest circuits.
So in the real world, "butterfly effects" of small, difficult to predict and model events can cascade into profound changes in quite large-scale systems. Digitization can help, by driving most such effects below the necessary thresholds to turn a bit "on" or "off", but it's not perfect. And mathematical models of mechanical systems are profoundly _not_ perfect: the actual shape of a piece of metal after manufacture, and especially after changes are made after the original design for expense or other manufacturing reasons, can profoundly change the behavior of the real system produced.
Even with software, unless people can follow the code end-to-end, it's prone to surprising errors. Rounding errors, for example, can creep in. Values that are not tested for because one computer scientist read the API one way, and the other read it another way, are rife, and can be be very difficult to avoid.
Re:What? (Score:1, Informative)
sum = input0 + input1 + ... + input400;
If inputN is constrained to be {0..3} then I now have a test space of at least 2^100 in order to prove addend uniqueness.
Re:falsely blaming the user (Score:5, Informative)
Professor Richard Schmidt says user reports are often unreliable: 'When the driver says they have their foot on the brake, they are just plain wrong." My '08 Prious has had three "surge" events. I was able to stop all three times. I challenge professor Richard Schmidt: If my foot was on the accelerator, how did I in fact stop? The Toyota people have told me they'll be reflashing the processors of all the Prius cars in a few months so any brake signal will shut down the engine. Why wasn't that done from the beginning? But anyway, I'm looking forward to the modification. In the meantime, I'm practicing quickly hitting the Neutral gear lever.
He's not saying every human report is wrong, it's just humans often think they saw or did one thing when they didn't. My experience conducting crew assessments in operational and simulator scenarios backs that up - someone will swear they did or say X when multiple observers and the event logger shows they didn't. It's not that they are lying just that we are often unreliable observers.
One of the hardest things in event investigation is sifting through eyewitness statements - which are often misleading or wrong; especially people seem not to be able to say what they saw; but rather interpret it. For example, instead of "I saw smoke" they say "the engine was on fire;" the former is a statement of what they saw, the latter conjecture.
Re:What? (Score:4, Informative)
Absolute 0 is the coldest a material can get. You can have a temperature lower than 0 Kelvin [wikipedia.org], but it doesn't mean what you think it means.
Re:Why? (Score:1, Informative)
Sorry guy I call Bullshit. Yes Toyota has a problem, but it pales in comparison to some of the major problems that all the other "Big 3" have had. (Exploding gas tanks, steering columns falling apart while driving). However, this would be a lot easier to believe "if" the current administration didn't just start to own GM. Now a company called Government Motors. I use to be a person who would almost always buy American, but I will rot in Hell before I EVER buy another American car. They got my @#$@# money for nothing, and they can fabricate all the stories they want about their competition now, but all that does is make me cement my decision to buy a Toyota for my next car.
Ford, you are somewhat forgiven for this, but your company did also ask for money.
Re:followup comments (Score:1, Informative)
Your advice would be great were it not for the fact that most driver education courses in America teach student drivers to use only relatively delicate applications of the brakes (to avoid skids), almost nobody has ever experimented in their own car to find out just how hard you have to press the pedal to achieve maximum braking, and studies have shown that many drivers are afraid to use maximum braking even in emergency conditions.
Oh, and also that fly-by-wire engine and transmission systems may well accept driver control inputs as mere suggestions rather than commands.
Re:Anyone else think it odd? (Score:3, Informative)
Not stop but most vehicles have a thing called limp mode, which causes the vehicle to kick into a safe state where it can only go upto 45-50mph and has very low acceleration. There was a time when limp mode only had a drivable range of 60mi to get you to a service center of some kind, but the distance is much larger now.
Re:followup comments (Score:4, Informative)
Re:tin.foil.hat (Score:4, Informative)
the prius is like a 5 year old car model and in all this time american "muscle" motor never came up with an answer.
The Prius is a car that, for a car of comparable size, is more expensive to build, more complex to repair, and nets out as more expensive over the general lifetime of a car. (Even if YOU don't own it for the whole time, most US cars run for a few hundred thousand miles before being scrapped.)
GM, who tried an electric car WAY back in the early 90's, decided to largely pass on the paralell hybrid tech of the Prius and its ilk, opting for only a small pseudo-hybrid option on a few of its models. (Essentially, a small electric motor/brake assist on the drive wheels.) Instead, they're rolling out an actually innovative serial hybrid this year. And if you take a moment to understand the difference, the change is profound.
The Prius and its ilk are "parallel hybrids." You have an underpowered classic internal-combustion motor driving the wheels via direct kinetic energy, with an electric motor also contributing kinetic energy from electrical power it gets from regenerative breaking or, for the modified ones, being plugged into a wall. It will NOT perform its full performance without any gas in the tank, and for most models you can't even drive it to a gas station 1 mile away if you don't have enough gas to start.
GM's Volt and its ilk are "serial hybrids", like diesel-electric trains. The wheels are powered ONLY by an all-electric drivetrain, and the internal combustion engine serves only to produce additional electricity. The engine only runs at its peak efficiency, and doesn't need to run at all if the batteries have enough of a charge in them. You could literally drain your fuel tank dry, top off the battery charge, and then drive to a gas station 40 miles away. (And with fewer moving parts, a mass-market volt should last longer and be easier to maintain than its paralell-hybrid ilk.)
Re:What? (Score:4, Informative)
If the software and control system of a modern passenger car does not allow for a complete verification of 2 pedal and 1 steering sensors, 4 brake and 1 steering actuator and 2 brake lights, then this software is unfit for its intended purpose. If the system does not allow specific subset of commands to be scientifically, mathematically verified to work as intended even in cases where non-verified parts of the software return any combination of valid and invalid values, then the subsetting structure of that system must be regarded as a complete failure.
You've forgotten about the numerous sensors INSIDE the engine, transmission, etc. I don't know what type of engine these cars had, but if its achieving anything like the levels of power, economy and reliability expected in modern cars it will have several hundred sensors inside the engine, and actuators firing many times per engine revolution to control fuel injection, ignition, valve timing, etc. as well as monitoring temperatures, oil pressure, air flow, exhaust composition, brake pad wear, wheel rotation, etc. Making an intenal combustion engine work at peak efficiency is NOT simple.
Book: Unsafe at Any Speed (Score:3, Informative)
Re:What? (Score:3, Informative)
Do you have any references that support your emphatic claims?