Toyota's Engineering Process and the General Public 345
Doofus writes "The Washington Post has published in today's paper an article titled 'Why it's so hard for Toyota to find out what's wrong' by Frank Ahrens on the Toyota situation and the difficulties of adequately conveying to Senators and Representatives — most of whom are non-technical — the debugging process. Ahrens interviews Giorgio Rizzoni, an 'expert in failure analysis' at Ohio State, who describes the iterations of testing that NHTSA will likely inflict on the Toyota sample cars they have purchased, and then moves into the realm of software and systems verification: 'He explained that each vehicle contains "layers of computer code that may be added from one model year to next" that control nearly every system, from acceleration to braking to stability. Rizzoni said this software is rigorously tested, but he added: "It is well-known in our community that there is no scientific, firm way of actually completely verifying and validating software."' Ahrens ends the piece with a quote from a 2009 LA Times interview with former UCLA psychology professor Richard Schmidt about how user reports are often unreliable: 'When the driver says they have their foot on the brake, they are just plain wrong. The human motor system is not perfect, and it doesn't always do what it is told.'"
Toyota is currently planning an event to challenge evidence presented by professor David W. Gilbert that called into question Toyota's electronic throttle system.
V&V (Score:1, Interesting)
From Wikipedia:
Verification and Validation (V&V) is the process of checking that a software system meets specifications and that it fulfils its intended purpose.
Since they already said the software is "rigorously tested" does this mean Toyota doesn't have specifications, or that their software doesn't fulfill its intended purpose?
Their software sounds like its written as a monolithic device driver (NVidia unified device model) comes to mind. Perhaps they should be looking for best practices in TDD, as well as dropping support for older models as time passes on.
tin.foil.hat (Score:3, Interesting)
come on, it's just a big conspiracy.
it's not like 100, 200, one thousand toyotas are
skidding of the highway and into a tree everyday.
there are like a handful of incidents.
-
naw, this is just a big PR campaign of american motor
industry to smear superior japanese tech.
the prius is like a 5 year old car model and in all this
time american "muscle" motor never came up with an answer.
-
big oil and big car a big happy american family.
-
the engine (sic) that drives the (u.s.) capitalistic machine needs
consumption and waste, not innovation and thriftiness.
here is the problem (Score:5, Interesting)
Less than 100 cars out of 8,000,000 have had this problem. That is a 0.001% failure rate.
Of those 0.001% of cars that had the problem, how many times did someone drive them before they failed?
I don't want to say this is user error, but I have seen some users do stupid stuff and not even know they did it.
Re:Why? (Score:2, Interesting)
Question: Why is there a congressional case about this?
Answer: The 911 call. Toyota not fixing the problem.
http://consumerist.com/2009/10/toyota-911-call-of-familys-fatal-lexus-crash-due-to-gas-pedal-stuck-on-floormats.html [consumerist.com]
Retort to conspiracy theory: This is a Toyota problem. They paid off the NHTSA people to get the scope of the investigation limited to accelerations of less than one second. This has nothing to do with GM, it has to do with Toyota fucking up and getting caught.These cases have been in the courts and Toyota keeps citing user error.
Re:Formal verification? (Score:3, Interesting)
Um ... did this guy ever heard of formal verification? Or is math proof not good enough for him?
How about this reformulation, then:
"It is well-known in our community that there is no scientific, firm way of actually completely verifying and validating a system that is Turing-complete."
And yes, there is a math proof for that. :)
Well, there is brute-force - just run the program start to finish for every possible combination of branch conditions. Just take 2 to the power of the number of if statements in the program and that's the number of tests you need to perform. Good luck doing that for anything more complicated than a thermostat, however...
Re:falsely blaming the user (Score:2, Interesting)
Re:Anyone else think it odd? (Score:4, Interesting)
btw - I hope your are right. I own a Prius, but not one with the problem, so I am unable to even try to help. If I did have one I would be disassembling the software system looking for potential overwrites of the variables that control the throttle calculation.
Re:falsely blaming the user (Score:3, Interesting)
I experienced a vehicle accelerating out of control in a late 90s Dodge Caravan. I had just gotten on to the highway and set the cruise control when the car started to accelerate. The floor mats were not on the pedal. Disengaging the cruise control had no effect. The car continued to accelerate.
I had to put both feet on the brake pedal and pull up on the steering wheel to slow down until I could get to an off ramp. I threw the car in neutral and turned the engine off. When I started it back up it was fine, and it never did it again, but I never used the cruise control in that vehicle again.
I don't think it was a mechanical linkage problem, as the vehicle was going at a steady speed when I engaged the cruise (I didn't engage it and then use it to accelerate). I think it was most likely the cruise control system, and to this day I'm hesitant to use one.
I think this type of thing probably happens more than we hear about, and it's not limited to any one manufacturer. As the guy who wrote the article said, cars are complex machines, with over 20,000 parts, and anticipating every possible failure is impossible.
But I also agree people are notoriously unreliable as witnesses, and agree a lot of incidents are more likely caused by the driver's own actions. I don't think that was the case with the incident I experienced, but being the only person there at the time, who's to say? I said earlier I didn't set my speed with the cruise control, but then I went through a few minutes of intense pressure as I tried to keep the vehicle under control until I could get it safely off the highway.
I'm sure there's a good chance I could get a detail like that wrong, which would greatly diminish the value of my anecdotal evidence.
Absolutely Impossible to Verify!!! (Score:3, Interesting)
Opinions on verifying code as a means to tell whether a Toyota will have 'sudden acceleration' above are UTTERLY, well, let us say, ill thought out in my opinion, in most cases. Code is only ONE part of an almost hopelessly complex system when ALL THE POSSIBLE VARIABLES are analyzed.
Failure analysis may start with code, but these systems then can encounter intermittent connections, power surges, static generated by multiple known and unknown items (including the rare intermittent connections), induced currents in parallel wires, temperature induced changes, faulty seals & water/condensation intrusion, etc. By the time an accident investigator looks at a vehicle that had a problem, the transients are long gone.
Intermittent Mechanical (& thus often electrical) changes & failures are an absolute bane of complex systems.
In my opinion, the only way you can find these rare transient problems is to find vehicles who have been reported to have these problems (& didn't crash) and then you load them up with data loggers and drive the hell out of them in all sorts of environments.
Personally, I really like a 1972 Blazer...with a manual transmission. Minimal plastic, no electronics beyond the turn signal module, fix it myself and I can start it with a bit of a downhill run. Yup, I drive my Highlander, but I'm thinking of putting a 72 Blazer back in as new shape.
Re:dismissing user reports? (Score:4, Interesting)
``Dismissing user reports is what got Toyota in trouble in the first place. Keep doing that. See how far it gets you.''
Right. Nobody I know about actually has a problem with there being a defect in the vehicles. The defect should not have been there and it's a great shame that it was, but everybody understands that it happens. If it happens too often, that gives you a poor reputation, but it doesn't happen to Toyota a lot so their reputation there is good.
Where Toyota went wrong is in how they handled the incident. What they should have done was err on the side of caution, notify people of a possible issue, and encourage them to be careful and report anything that might be related to Toyota to help them investigate the issue. Only after they would have done their best to confirm the issue could they have concluded that the issue does not actually seem to occur, and even in that case they should not have told people that there is no issue, especially not the people who report experiencing it.
What they did instead was deny that there was an issue before they had properly investigated it, and effectively called the reporters of the issue liars. Calling your customers liars is a very bad idea, and doing so with those who report a rarely occurring issue not only insults them, but also deprives you of an important source of information. It's probably the very worst thing they could have done.
Figuring out the parallel between this and full disclosure in computer security is left as an exercise to the reader.
Cars have brakes (Score:3, Interesting)
Car&Driver did some tests [motortrend.com] and found that even with the throttle wide open the brakes can still stop a car, even a 500hp muscle car. With a normal car the distance wasn't even significantly greater than with closed throttle.
Re:Yes, interesting. (Score:5, Interesting)
The gilbert problem is the reading from the toyota ECM when the two redundant APP (accln pedal position) signal circuits are shorted together (main and sub), From the toyota camry VSRM :
DESCRIPTION
This ETCS (Electronic Throttle Control System) does not use a throttle cable. The Accelerator Pedal Position (APP) sensor is mounted on the accelerator pedal bracket and has 2 sensor circuits: VPA (main) and VPA2 (sub). This sensor is a non-contact type, and uses Hall-effect elements, in order to yield accurate signals, even in extreme driving conditions, such as at high speeds as well as very low speeds. The voltage, which is applied to terminals VPA and VPA2 of the ECM, varies between 0 V and 5 V in proportion to the operating angle of the accelerator pedal (throttle valve). A signal from VPA indicates the actual accelerator pedal opening angle (throttle valve opening angle) and is used for engine control. A signal from VPA2 conveys the status of the VPA circuit and is used to check the APP sensor itself. The ECM monitors the actual accelerator pedal opening angle (throttle valve opening angle) through the signals from VPA and VPA2, and controls the throttle actuator according to these signals.
FAIL-SAFE
The accelerator pedal position sensor has two (main and sub) sensor circuits. If a malfunction occurs in either of the sensor circuits, the ECM detects the abnormal signal voltage difference between the two sensor circuits and switches to limp mode. In limp mode, the functioning circuit is used to calculate the accelerator pedal opening angle to allow the vehicle to continue driving. If both circuits malfunction, the ECM regards the opening angle of the accelerator pedal as being fully closed. In this case, the throttle valve remains closed as if the engine is idling.
If a pass condition is detected and then the ignition switch is turned off, the fail-safe operation stops and the system returns to a normal condition.
VPA and VPA2 are coming from the PCM with .5-1.1v at one of the sensors and 1.2-2.0v at the other when the pedal is at its relaxed position. When there's force at the pedal, one sensor will operate between 2.6-4.5v and the other at 3.4-5.0v.
Toyota specs normal voltage for both the VPA sensors between between .4-4.8v for VPA, and .5-4.8v for VPA2 with a .2v deviation between the 2 sensors. Anything out of those ranges will trigger a DTC
An internal short could occur within one or more of the paths from the circuits leading to the ecm. That could lead to a situation where the computer cannot detect its own failure.Therefore, when the system gets conflicting information, it arbitrarily ignores half the conflicting information. It does not know which of the circuits are lying or if they both are lying and shorted together. different resistance values will lead to arbitrary acceleration. Having the brake override it is a stopgap, but fixing the real problem (perhaps with a third circuit in voting mode which will require replacing the entire circuit path) or reversed sensors or log and opposing log sensors.
There might also be emi problems with induced magnetic fields in the CTS pedal assembly which detects induced emf as acceleration since it relies on induced emf to operate in the first place and is made of plastic. replacing with conventional denso rather than cts will also help.
Compare withmachi machine tools (Score:3, Interesting)
One of the design "features" of the Toyota product involved in the 2009 fatal accident in San Diego was that the driver needed to press the engine start button for three seconds to kill the engine. Can you imagine any machine tool company making a product that required the emergency stop switch to be depressed for three seconds to turn off the machine?
Another issue with that car was that getting the tranny into neutral was not trivial (sport shifting option).
Toyota screwed up big-time here.
Re:Little attention was given. Read Consumer Repor (Score:1, Interesting)
Does Not Safe at Any Speed ring a bell?
Mechanical linkages != automatically safer (Score:3, Interesting)
He wasn't discussing cars as a whole, just the aspects relevant to the Toyota fiasco[1].
No he wasn't. He said "The real problem is people who think that not having any sort of actual linkage is a good idea." That has nothing whatsoever to do with Toyota specifically.
On old cars there's nothing second guessing you.
That doesn't automatically translate to better or safer. It's simpler but that is all you can say for certain unless you want to compare specific cases. Just as newer is not always better, older is not always safer.
Yes, obviously some things are better on modern cars, but that's not the point here
No that's exactly the point. The grandparent post was implying that a mechanical linkage is intrinsically safer while providing no evidence to back up that assertion. If you are going to declare drive-by-wire to be more dangerous than the alternatives, you had better back up that declaration with data.
I've seen this "mechanical linkages are safer" argument before and I've never seen anyone making it actually back it up with facts. They just pre-suppose that the simpler, older technology is safer. It may be or it may not be but I've yet to see anyone prove it.