Follow Slashdot blog updates by subscribing to our blog RSS feed

 



Forgot your password?
typodupeerror
×
Bug Transportation Technology

Do Car Safety Problems Come From Outer Space? 437

Hugh Pickens writes "As electronic devices are made to perform more and more functions on smaller circuit chips, the systems become more sensitive and vulnerable to corruption from single event upsets. This is especially true of Toyota, which has led the auto industry in its widespread inclusion of electronic controls in the manufacture of their various car models. 'These circuit families store not just data, but their basic function electrically,' says Lloyd W. Massengill, director of engineering at the Vanderbilt Institute for Space and Defense Electronics at Vanderbilt University. 'In the unfortunate event of a particle flipping just the right bit, a circuit configured to carry out a benign action may be reprogrammed to carry out some unintended action.' Denise Chow writes in Live Science that some scientists are pointing to cosmic ray radiation as a plausible mechanism behind the sudden, unexplained acceleration reported to have occurred with the late model Toyotas."
"As the design of automobile systems continues to evolve from mechanical to electronic controls, relying more and more on various circuitry and chips, these electronic components may be vulnerable to being confounded by high-energy radiation writes Chow. Federal regulators were prompted to look into the possible role that cosmic rays played in Toyota's product recall fiasco after an anonymous tipster suggested the design of Toyota's microprocessors, software and memory chips could make them more vulnerable (PDF) to interference from radiation compared with other automakers. 'What's not known is what direction Toyota and other automakers are taking in terms of finding and correcting these issues,' says senior researcher Ewart Blackmore."
This discussion has been archived. No new comments can be posted.

Do Car Safety Problems Come From Outer Space?

Comments Filter:
  • by LostCluster ( 625375 ) * on Sunday March 28, 2010 @04:58PM (#31650752)

    Interference from radiation doesn't just come from outer space, it comes from cell phones, TV/radio stations, microwaves.... you see where this is going. I once worked in an office where there was a cell phone relay antenna too close to a PC, and we were constantly reinstalling the OS until I told them to move things around in the area.

    Thing is, when Windows gets a corrupted OS... it BSODs and we move on. Single-bit errors shouldn't send the car out of control... there should be some checksum that shouldn't add up. When a fault is detected, it should go to a backup program about safely shutting down the car.

    • by JoshuaZ ( 1134087 ) on Sunday March 28, 2010 @05:02PM (#31650788) Homepage
      That's almost exactly what I was going to say. You've managed to make an accurate first post that actually includes a suggestion for dealing with the problems in question. Are you sure you meant to post this comment on Slashdot?
      • Re: (Score:3, Insightful)

        by Cryacin ( 657549 )
        I think it's just trying to blame the little green men on a problem that has more terrestial origins.
        • by WrongSizeGlass ( 838941 ) on Sunday March 28, 2010 @05:26PM (#31651046)
          Tonight on CBS, a very special episode of Everyone Loves Space Ray:

          Space Ray: Hey, Deborah, did you hear what happened to my car?
          Deborah: Don't worry about it, Space Ray, you didn't cause it this time (simulated audience laughter)

          With a special guest appearance by Ace Frehley as "Just Another Confused Alien". Coming up right after "The Ghosts of Gilligan's Island"
      • by WaywardGeek ( 1480513 ) on Sunday March 28, 2010 @08:29PM (#31652254) Journal

        Radiation that can upset bits in an electronic circuit don't come from your cell phone, TV/radio stations or microwave oven. You may get enough EMI to interfere with your radio, but flipping individual bits in a chip pretty much requires an ion - basically a nucleus or neutron stripped of it's electrons flying through your chip. These come from two main sources. First, there's the Sun. Even with the magnetic shielding of the Earth, many fly through us all the time. Most common are single protons, but we occasionally are struck with gold nuclei, or even heavier. Older larger geometry chips were immune to single-event-upsets (SEUs) due to protons, but heavier elements could cause trouble. Newer, more advanced electronics are even sensitive to individual protons and neutrons. The other common source for radiation is neutrons from decays in lead used in electronic packaging. Ever hear of RohS compliance? Basically, a bunch of electronics companies around the world suddenly decided to "go green" and save us from lead poisoning by removing lead from their packaging. Ever wonder why? Do you really think they suddenly cared if they were killing our babies with lead poisoning? Uh... I'm afraid not. They removed the lead because of neutron radiation from lead decay.

        I'm guessing that studying radiation effects isn't very popular in Japan, possibly because we nuked them twice. However, they should get a clue and start learning about how to deal with rogue ions and neutrons.

        • by rickb928 ( 945187 ) on Sunday March 28, 2010 @08:38PM (#31652320) Homepage Journal

          I don't hear much about comsumer electronics being fritzed by cosmic rays, or microwave ovens, etc, though I suppose this might explain the random failurs. But comsmic radiation? That's a new one.

          But RHoS being forced by lead decay? I dunno, but tin whiskers is negating any advantage that offers.

          Give me good old eutectic 63/37 any day. It just works. Not a lot of kids usae circuit boards as pacifiers, ya know?

          • Re: (Score:3, Informative)

            by GooberToo ( 74388 )

            I don't hear much about comsumer electronics being fritzed by cosmic rays,

            Chances are you'll be hearing about this more and more over the next several decades or so. Scientists have discovered a large spot over the Atlantic (IIRC) where high levels of cosmic radiation are actually making it to the ocean's surface. Further investigation indicates this is because their Earth's magnetosphere is beginning to significantly weaken. Furthermore, its expected that not only will the the level of radiation exposure continue to drastically rise at this particular location, but that radiatio

          • Re: (Score:3, Interesting)

            by tlhIngan ( 30335 )

            I don't hear much about comsumer electronics being fritzed by cosmic rays, or microwave ovens, etc, though I suppose this might explain the random failurs. But comsmic radiation? That's a new one.

            It's quite common actually, and many documented studies have proven it does occur. You don't hear much because well, the effects are minimal in most cases. A flipped bit in RAM does nothing if it's just unused memory, for example. Or maybe it flips the bit in an unused register (that's getting reloaded with new dat

        • Re: (Score:3, Interesting)

          by TheLink ( 130905 )
          > Radiation that can upset bits in an electronic circuit don't come from your cell phone, TV/radio stations or microwave oven
          > You may get enough EMI to interfere with your radio, but flipping individual bits in a chip pretty much requires an ion

          You don't need to flip individual bits in a chip to cause problems with car electronics. I suspect if something flipped dozens or thousands it would still cause problems. So you shouldn't get so fixated on individual bit flips.

          From the perspective of car safet
        • by Kral_Blbec ( 1201285 ) on Sunday March 28, 2010 @10:30PM (#31652900)
          I'm a bit skeptical of your claims about lead decay in electronics. While some isotopes of lead are radioactive, those are products of uranium decay, which as any good geek knows, goes through alpha and beta decay until it ends as a stable particle of lead-206. In that pathway there is lead-214 and lead-210 that have half-lives of half an hour and 22 years respectively. However, unless they are putting uranium in your electronics, the only lead present is going to be from mined ores that have had plenty of time to decompose into a stable form.

          The best chart of lead isotopes I found is here http://education.jlab.org/itselemental/iso082.html [jlab.org]. I'm not sure why, but it lists a half life for lead-204 even though I thought it was supposed to be stable. Most half lives are a few minutes or hours.
    • by pushing-robot ( 1037830 ) on Sunday March 28, 2010 @05:05PM (#31650818)

      http://en.wikipedia.org/wiki/Non-ionizing_radiation [wikipedia.org]

      Granted, an unshielded circuit can be vulnerable to any EM field, but gamma rays affect electronics in a completely different way than microwaves do.

      • I was under the impression that gamma rays were much (orders of magnitude) less likely to have an effect on electronics as their wavelength was so much higher, but if they did, it would be (basically) a more drastic impact, because of the higher energy.
        • Re: (Score:3, Informative)

          by Anonymous Coward

          Nope, the exact opposite. Gamma rays [wikipedia.org] are short wavelength and high energy.

        • Gamma rays have a higher wavelength, which makes them less likely to interact, but a correspondingly high energy which makes the possible ionizing effect greater if they do interact.

          • Re: (Score:3, Informative)

            by hipp5 ( 1635263 )

            Gamma rays have a higher frequency,

            Corrected. And thus they have a shorter wavelength.

    • by pitchpipe ( 708843 ) on Sunday March 28, 2010 @05:14PM (#31650922)

      there should be some checksum that shouldn't add up. When a fault is detected, it should go to a backup program about safely shutting down the car.

      Or how about a computer redundancy system where a group of computers that are all capable of controlling the car watch the behavior of the computer that is actually controlling the car. Through a voting system they could decide to hand the control of the car over to a another computer in the event that the controlling computer doesn't act in a way that was deemed safe. This way the car could continue to operate normally while signaling that there is a problem that needs to be addressed.

      • by SeekerDarksteel ( 896422 ) on Sunday March 28, 2010 @06:13PM (#31651376)
        This is one of the most common methods of error tolerance, actually, N-modular redundancy [wikipedia.org] (typically either dual-modular or triple-modular). It's used in airliners and space shuttles, as well as a number of other critical applications. IBM actually sells servers (the system z series) which automatically runs two copies of everything and compares instruction results, so that failing processors can be detected and avoided.

        The proposal by the GP poster is actually much more difficult that it would seem at first glance. About the only place "checksum" style error detection is used is in memories/registers. The reason is that if I do a floating point addition, for example, the only way I know whether the addition gave me the right answer is to do the addition again and check.
        • by evanbd ( 210358 ) on Sunday March 28, 2010 @07:17PM (#31651836)

          You can build circuits that detect faults while operating. They're more complex than their normal counterparts, but the transistor count is less than 2x. On-line error detection [google.com] is a common name.

          Of course, such circuits get really expensive if you don't have a large market for them. But cars represent a fairly large market, so if it was the best approach they could probably use them. Of course, that assumes there's any market or regulatory pressure to use any sort of error detection at all.

    • by neiras ( 723124 ) on Sunday March 28, 2010 @05:40PM (#31651144)

      Single-bit errors shouldn't send the car out of control... there should be some checksum that shouldn't add up.

      What if the cosmic rays corrupted the checksum routine?

      The mind boggles!

      • There's no way for a SINGLE bit error to hit both the main routine and the checksum routine. Cosmic rays or other EMF based changes are rare events, so the mind boggles on the chance both can go wrong in the same instance.

    • Single-bit errors shouldn't send the car out of control... there should be some checksum that shouldn't add up. When a fault is detected, it should go to a backup program about safely shutting down the car.

      See, here's the problem with random errors that happen in the hardware from an outside source; It might happen after you did you sanity check...
      • A proper fault-tolerant design (which cannot be done entirely in software) would always fail safe on any single bit error.

    • by Jane Q. Public ( 1010737 ) on Sunday March 28, 2010 @05:52PM (#31651232)
      In order for it to interfere with a digital circuit, it first has to be radiation of the "ionizing" category, and then it has to get through whatever shielding the electronics are in. (I presume they are in some kind of can; no shielding at all would be plain stupid.)

      Cell phone radiation hardly qualifies. Nor, for that matter, do most terrestrial sources of radiation.

      "Cosmic rays", unlike most terrestrial-source radiation, are capable of penetrating shielding and disrupting electronics.

      However... striking just the right bit(s) to cause acceleration, in a large collection of cars, is so incredibly unlikely as to be in the "I don't f*ing think so" category.
      • Re: (Score:3, Interesting)

        More to the point they generate secondary showers of ionizing radiation when they transverse metallic shields so we should be careful not to make the problem worse by creating showers of particles with a greater cross section.

      • Re: (Score:3, Interesting)

        by mc6809e ( 214243 )

        In order for it to interfere with a digital circuit, it first has to be radiation of the "ionizing" category

        Neutron radiation isn't considered ionizing, yet interactions between the neutrons and the silicon in a typical chip will create charged particles that cause current surges. These current surges can interfere with the correct operation of a circuit and that includes individual transistors, not just bits in memory.

    • However, RF interference is well known and understood, and easy to protect against.

      Cosmic radiation is relatively new in regards to how well we understand the substantial impact it may actually have on modern technology. There are also fluctuations over time in the earth's magnetic field and how well it protects us from solar and cosmic radiation. With these two factors combined, we are seeing more and more warnings from scientists that solar and cosmic radiation have the potential to do massive damage to

      • No matter what the cause, I think this is a good indication that we need a real, physical kill switch that will absolutely halt the system if things go awry in these drive-by-wire systems. No software to depend on, because you're breaking a physical connection to do it. It should be easy and noticeable, but not something you're likely to grab by accident.

        Yeah I've said it before. My dad built a kill switch into his boat after he got knocked out of it by a big wave. He used a reed switch, a magnet and a short length of rope with a loop to go around the wrist. If thats too hard then a big red switch marked "EMG STOP" should not be.

    • Thing is, when Windows gets a corrupted OS... it BSODs and we move on.

      How do you move on from a BSOD in your car?? No, you won’t be dragged away in a bag. You will be dragged away in several bags!

      There is only one way to make bit-flips completely go away:
      Design every processing component with triple simultaneous execution, so a bit-flip can be detected properly. Also do mirroring on all data storages, and use checksums on them and on all data streams. Then do constant scrubbing (like in ZFS) on all storage systems.

      If you leave out even one of those things, the whole eff

    • by dwreid ( 966865 ) on Sunday March 28, 2010 @07:31PM (#31651898)
      At the risk of sounding like a geezer, I remember back in the late 70's when this was a problem in early designs of mini-computers. Then we used to see single bits get flipped and crash computers from a variety of sources including cosmic radiation and alpha particles that came from the spontaneous decay of elements in the ceramic chip housings. More recently, when I purchased my 2005 Cadillac CTS it experienced a variety of problems similar to this when I would drive through a toll station that was equipped with RFID ID systems. Behaviours including sudden acceleration, engine stalling, indicator lights on the instrument panel going "crazy", On-Star calling for help when nothing was wrong, causing the driver's seat to suddenly drive forward to the steering wheel (making it really hard to steer), etc. At the time the only solution was to pull over, shut off the car, remove the key, open the door, wait for everything to shut down and then restart. After many frustrating weeks of "we can't duplicate the problem" it was discovered that the car had faulty shielding on one of the cables that makes up the in-car network. Once fixed the "gremlins" went away. The real crime here is that, because the problem can't be replicated on demand, Toyota is blaming the behaviour on attention seeking owners. This bizare response was recently repeated on the floor of Congress by one of Toyota's congressional tools. (I mean duly elected government representative.)
      • by Anonymous Coward on Monday March 29, 2010 @03:45AM (#31654490)
        My dad was an IBM CE (Customer Engineer) specialist on one of the models in the IBM System/360 mainframe range. He used to like telling the story about how he and another engineer were out on a customer's site trying to determine an intermittent fault. They would bring the machine up and sure enough there would be this glitch at precise intervals. They just couldn't figure out what was causing it. That was, until the other CE took a look out the window.
        After a bit he said 'Tell me when it happens'. OK... '...now' my dad said. Then he said 'I'll tell you when the next one happens' and a few seconds later said '...now'. Which is exactly when it did glitch.
        It turned out that the customer's DP center was situated close to an airport. The CE could see the radar dish revolve at the end of the runway. When it pointed straight at him was when the glitch occurred. Needless to say the computer room received some RF shielding.
    • by rcamans ( 252182 ) on Sunday March 28, 2010 @07:52PM (#31652056)

      I worked on ECMs at GM (Delco Electronics) for 10 years at the start of their use (1980 to 1990). So if a cosmic ray came along and flipped a bit, it would have to be a specific bit. If it was a msb type bit in the accelerator position, then yes, acceleration. except that the bit would unflip right away because of pedal position update. Or if it was some engine feedback msb, again, yes, temporary acceleration, but again, only for a short time. Updates happen constantly.
      About EMI/EMC/RFI - the modules have been shielded and protected since day one against that. The engine is a very high disturbance environment in may ways. Sparks, for instance. The ECMs have been in almost all American cars since before 1980, because of the 1975 car air pollution reduction act Congress passed. The only way cars could meet the pollution restrictions was through ECMs. So If we have ECMs since nearly forever, and only just now one manufacturer has a bit flip problem? I don't think so. And these modules do not use the latest super-small feature processor technology. They use older temperature-resistant tech, Much larger features, far more radiation-resistant.
      No, the most likely problem is either a software routine with a bug, no error handler, or similar issue, or a mechanical,problem (less likely).

      • Re: (Score:3, Interesting)

        by putaro ( 235078 )

        The effect of random bit flips on software is going to be hard to define. Modern hardware probably has all of the code running in RAM, not ROM as it would have been back in the 80's. A bit flip in a register could cause very odd things to happen. Perhaps someone coded a loop like:

        for (i=0; i!=10; i++)
        do_something();

        Flip a bit in the register and that loop will not terminate until the register overflows.

        I don't think you can code so that random bit flips will not be a problem. The hardware

  • by LostCluster ( 625375 ) * on Sunday March 28, 2010 @05:02PM (#31650786)

    Since the biggest Toyota runaway story has turned out to be a problem exists between seat and pedals [aol.com] situation... is this all hype with no science behind it?

    • Re: (Score:2, Troll)

      by forkazoo ( 138186 )

      Since the biggest Toyota runaway story has turned out to be a problem exists between seat and pedals situation... is this all hype with no science behind it?

      Yeah, pretty much. Besides, error correcting systems are relatively well-uderstood technology. ECC hasn't been the best available option for RAM for ages, and even the imperfect gains of ECC will work around occasional single-bit corruptions in memory. Flash can be used with extensive checksums. Executables can have hashes like MD5 and SHA checked b

      • Re: (Score:2, Insightful)

        by Anonymous Coward

        >Executables can have hashes like MD5 and SHA checked before being allowed to execute, etc.

        That's a ONE TIME check when you load the program. Sure it can check if the data in the FLASH has start to corrupt or someone has tempered the firmware. However, It doesn't check the memory once the coding is running which is 99+% of the time the code is doing. Cosmic ray can be hitting your car ANYTIME and not just when it is parked.

        ECC checks the memory bits during access and you can have periodic scrubbing to

      • Re: (Score:2, Interesting)

        by MadShark ( 50912 )

        The problem is that many microcontrollers used in automotive systems don't have support for ECC or any other hardware based error checking mechanism. A lot of these systems only use the memory on the microcontroller chip. If there is external RAM on the unit, ECC memory isn't always used since it is more expensive. Flash is typically checksumed/CRCed/MD5 checked, but you don't typically see flash cells get flipped in the field. I've seen one unit get flash corrupted(out of many millions of possible units

    • Since the biggest Toyota runaway story has turned out to be a problem exists between seat and pedals situation..

      Ignorant alien between seat and pedals. Toyotas were designed for humans to drive. 'nuff said.

    • Re: (Score:3, Informative)

      Since the biggest Toyota runaway story has turned out to be a problem exists between seat and pedals situation...

      The article you linked to does not even begin to support that conclusion. Basically its a bunch of innuendo, like he [i]might[/i] have been late on payments on the car (since proven false) or that he should have shifted it to neutral (not an intuitive action for someone who has never driven a manual transmission - and certainly a last resort that does not negate the existence of a problem to begin with). Even information released after that article was published has been far from damning - basically toyot

    • by Venik ( 915777 )
      Some science is definitely behind this. The question is: how far behind? Physicists discussing software problems are not nearly as hopeless as programmers discussing physics. This is exactly how one gets gamma radiation from outer space appear in the same sentence with cell phones and microwave ovens as a possible cause for malfunctioning electronic circuits and sloppy coding.
  • by Anonymous Coward on Sunday March 28, 2010 @05:02PM (#31650790)

    I bet they still use C for these kinds of things, how about something safer, such as Eiffel?

    • Re: (Score:3, Insightful)

      by istartedi ( 132515 )

      If a cosmic ray flips a bit in the (insert safe language here) array boundary checker, then what?

  • No. (Score:4, Insightful)

    by stonecypher ( 118140 ) <`moc.liamg' `ta' `rehpycenots'> on Sunday March 28, 2010 @05:03PM (#31650804) Homepage Journal

    There's a reason that our entire modern world doesn't come crashing to a halt around us every 30 seconds. If every CPU was vulnerable to bit flips from random radiation, every part of your house would be on fire and arcing electricity. Times Square would look like the bridge of the 60s enterprise under attack.

    This is just some douchebag professor trying to ride the tragedies to fame. There's a reason it's always hitting the same system in the car. It's because the system is defective. There's a reason the professor has nothing but speculation to back himself up.

    This is the worst kind of charlatanry from someone who should know better. I hope his hosting school takes this very, very seriously.

    • Re:No. (Score:5, Insightful)

      by TheGeniusIsOut ( 1282110 ) on Sunday March 28, 2010 @05:12PM (#31650900)
      I can't even begin to calculate the probability of a single bit flip due to impact from a cosmic ray causing unintended acceleration in multiple vehicles. Possible? Certainly, nearly anything is. Plausible? Maybe in a very broad sense of the world. Likely? Not very.
      • by DingerX ( 847589 )
        It doesn't have to be likely. It just has to be a probable at approximately the same level as the incidence rate, and more probable than any competing explanation.

        Of course, more probable than a bit flip due to cosmic rays is a bit flip due to marginally bad RAM.

        I would think that Toyota's design process includes some sort of Byzantine fault tolerance. And I would think the automobile industry would have regulation regarding how safety-critical firmware is written. But then I think how the pressure from m
        • by adolf ( 21054 )

          You mean, the same marginally-bad RAM which seems to remember that the car is on and running? The same RAM that keeps the engine running properly? The same RAM allows the computer to throw an SES light if it detects that the engine is not running properly? The RAM that keeps track of the odometer, and controls the speedometer?

          That RAM? The one responsible for all these other problems that might be caused by by bad RAM, but which aren't happening?

          Hmm.

          Naah, don't think so.

    • Parent is not Flamebait. Disgusted? You bet. Angry that this type of crazy has made its way to the pages of /.? Indeed.

      I'm standing in line with SC on this one. This story needs to be tagged "unicorns, ponies and space rays".
    • Re:No. (Score:5, Informative)

      by SeekerDarksteel ( 896422 ) on Sunday March 28, 2010 @06:30PM (#31651530)
      There's a reason that our entire modern world doesn't come crashing to a halt around us every 30 seconds. If every CPU was vulnerable to bit flips from random radiation, every part of your house would be on fire and arcing electricity. Times Square would look like the bridge of the 60s enterprise under attack.

      Actually, every CPU _IS_ vulnerable to bit-flips from radiation. That part of it is not speculation. It does occur in commodity processors, and with probabilities large enough that we have ECC ram, and ECC and/or parity in caches. Some servers actually come with built in hardware fault tolerance methods, because when you run hundreds of servers non-stop for years, the probability that a particle strike screws up a register on chip is non-negligible. Now, still, the probability isn't _huge_. Definitely not high enough to be causing these specific problems, especially when the failure is always in the same manner. _That_ part of it is pretty much bullshit.
  • by nbvb ( 32836 ) on Sunday March 28, 2010 @05:05PM (#31650822) Journal

    Sounds a whole lot like the e-cache parity errors in the Sun UltraSPARC-II processors.

    If you were never affected by that, consider yourself a lucky person.

    particle-caused bitflips are very much real.

    • Re: (Score:2, Informative)

      by Anonymous Coward

      I work with someone who used to do tech support for Sun - those flips were due to a manufacturing error - tech support were just told to tells customers it was due to 'Sun Spots'.....

      • Re: (Score:2, Interesting)

        by Anonymous Coward

        Actually, it was due to a design error, as the cache wasn't ECC protected and occasional bit-flips weren't detected.
        http://www.sparcproductdirectory.com/artic-2001-dec-1.html

        • by asaul ( 98023 ) on Monday March 29, 2010 @05:31AM (#31654932)

          I wouldn't say error, it was designed with parity protection only, so was incapable of correcting single bit errors, only detecting them. Hence, the reason for the crashes (i.e it detected a bit flip). If two bits were flipped you would never know.

          I worked in the Sun front line call support during this time, and explaining this over and over to customers was somewhat painful. Never mind the years of mocking that still come from telling customers "it was a cosmic ray". Sun put massive effort into tracking, diagnosing and fixing this issue though. Some customers got versions of CPUs with "mirrored" SRAMs. Sad to say, I remember customers still getting errors with those.....

          The US-III chips came out with end to end ECC protection, but the problems remained. In the end it turned out to be a host of socket mounting, pin contact and design specification issues that caused the errors, mostly solved by the time the 1200MHz CPUs were out. I wouldn't be surprised if it was something similar with the US-II.

          As for Toyota, if they dont have end to end ECC they only have themselves to blame.

    • Re: (Score:3, Insightful)

      by dr2chase ( 653338 )
      Right, but then more of them would appear at higher altitudes.
      • Not necessarily, clouds absorb cosmic radiation - or more accurately water vapor absorbs cosmic radiation and forms clouds, so anywhere with a lot of cloud cover is going to have a lot of cosmic-ray cover too. Higher altitudes generally occur in hilly or mountainous regions (duh, that's what makes them high), and they also tend to have a lot more cloud cover because wind and moisture get blocked by the mountains.

        You'd probably be most likely to see lots of cosmic rays in dry, flat areas that usually have l

      • They do. This is a known phenomenon which has been measured. But the difference between, say, Denver and NYC isn't substantial enough that you would notice a difference with your personal electronics.
  • If this is true, recreate the phenomenon in a lab. Test your hypothesis by exposing the circuitry in question to similar radiation in a lab. While you can't test thousands of sets of circuitry, being able to recreate it by increasing the amount of radiation and testing or automating the testing and dosage cycle and letting it run until the malfunction is noted or another failure occurs.

    It's not out of the question, IBM noted in the 90s [scientificamerican.com]:

    Extensive background radiation studies by IBM in the 1990s suggest that computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month. If so, a superstorm, with its unprecedented radiation fluxes, could cause widespread computer failures.

    You have to fix this though. As a large manufacturer you have to accept this risk just like your competitors do. Airlines accept this risk and triple check their data because people's lives are at risk. As a car manufacturer, you are in the exact same position.

    I hope the fix they already rolled out as a recall includes triple checking data or -- if the article is correct -- we won't see a drop in these horrible accidents. I hope for drivers and public safety that it does. It's led to death and possibly wrongful incarceration [go.com]. Restitution is in order. Take testing motor vehicles seriously.

    • The UNKNOWN software/electronic fault theory has fallen over at the "prove it" stage, the cars that suffred sudden acceleration have been examined and the electronics found to be working. OTOH Toyota has recalled 3.8 million cars [google.com.au] to replace the floor pan so that the KNOWN problem of floor mats intefering with the pedal can be fixed.

      Finally, a wayward floor mat doesn't make a good news story unless you're writing it up for the Darwin awards.
    • Working in the space industry, we perform routinely those kind of integrated circuit tests with heavy ions (i.e, cosmic rays species). At sea level, you're more concerned with atmospheric neutrons coming from the decay of cosmic rays in the upper atmosphere, though.
      So, the bottom line is that :
      - the test facilities (heavy ion and neutron sources) to perform those tests are available
      - the single even effect theory and event rate predictions methods are well known (even if they are not perfect)
      Which means
  • by WrongSizeGlass ( 838941 ) on Sunday March 28, 2010 @05:08PM (#31650852)
    Whether you subscribe to Occam's razor, or just plain old common sense, rays from outer space are not Toyota's problem (though they may be the author's problem).

    This type of thing is just plain bat shit crazy. There is a problem somewhere in Toyota's system somewhere. Either a software bug or bad chips or something real and tangible ... but rays from outer space? Please.

    If someone here on /. had posted that in the last Toyota story they would have gotten a +5 Funny.
    • If there is a hard to define race condition locking up systems on the cars due to a software bug, it may be triggered by a bit getting flipped that is assumed to be an impossible event, this could be caused by a hardware glitch, a voltage spike, a cosmic ray strike or any combination.

      • > If there is a hard to define race condition locking up systems on the cars
        > due to a software bug, it may be triggered by a bit getting flipped that is
        > assumed to be an impossible event...

        That assumption is a design error.

  • 'These circuit families store not just data, but their basic function electrically,' says Lloyd W. Massengill, director of engineering at the Vanderbilt Institute for Space and Defense Electronics at Vanderbilt University. 'In the unfortunate event of a particle flipping just the right bit, a circuit configured to carry out a benign action may be reprogrammed to carry out some unintended action.'

    Shouldn't there then be a well-insulated ROM copy in the car that can replace corrupt values with reasonable def

    • "Check Chips at Mechanic" light that, well, tells the driver to send the car with its chips to the mechanic?

      I think there is a "Check Chips at Mechanic" light ... but it's only activated when the car is racing forward uncontrollably. Hey, who knows, maybe the car is just trying to get to a mechanic on its own? It's as likely as this "rays from outer space" theory.

    • Insulating the ROM would be much more expensive than just adding error correcting codes or having multiple copies of the ROM and comparing the contents periodically. The problem is no matter what you do, it's going to add cost and complexity, so unless you can show that single event upsets are indeed causing a problem there's no reason to prevent them.
  • This would be a shame. It is very well known that the size of the chips influences their susceptibility to charged particles. I am sure the people estimating the reliability have numbers about that. And there is no reason to use hi density electronics for this purpose, besides saving 10cents.

  • This sounds more like a cop-out for Toyota's design practices than anything. If it's not reliable enough for the road, then don't sell it! (safety laws and all).

    What's so wrong with simple and effective that good design philosophy gets thrown out in favor of industry buzzwords?

    • by Onyma ( 1018104 )
      I personally don't believe the engine control system in the Prius failed any safety tests that would have deemed it unsafe to sell when it was certified. I do think that the rising rate of cases (even after you factor out the money-grabbing scammers) signifies an age degradation issue of some component in the system. This is not an uncommon happening in engineering as it is truly impossible to perfectly rapid-age parts during testing the same way they will in real world scenarios.
  • Comment removed based on user account deletion
  • Oh, right. Hoods and bonnets. They already have those.

    They should start making them out of lead, maybe?

  • We had a system quit working that had not been modified in years. Upon investigation the problem was found in a Perl script. The date on file was years in the past. The error was due to a change to a single character and the character was changed by one bit. Someone suggested that this was caused by an "Oh-My-God" [fourmilab.ch] particle interaction - who knows?
       

  • If this were true then more electronics would go haywire at higher altitudes. They do not. I used to live in Leadville, CO and our computers (and cars) worked just fine. In fact, I'd say that a car receives more radiation from the trace amounts of Uranium in the asphalt than from the cosmos.

    As long as I can remember people have been blaming cosmic rays for all sorts of unexplained problems. It's just a convenient scapegoat for shoddy workmanship because few people understand comic rays or even what radiatio

    • Yes. Yes they do receive more radiation at higher altitudes. This is a known, measurable effect. That being said, the difference between sea level and ~5000 feet is not substantial enough that you would notice with personal electronics.
  • by jim_k_3038 ( 751126 ) on Sunday March 28, 2010 @07:32PM (#31651908)
    While working for Motorola, I worked on electronic throttle control (ETC). We spent a ton of time working to make the system "fail safe". I think we all had in the back of our minds that it was only a mater of time before we would have to testify as to our engineering decisions.

    My little part of ETC involved adding a sub processor which watch-dogged the main micro. The little micro asked a series of questions of the main micro. Both processors would need to agree on all the inputs and output of the system. The little micro would also ask question regarding real time OS (RTOS) of the main micro. The main micro would need to have tasks executing in the right order to satisfy the small micro. Lastly, the small micro would ask the main micro to perform math operations to verify accuracy. Oh, and the main micro was continuously checksumming it's memory too.

    Both micros had a direct hardware disable path to the H-bridge which was delivering power to the throttle plate. The throttle plate was spring loaded, so, with power cut, the throttle plate would snap to an idle position.

    Next came the electro / magnetic compatibility testing (EMC). We spent months inside huge chambers testing both radiation and susceptibility. One of the tests for susceptibility involved using a zap gun to spark a 20kV spark on each pin of our ECU. Not satisfied with that, our customer opened one of our modules and used a sparking spark plug to slowly zap our board to failure. Bottom line, that throttle plate better never stick one way, or the other.

    In the end, it always amazed me that the whole thing would work at all. Seemed to me that the system was always seconds away from going into some kind of fail safe mode.

    No, a stray bit flip is not going to facilitate a run away car. Least not on my system!
  • McMurdo (Score:5, Interesting)

    by Unxmaal ( 231 ) on Sunday March 28, 2010 @08:24PM (#31652220) Homepage

    When I was working for NASA, on the NISN network, we'd get these weird router crashes for the old Cisco router located at (or very near) the South Pole in Antarctica. It was always a memory problem, and I'd always have to call someone to get them to powercycle the router. It irritated me to keep bothering those guys, so I opened a case with Cisco TAC.

    The TAC guy sent a terse response, saying that particular crash was a "transient memory error" due to "alpha radiation or sun spots." That really pissed me off -- Cisco TAC just gave me a standard BOFH response! I escalated, and swung the NASA club around some, and finally got a senior engineer on the phone. "You said this router's at the South Pole, right? So that means it's at very high altitude, with very little ozone shielding, right?" "Umm, yeah." "Well there you go. There's a lot more radiation at that altitude than at sea level. Our stuff's only rated for sea level. See if they can .. I dunno, put a lead blanket over it or something."

    I relayed the info to my contact at McMurdo, and he laughed and said he'd figure something out.

    On a hunch, I checked the other two "high-altitude" routers we had, and sure enough, they both had a statistically higher failure rate for "transient memory errors".

    • Re: (Score:3, Insightful)

      by Shimbo ( 100005 )

      "You said this router's at the South Pole, right? So that means it's at very high altitude, with very little ozone shielding, right?" "Umm, yeah." "Well there you go. There's a lot more radiation at that altitude than at sea level.

      His explanation sounds a bit off; a few molecules of ozone may be good for stopping UV but I doubt it makes a lot of difference to cosmic rays.

      Just being at the South Pole is a much greater risk factor than mere altitude though, because it's where the magnetosphere funnels all the crap.

  • Weird (Score:4, Interesting)

    by AmonTheMetalhead ( 1277044 ) on Monday March 29, 2010 @04:33AM (#31654692)
    Having heard all these stories really makes me wonder, i live in Belgium where cars with manual gear boxes are the common norm, and i've had my car accelerate like nuts once (pedal got stuck because of the floormat) i shifted to neutral, turned of the engine & used my momentum to get to the side of the road where i could dislodge the mat.

    Are manual gearboxes that rare in the States?

Pause for storage relocation.

Working...