Google Wants To Test Driverless Cars In a Simulation 173
An anonymous reader writes Google has been testing its autonomous vehicles on U.S. roads for a while now. In fact, they're required to, by law. "California's regulations stipulate autonomous vehicles must be tested under "controlled conditions" that mimic real-world driving as closely as possible. Usually, that has meant a private test track or temporarily closed public road." It's easy enough to test a few prototypes, but whenever autonomous cars start being produced by manufacturers, it'll become a lot more complicated. Now, Google is lobbying to change that law to allow testing via computer simulation. Safety director Ron Medford said, "Computer simulations are actually more valuable, as they allow manufacturers to test their software under far more conditions and stresses than could possibly be achieved on a test track." Google spokeswoman Katelin Jabbari said, "In a few hours, we can test thousands upon thousands of scenarios which in terms of driving all over again might take decades." Shee adds that simulator data can also easily provide information on how human behavior creeps into driving. "It's not just about the physics of avoiding a crash. It's also about the emotional expectation of passengers and other drivers." For example, when one of Google's computer-controlled cars is cut off, the software brakes harder than it needs to, because this makes the passengers feel safer. Critics say relying heavily on simulation data is flawed because it doesn't take into account how other cars react to the computer's driving.
Simulations are limited by imagination (Score:2, Interesting)
Sure, but the article isn't taking about simulations vs real life. It's talking about simulations vs contrived but legally required tests on manufacturer test tracks. Both are limited by imagination but simulations are more thorough, at least according to Google
Re:Simulations are limited by imagination (Score:5, Interesting)
The problem with simulator testing is that you can't test scenarios that you didn't think of. This is particularly important to find problems arising from multiple simultaneous situations. For example, you might test the scenarios "front camera obscured by rain", "car ahead of you performs emergency stop", and "dog runs into street", but that doesn't necessarily tell you how the car will respond to a combination of the three.
Real life is far more creative than any scenario designer.
Which is why you should do both. A simulation can test millions of permutations -- including arbitrary combinations of events, and in far more variety than could be tested in a reasonable amount of time on real roads -- and can verify that software changes don't introduce regressions. Real-world testing introduces an element of randomness which provides additional insights for the simulation test cases.
Ultimately, governments should probably develop their own simulators which run the autonomous car through a large battery of scenarios, including scenarios which include disabling some of the car's sensors. Then autonomous vehicles from different manufacturers could be validated on a standard test suite before being allowed on the roads, and when real-world incidents occur in which an automated car makes a bad decision, those incidents can and should be replicated in the simulator and all certified vehicles tested. They should also do real-world testing, but I suspect that in the long run simulations will provide much greater confidence.
Re:so what is the problem? (Score:4, Interesting)
I'd flip it around. An automated car should be required to pass both a road test and a bevvy of simulated scenarios.
Certainly. But the question was whether automated testing should be considered sufficient. I think I would do my own flip-around. I think if Google wants to change the California law that requires road testing to make it so that simulation testing is sufficient, then I think Google should donate the simulator, and if an automated car passes the simulation but fails in the real world in a way real world testing would have uncovered but the simulator did not, Google should be held liable for all damages associated with that failure. Under that circumstance, I would be inclined to trust that Google's simulators are a sufficient match to reality to consider substituting simulation testing for road testing.
If Google doesn't want to subject itself to that criteria, then that's a tacit admission the simulation is not guaranteed to catch all the problems real world testing can catch, and I would consider their proposal to be invalid on its face.