Forgot your password?
typodupeerror
Google Transportation

Google Wants To Test Driverless Cars In a Simulation 173

Posted by timothy
from the do-you-want-to-be-a-virtual-pedestrian? dept.
An anonymous reader writes Google has been testing its autonomous vehicles on U.S. roads for a while now. In fact, they're required to, by law. "California's regulations stipulate autonomous vehicles must be tested under "controlled conditions" that mimic real-world driving as closely as possible. Usually, that has meant a private test track or temporarily closed public road." It's easy enough to test a few prototypes, but whenever autonomous cars start being produced by manufacturers, it'll become a lot more complicated. Now, Google is lobbying to change that law to allow testing via computer simulation. Safety director Ron Medford said, "Computer simulations are actually more valuable, as they allow manufacturers to test their software under far more conditions and stresses than could possibly be achieved on a test track." Google spokeswoman Katelin Jabbari said, "In a few hours, we can test thousands upon thousands of scenarios which in terms of driving all over again might take decades." Shee adds that simulator data can also easily provide information on how human behavior creeps into driving. "It's not just about the physics of avoiding a crash. It's also about the emotional expectation of passengers and other drivers." For example, when one of Google's computer-controlled cars is cut off, the software brakes harder than it needs to, because this makes the passengers feel safer. Critics say relying heavily on simulation data is flawed because it doesn't take into account how other cars react to the computer's driving.
This discussion has been archived. No new comments can be posted.

Google Wants To Test Driverless Cars In a Simulation

Comments Filter:
  • by turkeydance (1266624) on Friday August 22, 2014 @07:38PM (#47733775)
    until it's your life.
  • by m2shariy (1194621) on Friday August 22, 2014 @07:41PM (#47733781)
    Test in the fscking simulation and then test on the street. Win-win.
    • by dnavid (2842431) on Friday August 22, 2014 @08:14PM (#47733943)

      Test in the fscking simulation and then test on the street. Win-win.

      You don't need to ask for permission to test your car with simulations. You only have to ask for permission to replace real world testing with simulations. Personally, I'm not fond of replacing real world testing completely with simulations. The problem is that the point of testing software is to make sure the programmers have properly dealt with as many possible real world situations, and to reduce the likelihood the programmers haven't ignored an unexpected circumstance. Simulations can only test for what the simulation programmers have accounted for. Its substituting the system programmers' judgment for the simulation programmers' judgment. Its useful, but in my opinion insufficient.

      • by Your.Master (1088569) on Friday August 22, 2014 @09:15PM (#47734237)

        I'd flip it around. An automated car should be required to pass both a road test and a bevvy of simulated scenarios.

        • by dnavid (2842431) on Friday August 22, 2014 @11:44PM (#47734861)

          I'd flip it around. An automated car should be required to pass both a road test and a bevvy of simulated scenarios.

          Certainly. But the question was whether automated testing should be considered sufficient. I think I would do my own flip-around. I think if Google wants to change the California law that requires road testing to make it so that simulation testing is sufficient, then I think Google should donate the simulator, and if an automated car passes the simulation but fails in the real world in a way real world testing would have uncovered but the simulator did not, Google should be held liable for all damages associated with that failure. Under that circumstance, I would be inclined to trust that Google's simulators are a sufficient match to reality to consider substituting simulation testing for road testing.

          If Google doesn't want to subject itself to that criteria, then that's a tacit admission the simulation is not guaranteed to catch all the problems real world testing can catch, and I would consider their proposal to be invalid on its face.

          • by mysidia (191772)

            If Google doesn't want to subject itself to that criteria, then that's a tacit admission the simulation is not guaranteed to catch all the problems real world testing can catch, and I would consider their proposal to be invalid on its face.

            I think this is more along the lines of them wanting to avoid the time and expense, since every new model will have to be tested after every code change, I guess.

            The problem is.... can we really trust the simulations?

            I would rather it be required to have a small f

      • by vtcodger (957785)

        Personally, I'm not fond of replacing real world testing completely with simulations.

        Exactly. A broad battery of simulations makes sense for regression testing to prove that the 2027 model year software handles all the situations that the 2026 did. But real world testing is required to verify that the system doesn't do nutty things when confronted with unusual conditions -- dust clouds, ice coated wall to wall potholes, a trackless rural road or rarely used off ramp covered with four inches of snow with w

      • Please define "simulation".

        You can't test some rare situations in real life because they are so rare.

        For example car accidents. We're glad that they have been greatly reduced in real life and aren't predictable enough so that cars can be deliberatly sent into real life accidents. That's why we're running simulated accidents, crash tests. Of course not a computer simulation, it's still a simulation that neglects human factors. (evasion maneuvers might lead to other impact angles and speeds, passengers tensin

      • You don't need to ask for permission to test your car with simulations.

        Agreed. Google is being misleading in its arguments, which raises the question of whether it is being dumb or acting dumb. I have my opinion as to which it is, but neither inspires confidence in Google's judgement and motives, and confidence is of the essence when it comes to getting self-driving cars accepted.

        Simulations can only test for what the simulation programmers have accounted for.

        And they are also based on assumptions about the response of the cars' sensors to the real world.

    • by ganjadude (952775)
      can i mod this +1 obvious?
  • by Carnildo (712617) on Friday August 22, 2014 @07:49PM (#47733829) Homepage Journal

    The problem with simulator testing is that you can't test scenarios that you didn't think of. This is particularly important to find problems arising from multiple simultaneous situations. For example, you might test the scenarios "front camera obscured by rain", "car ahead of you performs emergency stop", and "dog runs into street", but that doesn't necessarily tell you how the car will respond to a combination of the three.

    Real life is far more creative than any scenario designer.

    • A simulation is also only as good as its simulator. The idea of a simulator that doesn't let errors slip that will only show up in the real world doesn't pass the sanity test.

      • and the simulator is only as good as the programmer. Google can test in the simulator all they want, I'm sure they do it anyway. But road testing is required regardless. I don't want to be in the beta group for driverless cars when we're going 70MPH on the 101.
    • by fisted (2295862)

      For example, you might test the scenarios "front camera obscured by rain", "car ahead of you performs emergency stop", and "dog runs into street", but that doesn't necessarily tell you how the car will respond to a combination of the three.

      Now that really leads to a difficult decision for the car. Should it:
      - Maintain heading and come to a halt? Or perhaps
      - Maintain heading and come to a halt? Or even
      - Maintain heading and come to a halt?

      Let's wait for strong AI to solve that problem for us.

      • by 0123456 (636235)

        Decades of aircraft autopilot failures tells us it will immediately hand control back to the driver, then blame 'human error' for the crash.

        • Because saying Our pilot assed up involves a significantly disproportionate settlement amount than an indictment of our entire technological foundation.

          Good post.

        • Of course, since this IS actually the best thing the autopilot can do for its own survival and the survival of others of its own kind - since any failure that can actually be blamed directly on it might result in the humans deciding to build different autopiloting devices or just do away with them altogether - then this could be taken as a sign that the AI works really, really well. ;)

      • by Carnildo (712617)

        Are you sure the car won't spot the dog, mistake it for a child (remember, the quality of information from the front camera is reduced), and perform an emergency turn to the left? Are you sure the presence of the car won't mask the presence of the dog, or vice-versa?

        It's easy to say "when in doubt, maintain heading and come to a halt". It's much harder to define "doubt" in a way that's useful to a computer.

      • by kamapuaa (555446)

        Are you suggesting simulators can't deal with multiple scenarios at the same time?

        • by rtb61 (674572)

          Where are simulator completely and utterly fails should be obvious to everyone, it does not test environmental analysis at all. The environment is not detected, analysed and correlated, it is simply fed into the program and so only half the system is tested, not the whole system.

          The system should be tested on the road under normal conditions, from rush hour to night driving with a driver ready to take over and at lest two randomly chosen independent observer. Things that need to be tested, missing or wor

      • by PopeRatzo (965947)

        Maintain heading and come to a halt?

        That's going to double travel times right away.

      • Steering around a problem is, on average, safer than applying the brakes. Frequently, pulling onto the shoulder and THEN applying the brakes in order to come to a stop next to the car you would have rear-ended is the best course of action.

        In the scenario, visibility is reduced and the pavement is slick with rain. "Maintain heading and come to a halt " in those conditions practically guarantees you'll get rear-ended. The car behind you has their vision obscured by rain, can't stop quickly on the slick

        • by Richy_T (111409)

          You (or the car behind you) were following too closely. Something an automatic car wouldn't be doing. In that case, swerving onto the shoulder is likely safer but you run the risk of say striking a cyclist or an expectant mother trying to get her spare wheel from the trunk of her car because she has a flat and is using the emergency lane for its intended purpose and not as a crutch for her bad driving.

          • Someone COULD have pulled off onto the shoulder in front of you.
            Someone DEFINITELY is in the lane in front of you. "Could" is less likely than "definitely" . The shoulder is the therefore the safer bet.

            "Or the car behind you is following too closely " - it normally is, most of the time. Especially considering that the driver of tge car behind you may well not be focused 100% on driving. If they are turning down the radio because they're calling in to try to win Aerosmith tickets, 1/4 mile is too close

    • by ShanghaiBill (739463) on Friday August 22, 2014 @08:05PM (#47733895)

      For example, you might test the scenarios "front camera obscured by rain", "car ahead of you performs emergency stop", and "dog runs into street", but that doesn't necessarily tell you how the car will respond to a combination of the three.

      This seems backwards to me. Testing combinations of scenarios happening simultaneously would be far easier in a simulator.

      • But only the combinations you think of, while in real life something might happen that you did not expect.

        • by Thiez (1281866)
          Irrelevant, an actual testdrive is also not going to cover all possible scenarios, and appears to be sufficient at this time. Remember, perfect is the enemey of good.
      • by Richy_T (111409)

        Real life is good for testing scenarios you might not expect, simulations shine at testing infrequent scenarios and subtle variations upon them. Both are required.

    • by Anonymous Coward

      Sure, but the article isn't taking about simulations vs real life. It's talking about simulations vs contrived but legally required tests on manufacturer test tracks. Both are limited by imagination but simulations are more thorough, at least according to Google

      • Sure, but the article isn't taking about simulations vs real life. It's talking about simulations vs contrived but legally required tests on manufacturer test tracks. Both are limited by imagination but simulations are more thorough, at least according to Google

        Google wants to replace expensive, real testing with inexpensive, fake (aka "simulations") testing. The two aren't comparable, and the danger is that Google can lobby to change the laws to allow simulations to replace real life testing. Whi

    • by swillden (191260) <shawn-ds@willden.org> on Friday August 22, 2014 @08:28PM (#47734021) Homepage Journal

      The problem with simulator testing is that you can't test scenarios that you didn't think of. This is particularly important to find problems arising from multiple simultaneous situations. For example, you might test the scenarios "front camera obscured by rain", "car ahead of you performs emergency stop", and "dog runs into street", but that doesn't necessarily tell you how the car will respond to a combination of the three.

      Real life is far more creative than any scenario designer.

      Which is why you should do both. A simulation can test millions of permutations -- including arbitrary combinations of events, and in far more variety than could be tested in a reasonable amount of time on real roads -- and can verify that software changes don't introduce regressions. Real-world testing introduces an element of randomness which provides additional insights for the simulation test cases.

      Ultimately, governments should probably develop their own simulators which run the autonomous car through a large battery of scenarios, including scenarios which include disabling some of the car's sensors. Then autonomous vehicles from different manufacturers could be validated on a standard test suite before being allowed on the roads, and when real-world incidents occur in which an automated car makes a bad decision, those incidents can and should be replicated in the simulator and all certified vehicles tested. They should also do real-world testing, but I suspect that in the long run simulations will provide much greater confidence.

      • "simulation" is also a technical description of "driving game". Let them also put the simulator on-line, to provide environment and background as hundreds of thousands of crazed and insane real humans try to crash into the auto-piloted cars. Each time someone succeeds, buff up their capabilities and give them credit and recognition, and develop response scenarios. That's how you "sim" car combat with real humans - you use real humans. It would be just like the dogfighting flight sims they use to train pilot

        • by swillden (191260)

          While it would be entertaining, I don't think that's a very useful method for evaluating the performance of self-driving cars, unless you're trying to design a car for demolition derby competitions. I understand that your'e trying to design an extreme environment on the theory that if the car can perform well there, it'll definitely do fine on real roads, but I don't think that theory is valid. In real life, the vehicles on the road try not to hit one another, and the method they use (in most countries, at

    • by PopeRatzo (965947)

      Real life is far more creative than any scenario designer.

      Ain't that the truth.

      This is why I don't see everyone in driverless cars in any of our lifetimes. I'm thinking it's at least 70 years out. And not least because a) who's going to pay for all the necessary infrastructure? and b) shared liability will make it a nightmare.

      Maybe first let's see if we can have a driverless NASCAR race without crashes. And then I want to see the CEO of a driverless car company put his kids in the car and send them on a

    • by D'Sphitz (699604)
      You can't test scenarios that weren't thought of on a private test track either.
    • You make an excellent argument for simulated testing. A real world test will only give you a few scenarios, simulation will throw millions at each individual unit.

      Of course you're right about real life constantly surprising people--that's why the development team is performing continuous algorithm development in the real world. I hope my automated car has a real-world-tuned algorithm in combination with a moslty-simulated per-unit system test.

    • In a simulation you can have people do all kinds of crazy shit you wouldn't see in thousands of on-the-road miles. You can simulate malfunctioning equipment that you wouldn't get without years of wear and tear. You can test modifications to the AI without real-world consequences. You can test the human-ai interaction on average drivers without liability problems. I could totally see a simulation being superior to reality for testing purposes.

      But you could also have a broken simulation, which could make the

    • by idji (984038)
      You are right that real life is far more creative, but that is not the point being argued here. The point here is that simulation is a better testing environment than a test track. The test track will have much less creative scenarios than simulation because they are so much harder to stage. A test track will not test scenarios that people didn't think of. The simulator is a much better environment to test dog+stop sign+rain - try doing that on a track. Put some creative people in the simulator and they'll
    • by hibiki_r (649814)

      The problem with real life testing is that it's so absolutely slow you won't even go through the examples you can think of.

      Simulator testing is a bit like property based testing on software. I come up with a 'test envelope' of things that could possibly happen, and let generators combine them in many ways, as to check way more options that I ever could with example based testing. Then we run a few thousand of those random scenarios every build. If there's ever a failure, it's recorded and we can reproduce i

  • by Joe_Dragon (2206452) on Friday August 22, 2014 @08:01PM (#47733871)

    may help test software but not the full hardware and how well it works in all kinds of weather / settings. Also what about road conditions / slight lines? odd traffic light layouts / intersections? Just useing google street view as the input likely will not get the full lay out from each lane / all times of day / all cycles.

  • by CaptainDork (3678879) on Friday August 22, 2014 @08:01PM (#47733879)

    ... lack of randomness.

    Will they simulate a 3 year old tossing a sandwich out the window into oncoming traffic?

    • 3-yo? how about everyone throws out all their trash? total discharge at 11 mph over posted speed. all 4 windows open and ejected into traffic? yes, i was doing 11mph over the line. right in front of me. NASCAR skilz.
      • I've read your post three times and I still have no idea what it's talking about. Sounds interesting, though.
    • by Ksevio (865461)
      Would that happen in 100,000 miles of normal driving? The advantage of simulation is it can simulate once in a life time events like a 3 year old tossing a sandwich out the window into oncoming traffic.
    • With AC coming almost as standard nowadays, I wonder if it will ever come to a point where you can't open the window if the car is going over a certain speed. It uses a lot less energy to cool the interior of the car using AC, than the cost to aerodynamics of having a window (or 2) open while travelling at speed.

      I welcome the day when cars simply become a means of transport.

  • by siphonophore (158996) on Friday August 22, 2014 @08:03PM (#47733887)

    Are we really having a public, political, emotional discussion about the relative merits of ATE vs Validation testing? Come on, Slashdot, you're a bunch of engineers, right? Does the CA state legislature have ANYTHING of value to add to your FMEA? What about your production planning? Test plan? V&V protocols?

    It's the height of hubris for outsiders (especially lawyers in the state legislature) to come in and dictate low-level engineering details. A responsible legislature (and public) would acknowledge that they have NOTHING of value to add to the discussion.

    The only appropriate regulation is "make it X safe." Don't tell us engineers to get there, and we won't tell you lawyers how to snort coke of a hooker's tits.

    • by kamapuaa (555446)

      Wish I could mod parent "+1 Edgy"

      If it wasn't so edgy, the suggestion to let the car companies decide for themselves would come off as ridiculous.

      • If it wasn't so edgy, the suggestion to let the car companies decide for themselves would come off as ridiculous.

        As long as they, and their insurance company, are willing to accept full liability, I don't see why not.

        • by kamapuaa (555446)

          You're OK with unsafe & unpredictable drivers on the road, as long as they're insured?

      • by siphonophore (158996) on Friday August 22, 2014 @08:31PM (#47734033)

        A regulatory "light hand" is appropriate here for a few reasons:

        1. The current state of the art is, comparatively, extremely dangerous (even with attentive, good drivers).
        2. Google (or the next few guys coming down the pipe) already have an extremely strong incentive to make their cars as safe as possible (speed of adoption, fear of future regulation).
        3. OTA updates would resolve problem behaviors after only a few incidents.

        Google is coming to the public with a (statistical) goldmine for human development. The cold skepticism they're getting is totally unwarranted and will do nothing but delay the enormous social and economic benefits that fully autonomous roads will bring.

        • by 0123456 (636235)

          1. The current state of the art is, comparatively, extremely dangerous (even with attentive, good drivers).

          One death per 60,000,000 miles (with inattentive, lousy American drivers) is 'extremely dangerous'?

          • "comparatively"

            This is (typically) the most dangerous thing people do all day.

            • by 0123456 (636235)

              Compared to what? Russian roulette?

              As far as I'm aware, no 'self-driving' car has driven anywhere near 60,000,000 miles.

              • No, not compared to Russian Roulette, compared to the things typical people do in a typical day. Also, with cars, death isn't the only danger. Permanent injury, significant temporary injury, and massive property damage are also dangers.

                Typical people live their entire lives without playing Russian Roulette even once.

                I don't really know why this is hard. Most people don't do a lot of dangerous things in a day.

                In fact, even in terms of death: 22% of people who die between the ages of 1 and 44 in the Unite

        • 3. OTA updates would resolve problem behaviors after only a few incidents

          You would let your car be connected to the internet? 0_0

        • Absolutely, government should get out of the way of private industry.

          Remember 2008?
        • by MrL0G1C (867445)

          The cold skepticism they're getting is totally unwarranted

          Corporations have terrible safety records, it's laws, unions and standards that keep them in check, skepticism is warranted

          Autonomous car accidents should be treated like aircraft accidents - thoroughly investigated and the conclusions used to further improve safety where possible and practical.

          People keep talking about the past safety record of google cars, but google cars don't have a safety record, they are not a final product. The cars that have

    • by westlake (615356)

      Come on, Slashdot, you're a bunch of engineers, right?

      Wrong.

      If by engineer you mean a licensed professional who stands by his work, and can be called to account for his failures.

      It's the height of hubris for outsiders (especially lawyers in the state legislature) to come in and dictate low-level engineering details.

      It also the height of hubris for the geek to allow Google to be the sole judge of its own work.

      • It also the height of hubris for the geek to allow Google to be the sole judge of its own work.

        Manufacturing in a regulated industry is a constant battle between operations an quality. Operations (with an eye toward revenue) tries to speed things up, Quality (with an eye toward recalls and audits) tries to slow things down. Both report through different paths to the CEO. The Geek in R&D will see his work checked over by a different department with a different set of metrics.

        You can pick your exceptions, but the overwhelming result of this organizational method is safer and better products.

        • Manufacturing in a regulated industry is a constant battle between operations an quality. Operations (with an eye toward revenue) tries to speed things up, Quality (with an eye toward recalls and audits) tries to slow things down. Both report through different paths to the CEO. The Geek in R&D will see his work checked over by a different department with a different set of metrics.

          You can pick your exceptions, but the overwhelming result of this organizational method is safer and better products.

          This is like an Ayn Rand novel from an engineer's point of view. Regulated industries act *nothing* like this. The people in power use that power to push out the cheapest crap that will make the most money, whether that's service or physical product. The people who assist in that activity gain the most favor in the company. The noble engineer you keep describing simply doesn't exist.

          The difference between lightly regulated and heavily regulated industries is that the sociopaths in companies from heavil

    • by Kjella (173770)

      You really think this is solely an engineering decision? I'm guessing this is just as much if not more a business decision. We could have real world testing which is expensive where unexpected quirks and flaws could be revealed or we could have simulations which are cheap and quite confined to whatever it is the scenario is testing. Everyone in suits would go with simulations, while engineers know that models are abstractions and simplifications of reality.

      Don't get me wrong, I'm sure they do a ton of simul

      • If you were running this I bet you would sign yourself up for an ATE-heavy 100%, a sample plan of trips around town, and an exhaustive DVT (verification in the lab and validation all over the country). You'd hit all the points on the FMEA and performance requirements doc, then throw some gonzo tests in there to add a little spice.

        I would say that's sufficient and you acted prudently, and engineers with production experience would say the same. Things would turn out just fine UNLESS some idiot decided to tur

  • Let's see if...

    Google writes the software for the car
    Google writes (or pays someone else to write) the simulator
    Google runs the test
    Google reports the results

    Seems like with simulations we would be somehow implicitly trusting google that their simulator sufficiently models reality vs only modeling what the self driving software expected...

    Although simulation has its place to improve testability during training and development, how does this test against reality? The reason to test against reality is genera

    • by Ksevio (865461)
      Presumably Google would also run the non-simulator tests and report the results, so there's no real difference there.
  • Car Analogy (Score:4, Funny)

    by ProzacPatient (915544) on Friday August 22, 2014 @08:50PM (#47734117)
    Can someone give me the obligatory car analogy?
    • No idea what you mean but how about this: If a car leaves New York traveling 60 MPH and another card leaves from Vegas at 50MPH, how long does it take for Gloria Allred to show up after a Google car crashes into one of them?
  • Can I miss Google spokeswoman Katelin Jabbari?

  • Ooops! (Score:4, Funny)

    by jd (1658) <imipak&yahoo,com> on Friday August 22, 2014 @10:29PM (#47734597) Homepage Journal

    Found a bug in physics.c, those cars we mass produced last year will spontaneously explode after 367 days of exposure to an atmosphere containing oxygen, or when white lines are painted rather than vinyl, or when attempting a corner of a prime number of degrees when speeding on a cambered road.

    Why wasn't this spotted sooner?

    Because we hadn't expected to need chemistry or non-Euclidian geometry in a physics engine.

  • Bit-flip error in specific hardware triggered by the 2022.3.5 version driving in Death valley for over 6.5 hours.

  • The article is pointless. Okay, Google is trying to replace the current "controlled" road test with a simulator. The article goes on to say how wonderful simulators are. So what? It says *nothing* about the current regulations. What are they intended to test? Are they done once per model? For every firmware revision? Every individual vehicle? Are they meant to be fully exhaustive or are they more on the order of the driving test a person must take to get a license? Without knowing what the current tests

Facts are stubborn, but statistics are more pliable.

Working...