Please create an account to participate in the Slashdot moderation system

 


Forgot your password?
Close
typodupeerror
×
Transportation

Rivian Software Update Bricks Infotainment System, Fix Not Obvious (electrek.co) 111

Posted by BeauHD from the someone's-getting-fired dept.
An anonymous reader quotes a report from Electrek: On Monday, Rivian released an incremental software update 2023.42, which bricked the infotainment system in R1Ses and R1Ts. The company is frantically working on a fix, but it might not be an OTA. [...] The vehicles are drivable, but software and displays go black. It appears that the 2023.42 software update hangs at 90% on the vehicle screen or 50% on the app screen, and then the vehicle screens black out. All systems appear to still work except for the displays. At the moment, it appears that Amazon vans are not impacted. Update: The company has acknowledged the issue with affected customers but has yet to issue a fix or plan to fix. Rivian's vice president of software engineering, Wassim Bensaid, took to Reddit to update users on the situation, writing: "Hi All, We made an error with the 2023.42 OTA update -- a fat finger where the wrong build with the wrong security certificates was sent out. We cancelled the campaign and we will restart it with the proper software that went through the different campaigns of beta testing. Service will be contacting impacted customers and will go through the resolution options. That may require physical repair in some cases. This is on us -- we messed up. Thanks for your support and your patience as we go through this.

*Update 1 (11/13, 10:45 PM PT): The issue impacts the infotainment system. In most cases, the rest of the vehicle systems are still operational. A vehicle reset or sleep cycle will not solve the issue. We are validating the best options to address the issue for the impacted vehicles. Our customer support team is prioritizing support for our customers related to this issue. Thank you."
This discussion has been archived. No new comments can be posted.

Rivian Software Update Bricks Infotainment System, Fix Not Obvious More

Rivian Software Update Bricks Infotainment System, Fix Not Obvious

Comments Filter:

  • Software in cars is awesome! (Score:5, Insightful)

    by Anonymous Coward on Tuesday November 14, 2023 @05:06PM (#64005885)
    Manufacturers can't even get in-car entertainment system updates right, and you want us to trust our lives to them with self-driving cars? Insanity!

    • Re: (Score:3)

      by Xenx ( 2211586 )
      Human error also exists with human drivers, and the testing there is generally less rigorous. I'm not saying self-driving tech is without concerns, but I wouldn't point to this kind of update issue as proof for why it's bad.

      • Re: (Score:3, Insightful)

        by Opportunist ( 166417 )

        The difference is maybe that human error in drivers causes one car to cause one accident, while a software error in self-driving cars affects thousands, if not tens of thousands of cars.

        • Re: (Score:2)

          by Xenx ( 2211586 )
          Yes, but it's also not that simple as well. There is the matter of severity vs frequency to consider. Coupled with that, there would usually be multiple check points before self-driving software was updated. Failures can still happen, but would require multiple related failures.

          • I don't believe it would require multiple related failures at all, all it would require is one failure that wasn't tested under a certain condition.

            In a sense screen going blank on startup is easy to detect and should have been caught by basic testing, however a race condition that cause the car to do something stupid one every 100km on with a particular environment would not be easily detectable. The thing is the didn't even test installation, so if they are not doing that them what are the chances somethi

            • Re: (Score:2)

              by Xenx ( 2211586 )

              I don't believe it would require multiple related failures at all, all it would require is one failure that wasn't tested under a certain condition.

              That would still be two failures. A failure in the code, and a failure in the testing. The error being referenced in the story would/could have been caught by someone paying attention. Yes, there is potential for a rare/hidden error to affect self-driving. Nobody is perfect, and thus a group would also not obtain perfection. What it would do is minimize the risk of that happening. Just like with human drivers, we know we cannot eliminate risk. Thus, the goal has to be to minimize it.

              • Re: (Score:2)

                by Hank21 ( 6290732 )

                I don't believe it would require multiple related failures at all, all it would require is one failure that wasn't tested under a certain condition.

                That would still be two failures. A failure in the code, and a failure in the testing. The error being referenced in the story would/could have been caught by someone paying attention. Yes, there is potential for a rare/hidden error to affect self-driving. Nobody is perfect, and thus a group would also not obtain perfection. What it would do is minimize the risk of that happening. Just like with human drivers, we know we cannot eliminate risk. Thus, the goal has to be to minimize it.

                I'd argue 3 or more failures: Failure in the code Failure in QA Failure in Deployment(they mentioned certs did not match code deployed) Failure in Catching the error on failed deployment And the ultimate failure - no back-out procedure at the target to revert to known good code - at the very least a watchdog timer should have triggered a reversion. Hire some older coders with more experience - no offense to the experienced 22 year olds out there - just seems rookie mistakes at multiple levels!

                • Flash is relatively cheap now so there is absolutely zero excuse for not using an A/B system for the infotainment system software, OR the PCM software for that matter. No system doing remote updates should be without such a safeguard. Then if there is an error you can revert to the prior system easily - the watchdog system can do it automatically in fact, but you can also give the user a way to trigger it.

                  • Are there really people here against having a system backup to revert to? Who do you losers work for? I want to be sure I don't use their products.

          • self-driving can fail with bad map data and map data is hard to test.

      • Re: (Score:2)

        by taustin ( 171655 )

        It's a pretty good sign that the technology isn't as mature as it should be before mass deployment.

        • Re: (Score:2)

          by Xenx ( 2211586 )
          It really isn't, however. This wasn't self-driving software being updated. You can't just draw a direct comparison. People would generally be more careful with safety related software, though humans are humans. There would also usually be a verification process that people are less likely to sidestep.
      • Why not? There's more code and more dependence on machine learning models which we basically don't understand and are more praying to the beheaded chicken like "THE ECONOMY" in that south park episode rather than writing and understanding coherent human readable programming instructions. We're all fucked.

    • Re: (Score:3, Insightful)

      by thegarbz ( 1787294 )

      It's almost like the systems that play Spotify are not the same systems that control your engine, breaking, or any other automated function of the cars. Typically they aren't event programmed by the same departments, let alone have common people working on them.

      Please get a clue.

      • Re:Software in cars is awesome! (Score:4, Interesting)

        by Ceseuron ( 944486 ) on Wednesday November 15, 2023 @10:46AM (#64007271)

        It's almost like the systems that play Spotify are not the same systems that control your engine, breaking, or any other automated function of the cars. Typically they aren't event programmed by the same departments, let alone have common people working on them.

        Please get a clue.

        Unless you have knowledge of their vehicle system architecture as well as their company org chart and software development cycle, then you have no idea how their systems are are structured or who works on what aspect of those systems. Absent that knowledge, your comment is pure conjecture based on what you think "typically" goes on at a car company.

        Also, "breaking" is what Rivian's OTA update did to their cars. "Braking" is what causes a vehicle to slow down. Give that the difference between the two eludes you, I'm going to suggest you follow your own advice.

        • Re: (Score:2)

          by nazrhyn ( 906126 )
          Don't make me loose my temper...

        • Unless you have knowledge of their vehicle system architecture as well as their company org chart and software development cycle, then you have no idea how their systems are are structured or who works on what aspect of those systems.

          No actually I do know how they are structured and how they are separated. The differentiation and independence of code as well as departments and software programming is covered under ISO 26262 - the functional safety standard for road vehicles which is applied globally by all car manufacturers.

          Also, "breaking" is what Rivian's OTA update did to their cars. "Braking" is what causes a vehicle to slow down.

          Thanks for the correction. I only speak 5 languages so sometimes I make mistakes since English isn't my first. Fun fact in one of the languages I speak we call language pedants "ant fuckers".

          Give that the difference between the two eludes you, I'm going to suggest you follow your own advice.

          But in all languages I sp

          • No, you actually do not know how their systems are structured or how their architecture is designed. You also do not know what their software and firmware development cycle consists of. Googling for ISO standards does not make you an expert on the matter.

            When you are an actual employee of Rivian, and you are actually working on the systems you claim to have knowledge of, then you will have some credibility. Otherwise you are just spouting conjecture backed up by something you spent five minutes on Google lo

      • I don't recall the details from the video I watched on the rivian but usually these days there are vehicle features you cannot control any way but through the infotainment system. It may not include driver assistance or being to operate the vehicle, but it still can affect more than music and navigation.

    • Re: (Score:2)

      by AmiMoJo ( 196126 )

      The same reasoning applies to all the mechanical engineering in the car. It's only relatively good now because we have been refining it for decades.

    • Re: (Score:2)

      by mjwx ( 966435 )

      Manufacturers can't even get in-car entertainment system updates right, and you want us to trust our lives to them with self-driving cars? Insanity!

      This is the car company that nearly killed Richard Hammond... I mean, one of the times where Richard Hammond was almost killed on Top Gear.

  • Does it blank out the speedometer?

    • Re:except for the displays (Score:4, Interesting)

      by Fly Swatter ( 30498 ) on Tuesday November 14, 2023 @05:45PM (#64005965) Homepage
      If it blanks out the driver's dash, the NHTSA will come calling. Presumably there would be no way to confirm which gear it is in (yes I said gear, which is still what we call PRND), at that point knowing the speed would then be only a relatively minor issue. For how far you can go on a charge waiting for the beeps would not be ideal.

      • If it blanks out the driver's dash, the NHTSA will come calling. Presumably there would be no way to confirm which gear it is in (yes I said gear, which is still what we call PRND), at that point knowing the speed would then be only a relatively minor issue. For how far you can go on a charge waiting for the beeps would not be ideal.

        PRND? My gear says R 1 2 3 4 5 6

        Seriously, this points out a serious issue with OTA updates. Not being able to list to Taylor Swift may be life changing, but a bug that only manifests itself in an emergency situation or edge case could be life ending. "Opps, we messed dup," won't cut it. I could see NHTSA requiring testing closer to what aircraft have, which manufacturers will fight since it will raise their costs.

        Then there is the issue of integrating third party software in a vehicle where a change by o

    • No. Speedometer etc are all on their own independent display behind the steering wheel.

  • It's not dead ... (Score:3)

    by fahrbot-bot ( 874524 ) on Tuesday November 14, 2023 @05:19PM (#64005927)

    In most cases, the rest of the vehicle systems are still operational.
    A vehicle reset or sleep cycle will not solve the issue.

    It's tired and shagged out following a prolonged squawk -- I mean, update. :-)

  • is it to test it before releasing it....

    • Re: (Score:2)

      by gweihir ( 88907 )

      Not hard. But expensive. This seems to be a classical case of "save a penny, lose a million". It also is expensive to hire people that actually know what they are doing so that gets avoided as well.

    • Re: (Score:2)

      by Anonymous Coward

      You can test it all you want, track all the deployment issues, and get management to sign off on the release, but if some sleep deprived engineer selects the wrong file to push to the vehicles, all your testing is for naught.

      What they didn't do was have a test group that would receive the update a week before it went mass distribution. The test group usually catches deployment issues that are hard to test in the lab.

  • whoops (Score:3)

    by Osgeld ( 1900440 ) on Tuesday November 14, 2023 @05:21PM (#64005931)

    This is on us -- we messed up.

    Well that's nice, do they have any clue on how much of a pain in the ass it is to get something done at a dealership / repair center? Are they paying for Ubers to and from work? what about picking up the kids at school?

    That's already a frustrating thing to have to deal with at a normal dealership that's in any town (like Ford or whatever), but the closest one to me is over an hour drive... without much traffic, during rush hour its more like 2 or more, gotta get there early or else it might be there a couple days!

    • Dude, if you can afford a 6 figure high end luxury vehicle, I think you can afford to take some time to travel in the shuttle virtually ever dealership has available.

      • Also once you reach that level, free loaner drop off with free pick up of your bricked paperweight should be a thing.

      • Re: whoops (Score:5, Insightful)

        by Powercntrl ( 458442 ) on Tuesday November 14, 2023 @06:11PM (#64006039) Homepage

        Dude, if you can afford a 6 figure high end luxury vehicle, I think you can afford to take some time to travel in the shuttle virtually ever dealership has available.

        Or just drive one of your other vehicles. Hell, I'm constantly bitching about a lack of funds and even I have two other vehicles in my household which aren't the EV (a Bolt EUV, not one of those crazy expensive high-end EVs). That's kind of useful, because one one of my gas-burning pieces of shit is broken, I drive the other one to get parts.

        And if my partner happens to be off work that day, I get to drive his car to go get parts while I also can lament the fact that my vehicles still use gas. We live in interesting times.

        • ...and when I make a typo and don't bother to proofread, I get to lament the fact that /. lacks an "Edit" button. As I said, interesting times.

          • I get to lament the fact that /. lacks an "Edit" button

            I've been here a while and I remember reading a Q and A years ago where someone asked about this. From what I remember it's something about how they wanted to prevent people from going back and edit their comments after receiving "negative" responses, making it look like they're being attacked for "nothing". Essentially if you're going to say something then own it, don't try to weasel your way out later when others hold you accountable for what you said.

            That said I've always wished for a 10-15 minute wind

      • Re: (Score:2)

        by taustin ( 171655 )

        Or have the butler take care of it.

      • I own 2 2013 compacts thanks that's not the point

    • Re: (Score:2)

      by AmiMoJo ( 196126 )

      Mistakes happen, and I'm okay with that as long as they make it right.

      The real issue is manufacturers that release alpha software and never fix it. I'm looking at you, Tesla.

      They haven't even managed to get the auto wipers to work right, after all these years. They made some progress, then Elon took the rain sensor away and told them to use the camera instead, and it was broken all over again.

      They are basically the AAA game developer of the car world, releasing complete crap and promising to patch it later,

  • Beyond the pathetic engineering practices (Score:5, Insightful)

    by Rosco P. Coltrane ( 209368 ) on Tuesday November 14, 2023 @05:28PM (#64005939)

    A car maker that says "Oops we pressed enter too fast, sorry!" really doesn't entice me to buy one of their vehicles.

    But what entices me even less is that I'm pretty sure there was no way to disable or opt out of automatic updates, or keep the car offline.

    If I ever buy an electric car to replace my 20 year old dumb diesel econobox, it will have to be offline and not updating anything. If it works, don't fix it, and don't touch my fucking property without my consent.

    At this point though, it looks very much like all the options for a new car will steer me towards riding the bus instead. Because having a non-internet-connected, non-mothership-contacting car these days is nigh impossible.

    • Re:Beyond the pathetic engineering practices (Score:5, Informative)

      by dfghjk ( 711126 ) on Tuesday November 14, 2023 @05:34PM (#64005943)

      It's not just electric cars, in fact it has nothing to do with the cars being electric.

    • A car maker that says "Oops we pressed enter too fast, sorry!" really doesn't entice me to buy one of their vehicles.

      Perhaps they figured 'insider threat' was a bit too honest for the stock price.

      Still struggling as to how the 'wrong' security certificates were in fact right enough for the vehicles to accept the update...not that we're pointing fat fingers or anything...

    • But what entices me even less is that I'm pretty sure there was no way to disable or opt out of automatic updates, or keep the car offline.

      Disconnect the antenna.

    • If I ever buy an electric car to replace my 20 year old dumb diesel econobox, it will have to be offline and not updating anything.

      I agree but I don't think such a thing will exist.

      • Re: (Score:2)

        by gweihir ( 88907 )

        It is enough for a vehicle to continue to work fine when losing connection. Then just unplug the antenna.

    • Because having a non-internet-connected, non-mothership-contacting car these days is nigh impossible.

      In the old days we used to joke about "What if cars ran Windows?" I guess it's the auto industry that is laughing now.

      If you have the budget for it, there's always kit cars and then paying someone else to put it together for you if you're not mechanically inclined. That plan probably gets a bit more challenging (and expensive) if you've really got your heart set on an EV though.

    • The updates are not automatic and there is no way to make them automatic. Everyone who got bit by this pressed the âoeupdateâ button.

    • Re:Beyond the pathetic engineering practices (Score:4)

      by thegarbz ( 1787294 ) on Wednesday November 15, 2023 @12:19AM (#64006531)

      A car maker that says "Oops we pressed enter too fast, sorry!" really doesn't entice me to buy one of their vehicles.

      Yeah I know. They should just cover up the problem and not admit fault like all the other companies making vehicles that you no doubt support - mostly out of ignorance of what goes on.

      Me, I'm more enticed to buy vehicles from companies that are open, honest, and communicate with customers. Enjoy your Ford that you're inexplicably warned not park in the garage because it'll burn your house down, or the Telsa which is not covered by warranty because you drove it in the rain (that happened to a friend of mine before a complaint to regulators got them to back down and repair the battery which had known widespread issues with water ingress), or maybe you prefer Toyota, the ol' reliable car that toooootally didn't have a spurious acceleration issue - oh and by the way there's a recall notice out on your car, don't ask us why we won't tell you.

      If you actually knew what was going on behind the scenes you wouldn't own your dumb 20 year old econobox either.

  • Bad OTA update needs an dealer reload? (Score:5, Interesting)

    by Joe_Dragon ( 2206452 ) on Tuesday November 14, 2023 @05:34PM (#64005941)

    No backup / recovery firmware?
    No local usb port to load from (or that we only give the images to the dealer?)
    Needs an dealer only installer plunged into the obd2 port?
    Why did the car not check the certificates of the update before install?

    Can the same thing happen with an network or battery drop out in the middle of any update install?

    • Re: (Score:2)

      by xxdelxx ( 551872 )
      More to the point - how open is it to 3rd party (assumed hostile) updates? Clearly there's no on-device checking for validity.

    • No backup / recovery firmware?

      Par for the course with any computer I've used for the last few decades. Hey, why should a $100K need 10 cents worth of emergency recovery firmware?

      Seriously, I had a problem updating a Gigabyte mobo with "DualBIOS". The company literally uploaded the wrong firmware for my board on their web site. DualBIOS did not work as advertised and only gave me a black screen. The only way I was able to fix it was to de-solder and swap the firmware chips.

      Can the same thing happen with an network or battery drop out in the middle of any update install?

      I have no idea about Rivian, but when my dad wanted to update

  • Try turning it off and on again.

  • Wrong...security? (Score:5, Insightful)

    by geekmux ( 1040042 ) on Tuesday November 14, 2023 @05:43PM (#64005955)

    "a fat finger where the wrong build with the wrong security certificates was sent out."

    Maybe we can understand the 'wrong' build going out, but what the hell do you mean the 'wrong' security certificates? They certainly seemed to be right enough for the cars to accept the update. If it were in fact 'wrong', then something would have likely rejected it.

    Otherwise known as one of the main reasons you use security certificates.

    Seems that would take a hell of a lot more than a lone fat finger...

    • Maybe they have more than one valid certificate chain?

      • Re: (Score:2)

        by taustin ( 171655 )

        Almost like they have separate dev and production networks, eh?

        • Almost like they ignored the entire point of that segregation, eh? Prod should be cloned to Test if you’re gonna replicate anything to include cert chains. A Development network should be exactly that, especially for products this critical. No Dev signed code should land on Prod without extensive Test(ing) and validation. Hell, give cars away to sales staff to be beta testers if that’s what it takes. Lot cheaper than this embarrassing publicity.
          • Having production users beta test bespoke code to fix their obscure issues is a thing. Not saying that happened here but also won't jump to conclusions based on limited info either.

            • Having production users beta test bespoke code to fix their obscure issues is a thing. Not saying that happened here but also won't jump to conclusions based on limited info either.

              It may be a "thing", but that doesn't mean it's the right thing. A fat finger is an admitted mistake. The fault isn't really clouded in obscurity.

              I'd say bricking a part of the car is something that is easily avoidable by using an actual beta group of cars to throw updates to first. If a factory cannot afford to keep such a group of cars on hand (unlikely given what they're charging for the product), then at least keep first-round software updates contained to the dealership inventory used by managers an

              • I work primarily with Cisco equipment. Occasionally enterprise users have very obscure bugs due to very specific configurations, and Cisco will actually create patches that are outside the normal release trains and will never be generally available. Microsoft does this sometimes too, releasing patches that are not pushed to everyone but made available only for people who have very specific issues.

                No matter how big your suite of beta testers, there are going to be edge cases where fixes can only be prac

    • Perhaps they meant expired?

  • The problem is not... (Score:3)

    by MpVpRb ( 1423381 ) on Tuesday November 14, 2023 @05:45PM (#64005969)

    ...that they rolled out a corrupted update
    The problem is that they had no recovery plan other than physical access

    • The problem is that they had no recovery plan other than physical access

      Physical access is a recovery plan.

      You know what we call remote access to install custom firmware even when the main image is incapable of starting? A backdoor.

  • Was going to happen eventually (Score:5, Interesting)

    by bhcompy ( 1877290 ) on Tuesday November 14, 2023 @05:49PM (#64005975)
    This has been a concern among pundits, Consumer Reports, etc for a long time. The NHTSA needs to step up and put constraints around OTA updates to make sure they're thoroughly tested before being released to the public. People have been ringing this bell since at least when Tesla released a major braking update within a few days of Consumer Reports blindsiding them with a negative report on brake performance, as there's no way the update to a major system was tested as much as it should have been before being deployed to every vehicle in the fleet. We can't rely on businesses to do the proper QA on their own. We have vehicle safety regulations because this has already been proven time and time again since the advent of the vehicle.

  • We put SaaS in your car, we're calling it CaaS. You've got an update, like it or not!

    Your Car as a Service isn't working? Hmm, I guess we'll come to you!

    This kind of thing is going to kill the company if it keeps up.

    • This kind of thing is going to kill the company if it keeps up.

      One can only hope.

      And one can also hope that this serves as a lesson to the rest to not try to cram CaaS down their customers' throats.

  • I'm sure we can think of more than a few product failures where we would hear nothing but crickets from a failed product, but here Rivian was upfront and took responsibility for it.

    • Re:Good on Rivian for being up front about the pro (Score:4, Informative)

      by Opportunist ( 166417 ) on Tuesday November 14, 2023 @06:30PM (#64006083)

      What alternative do you think they had? I kinda could imagine that people would have noticed if their cars don't work, and that it wouldn't even take a day 'til people notice "hey, it's not just my car".

      • What alternative do you think they had?

        All of them. Blame the user. Say it was an isolated incident. Deny everything. You know, basically what most car companies do when something goes wrong.

        • That might have worked back in the 90s, today everyone has the ability to tell everyone on the planet in an instant about their experience with your product and there's plenty of ways to do so.

          Welcome to the internet.

      • What alternative do you think they had? I kinda could imagine that people would have noticed if their cars don't work, and that it wouldn't even take a day 'til people notice "hey, it's not just my car".

        Yea, kinda hard ti hide in today's world. What boggles my mind is why they didn't push the OTA to some test mules and discover this. Either they did, in which case their testing and update process is broken, or they didn't and they are negligent.

        I purpose this test: Before every update, the CEO is required to drive a test mule at full speed around the test track to ensure everything works properly. Repeat in self driving mode.

        • Re: (Score:3)

          by pjt33 ( 739471 )

          I'm not sure that driving at full speed around a test track would have helped to identify a problem in the "infotainment system".

          • I'm not sure that driving at full speed around a test track would have helped to identify a problem in the "infotainment system".

            No, but the CEO sure would be interested in making sure any update worked properly...

  • and should be fired. oh. he is a C suite. clearly someone elses fault then.

  • They gave their owners a dirty fat fingering update.

  • Here's the problem: Infotainment (Score:3)

    by Required Snark ( 1702878 ) on Tuesday November 14, 2023 @08:02PM (#64006223)
    It's not like designing the breaks or powertrain or steering, it's just some half-assed software right? If it fails, who cares? So you can't get your overpriced streaming, just turn on the smartphone instead. No biggie.

    Outside of a few highly regulated industries (aerospace, healthcare) there are no enforceable standards for software. Look at how many huge companies routinely suffer data breaches, and there is zero accountability. Some companies are repeat offenders and they never get it right. (I'm looking at you, T-Mobil.)

    This will never change until there are severe consequences when software breaks. How about a $1000 fine for each individual who has their data leaked, othe their car be bricked? On the second offense, $10,000 per. It is 100% certain that if that was the consequence it would happen exactly once, a name player would go down in flames, and it would never happen again.

  • It's broken until they redesign OTA updates so that this isn't possible.
    What? Short on memory? Can't bother with redundancy? Constrained with crap hardware?

    But I hear you get a lot of truck for $80,000.00

Slashdot Top Deals

Be sociable. Speak to the person next to you in the unemployment line tomorrow.

Close