Volkswagen Has 'massive' Software Problems With New ID3 Electric Vehicles (electrek.co) 107
Socguy shares a report from Electrek: Germany's Manager Magazine reports today that Volkswagen is struggling with software problems for its ID3 all-electric car. According to the report, the ID3 will be built for months with an incomplete software architecture that could affect up to 20,000 electric cars. These units, intended for sales in Europe and not the U.S., will require a manual software update.
The nature of the software problem -- how it affects vehicles or the sales timeline -- was not disclosed. The magazine reported the issue as "massive." And unfortunately, the fix is cumbersome. Thousands of ID3 cars will be parked in dedicated rented spaces until the spring when service teams will be deployed with mobile computer stations. New software will be manually installed in this manner for the first 10,000 or so ID3s. A total of 20,000 ID3 vehicles will need to be reworked until the second wave of production begins in May. At that time, further software updates can be deployed over-the-air.
The nature of the software problem -- how it affects vehicles or the sales timeline -- was not disclosed. The magazine reported the issue as "massive." And unfortunately, the fix is cumbersome. Thousands of ID3 cars will be parked in dedicated rented spaces until the spring when service teams will be deployed with mobile computer stations. New software will be manually installed in this manner for the first 10,000 or so ID3s. A total of 20,000 ID3 vehicles will need to be reworked until the second wave of production begins in May. At that time, further software updates can be deployed over-the-air.
Re:Overair or Online download and USB (Score:5, Informative)
So, apart from needing to calm down a bit, that's not how it works, at least with Tesla.
The Tesla app on your phone informs you that there is an update available to install. You can look at the change log. It is your choice if and when you update your car.
I personally don't know the details about the security of the download itself, but that's because I don't care. It could be encrypted in transit (I imagine it to be SFTP or something like that, but as I said, I don't know).
People seem to think that their cars are updating themselves without their involvement. I dunno, maybe Nissan does that. Tesla doesn't. I don't see the difference between an encrypted download and a option to install and what you are suggesting except the inconvenience of what you are proposing and the risk you are introducing into the process (such as a corrupted file system on the USB stick you are using, etc).
Re: (Score:3)
With Teslas, you also have the option to select whether you want to be closer to the bleeding edge or not.
You can wait for any issues to surface in a rollout before allowing your car to be updated.
Can you roll back? (Score:3)
Can you roll back when it breaks something that matters to you?
Re: (Score:2)
Re: (Score:1)
If it breaks, you take it to the service center.
Have you ever tried reading your email when the Internet is down?
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
I can't imagine myself volunteering to beta test automobile control software. Maybe I've spent too many years watching the sausage being made.
Re: (Score:2)
So, apart from needing to calm down a bit, that's not how it works, at least with Tesla.
People seem to think that their cars are updating themselves without their involvement. I dunno, maybe Nissan does that.
From my perspective this is all irrelevant. The existence of unnecessary real-time bidirectional communication channels with full control over vehicle baked into vehicles (e.g. Tesla) is a huge potential completely avoidable risk given total ubiquitous failure of industry to develop and manage secure systems.
Just look at TFA vehicle manufacturers can't even be bothered to write reliable software which most certainly means its swiss cheese from a security perspective.
Re: (Score:3)
It's not really a free choice to update. If there is a problem with your car the first thing they tell you to do is update. There is no separation of bug fixes and feature updates. The behaviour of your car changes and there isn't much you can do about it, realistically.
Some people like that, some don't.
There are safety issues. People have died because they trusted autopilot in a particular area and then an update broke it.
Last I heard Japan was way behind on updates because the regulator wasn't happy with
Re: (Score:2)
Unless Nissan's engineers are morons, there is no risk of copying the file via usb stick. Any rational update file format would contain an internal checksum. Try "unzip -t ". I suppose a corrupted file would waste time, but shouldn't be accepted.
Re: (Score:2)
Microsoft has been known to disrupt surgeries with forced Windows 10 updates.
No, that would be the fault of an incompetent IT staff at the hospital. Updates should be rolled out in any enterprise of any size with WSUS, Altiris, or some similar product, that is controlled by policies and registry entries and Microsoft provides voluminous documentation as to how to do it correctly. If a Windows computer is automatically updating itself from updates.windows.com on Patch Tuesday then the IT staff are morons the deserve to lose their jobs.
Re: (Score:2)
No, that would be the fault of an incompetent IT staff at the hospital. Updates should be rolled out in any enterprise of any size with WSUS, Altiris, or some similar product, that is controlled by policies and registry entries
Yea sure it's the users fault there is no off switch and you have to run your own update server and hack the registry to get it to stop on some editions of windows.
and Microsoft provides voluminous documentation as to how to do it correctly.
Nothing short of voluminous documentation is required precisely because its completely inaccessible to mortals. I recall quite a large number of "IT" people scratching their heads because even with WSUS deployed Windows 10 clients were STILL getting updates from Microsoft.
If a Windows computer is automatically updating itself from updates.windows.com on Patch Tuesday then the IT staff are morons the deserve to lose their jobs.
Nothing like living on the edge. Some people like navigating forests of b
Re: (Score:2)
> WSUS
Even that doesn't guarantee some updates won't slip through. Our main product is a Windows app that won't run on 1903 since we're using a .NET framework that Microsoft hasn't bothered to update, but every so often a desktop will just update on its own. We haven't found a pattern yet as to which machines Microsoft decides to force an update.
Re: Overair or Online download and USB (Score:2)
. For example, Microsoft has been known to disrupt surgeries with forced Windows 10 updates.
Nobody in IT or medicine should care as no one's dumb enough to use general purpose, consumer-grade code for mission critical tasks, right?? (Please, no one answer; it's early and I want to pretend for a liittle longer.)
A phone? (Score:1)
Re: (Score:2)
So... no hands free kit for you then? No listening to your phone playlist while driving either then? How is the car supposed to get the software update? How is the car supposed to even know there is a software update? Should they build a separate phone into the car purely for the purpose of downloading the update? Or maybe wifi, so eve
Re: (Score:2)
Fun fact: Malware does not go away ... (Score:2)
if you carry it over on a physical device. It will gladly download to your USB stick just as well. ^^
So as long as there is a firewall, and you only open the port for that source and that connection, going the direct route is very much equivalent.
OK, with how insane the new generation of software development, you may actually be actively blocked from doing it that way, and you may have a point.
Re: (Score:3)
It uses the cell phone network to make outbound connection to known authorized server, handshake and authenticate each other. Then download instructions, packages if necessary and close the connection for a set period.
There are no WAN facing ports, Tesla does not accept any inbound connection from any server. AFAIK
Re: (Score:1)
Tesla is a not bloody modem connected to the general internet with open ports allowing random servers to send packets. It uses the cell phone network to make outbound connection to known authorized server, handshake and authenticate each other.
The "cell phone network" is insecure by design.
There are no WAN facing ports, Tesla does not accept any inbound connection from any server. AFAIK
Is this supposed to mean something?
Re: (Score:2)
Tesla is a not bloody modem connected to the general internet with open ports allowing random servers to send packets. It uses the cell phone network to make outbound connection to known authorized server, handshake and authenticate each other.
The "cell phone network" is insecure by design.
There are no WAN facing ports, Tesla does not accept any inbound connection from any server. AFAIK
Is this supposed to mean something?
Yes it does. Are you not familiar with internet protocols?
Re: (Score:2)
Yes it does.
No it does not. Not accepting inbound connections says nothing substantive about the security of the system and susceptibility to remote compromise.
Are you not familiar with internet protocols?
Are you not familiar with reverse shells?
Re: (Score:2)
You asked where it means something. I wrote that it does, because it clearly means something.
If you intended to convey a different messages like "Not accepting incoming connections is not a complete solution to securing a computer" why didn't you write that?
Re: (Score:2)
You asked where it means something. I wrote that it does, because it clearly means something.
It clearly means nothing.
If you intended to convey a different messages like "Not accepting incoming connections is not a complete solution to securing a computer" why didn't you write that?
I stand by my remarks. Not accepting "inbound connection" is a meaningless indication of system security.
For some perspective all Windows clients since XP SP3 come with a stealth mode firewall enabled by default.
Re: (Score:2)
Most likely Tesla cars join Tesla network using a direct channel to Tesla servers and nothing else. No one can even see the ports of the MCU other than Tesla servers. The packets travel on LTE network, but fully encrypted, like a VPN.
This OTA is probably way more secure than the USB stick you plug into a generic windows xp
Re: (Score:2)
Is Tesla running WinXP SP3?
What is the relevance? Would my remarks be any more or less valid if they were?
Does it allow users to install and run apps?
After spending 5 seconds on Google apparently the answer is YES. Not that this is in any way relevant or responsive to my remarks.
It has a infotainment subsystem running on a sandbox. It has one browser, locked down, running in a sand box. Most likely Tesla cars join Tesla network using a direct channel to Tesla servers and nothing else. No one can even see the ports of the MCU other than Tesla servers. The packets travel on LTE network, but fully encrypted, like a VPN. This OTA is probably way more secure than the USB stick you plug into a generic windows xp sp3 computer you are most familiar with.
Tesla vehicles have been successfully compromised in the past remotely via Cellular interface with successful injection of arbitrary messages into the vehicles CAN.
Tesla sandboxes have been successfully bypassed.
At the end of the day Millions of people have physical access to systems connected to Tes
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
So you are going to stop banking then? Stop getting on planes, trains and boats?
Certainly tried and true. Ascribe an absolutist position and let the derisive ridicule fly.
Oh you don't like your car tracking you everywhere you go?!?... Privacy is dead don't like it go live in the woods.
Re: (Score:3)
Tesla uses the industry standard practice of establishing a VPN with their secure server. The car then accepts incoming connections from the secure network, which is how they can do remote diagnostics.
The main danger is if someone gets into the Tesla network.
Re: (Score:2)
Tesla uses the industry standard practice
Industry standard practice yields industry standard results.
The main danger is if someone gets into the Tesla network.
What's not secure about a computer network of millions of vehicles?
Re: (Score:2)
Re: (Score:2)
It's actually an ARM based SoC which runs software from an MMC memory chip. If I were attacking it I'd start by attacking that as a way to connect a VPN to the Tesla network and start looking around.
You can root Tesla cars and look around their filesystem or make changes. People do it to fix cars that Tesla won't touch and enable features without having to pay.
Re: (Score:2)
Re: (Score:2)
They might want to protect their own network from those people though.
Re: (Score:2)
Re: (Score:2)
"Oh please oh please God who is all merciful and all knowing please take my paltry lines of code and bless them before they runneth on CPU pastures!"
I like to think that occasionally it works, and my usual response is, "I'll be damned".
The only person who thinks that there is software that doesn't have bugs is someone who hasn't coded, or has coded the equivalent of "Hello World".
Re: (Score:1)
OK Boomer.
Re: (Score:2)
VW probably wants you to take the vehicle to the dealership for an update. They might even charge you for it. VW != Tesla.
Re: (Score:2)
correction: article seems to think VW is going to do over-the-air updates. Hmm. How un-VW of them.
Re: (Score:2)
If you control a car anything like you control a keyboard, is it any wonder they don't want you having any control over it?
All Parking Spaces are Full - of 737s (Score:5, Funny)
Re: (Score:2)
I'm curious as to how you think downloading and installing via USB is going to be any different than downloading directly to the unit, besides being a bunch of extra time consuming steps..
It *will* come with very kampfy chairs, though! (Score:2)
Very kampfy!
https://i.kym-cdn.com/photos/i... [kym-cdn.com]
Orthographic error in the headline -Wolkswagon- (Score:4, Informative)
A new low for an editor.
And yes, i make orthographic mistakes too, but English is not my mother tongue (is spanish, I also speak french), and my job title is not editor.
Is not about being a grammar nazi (I kinda sorta hate those), but an editor in a headline is kinda low....
Re: (Score:2)
But then you replaced the *correct* "V" with a "W".
Credibility == gone.
Re: (Score:2)
And yes, i make orthographic mistakes too
... In your own very post. Please take right plam, apply it to face, and let out a deep sigh while you question the life you live, pointing out the mistakes of others while you yourself aren't able to get the very word you complain about correct.
Re: (Score:2)
Where can I get a plam?
Re: (Score:2)
Is not about being a grammar nazi (I kinda sorta hate those)
No, it's about being a typo Nazi, the lowest of the low.
Just to go all meta-nazi, there are typos and then there are serious lapses of thought. This doesn't look like a typo to me, because "o" is nowhere near "e" on a keyboard. The headline is using English "wagon" instead of German "Wagen". Well, if you insist on translating foreign names, at least be consistent and say something like "Folk wagon".
Nope, people glorifying stupidity are the lowest. (Score:3)
There is nothing more degenerate and retarded, than a grown man who is not only literally too stupid for literacy, but who somehow acts like that is a cool thing, and you are uncool if you are ... eww ... *smart*.
You know, like those 80s "sports jock" bros going "Neeeeeeeeeerds!". Before being smart became the new cool outside of a site formerly claiming "News for nerds.".
Like those Jack Packard types that go "I too, do not like to think.".
Reminds me of this Idiocracy scene: https://youtu.be/Mif5anwZeXY [youtu.be]
No..
Re: Nope, people glorifying stupidity are the lowe (Score:2)
Re: Nope, people glorifying stupidity are the low (Score:2)
Re: (Score:1)
Two words: in competent.
This is supposed to be a professional site. If you can't even fucking spell "Volkswagen," go run a Wordpress blog or something.
Designed to cheat emissions tests. (Score:5, Funny)
Lol. Replaced $90/hr engineers with $15/hr coders (Score:2)
Or their near senior engineers recently completed a coding bootcamp in Whatthefuckistan. They saved thousands of dollars in engineering costs.
Re: (Score:1)
So much of that, but how would they know? (Score:2)
I've done a lot of "great you got this working, Zach. This file can he replaced with the word 'join'. Dan, awesome work covering all of the corner cases. That's a useful functionality you added. SQL calls that a 'view' - it's built-in."
Newer peope often do a good job re-inventing the wheel. It worked well that the company kept me around to point out where the wheels were BEFORE they re-invented them most cases.
Software is Hard (Score:5, Interesting)
We know that a software company can build an electric car. What we are learning is if a car company can build software.
The guy in charge of VW effort in this regard is Christian Senger who talked about VWs effort in an interview [yourstory.com] in 2017. From the article:
"The other thing is that the car is now an operating system. The Volkswagen operating system will be available with a whole range of services from startups and partners. By 2020, we will have all this in place and we will be one of the largest producers of electric vehicles in the world. By then, I must also prove to the board of Volkswagen that this can be a sustainable business model."
I suspect he is under a great deal of stress at the moment, but it is a massive undertaking.
Re: (Score:2)
Well, VW did have software experience programming their emission controls to maximum pollution except when they were being tested. Maybe their current software problems are due to them adding in code to cheat the emission tests.
Re: (Score:1)
Silly. They maximized torque/horsepower, but not pollution. Maximizing pollution would have meant running a lot richer than that. Unburned fuel is a horrific pollutant.
The "cheat" software will not affect their all-electric vehicles (there are no emissions standards to cheat). It was funny once. Sort of. Now it's just repetitive and stupid.
Re: (Score:2)
Deluded. Granted you could produce as much pollutants as you like with bad combustion settings. However even with optimal combustion settings maximizing torque/horsepower means maximizing pollution. Pollutants are a residue of the combustion process. So VW knew they were maximizing pollution.
VW are no more able to innovate in the ICE area. They are stuck so they cheat. IMHO they should do less marketing, less advertising, less product placement and invest more in R&D.
Re: (Score:3)
Again with the word "maximizing". Are you the one who is deluded?
Yes, running lean improved fuel economy and increase NOx production. It didn't "maximize" it. It's not like VW is some Captain Planet villain that pollutes just for the sake of pollution. And you're still daft for trying to make stupid jokes about the "cheat" software ruining their electric cars. Like I said . . . sort of funny once, but it's done and dead.
VW is done with ICE engines. They won't be making any as much for PR reasons as an
Re: (Score:2)
Well, I'll go way out on a limb and speculate that if they had spent less time thinking about adding "a whole range of services from startups and partners", and had instead just focused on implementing the embedded logic required to control the electric drivetrain, then they wouldn't be in this pickle.
Re: (Score:2)
Sounds like they haven't got all the advertising, "value added" bloatwear and of course spyware in yet.
That's presumably what he means about "business model". Monetising the data and selling you shit in your own car for additional revenue.
Re: (Score:2)
I have a basic Seat Ibiza which is made by a company owned by VW.
The car is really basic, like 1980's basic, but it has a port on the dash where you can attach the optional extra flat screen/tablet thing. Suddenly you get access to all the stats of the car, maintenance cycles, temps, fuel consumption, etc.
The whole car is run by an OS, it has been like that for years, because it is much cheaper, more reliable and more accurate to run everything like that.
So no software is not hard for engineers who know how
Re: (Score:2)
The current Linux kernel CAN bus stack was written by Volkswagen, so I suppose they can build software.
German software sucks. (Score:1)
MP3 format has redundant data, both frame count and end of frame markers. Both must be legal and both must be consistent. Else it crashes. It takes forever to reboot the entertainment system. It skips tracks, gets confused, ...
BMW service blames my tracks. Your file is not correct. Not our problem.
Re: (Score:2)
Both must be legal and both must be consistent. Else it crashes. It takes forever to reboot the entertainment system. It skips tracks, gets confused, ...
Odd. I never have that problem with the CD player in my car. It just works. I must be doing something wrong.
Re: (Score:2)
Re: German software sucks. (Score:2)
Files? My BMW connects to BMW Music / Napster. It does have a hard drive, but I'll have to try it someday first. When I find the time to rip my CDs.
Re: (Score:2)
MP3 really needs to die. It was revolutionary when it was new, but it's seriously dated now - there are many newer, most advanced codecs now, all of which provide a more accurate reconstruction at any bitrate. That means smaller files.
Re: German software sucks. (Score:3)
Do cars really have a SIM card, nowadays? (Score:2)
Non-car-using Europan city dweller here. I never owned a car and it is usually months before I even sit in one. And my question is this:
How do modern cars get updates over the air?
Do you need to buy a SIM card for them, and stick it into the dashboard somewhere? Costing you money too? Or does the manufacturer have his own systems and just does it behind your back? What if I want to skip an update that is known to break my car? What about privacy and tracking? Do I need to cut the antenna or can I just insta
Re: (Score:2)
Good for monitoring the engine but also a recognised privacy problem, I am sure DuckDuckGo can tell you more.
Re: (Score:2)
Several of them will pair with your phone, and download updates via your cellphone over a secure VPN. In the case of Tesla you can choose to install or not, and apparently you can even choose to be in an "early adopter" group of firmware testers if you want. In the case of my wife's older Audi they plugged a unit into the CAN bus to install updates over the dealership's wireless LAN when it was in for scheduled maintenance (and again we could choose to update or not.)
Re: (Score:2)
Re: (Score:2)
In the case of Tesla, all of their cars (other than the original Roadster) have 3G (in older cars) or 4G LTE connectivity built in. (The old ones can be upgraded to 4G LTE; it requires a hardware installation and is not free.) An account comes with the car; you do not need to install your own SIM. Since July 2018, they have divided their 4G features into two tiers: Standard Connectivity and Premium Connectivity. Standard is free, and includes navigation and OTA updates. Premium costs $10/month, and adds liv
Maybe hire some real engineers next time? (Score:2)
This is undoubtedly another effect from trying to do software (which is about the most complex engineering discipline known to man) on the cheap.
Re: (Score:2)
This is undoubtedly another effect from trying to do software (which is about the most complex engineering discipline known to man) on the cheap.
I don't think software is anywhere near the most complex engineering discipline known to man. I suspect much more discipline is needed to build a rocket booster which can return to its launch site and land on its tail. Yes, that requires a lot of software that is hard to test, but it requires many other engineering skills too.
Re: (Score:2)
I don't think software is anywhere near the most complex engineering discipline known to man.
You do not seem to know a lot about it then. Sure, writing simple business logic is not that hard. But do some real computing, add reliability, performance and security and things look a bit different. Of course, not that many coders ever go into these depths, because most will completely fail to get anything working there.
Re: (Score:2)
I don't think software is anywhere near the most complex engineering discipline known to man.
You do not seem to know a lot about it then. Sure, writing simple business logic is not that hard. But do some real computing, add reliability, performance and security and things look a bit different. Of course, not that many coders ever go into these depths, because most will completely fail to get anything working there.
Actually, I do know something about it. I was a Principal Software Engineer at Digital Equipment Corporation, and my son used to work at the Redstone Arsenal in Huntsville, AL.
What a difference a name makes (Score:2)
Imagine, if you will, what this list of comments would look like if this were a story about Tesla instead of a story about VW.
Once again would be be ruled by derp galore and confronted with proof yet again how delusional Elon Musk is.
Re: (Score:2)
Re: (Score:2)
ID3? (Score:2)
I'm a music nerd, I know little about cars. Glancing at this headline briefly, my first thought was: ID3 tags? What do metadata containers have to do with cars? Heh.
OMFGIH (Score:2, Insightful)
Fuckwit, it's volks-wag-EN!!!!
When you say it like vollks-wag-ON you come across as a che-guevara-t-shirt-wearing jouno hack wannabe!
Fuck!
Oi, can we trade this lot of eds for live, real humans? I'm convinced they're all bots with names like Che, Fidel, Joe.
Stalin, not Biden.
Fucking wake up, eds!
Re: (Score:2)
A common problem with multinational brands. SEAT have it worse.
Emissions evasion sub-routine confused (Score:2)
That's normal (Score:2)
Their engineers concentrated on emission fraud the last 10 years.
This is the kind of thing.... (Score:1)
This is the kind of thing you would fire 100 the entire middle management team over. There is no excuse for this... at... all...
This was some middle management VPs trying to meet a schedule to get their bonuses... but let's face it, they've probably already left the company to rinse and repeat somewhere else.
Seen it before... (Score:2)
I've seen this sort of problem before, but with satellite receivers. And unfortunately, they were stationed all around the world. The company had to send techies to every location to re-flash the receivers. Why? Because someone accidentally sent out an over-the-air update that broke the receiver's ability to do over-the-air updates, or receive anything at all, for that matter. There was no local ability to do a rollback (this was the 90's, and the equipment was from the 70's).
And in true Dilbert fashio
OTA update is probably broken (Score:2)
Re: (Score:2)
SAP (Score:2)
I was going to mention SAP, but it is more than 30 years old.
Re: (Score:2)
And although developers are found world wide the best Linux desktop KDE (Plasma) originated in Germany.
Re: (Score:2)
Siemens is headquartered in Germany, and create some of the best real-time software out there. It's not flashy stuff, but if you're going to monitor a nuclear power plant, a rocket engine, or a refinery you *want* Siemens hardware and software.