Tesla Unveils New Dojo Supercomputer So Powerful It Tripped the Power Grid (electrek.co) 106
An anonymous reader quotes a report from Electrek: Tesla has unveiled its latest version of its Dojo supercomputer and it's apparently so powerful that it tripped the power grid in Palo Alto. Dojo is Tesla's own custom supercomputer platform built from the ground up for AI machine learning and more specifically for video training using the video data coming from its fleet of vehicles. [...] Last year, at Tesla's AI Day, the company unveiled its Dojo supercomputer, but the company was still ramping up its effort at the time. It only had its first chip and training tiles, and it was still working on building a full Dojo cabinet and cluster or "Exapod." Now Tesla has unveiled the progress made with the Dojo program over the last year during its AI Day 2022 last night.
The company confirmed that it managed to go from a chip and tile to now a system tray and a full cabinet. Tesla claims it can replace 6 GPU boxes with a single Dojo tile, which the company claims costs less than one GPU box. There are 6 of those tiles per tray. Tesla says that a single tray is the equivalent of "3 to 4 fully-loaded supercomputer racks." The company is integrating its host interface directly on the system tray to create a big full host assembly. Tesla can fit two of these system trays with host assembly into a single Dojo cabinet. That's pretty much where Tesla is right now as the automaker is still developing and testing the infrastructure needed to put a few cabinets together to create the first "Dojo Exapod."
Bill Chang, Tesla's Principal System Engineer for Dojo, said: "We knew that we had to reexamine every aspect of the data center infrastructure in order to support our unprecedented cooling and power density." They had to develop their own high-powered cooling and power system to power the Dojo cabinets. Chang said that Tesla tripped their local electric grid's substation when testing the infrastructure earlier this year: "Earlier this year, we started load testing our power and cooling infrastructure and we were able to push it over 2 MW before we tripped our substation and got a call from the city." Tesla released the main specs of a Dojo Exapod: 1.1 EFLOP, 1.3 TB SRAM, and 13 TB high-bandwidth DRAM.
The company confirmed that it managed to go from a chip and tile to now a system tray and a full cabinet. Tesla claims it can replace 6 GPU boxes with a single Dojo tile, which the company claims costs less than one GPU box. There are 6 of those tiles per tray. Tesla says that a single tray is the equivalent of "3 to 4 fully-loaded supercomputer racks." The company is integrating its host interface directly on the system tray to create a big full host assembly. Tesla can fit two of these system trays with host assembly into a single Dojo cabinet. That's pretty much where Tesla is right now as the automaker is still developing and testing the infrastructure needed to put a few cabinets together to create the first "Dojo Exapod."
Bill Chang, Tesla's Principal System Engineer for Dojo, said: "We knew that we had to reexamine every aspect of the data center infrastructure in order to support our unprecedented cooling and power density." They had to develop their own high-powered cooling and power system to power the Dojo cabinets. Chang said that Tesla tripped their local electric grid's substation when testing the infrastructure earlier this year: "Earlier this year, we started load testing our power and cooling infrastructure and we were able to push it over 2 MW before we tripped our substation and got a call from the city." Tesla released the main specs of a Dojo Exapod: 1.1 EFLOP, 1.3 TB SRAM, and 13 TB high-bandwidth DRAM.
Embarrassing for Tesla (Score:5, Insightful)
Re: (Score:2)
They probably just shorted the circuit, I can do that too.
Re: (Score:2)
Yeah I think it's more likely that Musk just didn't adequately coordinate with the local utility.
Re: Embarrassing for Tesla (Score:2)
I donâ(TM)t think Musk personally would be doing that.
Or is that how they the logic works? Musk is never personally responsible, unless itâ(TM)s something negative, then itâ(TM)s all on him?
Re: Embarrassing for Tesla (Score:2)
Isn't this something negative?
Re: (Score:2)
Yes, that's my point. The narrative goes: Anything good is due to the team. Anything bad is because of Musk.
Re:Embarrassing for Tesla (Score:5, Informative)
Re: (Score:2)
That they would trip breakers outside their infrastructure shows poor planning.
Why? If Tesla's system is designed for 2 MegaWatts and all the breakers are sized accordingly, but powering up their systems to 1.5 MegaWatts caused the local grid to exceed its capacity to both Tesla and all its other customers, then everything is working as intended. In a made up example, if the local substation is designed for 6MW and it was already supplying 5MW, then Tesla's additional 1.5MW would cause it to trip as designed.
It could be that Tesla needs to find a supplier that can supply 2MW without
Re: (Score:3)
Embarrassing for the headline writer too. Ancient mainframes used vast amounts of power for their vacuum tubes, it didn't make them powerful in terms of performance.
We need to some some performance/Watt numbers for this thing to know if it's really any good.
Re: (Score:2)
Isn't this kind of basic power engineering that is well solved for every data center and industrial users. You should know what your power draw is and the voltage/current of your feeds. It's not completely simple math, but a well-solved engineering problem.
Doesn't show a lot of confidence in planning if you can't do electrical 101. Also, data center in Palo Alto? Doesn't seem like the ideal locations as the data center can be located almost anywhere. Real estate costs, grid capability and costs per megawatt
Alternate headline: (Score:5, Insightful)
"Tesla Unveils New Dojo Supercomputer So Incompetently Run It Tripped the Power Grid"
Doesn't matter how big your thing is. Tripping the substation in the city exposes your incompetence and inability to manage your contractually agreed load.
Nice computer, but not a shining review of Tesla's business practices.
Re: (Score:2, Interesting)
At least nobody probably died this time, but powering things in Texas comes with a known risk. Their power grid is not reliable.
https://en.wikipedia.org/wiki/2021_Texas_power_crisis [wikipedia.org]
Re: (Score:2)
Re: (Score:3)
Re: (Score:2)
What about Palo Alto, Virginia or Palo Alto, Pennsylvania?
Re: (Score:1)
"Texas power grid continues to prove it can't handle supplying power".
Except that isn't the case here and has nothing to do with the topic at hand. Save your comment for actual grid problems rather than a customer triggering circuit protection.
Re: (Score:2)
Imagine a Beowulf cluster of these. ;-)
They could take down the whole western grid!
olde school slashdot! (Score:2)
1 Use old meme ...
2
3 Profit!
Re: olde school slashdot! (Score:1)
Re: (Score:2)
All your Dojo are belong to us.
Re: olde school slashdot! (Score:2)
Someone set us up the Dojoâ¦
Re: (Score:2)
In America, people kill power station. In Russia, power station kill you!
Am I doing this right?
Re: (Score:2)
Re: (Score:2)
Would have warmed my cold, dead, heart if those kids on my lawn hadn't stolen it along with my gnomes years back.
What happened to the Star Trek Red Shirt dude? (Score:2)
who tried to disconnect Dojo's power conduit to stop it from crashing the Palo Alto grid?
Re: (Score:2, Insightful)
It's the kind of statement that pseudotechie Musk fanboys just love. Haha we just blew up another extremely expensive rocket, isn't science great!
Re:Alternate headline: (Score:5, Interesting)
I may be a complete waste of time and energy anyway.
Look at how Waymo's self driving cars work, compared to Tesla's. Waymo uses lidar and high end cameras to capture a huge amount of information about the environment. While some machine learning is used to identify objects, it is working with a lot more information than Tesla's camera only solution.
Recently there have been multiple incidents of Tesla cars ramming into motorcyclists at high speed because they see two tail lights and assume from the distance apart that they are seeing a car far off in the distance. Waymo's cars have lidar too, so they can see that it's a bike even in complete darkness and poor weather when their cameras cannot. They have algorithmic checks in place so that even if the machine learning thinks it sees a far off car, the algorithm can see that it's too narrow and the camera got the distance completely wrong thanks to the lidar, so ignore that.
Secondly, Waymo uses algorithms to determine the path the car will take. It uses algorithms to determine the possible paths that other vehicles, cyclists and pedestrians may take. Tesla is trying to teach machine learning to drive. An algorithm can be audited and humans can make decisions about what kind of driver it should be by directly controlling its decision making logic. Machine learning is a black art where even the most accomplished developers can't stop their chat bots becoming Nazis within 5 minutes of meeting Twitter, or remove racial factors from decision making. Some of them can't even get their cameras to recognize darker skin, kinda important when it's your car's vision system.
The big Tesla difference is moxie (Score:2)
Tesla's autonomous systems are obviously not as good as Waymo's. However, they have the only commercial system past Level 2 on the road (not counting the very few Honda and Mercedes official Level-3 cars) because they have enough moxie to pass off an attempted Level-3 system as a legal Level-2 while marketing as a Level-3. Well, at least until the government starts doing its job and calling Tesla on their marketing fraud. The government has shirked its responsibility for many years, so it's still not cle
Re: (Score:2)
I may be a complete waste of time and energy anyway.
Don't be so hard on yourself, bro.
Re: (Score:3, Insightful)
Look at how Waymo's self driving cars work
I have and I really root for Tesla in this case. We should be putting every effort into decent machine learning. Throwing very sensitive, expensive, and really high end sensing technology into every car is not a sustainable future.
We live in a world where people simultaneously complain about things being both too dumb and too expensive. Heaping more costs on it isn't the answer.
Tesla has a way to go but at they are at least leading towards a reasonable solution. Waymo's (and indeed basically every other com
Re: (Score:3)
The tech will get cheaper as commercial users adopt it. Makes financial sense for taxis and haulage.
That's the same plan Musk had, except he started with luxury consumer models instead of commercial ones. Interestingly rivals like BYD and CATL went with commercial vehicles, and now have the best battery tech.
Re: (Score:2)
The tech will get cheaper as commercial users adopt it. Makes financial sense for taxis and haulage.
There is a fundamental floor to tech costs. LIDARS are highly complex pieces of gear. They won't ever be as cheap as not having them at all (LIDAR units still come with advanced cameras too so you don't even get to substitute the cost in this case).
LIDARS in their current form will never make it to general purpose mass purchased consumer vehicles. They may find a home in the luxury vehicle market, but something needs to fundamentally change in the technology to make it viable.
Re: (Score:2)
Radar used to be expensive and highly complex, now you can buy single chip solutions and it is mandatory on new cars.
Re: (Score:2)
Radar used to be expensive and highly complex, now you can buy single chip solutions and it is mandatory on new cars.
Indeed it did. However radars are still quite expensive and still more expensive than nothing at all. Speaking of Radars are nothing like they were in the past so when you are thinking about your next reply consider the fact that I said "LIDARS in their current form"
But congrats on moving the goalposts. You've jumped from economies of scale to magic technology changes to make your argument, and for the latter I absolutely agree with. I just caution you to not hold your breath, especially considering LIDARS
Re: (Score:2)
I'm curious what's going to happen when there's thousands of these radars on every street, driving around picking up thousands of stray signals.
Hopefully the DSPs will be able to pick their coded signal out of the noise in time to make use of it.
Has anyone seen any research into this area?
Yeah, but marketing (Score:2)
In the old days it was something like, "Car so powerful it will shake the doors off your neighbor's house."
Of course if it were true it would just be a problem caused by a malfunctioning exhaust, poorly balanced engine, faulty motor mounts or any number of--DID I MENTION IT BLEW THE DOORS OFF MY NEIGHBOR'S HOUSE???
Sales are through the roof. Which now needs to be patched. ALL RIGHT!!! NEW SHINGLES!!!
Re: (Score:2)
Indeed. Gross incompetence, and maybe even criminal malfeasance.
Re: (Score:2)
Yes to the former, no to the latter. When you're a high volume customer you will have a dedicated feeder (more likely two) from the substation. There's nothing criminal about breaching a contract, and if something were to have gotten damaged here it would certainly have been the fault of the energy provider not setting up the feeder protection correctly.
Bottom line is you as an end user downstream of any protection (be it large 110kV feeder or the service fuse on your house) can never do anything on your si
Re: (Score:1)
Bottom line is you as an end user downstream of any protection (be it large 110kV feeder or the service fuse on your house) can never do anything on your side of the electrical line to be considered criminal.
Sure you can. Intentionally connect a generator back-feeding power that's significantly out of phase.
Re: (Score:2)
Bottom line is you as an end user downstream of any protection (be it large 110kV feeder or the service fuse on your house) can never do anything on your side of the electrical line to be considered criminal.
Sure you can. Intentionally connect a generator back-feeding power that's significantly out of phase.
For example. Or probe for resonances with a switched load.
Re: (Score:3)
Sure you can. Intentionally connect a generator back-feeding power that's significantly out of phase.
Have you done that before? Or seen it done? I have*. The outcome was sending the generator and turbine back to Siemens for a $5million repair job after the sheer coupling sheered and the bearings couldn't handle the force causing the rotor to contact the stator (luckily only slightly) in the generator. Alstrom also replaced our breaker which is standard practice for that gear when it trips.
The utility provider? Well nothing happened to them, they just gave us a call to say a breaker on our feeder tripped op
Re: (Score:2)
Poor planning (Score:5, Insightful)
They're not the least bit embarrassed by this?
Re: (Score:3, Insightful)
Much like the republican party, you can’t embarrass someone without shame.
Re: (Score:3)
Tesla probably did it as a publicity stunt.
Also much like the party.
Re: (Score:2)
Tesla probably did it as a publicity stunt.
Also much like the party.
Yep. In any reasonable country, doing this intentionally would result in a huge fine and may well result in criminal charges for endangering critical infrastructure. There is absolutely no need for this type of "test". You ask. The power station people can tell you everything, down to the impedance of the network, because they need to know in ordert o do their jobs.
Re: (Score:1)
Its quite hilarious that you attempt to pin this on any entity other than CA's power grid itself which is very well known to be unstable.
Re: (Score:2)
Even as of "right now", the CAISO is reporting 75% of the load is being served by renewables.
They also tripped a sub-station, not the grid. And probabl
Re: (Score:1)
The grid is national, news for you. 80 percent fossil fueled is what makes and powers everything in these united states.
Telsa should locate their high energy suck stuff to where it can be green powered.
What a disgrace.
Re: so it isn't so (Score:2)
I can trip the power grid with copper pipe (Score:2)
put in the right spot. That copper pipe must be doing zillions of floating point computations per microsecond!
Re: (Score:2)
Wow, that's awesome. What do your charge for CPU time on "Copper Pipe"? My VC buddies and I are very interested...
Re:I can trip the power grid with copper pipe (Score:4, Interesting)
True story: Downtown Santa Cruz, CA was once blacked out for several hours due to a mylar balloon contacting power lines. You should invest in that. Not only has it demonstrated the power to blow the grid, some of the balloons are "in the cloud".
Re: (Score:2)
Shut up and take my money!!
Re: I can trip the power grid with copper pipe (Score:1)
I'll take your money...but I won't plow your driveway.
Intentional probably (Score:3)
"Hey look at us we made a computer so powerful it tripped the breakers! We are doing so much sci fi shit over here."
But I am curious what they're doing with it.
Re: (Score:2)
they explained it: the main use is training their neural network models
Re: (Score:2)
I'm aware, just was curious about details more than anything. I know, we'll not get them just saying.
Re: (Score:2)
Watch the Tesla AI Day video from last Friday. They are pretty detailed in explaining what they are doing with this.
Tesla Corporate charter (Score:2)
1. YOLO
2. LOL
Re: (Score:2)
Re: (Score:2)
Re:Fucking Slashdot (Score:5, Insightful)
Had you bothered to read the article from which the headline was taken word for word, you would have found this:
Bill Chang, Tesla’s Principal System Engineer for Dojo, said during
“We knew that we had to reexamine every aspect of the data center infrastructure in order to support our unprecedented cooling and power density.”
They had to develop their own high-powered cooling and power system to power the Dojo cabinets.
Chang said that Tesla tripped their local electric grid’s substation when testing the infrastructure earlier this year:
“Earlier this year, we started load testing our power and cooling infrastructure and we were able to push it over 2 MW before we tripped our substation and got a call from the city.”
That is hardly an endorsement or marketing for Tesla. It means they failed on a number of issues, not least of which was to notify the local power company what they (Tesla) were planning to do, didn't coordinate with said power company on the timing of load testing, didn't have a back up plan despite touting their battery backups for municipalities, and didn't have a fallback plan in case something like this should happen. That's called poor project management and a failure.
Put another way, "We'll just fuck around and if something goes wrong say, "Oops!"" Not someone you want to do business with.
Re: (Score:2)
I wonder if it's less scandalous than it looks. Substations are presumably sized to be economical for expected peak load plus a safety margin. Adding two megawatts could break that assumption with only normal incompetence.
Re: (Score:3)
In order for Tesla to have done this they would had to remove every breaker between their system and the grid, and likewise the power company would have to have nothing in place to protect their grid from rogue users like Tesla.
They only tripped the incoming feeder from the city substation. Literally nothing needed to be removed or changed and absolutely everything worked as expected (electrical protection wise). It was gross incompetence on the part of Tesla to draw more than their contractually agreed upon demand, and even more so that a load testing activity wasn't co-ordinated with the grid operator, but the power company very much had a thing in place and it worked exactly as designed to protect the grid from Tesla's shenanig
Power grid trippin' (Score:2)
What the hell kind of measurement is that? Maybe it was a hot day, and the load was around peak already. Also, how big was the power grid? The size of a football field or a banana? Maybe it was a weak-ass sub-station? Jeeze, it has so many variables.
Similar headlines (Score:5, Funny)
"Engineers build road so powerful it causes a rock slide!"
"Mr. Christie builds Oreo so powerful it breaks all the cookie-making machines in the factory!"
"Putin starts war so powerful that it causes Russia to run out of military equipment!"
What Bullshitting Hype (Score:4, Interesting)
Substations belong to the electricity company. A factory like Tesla's will have a substation dedicated to it, probably several. Jeez, my suburban street has a substation, it's right next to my back garden. Every substation will have some kind of fuse or circuit breaker to limit maximum current. In turn, every property served by the substation will have some kind of fuse or circuit breaker to limit maximum current, to something less than that in the substation - unless the user puts a copper rod in it. In the UK it is illegal to mess with your property's main fuse.
In this case, allowing for confused journalists, either the substation breaker or the downstream Tesla breaker was tripped. It was Tesla's incompetence that their computer's local supply's fuse was such that didn't blow first (it did have one didn't it?).
Believe it or not, the electricty supply company guys do monitor consumption in the various parts of their system. No doubt they saw the sudden drop in the supply to Tesla and phoned them up to ask if there was a problem - not that is would have been an issue, but because their watch is so fucking boring without these little events. I don't suppose the rest of their grid was affected in the slightest unless the design of it is really fucked up.
Re: (Score:2)
This is just Tesla seeking publicity. They love it.
Only because people are stupid. What Tesla demonstrated here was _incompetence_.
Re: (Score:2)
Jeez, my suburban street has a substation, it's right next to my back garden.
A more apt comparison between you and Tesla would be your service fuse. When you get up to shenanigans your service fuse pops. When Tesla gets up to shenanigans the substation trips their (dedicated) feeder.
Re: What Bullshitting Hype (Score:2)
> This is just Tesla seeking publicity. They love it.
It was a brief line in a 4 hour technical presentation.
The news sites which will use anything âoeElonâ or âoeTeslaâ in a headline for clicks are the ones who love it.
Basically incorrect (Score:3, Insightful)
Tesla was not boasting about how much power their computers used but how thermally efficient their cabinet and cooling systems were, allowing them to pack more computing power into less space without it failing.
Re:Basically incorrect (Score:4)
The headline and most comments are basically incorrect.
Most comments here are shaming Tesla for not co-ordinating a test with the power company and for drawing enough power that they tripped the feeder from their utility provider. Those comments are completely correct, because what they did was so stupid that it completely overshadows the stupid headline.
Re: (Score:2)
RTFA before slamming Tesla (Score:2, Informative)
“Earlier this year, we started load testing our power and cooling infrastructure and we were able to push it over 2 MW before we tripped our substation and got a call from the city.”
The computer did not overload the circuit, they were testing to determine how much power would be reliably available. This seems like a good idea to me as you would not want to have power problems when doing a training run on a huge dataset. Most substations from what I se
Re: (Score:2)
No one is slamming Tesla because their computer tripped, it's that the article treats it as a badge of honor that Tesla tripped a substation.
When you do have an operation of this scale, you coordinate with the utility. You don't unilaterally do a test and surprise the utility and *then* 'got a call from the city".
I consult on this scale of things frequently and this sort of outcome as characterized by Tesla would be a sign of doing it wrong.
But either way, even if it was a coordinated test done correctly a
Re: (Score:2)
we were able to push it over 2 MW before we tripped our substation and got a call from the city.
Still a lack of coordination from Tesla. "getting a call from the city" -- WTH are you guys doing going from 0 to 2 MW in a minute without letting us know in advance? If it was a fault from the city, the factory would call the city, not the opposite. At least at my place when power goes down for no good reason, it's us (the factory) calling the electric company, asking them what is wrong on their side, and let them know we'll ask for compensations for our business losses.
Re: (Score:2)
This is actually terminally stupid. What you do is you ask your supplier how much power you can safely draw, because they _know_.
If you did something as demented here, you would probably get hit with a huge fine an a credible threat to disconnect you. Endangering critical infrastructure is not something smart people do.
Re: RTFA before slamming Tesla (Score:2)
Re: (Score:2)
This seems like a good idea to me
Really? Can you not think of other, possibly less sociopathic, ways in which they may have obtained this information? Even if it was not available from the utility, do you think perhaps it might be a good idea to give them a heads-up that you are attempting to intentionally break their infrastructure before doing so? Have you lived in a civilized society long, or are you new here?
Power Usage Not Equal to Computing Power (Score:2)
Re: (Score:2)
Just imagine... (Score:2)
Re: (Score:2)
...a beowulf cluster of these!
I imagine that *way* back, "powerful" would be a cluster of Beowulfs [wikipedia.org] ... :-)
elon haters coming out the wood work (Score:1, Informative)
elon creates yet another amazing product and somehow all the top comments are nothing but derision, on a tech site no less.
this site has become festered with the woke mob, blinded by politics.
Re: elon haters coming out the wood work (Score:2)
Okay, what exactly is the amazing product?
Re: (Score:2)
Umm ... (Score:2)
Tesla Unveils New Dojo Supercomputer So Powerful It Tripped the Power Grid
You mean a supercomputer that *uses* so much power it tripped the grid. "Powerful" in context like this implies computing power, which isn't necessarily related to power consumption -- though it often is, but there's not a hard, direct correlation..
More like "Tesla too stupid to calculate power" (Score:2)
It is not a great accomplishment to have a computer or really anything consume a lot of power. It is a great demonstration of incompetence though to overload a local power grid.
California? (Score:2)
Isn't California the place with the most stressed power grid anywhere? Why not next to a hydro station in Oregon? Or in the Texas desert with ten acres of solar panels?
"So powerful it cut electricity to humans" (Score:2)
And it didn't ask permission, or say it was sorry, which shows it has studied well the lessons to be learned from politics.
Not impressive. (Score:2)
Something missing here (Score:1)
Continuing the tradition. (Score:1)
Re: (Score:1)
Re: (Score:2)
For a sec I wondered if it was a dam that hovers over water and thought... wow, this sounds really cool. I am now all disappointed :p
Re: (Score:2)
Now there's a title.
They suck at the datacenter (Score:1)
Hmmm (Score:1)
So, the Dojo drawing 2 MW for the 1.1 EFLOPS seems quite a bit impressive.