'AI Ambition is Pushing Copper To Its Breaking Point' (theregister.com) 40
An anonymous reader shares a report: Datacenters have been trending toward denser, more power-hungry systems for years. In case you missed it, 19-inch racks are now pushing power demands beyond 120 kilowatts in high-density configurations, with many making the switch to direct liquid cooling to tame the heat. Much of this trend has been driven by a need to support ever larger AI models.
According to researchers at Fujitsu, the number of parameters in AI systems is growing 32-fold approximately every three years. To support these models, chip designers like Nvidia use extremely high-speed interconnects -- on the order of 1.8 terabytes a second -- to make eight or more GPUs look and behave like a single device.
The problem though, is that the faster you shuffle data across a wire, the shorter the distance at which the signal can be maintained. At those speeds, you're limited to about a meter or two over copper cables. The alternative is to use optics, which can maintain a signal over a much larger distance. In fact, optics are already employed in many rack-to-rack scale-out fabrics like those used in AI model training. Unfortunately, in their current form, pluggable optics aren't particularly efficient or particularly fast.
Earlier in 2024 at GTC, Nvidia CEO Jensen Huang said that if the company had used optics as opposed to copper to stitch together the 72 GPUs that make up its NVL72 rack systems, it would have required an additional 20 kilowatts of power.
According to researchers at Fujitsu, the number of parameters in AI systems is growing 32-fold approximately every three years. To support these models, chip designers like Nvidia use extremely high-speed interconnects -- on the order of 1.8 terabytes a second -- to make eight or more GPUs look and behave like a single device.
The problem though, is that the faster you shuffle data across a wire, the shorter the distance at which the signal can be maintained. At those speeds, you're limited to about a meter or two over copper cables. The alternative is to use optics, which can maintain a signal over a much larger distance. In fact, optics are already employed in many rack-to-rack scale-out fabrics like those used in AI model training. Unfortunately, in their current form, pluggable optics aren't particularly efficient or particularly fast.
Earlier in 2024 at GTC, Nvidia CEO Jensen Huang said that if the company had used optics as opposed to copper to stitch together the 72 GPUs that make up its NVL72 rack systems, it would have required an additional 20 kilowatts of power.
Speed of light (Score:5, Informative)
Re:Speed of light (Score:5, Informative)
Re: (Score:2)
Re:Speed of light (Score:5, Interesting)
I suspect the human brain uses something similar to a LLM because of the similar things it does. But it also does stuff the LLMs don't, and the wetware is very different from hardware. Instead of picturing a 4096 node transputer network or even a million 4004s which is probably closer, imagine 80+ billion nodes each capable of storing and processing a bare handful of floats but each with connections not just to immediate neighbors [nih.gov] in every direction but also to other nearby neighbors on the other side of them, and maybe with storage in the connections themselves as well. And we're trying to simulate it on not only fundamentally but also massively different equipment and with a fractional understanding, it's pretty funny really when you think about it, and then think about what thinking means some more :)
The human brain achieves its relatively low power dissipation with voltages commonly measured in tens of millivolts, and yet it is thermally sensitive and also produces substantial heat [nih.gov], some of which is removed by air cooling of blood passing through the sinuses [physiology.org].
Re: (Score:2)
> relatively slow connections
The neuronal connections are slow but we're just now beginning to measure the terahertz waves coming off the microtubules in the neurons themselves which appear to have tubulin crystals that oscillate and send information through entanglement across the brain network.
Like in chips we can imagine a power-side bus and a data-side bus and both are valuable.
Both anaestheics and psychedelics are shown to dampen or modulate the resonance of these crystals in the Layer-5 pyramidal n
Re: (Score:1)
I've made the following comment numerous times to my teams when waiting for software or patches or updates to install: Considering this is moving at the speed of light, it's taking an awfully long time to finish.
However, since the signal is moving at only 70% of the speed of light, it makes sense why it takes so long. After all, its only moving at 130K/kps rather than the standard 186K/kps.
For those who are wondering, yes, I am being sarcastically facetious.
Re: (Score:2)
I've made the following comment numerous times to my teams when waiting for software or patches or updates to install: Considering this is moving at the speed of light, it's taking an awfully long time to finish.
However, since the signal is moving at only 70% of the speed of light, it makes sense why it takes so long. After all, its only moving at 130K/kps rather than the standard 186K/kps.
For those who are wondering, yes, I am being sarcastically facetious.
RF velocities in wires is just plain weird. A good example is antennas, which must take into account the coax or ladder line, then of course the antenna itself.
The velocity factor even has a difference if the antenna is bare metal, or insulated. On a wire antenna, if you use insulated wire, the antenna will be a slightly different length shorter than a bare wire.
It is a real trap for chip design. Using a rule of thumb 1 foot per nanosecond, and trying to get sub nanosecond performance, we end up with c
Re: (Score:2)
Clock distribution networks are all about making sure the clock edges stay aligned enough across the circuit. ("Enough" normally means within the tolerance of setup and hold times.)
Are people still trying to build self-clocking data flow processors, or is the overhead too high compared to traditional clocked designs?
Re: (Score:2)
David Gerrold in "when Harlie was 1" mentioned this in respect to the Graphic Omniscient Device
That's not how the world works. (Score:2)
Yeah, the original article is full of shit. Datacenter size and needs have been growing for a lot longer than this new fake AI schtick has been going up. The BS AI chips that just chew raw power to find exponential interactions between n^1 of data are just today's hot air. When real AI gets here it will be real AI, much more useful, and much more efficient. And yes, at that point those companies still doing the AI Schtick will be done. So will NVIDIA's stock price.
Previous poster says:
> ... the spee
Re: (Score:2)
Re: (Score:2)
It all gets crazier, when you remember the speed of light, and that an electrical signal only travels at about 70% of that. If you have, say, a 5GHz signal, that means that in one complete period (which takes 1 / 5*10 second), the signal will only travel about 4cm. Assuming you are trying to get a bunch of separate components to work in sync, well, that's not going to work. Work in parallel, maybe, but never synchronously.
Yup, and even at the unemcumbered speed of light in free space, there is a hard speed limit.
AI is going to be a bubble like no other. My guess is that the incredible amount of power they will suck out of the grids will affect the price of electricity to the point where normies will just install solar and go off the grid. If you don't live in a development, it is already often cheaper to go to a discrete solar array.
Then when the AI bubble bursts and the sunk money and the gigawatt power isn't neede
Re: (Score:3)
Work in parallel, maybe, but never synchronously.
Of course it will work, it's just an engineering problem to make it happen. Digital signalling has worked with this problem for an eternity, it is especially important when signals are sent differentially. Many things by necessity on your motherboard need to happen in sync and despite components being close together you may find that the actual signals arrive at wildly different times.
This is why when you look at a high speed parallel interface you often see squiggly lines, that is an engineering attempt to
The BS is strong in this one, Luke (Score:3)
With 20kW for 72 devices, it would come down to an extra 270 watt per device.
Only a hefty CO2 laser can deliver that amount of power.
Not to forget that copper interconnects don't go from chip to chip directly in between racks.
Re: (Score:2)
Well, the liar-in-chief at Nvidia obviously cannot do math.
They're way into la-la land (Score:4, Insightful)
Fundamentally, the total power required is already stupid high with nothing to show. Spreading it out isn't any sort of fix. All you're doing is upping the total power needs even further.
Re:They're way into la-la land (Score:4, Informative)
with nothing to show
AI has plenty to show. You have just fallen into the trap of thinking AI = ChatGPT / CoPilot. In reality we've used large models of trained neural networks for many years doing many different things and the output of it is absolutely amazing.
For example one tool I have uses an AI model trained only on astronomy images to correct for motion and blur in stars. It achieves in seconds orders of magnitude better results than some people whose profession is to work with deconvolution can do in many hours. Even in industry such tools are heavily used such as supercomputers you find at some oil companies - what once was sitting solving wave equations is now sitting training AI models because it turns out it is actually better at classifying oil fields than people were at throwing equations at the problem. We've blown a lot of computing power and time at training models to identify how control schemes change as well pressure depletes, we have trained models to identify cancer on MRIs with accuracy better than professional diagnosis for some cancers.
To say it has nothing to show, is to say "I've not looked - and I haven't found anything". AI is far bigger than a shitty chatbot that most of Slashdot associate with it.
Re: (Score:2)
For example one tool I have uses an AI model trained only on astronomy images to correct for motion and blur in stars.
Can you share the name of the tool?
speed vs. power vs. distance (Score:2)
>"Unfortunately, in their current form, pluggable optics aren't particularly efficient or particularly fast."
It is every bit as fast as anything you can do with copper, and potentially very much faster (and not because of the "speed of light" but due to the properties of light, like lack of EMI). As far as power efficient, I would say that is mostly because of distance. Fiber modules, in the current form, are designed to push a readable signal for long (hundreds of meters) or very long distances (kilom
Re: (Score:2)
If you RTFA that's what this article is actually about. There's a few paragraphs on how copper is currently a limiter and then a whole bunch of them on a startup called "Ayar Labs" which has developed and is continuing to develop optical interconnects which are going into products from Intel and Fujitsu.
Re: (Score:2)
Agreed that a different fibre standard could help - but... it's been a long time since I really knew this stuff... in an optical connection, you're converting an electrical signal to light, which is inherently inefficient. What's more, the power of the light isn't used at the other end, in effect it's thrown away and new power is used to convert it to an electrical signal again. Contrast to a copper cable, where the electrical signal is (probably) boosted a bit to send it out, but the energy contained withi
What need to support larger AI models? (Score:2)
"Much of this trend has been driven by a need to support ever larger AI models."
What need? It's a want, not a need, and it's the direction because tech bros can't think of anything else.
Just because "the number of parameters in AI systems is growing 32-fold approximately every three years" doesn't mean it should or that it will continue to. One good reason is to avoid "pushing copper to its breaking point". Hell, even Intel recognizes this 25 years ago. There comes a time when turning the knob up stops
Re: (Score:2)
What need? It's a want, not a need, and it's the direction because tech bros can't think of anything else.
Exactly. But these assholes always like to pretend what they are doing is critical and will save the world. In this case, very obviously not so.
That said, there seems to be some value in small, specialized LLMs.
Re: (Score:2)
What need? It's a want, not a need, and it's the direction because tech bros can't think of anything else.
The accuracy and usability of a model is dependent on its size. You can see that in the development of models even over the past few years. The bigger the data set the better the result.
You may be happy that your AI assisted cancer diagnosis is basically no better than a coin flip, but other people may not want to stake their lives on it.
Re: (Score:2)
Just because "the number of parameters in AI systems is growing 32-fold approximately every three years" doesn't mean it should or that it will continue to.
Well apparently they have hit a wall, throwing more parameters into the models is not significantly producing better models. The only measurable increase in ChatGPT-5 is in language skills. Logic, coding etc. are the same, despite having significantly more parameters. So if more parameters is not making better models we might have reached the peak ability of large language models. Unless they pull a bunny out of a hat ChatGPT-4 might be as good as it gets.
"Bill Gates feels Generative AI has plateaued,
And all for "better crap" (Score:4)
Does the human race not have any real problems to solve?
Re: (Score:2)
Well, current gen AI is rapidly solving the problem that we have struggled and spend incredible amount of resources to solve since the first meeting between two different tribes. That of a language barrier.
This alone elevates LLM into the niche of "uniquely valuable tool that is actually solving a problem we failed to solve for entire existence of our species".
Re: (Score:2)
Not really. Automated translation _is_ has become pretty good, but it does not use LLMs. The neural networks used for automated translations use proper supervised learning, which is vastly superior if you can afford it. LLMs, on the other hand, do only use weak forms of supervised learning or something that is called "self-supervised". Hence LLMs do not actually play a role here.
Re: (Score:2)
Scientists and engineers are working on developing AI that can do useful stuff. Alpha Fold is a good example
Unfortunately, most of the consumer facing AI is just crap generators
Re: (Score:2)
AlphaFold is not an LLM.
Green New Deal (Score:2)
They better figure out efficient opamps and transducers soon because the Green New Deal needs all of the world's copper production through 2169 by 2030.
Scratch that - that estimate was before environmentalists shut down the world's largest copper mine in Guatemala.
Copper is going to get crazy expensive if those regulations are rammed through. Many economists will have black eyes along the way. And Jaguar shareholders.
I thought the photonics chaps had something close to 1:1 electron/photon transduction a fe
Cray (Score:2)
All the AI bullshit aside, the technical problem sounds exactly what Seymour Cray was running up against when trying to build the CDC 8600, and prompted him to completely rethink supercomputer design. The results was the Cray-1.
It seems to my ill-informed brain that we may be at a similar inflection point: either we are on the brink of a new computer design breakthrough, or the LLM approach will be scrapped (but the vast array of resources now available to AI research will let them try totally unique ideas
Problems due to not understanding bottlenecks (Score:2)
When you hit bandwidth issues, then increasing the speed is your only solution. Now, too many companies just think about doing things the same way, just faster. When it comes to many of these things, putting real work into things like increasing the bandwidth for interconnects would be a better solution, but NVIDIA doesn't want to put in the hard work of coming up with things that allow more to be done in parallel. If you have memory bandwidth issues, and HBM with 512 bit isn't enough, then look for w
Re: (Score:2)
NVIDIA doesn't want to put in the hard work of coming up with things that allow more to be done in parallel
What? That's all they do. The whole nature of the modern GPU and/or GPGPU is that you have tons of little cores which are aggressively parallel and have special memory to improve the parallelism.
Can things be broken up so you don't need as high "speeds" to get the job done?
My guess is that we will eventually get better at switching functional units on and off (or even just parts of them) so that we can turn P-cores into E-cores whenever we want, but before then we will get schedulers which can intelligently feed both P-cores and E-cores at the same time. But GPUs seem to have so many
Why use fiber? (Score:3)
Put a bunch of processing units on a giant inverted concave surface, and let them communicate directly with each other by laser.
The bowl can be a heat sink, and you can cool it from above. Make the bowl a mesh so any hot air accumulating below can rise through. Put it all in a clean room so the lasers don't have to deal with dust. Keep the lights off so you have maximum contrast and can use less powerful lasers.
Now your system size / node quantity is limited by the number of laser transceivers you can mount on a processor. You're never going to make it big enough to worry about speed of light delay like you would with a copper data bus.
Grace Hopper's Nanoseconds (Score:2)
Anyone else get one of Grace Hopper's nanoseconds?