Google's Claims of Super-Human AI Chip Layout Back Under the Microscope (theregister.com) 56
A Google-led research paper published in Nature, claiming machine-learning software can design better chips faster than humans, has been called into question after a new study disputed its results. The Register reports: In June 2021, Google made headlines for developing a reinforcement-learning-based system capable of automatically generating optimized microchip floorplans. These plans determine the arrangement of blocks of electronic circuitry within the chip: where things such as the CPU and GPU cores, and memory and peripheral controllers, actually sit on the physical silicon die. Google said it was using this AI software to design its homegrown TPU chips that accelerate AI workloads: it was employing machine learning to make its other machine-learning systems run faster. The research got the attention of the electronic design automation community, which was already moving toward incorporating machine-learning algorithms into their software suites. Now Google's claims of its better-than-humans model has been challenged by a team at the University of California, San Diego (UCSD).
Led by Andrew Kahng, a professor of computer science and engineering, that group spent months reverse engineering the floorplanning pipeline Google described in Nature. The web giant withheld some details of its model's inner workings, citing commercial sensitivity, so the UCSD had to figure out how to make their own complete version to verify the Googlers' findings. Prof Kahng, we note, served as a reviewer for Nature during the peer-review process of Google's paper. The university academics ultimately found their own recreation of the original Google code, referred to as circuit training (CT) in their study, actually performed worse than humans using traditional industry methods and tools.
What could have caused this discrepancy? One might say the recreation was incomplete, though there may be another explanation. Over time, the UCSD team learned Google had used commercial software developed by Synopsys, a major maker of electronic design automation (EDA) suites, to create a starting arrangement of the chip's logic gates that the web giant's reinforcement learning system then optimized. The Google paper did mention that industry-standard software tools and manual tweaking were used after the model had generated a layout, primarily to ensure the processor would work as intended and finalize it for fabrication. The Googlers argued this was a necessary step whether the floorplan was created by a machine-learning algorithm or by humans with standard tools, and thus its model deserved credit for the optimized end product. However, the UCSD team said there was no mention in the Nature paper of EDA tools being used beforehand to prepare a layout for the model to iterate over. It's argued these Synopsys tools may have given the model a decent enough head start that the AI system's true capabilities should be called into question.
The lead authors of Google's paper, Azalia Mirhoseini and Anna Goldie, said the UCSD team's work isn't an accurate implementation of their method. They pointed out (PDF) that Prof Kahng's group obtained worse results since they didn't pre-train their model on any data at all. Prof Kahng's team also did not train their system using the same amount of computing power as Google used, and suggested this step may not have been carried out properly, crippling the model's performance. Mirhoseini and Goldie also said the pre-processing step using EDA applications that was not explicitly described in their Nature paper wasn't important enough to mention. The UCSD group, however, said they didn't pre-train their model because they didn't have access to the Google proprietary data. They claimed, however, their software had been verified by two other engineers at the internet giant, who were also listed as co-authors of the Nature paper. Separately, a fired Google AI researcher claims the internet goliath's research paper was "done in context of a large potential Cloud deal" worth $120 million at the time.
Led by Andrew Kahng, a professor of computer science and engineering, that group spent months reverse engineering the floorplanning pipeline Google described in Nature. The web giant withheld some details of its model's inner workings, citing commercial sensitivity, so the UCSD had to figure out how to make their own complete version to verify the Googlers' findings. Prof Kahng, we note, served as a reviewer for Nature during the peer-review process of Google's paper. The university academics ultimately found their own recreation of the original Google code, referred to as circuit training (CT) in their study, actually performed worse than humans using traditional industry methods and tools.
What could have caused this discrepancy? One might say the recreation was incomplete, though there may be another explanation. Over time, the UCSD team learned Google had used commercial software developed by Synopsys, a major maker of electronic design automation (EDA) suites, to create a starting arrangement of the chip's logic gates that the web giant's reinforcement learning system then optimized. The Google paper did mention that industry-standard software tools and manual tweaking were used after the model had generated a layout, primarily to ensure the processor would work as intended and finalize it for fabrication. The Googlers argued this was a necessary step whether the floorplan was created by a machine-learning algorithm or by humans with standard tools, and thus its model deserved credit for the optimized end product. However, the UCSD team said there was no mention in the Nature paper of EDA tools being used beforehand to prepare a layout for the model to iterate over. It's argued these Synopsys tools may have given the model a decent enough head start that the AI system's true capabilities should be called into question.
The lead authors of Google's paper, Azalia Mirhoseini and Anna Goldie, said the UCSD team's work isn't an accurate implementation of their method. They pointed out (PDF) that Prof Kahng's group obtained worse results since they didn't pre-train their model on any data at all. Prof Kahng's team also did not train their system using the same amount of computing power as Google used, and suggested this step may not have been carried out properly, crippling the model's performance. Mirhoseini and Goldie also said the pre-processing step using EDA applications that was not explicitly described in their Nature paper wasn't important enough to mention. The UCSD group, however, said they didn't pre-train their model because they didn't have access to the Google proprietary data. They claimed, however, their software had been verified by two other engineers at the internet giant, who were also listed as co-authors of the Nature paper. Separately, a fired Google AI researcher claims the internet goliath's research paper was "done in context of a large potential Cloud deal" worth $120 million at the time.
second step of Skynet creation (Score:3)
Re:Dreamweaver all over again. (Score:5, Insightful)
Machine generated logic will always be inherently, inescapably un-optimized.
What basis do you have to make this claim? It may take centuries but I have no doubt that humans will become second fiddle to the optimization capabilities of AI.
Re:Dreamweaver all over again. (Score:4, Insightful)
A deep conviction that their brain is magic.
Re: (Score:2)
You don't really think that's a winning argument, do you?
From what I can tell, the only people who believe in magic here are the newly-minted futurists who believe AI is going to solve all of the world's problems.
Re: (Score:2)
The idea that the human brain is somehow unique and non-duplicable, or that what it does is non-computable, is basically a claim that it is supernatural. Magic. You can make your layman arguments from emotion all you want.
Re: (Score:2)
or that what it does is non-computable, is basically a claim that it is supernatural.
Nonsense! We have very good reason to believe that whatever it is that brains do, computation alone is insufficient.
The only one here filling in that gap with the 'supernatural' is you.
See, you only came to the "conclusion" the brain is a computer because computers are the most advanced technology that we have at the moment. Because you can't stand the idea that we simply don't know, you're willing to accept any explanation, without evidence, as long as it doesn't contradict any of your other cherished be
Re: (Score:1, Troll)
(Modded down by "Big AI" for exposing the flaw in their business plan.)
Re: (Score:1)
HAHA burn all your sock-puppet mod points on me you fucking Russians!
Re: (Score:2)
A deep understanding of nature and history, and what evolutionary algorithms both are and aren't good at.
Our brains are evolutionary algorithms. This is actually very visible in your own thought processes, if you pay attention. You generate ideas and then apply selection pressures to them, both internally and (for greater effectiveness) in collaboration with other people.
In "The Beginning of Infinity", David Deutsch makes a compelling argument that the process of variation and selection is the only process by which knowledge is created, in any context. Memetic evolution is not just analogous to genetic evolu
That's nonsense. (Score:3)
First off, brains are not algorithms - if it were so anyone could be taught ANY skill or knowledge to same capacity.
I.e. "Intelligence" would not be a category. Neither would aptitude.
Similarly, various animals on a similar level of brain development could be taught exact same set of skills.
E.g. Birds could be taught say... elementary math. Lower mammals division and square roots, higher mammals geometry...
Also, brains are not there to "generate ideas and then apply selection pressures to them" - if it were
Re: (Score:2)
We could look at compilers to get an idea of how this might work. Of course they are not using AI, but they are able to apply various algorithmic optimizations. Not just for software, for Hardware Description Languages (HDLs) like Verilog too.
On the software side, a human can still out-optimize a C compiler in many cases. I don't know enough about HDLs to say for sure, but I think the answer is we probably don't know because the manufacturers keep their inner workings secret. Certainly the move from hand de
Re: (Score:2)
On the software side, a human can still out-optimize a C compiler in many cases.
I don't think that's generally true, and hasn't been true for a long time. Deeply-pipelined, superscalar processors with out of order execution and complicated branch prediction logic, fed by complicated layers of caching create a situation where humans find it very difficult to keep track of all the relevant state needed to optimize execution.
Re: (Score:2)
For CPUs where the machine code is little more than an intermediate language, sure. Even compilers can't keep up with them, not least because a lot of the information on exactly how they work is a trade secret.
There are billions of ARM CPUs in the world that don't have long pipelines or superscalar capabilities though. ARM has put a lot of effort into getting them optimized, but they still don't produce code that's as good as hand tuned assembler in many cases. Sometimes you can help them a bit by changing
Re: (Score:2)
>humans find it very difficult to keep track of all the relevant state needed to optimize execution.
I work in a place where people design CPUs. I assure you that there are people who comprehend it very well.
Re: (Score:2)
>humans find it very difficult to keep track of all the relevant state needed to optimize execution.
I work in a place where people design CPUs. I assure you that there are people who comprehend it very well.
Comprehend the architecture is a very different thing from being able to manually optimize effectively.
Re: (Score:2)
>humans find it very difficult to keep track of all the relevant state needed to optimize execution.
I work in a place where people design CPUs. I assure you that there are people who comprehend it very well.
Comprehend the architecture is a very different thing from being able to manually optimize effectively.
I'm talking about going the other way - understanding what code people write and designing the hardware to execute it fast.
In the real world this goes both ways, but comprehension of the architecture is a big part of being able to optimize for it.
I know how to get optimal use from my instructions because I know the logic behind it inside out and we test it on each new product. But my stuff is narrowly focused compared to the CPU core designers.
Re: (Score:2)
>I don't know enough about HDLs to say for sure,
Synthesizers and optimizers only get you so far with optimization. There is often a case of someone coming up with a mathematical simplification or invariant or something that lets you then write the HDL that gets the synthesizer to make a better (read 'better' as smaller or faster or whatever metric you care about) netlist.
Check out the advances in AES SBOX minimization. The Canwright equations, then the tower field architecture from Boyar-Peralta.
Compared
Re: (Score:2)
I have no doubt that humans will become second fiddle to the optimization capabilities of AI.
That sounds an awful lot like blind faith to me. With a qualifier like "it may take centuries", I have to wonder if you even believe it yourself.
What basis do you have to make this claim?
AI, as we understand it today, can't create new information. Even the fancy transformers that every one is excited about are strictly bound by whatever happens to be retained by their model. More than that, their output merely probabilistic, meaning that things like novelty and creativity are off the table. Lacking anything like understanding, the same is neces
Re: (Score:2)
Much easier to trust the output when it has the unoptimised code to directly compare against. And if it's okay for a human to use a genetic or evolutionary algorithm, then it should be fine for an "AI" to use those same algorithms.
It may not be able to make some intelligent leap as to an alternate way to solve the problem (I.E. an entirely different algorithm) but in terms of optimization I have no doubts an "AI" could go through all of the low-hanging-fruit optimization methods much faster than a person c
Re: (Score:2)
The claim was that it would exceed our ability. Now the claim is that it will just help out a little?
Re: (Score:2)
That sounds an awful lot like blind faith to me. With a qualifier like "it may take centuries",
Not at all. Humans have been talking about doing certain things for millennia (through stories from many different cultures) which has subtly made them into the goals of humanity. We've slowly been progressing (or attempting to) for centuries and there is no reason to believe we are going to stop until we reach them. Big ones include going to other worlds, quasi-immortality (eternal youth), and constructing an intelligent being. These things are impossible but they all require sophisticated science.
Faith th
Re: (Score:2)
This is not logic though, it is finding the optimal physical layout of predetermined logic. It is more like winning a chess game than coming up with logic. Frankly it is surprising humans can be better than computers at it. This is like how it took a while for computers to beat humans in Go or Chess. There are a limited set of rules, computers should be able to ace this stuff. It appears we are at the stage similar to how Chess programs rely on a predeterminied set of opening moves.
Re: (Score:2)
The human brain is billions of neurons with *each neuron* having 100's of millions to billions of connections. Not the '10's' shown in simplified diagrams. Each connection does not have a single weighting, but can exist along a multi-variate range of weights.
All of this contained in a package that turns Oreo's and coffee (and maybe some bean sprouts) into energy.
The juice becomes a stain; the stain becomes a warning.
It is by will alone I set my mind in motion.
Re: Dreamweaver all over again. (Score:3)
Re: (Score:2)
Frankly it is surprising humans can be better than computers at it. This is like how it took a while for computers to beat humans in Go or Chess. There are a limited set of rules, computers should be able to ace this stuff.
Turns out they can't yet. Now that go players have had a chance to examine the playing style of AI players, turns it it can be beaten 97-99% of the time by not even top amateurs:
https://www.vice.com/en/articl... [vice.com]
If you play like a human, the AI will massively pwn you, but one thing is show
Re: (Score:2)
Did you read the article you linked to? The human won by using an AI program against it. From the article "Pelrine memorized the tactic suggested by the program and was able to beat the game 14 out of 15 times." It just means we have to keep training the AIs .. maybe pit them to fuzz against each other. What's it called? Generative adversarial network? An AI came up with the tactic obviously it can learn to recognize when the tactic is being deployed.
Re: (Score:2)
Did you read the article you linked to? The human won by using an AI program against it.
Kinda, but then played as a person, right?
From the article "Pelrine memorized the tactic suggested by the program and was able to beat the game 14 out of 15 times."
I took that to mean that during the games he was not assisted by the AI.
It just means we have to keep training the AIs .. maybe pit them to fuzz against each other. What's it called? Generative adversarial network?
That's more or less how the AIs work for this
Re: (Score:2)
Re: (Score:3)
Machine generated logic will always be inherently, inescapably un-optimized.
People used to say that about compilers, too.
Re: (Score:2)
People used to say that about compilers, too.
People still say that about compilers, and well they should. Compilers didn't get better at optimising than humans. They did get better than themselves, yes, LLVM generates code significantly better than Borland Turbo C, but a human can still beat a compiler. Sure, ordering the instructions for efficiently pipelined, cache optimised, vectorised and multi-threaded code by hand is a damn sight harder today than when the most you had to do was cycle count your 80386 code, but we aren't talking about it being e
Re: (Score:2)
Those people were right. Compilers aren't magic, you know. A dedicated human can still beat the compiler. After all, the human has access to a lot more information about the task than the compiler does.
With the nonsense people produce these days thinking the compiler will magically fix their terrible code, a human could probably "beat the compiler" just by improving the code and compiling again...
Maybe (Score:1)
Maybe... (Score:5, Insightful)
Re: (Score:1)
what's particularly ironic is how Nature has been going on and on about how there's a reproducibility crisis in science.
Along comes Google with an extraordinarily obvious conflict of interest, and a stacked deck of reviewers with critical data missing to prove reproducibility to demonstrate just how much Nature cares about the crisis.
Nature only cares as much as the for profit company sitting behind them cares.
Re: (Score:2)
Cart before the horse. You're not wrong but...
Nature is primarily a biosciences journal, and a very high profile one, arguably the top journal in the world. They also don't have a clue how to review anything substantially outside their area. Unfortunately now there is a larger and larger intersection with deep learning making its way into the biosciences and they are not well placed to review properly. They've been letting in cool looking but shitty computation based papers for fucking ever.
So yeah sure rep
Re:Maybe... (Score:4, Informative)
They keep accepting papers from Ranga Dias who makes big claims about room temperature superconductors though none of his stuff has ever been reproducible by anyone. First it was for creating a superconducting metallic hydrogen, which they subsequently claimed they lost. Then it was superconductors at room temperature under pressure, which Nature later retracted when it was proven that the data was falsified. Fool me twice? No. They accepted his latest paper even though they were forced to retract his previous one. As usual, nobody can replicate the results of his latest paper either. He needs to either publish everything properly or lose the funding and top postdocs that are attracted to his lab.
References:
https://undark.org/2023/03/27/... [undark.org]
https://www.newscientist.com/a... [newscientist.com]
Amazing piece of work (Score:5, Insightful)
Re: (Score:2)
It's amazing that the professor and students spent months in mounting an attempt to replicate this work. Replication is a thankless job, and often (as in this case) the result is not something clear-cut like a rocket booster landing safely on a recovery pad.
Agreed, although the professor almost certainly off-loaded 100% of the actual grunt work to his grad students.
Ah, Google.. (Score:2)
The joke that keeps on giving.
Misuse of Nature (Score:2)
Re:Misuse of Nature (Score:4, Informative)
How can some paper get published if all aspects are not open and replicable?
Many AI paper are like that. Training data and weights are kept secret. And training may be so expensive that nobody can replicate.
Re: (Score:2)
Indeed.
I'm told that commercial organisations are now out-stripping academia in the amounts of research they perform. That a commercial organisation can make claims in excess of the verifiable facts should really surprise no one (and that they'd keep some aspects of their work secret, likewise should be no surprise).
We generally assume the likes of Nature would be wise to this and would be fact-checking very carefully - but clearly that would take months of research to achieve, so it's not really practical.
ALL chip layouts are under the microscope (Score:2)
You can't see anything without.
Impossible to replicate AI papers (Score:2)
Note that AI papers are impossible to replicate, since you need the actual weights to reuse the same network. Even in such case, there are models (such as GANs) that rely on random noise to generate an output, which differs from case to case.
Unlike an algorithm, which can be proven right or wrong, AI is completely dependent on the weights, which are completely dependent on the training data and the training process, which is often impossible to replicate.
And don't touch my weights!!
Is any chip layout done by a human? (Score:2)
Re: (Score:2)
Nature (Score:2)
I've found that Nature frequently publishes bogus nonsense, particularly when it is not biology.