Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
×
Google AI

Google's Claims of Super-Human AI Chip Layout Back Under the Microscope (theregister.com) 56

A Google-led research paper published in Nature, claiming machine-learning software can design better chips faster than humans, has been called into question after a new study disputed its results. The Register reports: In June 2021, Google made headlines for developing a reinforcement-learning-based system capable of automatically generating optimized microchip floorplans. These plans determine the arrangement of blocks of electronic circuitry within the chip: where things such as the CPU and GPU cores, and memory and peripheral controllers, actually sit on the physical silicon die. Google said it was using this AI software to design its homegrown TPU chips that accelerate AI workloads: it was employing machine learning to make its other machine-learning systems run faster. The research got the attention of the electronic design automation community, which was already moving toward incorporating machine-learning algorithms into their software suites. Now Google's claims of its better-than-humans model has been challenged by a team at the University of California, San Diego (UCSD).

Led by Andrew Kahng, a professor of computer science and engineering, that group spent months reverse engineering the floorplanning pipeline Google described in Nature. The web giant withheld some details of its model's inner workings, citing commercial sensitivity, so the UCSD had to figure out how to make their own complete version to verify the Googlers' findings. Prof Kahng, we note, served as a reviewer for Nature during the peer-review process of Google's paper. The university academics ultimately found their own recreation of the original Google code, referred to as circuit training (CT) in their study, actually performed worse than humans using traditional industry methods and tools.

What could have caused this discrepancy? One might say the recreation was incomplete, though there may be another explanation. Over time, the UCSD team learned Google had used commercial software developed by Synopsys, a major maker of electronic design automation (EDA) suites, to create a starting arrangement of the chip's logic gates that the web giant's reinforcement learning system then optimized. The Google paper did mention that industry-standard software tools and manual tweaking were used after the model had generated a layout, primarily to ensure the processor would work as intended and finalize it for fabrication. The Googlers argued this was a necessary step whether the floorplan was created by a machine-learning algorithm or by humans with standard tools, and thus its model deserved credit for the optimized end product. However, the UCSD team said there was no mention in the Nature paper of EDA tools being used beforehand to prepare a layout for the model to iterate over. It's argued these Synopsys tools may have given the model a decent enough head start that the AI system's true capabilities should be called into question.

The lead authors of Google's paper, Azalia Mirhoseini and Anna Goldie, said the UCSD team's work isn't an accurate implementation of their method. They pointed out (PDF) that Prof Kahng's group obtained worse results since they didn't pre-train their model on any data at all. Prof Kahng's team also did not train their system using the same amount of computing power as Google used, and suggested this step may not have been carried out properly, crippling the model's performance. Mirhoseini and Goldie also said the pre-processing step using EDA applications that was not explicitly described in their Nature paper wasn't important enough to mention. The UCSD group, however, said they didn't pre-train their model because they didn't have access to the Google proprietary data. They claimed, however, their software had been verified by two other engineers at the internet giant, who were also listed as co-authors of the Nature paper.
Separately, a fired Google AI researcher claims the internet goliath's research paper was "done in context of a large potential Cloud deal" worth $120 million at the time.
This discussion has been archived. No new comments can be posted.

Google's Claims of Super-Human AI Chip Layout Back Under the Microscope

Comments Filter:
  • by kiviQr ( 3443687 ) on Monday March 27, 2023 @08:32PM (#63404908)
    AI self-designing GPU for AI ... what could go wrong? Only if Hollywood had some ideas....
  • It's impossible to prove that someone wouldn't have used ai to design a better human designed chip. Maybe they get credit but prove you didn't use ai to come up with you better design. Proving a negative? That's press worthy.
  • Maybe... (Score:5, Insightful)

    by Anonymous Coward on Monday March 27, 2023 @09:43PM (#63405000)
    ... Nature shouldn't be accepting papers for publication if all of the required algorithms and training data are not available for others to replicate the results. Otherwise anyone, like Google, could get fraudulent papers published and there'd be no way anybody could disprove their results or hold them to account.
    • by cats-paw ( 34890 )

      what's particularly ironic is how Nature has been going on and on about how there's a reproducibility crisis in science.

      Along comes Google with an extraordinarily obvious conflict of interest, and a stacked deck of reviewers with critical data missing to prove reproducibility to demonstrate just how much Nature cares about the crisis.

      Nature only cares as much as the for profit company sitting behind them cares.

    • Cart before the horse. You're not wrong but...

      Nature is primarily a biosciences journal, and a very high profile one, arguably the top journal in the world. They also don't have a clue how to review anything substantially outside their area. Unfortunately now there is a larger and larger intersection with deep learning making its way into the biosciences and they are not well placed to review properly. They've been letting in cool looking but shitty computation based papers for fucking ever.

      So yeah sure rep

    • Re:Maybe... (Score:4, Informative)

      by backslashdot ( 95548 ) on Tuesday March 28, 2023 @06:30AM (#63405564)

      They keep accepting papers from Ranga Dias who makes big claims about room temperature superconductors though none of his stuff has ever been reproducible by anyone. First it was for creating a superconducting metallic hydrogen, which they subsequently claimed they lost. Then it was superconductors at room temperature under pressure, which Nature later retracted when it was proven that the data was falsified. Fool me twice? No. They accepted his latest paper even though they were forced to retract his previous one. As usual, nobody can replicate the results of his latest paper either. He needs to either publish everything properly or lose the funding and top postdocs that are attracted to his lab.
      References:
      https://undark.org/2023/03/27/... [undark.org]
      https://www.newscientist.com/a... [newscientist.com]

  • by timeOday ( 582209 ) on Monday March 27, 2023 @09:56PM (#63405020)
    It's amazing that the professor and students spent months in mounting an attempt to replicate this work. Replication is a thankless job, and often (as in this case) the result is not something clear-cut like a rocket booster landing safely on a recovery pad.
    • It's amazing that the professor and students spent months in mounting an attempt to replicate this work. Replication is a thankless job, and often (as in this case) the result is not something clear-cut like a rocket booster landing safely on a recovery pad.

      Agreed, although the professor almost certainly off-loaded 100% of the actual grunt work to his grad students.

  • The joke that keeps on giving.

  • How can some paper get published if all aspects are not open and replicable? If you do this just to get some big money deal agreed on with some third party and want a commercial secret kept back, then get off my lawn! Don't waste peoples time and journals reputation. Your own has long time ago sunk below bottom.
    • Re:Misuse of Nature (Score:4, Informative)

      by real_nickname ( 6922224 ) on Tuesday March 28, 2023 @12:54AM (#63405196)

      How can some paper get published if all aspects are not open and replicable?

      Many AI paper are like that. Training data and weights are kept secret. And training may be so expensive that nobody can replicate.

    • Indeed.

      I'm told that commercial organisations are now out-stripping academia in the amounts of research they perform. That a commercial organisation can make claims in excess of the verifiable facts should really surprise no one (and that they'd keep some aspects of their work secret, likewise should be no surprise).

      We generally assume the likes of Nature would be wise to this and would be fact-checking very carefully - but clearly that would take months of research to achieve, so it's not really practical.

  • You can't see anything without.

  • Note that AI papers are impossible to replicate, since you need the actual weights to reuse the same network. Even in such case, there are models (such as GANs) that rely on random noise to generate an output, which differs from case to case.

    Unlike an algorithm, which can be proven right or wrong, AI is completely dependent on the weights, which are completely dependent on the training data and the training process, which is often impossible to replicate.

    And don't touch my weights!!

  • As far as I know, it's all done by AI. No one is doing the layout for millions of gates. You might provide some hints to the layout software, but that's about it. It's all super human.
    • It's done by algorithms. Whether those algorithms are "AI" or not is a different discussion. Starting with a really good design and letting AI make some tweaks might get you a better design. If it does, that's great. You can use the design and publish a paper. If not, you already have a really good design that you can take to manufacturing.
  • I've found that Nature frequently publishes bogus nonsense, particularly when it is not biology.

"I got everybody to pay up front...then I blew up their planet." "Now why didn't I think of that?" -- Post Bros. Comics

Working...