DeepMind Chief Dismisses DeepSeek's AI Breakthrough as 'Known Techniques' (cnbc.com) 29
Google DeepMind CEO Demis Hassabis downplayed the technological significance of DeepSeek's latest AI model, despite its market impact. "Despite the hype, there's no actual new scientific advance there. It's using known techniques," Hassabis said on Sunday. "Actually many of the techniques we invented at Google and at DeepMind."
Hassabis acknowledged that Deepseek's AI model "is probably the best work" out of China, but its capabilities, he said, is "exaggerated a little bit."DeepSeek's launch last month triggered a $1 trillion U.S. market sell-off.
Hassabis acknowledged that Deepseek's AI model "is probably the best work" out of China, but its capabilities, he said, is "exaggerated a little bit."DeepSeek's launch last month triggered a $1 trillion U.S. market sell-off.
hoping this is the straw that broke the camel's ba (Score:5, Insightful)
I guess Sam the Man didn't. In any case, is this the death knell of the trillions being flushed down the toilet?
In my crystal ball I see a rapid build out of power generation, and then a glut of cheap energy..
Re: (Score:2)
Even if everything we're hearing from Google is true.
I'm not sure what that gets them. They can spend a billion dollars to get an edge that can be erased for a few million? Just because they knew that doesn't really mean that billion is a good investment.
Re: (Score:3)
No, you are mixing OpenAI and Google Deepmind. OpenAI created a chatbot and their whole businessmodel is based on that. OpenAI is very much hurt by Deepseek.
Deepmind on the other hand is famous from AlphaFold, which even got the Nobel for Hassabis. Deepmind is more focused on medical applications, like drug development and Deepseek did not offer anything on there. I still think that Google made the chatbot just because the bosses demanded it and they are not very interested in developing in further.
Even if
Re: (Score:2)
Re: (Score:3, Interesting)
You need at least one genius to make it. It is currently the only AI we have that is actually smarter than humans, as it can make new scientific discoveries that humans were unable to do even they spent decades trying.
Regarding this, multiple fields (CS and structural biology) disagree with you. The model is not smarter than humans in anyway, heck it is not even smart: it is a statistical model that is able to interpolate very well, but is incapable of extrapolating (not too much, at least), and that's not a limitation, just part of its design.
There are a ton of papers already that are tackling the many limitations of AlphaFold, but that's because the whole field is working together to improve it (as it should be with S
Re: (Score:2)
ck... So the AI models can "learn" from each other. Who knew? I guess Sam the Man didn't. In any case, is this the death knell of the trillions being flushed down the toilet? In my crystal ball I see a rapid build out of power generation, and then a glut of cheap energy.. .and AI companies trying to explain why their pants are down.
I guess he has to say something. But people need to make no mistake. If the model of AI requires trillions of dollars, and their own nuclear power plants - it ain't gonna work. MOst thinigs become a lot less expensive over time, and the present AI model is not immune from that. I'm not certain when the bubble is going to burst, but it will, and trillions of dollars will disappear in less than a second. We've seen the rumbling already.
He should know (Score:2)
"Actually many of the techniques we invented at Google and at DeepMind."
Demis's thoughts would have been: "and I'm really kicking myself that we did not do more to apply RL to LLM training."
Re: (Score:2)
https://huyenchip.com/2023/05/... [huyenchip.com]
Misdirection (Score:5, Insightful)
Re: (Score:1)
I wish I had mod point to mod this up
Re: (Score:2)
The cost all people report about (and mostly misunderstand) is the cost to train the model. It neither includes buying hardware, paying researchers, founding a company or running the webservice. They trained a model, documented how they did it, and gave an estimate what the training costed.
And that's reasonable. Because if you have your hardware and pay your researchers, you're interested in what training a model costs and not in what the salary of a Chinese data scientist is.
Re: (Score:2)
It's also reasonable because you don't need to buy hardware, there's a shitload for rent. DeepMind "rents" theirs from Google and OpenAI was gifted a lot of theirs from Microsoft. You also don't need to pay researchers if you don't want to do research. If you just want to train something that does what DeepSeek does, you just follow their paper or pretty soon the open source community will have it all stuck together in some convenient download-and-run package for you.
The GP has it right. DeepSeek is touted
Just more wild gesticulations (Score:4, Insightful)
From the guy salty that his golden goose just became a lead turkey. Deepseek just democratized frontier AI for everyone and they're pissed they don't own it and forever more never will. It's out there now. TS.
Re: (Score:3)
Hassabis has not indicated that he is interested in money. He has said that he would prefer to work in academy, but it would be impossible to get similar funding from there. And I think this hurts OpenAI, not Google. Google is clearly focusing on medicine with the AlphaFold-line and other medical related AI projects. They would have not made the chatbot (which they put together quite quickly) at all if they weren't forced to do it by investors.
Quibbling on semantics (Score:5, Insightful)
What types of advances are scientific versus engineering? Is one more valuable than the other? The two DeepSeek advances were (1) multi-head latent attention and (2) writing PTX to convert some SMs from compute to communications. These were definitely innovative advances. Were they scientific or engineering? The real answer is that is doesn't matter because they both are innovative and helpful. Both also were directly motivated by US sanctions.
Multi-head latent attention or variants or improvements on it will likely make it into some non-Chinese models. For models where communication bandwidth is a bottleneck, perhaps the PTX technique might be used, although I imagine that Nvidia would just make the adjustments in the their software tools and eventually in their future hardware architectures.
Re:Quibbling on semantics (Score:5, Interesting)
What types of advances are scientific versus engineering? Is one more valuable than the other? The two DeepSeek advances were (1) multi-head latent attention and (2) writing PTX to convert some SMs from compute to communications. These were definitely innovative advances. Were they scientific or engineering? The real answer is that is doesn't matter because they both are innovative and helpful. Both also were directly motivated by US sanctions.
Multi-head latent attention or variants or improvements on it will likely make it into some non-Chinese models. For models where communication bandwidth is a bottleneck, perhaps the PTX technique might be used, although I imagine that Nvidia would just make the adjustments in the their software tools and eventually in their future hardware architectures.
Sure, and a whole lot of shit Google bases its empire on was invented and written by other people. Sounds like this guy is butt-hurt because a bunch of Chinese managed to catch up with the West when China was supposed to be ten years behind the US on AI (according to the US). To make it worse it wasn't some multi billion dollar CCP mega project that did it (that would lessen the humiliation) it was a bunch of nerds working for an obscure two bit hedge fund in Hangzhou using a datacenter with a few tens of thousands of intentionally crippled NVIDA export chips. Of course he has to piss on everything DeepSeek did, it's not just Asians that like to save face after they have been comprehensively humiliated by their own hubris and complacency. If these were all 'known techinques' you'd think the US AI bros would have done everything DeepSeek just did years ago?
Re: (Score:2)
Sure, and a whole lot of shit Google bases its empire on was invented and written by other people. Sounds like this guy is butt-hurt because a bunch of Chinese managed to catch up with the West when China was supposed to be ten years behind the US on AI (according to the US).
Much of the current AI research is published, and there are a lot of open models. DeepSeek arguably copied or at the very least distilled existing models. So, it's not so much that China independently "caught up" through sheer genius and hard work. If China had to independently replicate US progress, then China would indeed have been many years behind. Fortunately for China, US companies made it easy for everyone to "catch up." This openness is a good thing for the total rate of progress and for each i
They knew this technique, but chose not to use it? (Score:4, Insightful)
I mean, DeepSeek is kicking ass in reasoning, did it on the cheap, so why did it take their effort to get OpenAI to spring a new reasoning model to the public (and make it cheaper?).
This is the fundamental issue with these companies... closed source AI, with the biggest, best tech kept behind closed doors. That's the BEST reading of all of this.
At the core of it all is the notion that these companies - who WILL need to spend lots of money - are spending money on the wrong things. Bigger models aren't really going to crack the AGI/ASI barrier, and the current models cost a lot on inference. There are lots of gains in efficiency to be made, and that is what is key about DeepSeek. Inference needs to be cheaper for these "Frontier" AI companies.
Even if you had an "ASI model" with more data than the entire internet could provide, the resulting queries would be unsustainable, given the efficiencies in the o1-level models, for example. Likewise, training up such a model is ridiculously expensive (in compute, energy, infrastructure) still, without getting those gains in efficiency.
I see a world where we get open source models that will run well and cheap - locally on your phones, even... and do a pretty reasonable, accurate, and hallucination-free job. A mid-tier where queries are offloaded to server farms for more complex agentic work (the new employees), and a god-tier AI level, ASI with tendrils sniffing into every nook and cranny for information, that will cost the big bucks, but solve big problems ("cure cancer", "cold fusion") and still be cheaper than human research.
The problem is that a lot of these CEOs lack the vision, often caught up in their own egos or company pride to see the forest for all the trees.
Re: (Score:2)
It actually wasn't that cheap. Maybe cheap compared to what OpenAI/Google/MS spend but not $5.6 million cheap.
The bigger issue is the underlining tech (Score:2)
Chat GTP and the other big American AI companies can squeeze out competitors using high interest rates and by having private conversations with the private equity companies that would normally be investing in their competitors and telling them to lay off (plus a few other nasty little antitrust violations)
DeepSeek is not about "scientfic advance" (Score:5, Interesting)
From the start, it was clear that there is no real scientific breakthrough with DeepSeek. It was described as using several known optimization techniques, and it worked as expected, which, by the way, is a significant advance by itself.
The real breakthrough is that DeepSeek has shown that the state of the art in AI is not that expensive, not that complicated, and that OpenAI and friends do not really have a moat. They have burst a bubble.
As for DeepMind, how can they be so good scientifically and so bad at production? What is Google doing? They essentially invented modern AI, and continue doing so, and yet, get beaten to market by everyone else.
Doesn't mean much (Score:2)
"Actually many of the techniques we invented at Google and at DeepMind."
And many of the techniques Apple used in creating the Macintosh were invented at Xerox. You may observe for yourself how much of the PC market Xerox owns today.
Secret ingredients (Score:2)
MoE (as seen for example in Mixtral or GPT-4 a year ago) and ChainOfThought (as seen in simple CoT prompts a year ago and recently baked into OpenAI's o1 and o3 models).