Google Shared AI Knowledge With the World - Until ChatGPT Caught Up (washingtonpost.com) 33
For years Google published scientific research that helped jump-start its competitors. But now it's lurched into defensive mode. From a report: In February, Jeff Dean, Google's longtime head of artificial intelligence, announced a stunning policy shift to his staff: They had to hold off sharing their work with the outside world. For years Dean had run his department like a university, encouraging researchers to publish academic papers prolifically; they pushed out nearly 500 studies since 2019, according to Google Research's website. But the launch of OpenAI's groundbreaking ChatGPT three months earlier had changed things. The San Francisco start-up kept up with Google by reading the team's scientific papers, Dean said at the quarterly meeting for the company's research division. Indeed, transformers -- a foundational part of the latest AI tech and the T in ChatGPT -- originated in a Google study.
Things had to change. Google would take advantage of its own AI discoveries, sharing papers only after the lab work had been turned into products, Dean said, according to two people with knowledge of the meeting, who spoke on the condition of anonymity to share private information. The policy change is part of a larger shift inside Google. Long considered the leader in AI, the tech giant has lurched into defensive mode -- first to fend off a fleet of nimble AI competitors, and now to protect its core search business, stock price, and, potentially, its future, which executives have said is intertwined with AI. In op-eds, podcasts and TV appearances, Google CEO Sundar Pichai has urged caution on AI. "On a societal scale, it can cause a lot of harm," he warned on "60 Minutes" in April, describing how the technology could supercharge the creation of fake images and videos. But in recent months, Google has overhauled its AI operations with the goal of launching products quickly, according to interviews with 11 current and former Google employees, most of whom spoke on the condition of anonymity to share private information.
Things had to change. Google would take advantage of its own AI discoveries, sharing papers only after the lab work had been turned into products, Dean said, according to two people with knowledge of the meeting, who spoke on the condition of anonymity to share private information. The policy change is part of a larger shift inside Google. Long considered the leader in AI, the tech giant has lurched into defensive mode -- first to fend off a fleet of nimble AI competitors, and now to protect its core search business, stock price, and, potentially, its future, which executives have said is intertwined with AI. In op-eds, podcasts and TV appearances, Google CEO Sundar Pichai has urged caution on AI. "On a societal scale, it can cause a lot of harm," he warned on "60 Minutes" in April, describing how the technology could supercharge the creation of fake images and videos. But in recent months, Google has overhauled its AI operations with the goal of launching products quickly, according to interviews with 11 current and former Google employees, most of whom spoke on the condition of anonymity to share private information.
Sad what Google has become (Score:2, Insightful)
Re: (Score:2)
They could always do what Xerox Parc did and give tours to Apple. See how well that worked for them.
Re: (Score:2)
Xerox, sadly for them, couldn't figure out how to successfully commercialize their ideas. The only computer system they made based on them was $INFINITY dollars. OK, not really, but the Alto series really was grossly overpriced, you could really only process documents with it and it cost much more than the Unix workstations with similar spec which followed — and it's not like they were much later, either.
Re: (Score:2)
From interviews I've read the problem was more that the suits in charge didnt understand what the value in those products was and just wouldnt part with the resources to take ideas from lab to showroom. The Parc boy where coming up with technological marvels and the cigar chompers on the board where all "Who the fuck wants a computer? We sell photocopiers!"
Something I suspect is familiar to many of us who've come up with a neat idea only for the boss to be mystified by it.
Re: (Score:2)
education.
the kryptonite of monopolies.
what now are the possibilities
Bitten by their hatred of copyright again. (Score:5, Insightful)
I've said it before with their browser. I said it with Android. Now it's happened again. Google hates copyleft and has tried to destroy it and now that's come back to bite them. Just at the moment they could get the greatest benefit, and use aggressive open sourcing to keep them ahead of the competition, their own code is being used to destroy them and they have to give up their greatest potential advantage by shutting their community out.
Well deserved, in this case.
Re: (Score:2)
copyright / copyleft? Swapped over in my title, but it's the lack of the combination that will kill them.
Re: (Score:3)
Except in the case of AI it prospers with a strong hardware base. More so as the models get bigger and bigger and the training times get longer.
Re: (Score:2)
Except in the case of AI it prospers with a strong hardware base. More so as the models get bigger and bigger and the training times get longer.
Absolutely correct, though please read the memo [semianalysis.com] that Okian Warrior linked to [slashdot.org], that belief has been a trap for Google which open sourcing their secret sauce would have saved them from. However, that hardware element is the same as a mobile operating system prospering with careful hardware/software integration. They could have used a very strong copyleft which forced everyone to share code and still have kept their advantage based on having the biggest hardware and ensuring that the software was always upgrad
Re: Bitten by their hatred of copyright again. (Score:2)
The code isn't really the value equation though, the ideas are. You can write a transformer LLM that can do most of what ChatGPT does in a few hundred lines of code with PyTorch or whatever. It was the *idea* behind transformer LLM that changed everything in ML and that came from a Google journal article (specifically "Attention is all you need"). Well that and the actual trained model, but Google and OpenAI aren't sharing those anymore. Curiously, Meta actually did (the llama model which is roughly as powe
Re: (Score:2)
The barrier to entry is the enormous need for hardware to back these AI systems.
Google had ample time to expand their services but they got distracted by the sweet ad money.
Google was caught flat-footed (Score:5, Interesting)
A recent leaked memo [semianalysis.com] from Google shows this quite nicely.
In summary, LLaMA (from Meta) was leaked to the public, and then the public jumped on it and made tremendous advances in the technology - far more than Google could have ever imagined, in the space of two months.
Yep. LLaMA was leaked at the beginning of March, and already people have got the large language model running on phones, running personalized versions, and training variants in a few hours.
Someone got ChatGPT running on a RasPi, by restricting the floating points to 4 bits of precision (if I understand it correctly, didn't look in detail).
The Google memo (from the link above) is a pretty interesting read. They *wanted* to have a "moat" around their software, a secret-sauce that no one else had, to give them dominance in the field. What happened instead is that the public jumped leapt ahead of them by leaps and bounds, and now they are trying to play nice with the community in an attempt to catch up.
Re: (Score:2)
Re: Google was caught flat-footed (Score:2)
Re: (Score:3)
Has anyone been able to verify the authenticity of the "leaked" memo? It reads like fan fiction.
Someone got ChatGPT running on a RasPi, by restricting the floating points to 4 bits of precision (if I understand it correctly,
The term you're searching for is quantization. No, no one got ChatGPT running on a Raspberry pi, though you can get a similar chat thing running on one.
You should probably know that you lose a ton of information when you use a 4-bit quantization. The model, consequently, will not perform nearly as well.
Quantization FTW (Score:2)
The term you're searching for is quantization. No, no one got ChatGPT running on a Raspberry pi, though you can get a similar chat thing running on one.
You should probably know that you lose a ton of information when you use a 4-bit quantization. The model, consequently, will not perform nearly as well.
You're right, good catch. Thanks.
Note that the quantization in the human model is way lower than what we imagine it would need. For example, the human eye registers only 12 levels of brightness. The sensitivity range is very wide (starlight up to direct sunlight), but within any specific environment the eye only notes 12 levels of brightness.
Similarly, there are only about 1 million nerves in the optic bundle, so maybe image recognition could get away with less spacial resolution. The human eye has fewer bl
Re: (Score:2)
it would be really really interesting if you could publish that chat or at least tell what prompts you used to get to that.
Re: (Score:2)
Google may have been bitten but the world has prospered, or at least the world has advanced in AI and phones because of the completion android added.
For me life isn't about screwing the every last cent out you can out of every one you can, but making a better world for everyone.
The question is if better AI or more cell phones has made the world better is debatable, but the fact that google shared the knowledge accelerated their development is clear.
Google should be worried (Score:5, Interesting)
They've been resting on their laurels for over 20 years now. It's about time someone else finally gave them some competition.
No joke, just the other day, I was at a tech workshop where we were playing around with ChatGPT. A bunch of colleagues were typing in "2+2=5" just to see if they could get a rise out of it. I told them to stop, and instead put in the prompt, "Can 2 +2 = 10?" Now, of course it can, in base-4 math, but I truly expected ChatGPT to fail at the inquiry. But I was darned, ChatGPT gave a paragraph response to the tune of "Under most circumstances, 2+2 equals 4, not 10. But if we were to use Base-4 numbers, then 2+2 does equal 10.
Then, I entered "Does 2+2 equal 10" into Google. The results were atrocious [google.com]. But, if I changed the inquiry to "does 2+2=10", then it finds a Quora result that does answer the question. But strange how Google can't find the answer when I write the world "equal". [google.com]
Re: Google should be worried (Score:4, Insightful)
Re: (Score:2)
Oh don't worry, Microsoft will find a way to clutter ChatGPT with ads that are hard not to click.
Re: (Score:2)
When doing do, Bard's answer to "Can 2 + 2 = 10?" is
"2+2=4 in base 10, the most common number system. However, in base 4, 2+2=10. This is because in base 4, there are only four digits: 0, 1, 2, and 3. Therefore, 10 in base 4 is equal to 4 in base 10."
And then it goes on a long explanation on how to convert between base 4 and base 10
Closing the barn door after the horse has bolted (Score:5, Insightful)
At this point, it's too little, too late. Google released their paper on transformer architectures in 2017, and OpenAI announced GPT-1 in 2018. You could argue that OpenAI continued leaning on Google research as they developed GPT-2, GPT-3, GPT-3.5, and GPT-4, but the most important thing from a business perspective is not the advancements of the LLMs themselves, but of the integration. OpenAI and Microsoft have certainly not been cribbing any of that from Google, since Google appears to have done little to nothing in that regard.
While Google is busy showing off their LLM that seems to be on-par with GPT-3 at best (and if everybody has been cheating off Google's homework, why is Google a generation or two behind), OpenAI and Microsoft have been showing integration with applications and real-time data lookups. Microsoft is now well on their way to having GPT-4 integrated into their search engine, their web browser, their office suite, their business communications platform.
The whole "OpenAI is only successful because Google did all the hard work first" thing always rang hollow to me in the same way that you could say that computer GUIs were only successful because Xerox did all that work at PARC. I mean, sure, they did, but they didn't do anything with it and quickly became irrelevant in that market.
Re: (Score:1)
The whole "OpenAI is only successful because Google did all the hard work first" thing always rang hollow to me
Bro, I remember the same thing in the 1990's with Linux versus commercial UNIX. Folks claimed Linux just "copied" all their ideas and it was really inferior to Solaris or AIX. I mean people said that shit with a straight face and told me Linux & *BSD was just a flash in the pan. What a load of shit. The truth is Linux and *BSD did copy a lot of their ideas. That's how you build something better! You take the best of what's out there and add some more good stuff to it. Of course the actual problem was th
Re: (Score:3)
if everybody has been cheating off Google's homework, why is Google a generation or two behind
Because that is how humans innovate and develop we build on the work of people who came before us. Just because one person came up with an idea doesn't mean the next person cannot improve on it or the next person is more or less smarter than the last.
This is what bothers me about intellectual property we all learn from others society works together for the betterment of all. However claiming patent or copyright just says mine. As for the homework analogy there is copying homework from a learning point of vi
Re: (Score:2)
Don't count Google out. They weren't the first successful search engine, they weren't the first successful email provider, the first successful map provider, or the first smartphone maker. If they are good at anything, they are really good at being a *second* mover.
Re: (Score:2)
What was the last in-house Google product that became a successful major product and is still around today? Chromebooks in 2011? Chromecast in 2013? Has Google launched a single major in-house product in the last ten years that is a success?
Google has launched and discontinued several hundred products in the past ten years, but that doesn't help the argument of not counting them out.
Re: (Score:2)
Research is not the same as development. In a wealthy monopoly environment like Google, the bean counters like the cache of unbridled research. But when it comes to development, they want proof it will make money. That is what has happened over and over in these monopoly places: Bell Labs (AT&T), IBM, RCA, Xerox, Kodak, ...
Re: (Score:2)
cachet not cache
Google contributes back to tha community (Score:1)
Everyone should be FORCED to share their work (Score:1)
Maybe they give themselves too much credit (Score:2)
They might have great research ideas but others implemented them and now are getting rewarded for their efforts.
Why didn't they follow through? OpenAI was also publishing their research.