OpenAI and Others Seek New Path To Smarter AI as Current Methods Hit Limitations (reuters.com) 42
AI companies like OpenAI are seeking to overcome unexpected delays and challenges in the pursuit of ever-large language models by developing training techniques that use more human-like ways for algorithms to "think." From a report: A dozenAI scientists, researchers and investors told Reuters they believe that these techniques, which are behind OpenAI's recently released o1 model, could reshape the AI arms race, and have implications for the types of resources that AI companies have an insatiable demand for, from energy to types of chips.
After the release of the viral ChatGPT chatbot two years ago, technology companies, whose valuations have benefited greatly from the AI boom, have publicly maintained that "scaling up" current models through adding more data and computing power will consistently lead to improved AI models. But now, some of the most prominent AI scientists are speaking out on the limitations of this "bigger is better" philosophy. Ilya Sutskever, co-founder of AI labs Safe Superintelligence (SSI) and OpenAI, told Reuters recently that results from scaling up pre-training -- the phase of training an AI model that uses a vast amount of unlabeled data to understand language patterns and structures -- have plateaued. Sutskever is widely credited as an early advocate of achieving massive leaps in generative AI advancement through t he use of more data and computing power in pre-training, which eventually created ChatGPT. Sutskever left OpenAI earlier this year to found SSI. The Information, reporting over the weekend that Orion, OpenAI's newest model, isn't drastically better than its previous model nor is it better at many tasks: The Orion situation could test a core assumption of the AI field, known as scaling laws: that LLMs would continue to improve at the same pace as long as they had more data to learn from and additional computing power to facilitate that training process.
In response to the recent challenge to training-based scaling laws posed by slowing GPT improvements, the industry appears to be shifting its effort to improving models after their initial training, potentially yielding a different type of scaling law.
Some CEOs, including Meta Platforms' Mark Zuckerberg, have said that in a worst-case scenario, there would still be a lot of room to build consumer and enterprise products on top of the current technology even if it doesn't improve.
After the release of the viral ChatGPT chatbot two years ago, technology companies, whose valuations have benefited greatly from the AI boom, have publicly maintained that "scaling up" current models through adding more data and computing power will consistently lead to improved AI models. But now, some of the most prominent AI scientists are speaking out on the limitations of this "bigger is better" philosophy. Ilya Sutskever, co-founder of AI labs Safe Superintelligence (SSI) and OpenAI, told Reuters recently that results from scaling up pre-training -- the phase of training an AI model that uses a vast amount of unlabeled data to understand language patterns and structures -- have plateaued. Sutskever is widely credited as an early advocate of achieving massive leaps in generative AI advancement through t he use of more data and computing power in pre-training, which eventually created ChatGPT. Sutskever left OpenAI earlier this year to found SSI. The Information, reporting over the weekend that Orion, OpenAI's newest model, isn't drastically better than its previous model nor is it better at many tasks: The Orion situation could test a core assumption of the AI field, known as scaling laws: that LLMs would continue to improve at the same pace as long as they had more data to learn from and additional computing power to facilitate that training process.
In response to the recent challenge to training-based scaling laws posed by slowing GPT improvements, the industry appears to be shifting its effort to improving models after their initial training, potentially yielding a different type of scaling law.
Some CEOs, including Meta Platforms' Mark Zuckerberg, have said that in a worst-case scenario, there would still be a lot of room to build consumer and enterprise products on top of the current technology even if it doesn't improve.
Jam tomorrow lads (Score:5, Insightful)
Re: (Score:3)
Obviously. These assholes are now using common scam tactics to keep the marks interested and believing.
How many solar panels so far? (Score:2)
Need a calculation estimate for how many thousands of homes could be powered for 4 hours a day by purchasing and installing solar panels costing the same as 10% of the money and energy used to to train AI models and process AI prompts requests in the last 3 years.
Re: (Score:2)
I just asked ChatGPT that. It says:
Cost of AI model training is (conservatively estimated) "3 years at about $10 billion" and then concludes "200,000homes".
This is essentially a result of "better search" and probably a tiny bit of Wolfram Alpha in the background. The statement is completely unverified, but seems plausible to me.
Comment (Score:3)
Re:Comment (Score:4, Funny)
If they really believed in what they were doing, they would have let ChatGPT comment on their behalf.
Re: (Score:3)
Several comments around here deserved to be modded Funny, but the discussion as a whole scored zero Funny.
Bingo (Score:3, Insightful)
LLMs are only a PART of intelligence. They're a necessary part, but other parts are needed. I saw an article yesterday saying that they had "no coherent model of the world", and that's right too. They don't really have any model of the world, only of language.
So my model still forecasts a basic AGI around 2035. But we'll get lots of really useful (for some purpose or other) things on the way.
Re:Bingo (Score:4, Insightful)
LLMs do not even have a model of language. All they have is a lot of detail connections, and that is it. No model and no understanding.
As to AGI, any predictions at this are pure hallucinations. We do not even have a theory how it could be done. The only thing known that is remotely like it (automated theorem proving) dies from combinatoric explosion before it can get to any real depth.
Re: (Score:3)
We've got LOTS of theories about how AGI could be done. Maybe several of them are correct. And we don't know whether any of them are correct, and won't until we get to the point of trying them.
OTOH, it's also possible that we'll get an AGI without any good theory of how it works. (Theories for parts of it, yeah, but not for the complete thing.)
FWIW, I sort of hope that the AGI is developed out of a hospital management system, but don't really expect that. But that one would have "taking care of people"
Re: (Score:2)
We've got LOTS of theories about how AGI could be done
We don't have any "theories". All we have is a bunch of crackpot nonsense. We don't understand the problem well enough to even know what questions we should be asking. Anyone claiming otherwise is lying, delusional, or both.
it's also possible that we'll get an AGI without any good theory of how it works
Quite a few crackpots also believe that AGI will be achieved spontaneously by magic, though they prefer to use the term 'emergence'. It's silly science fiction nonsense that can and should be ignored.
Re: (Score:2)
LLMs do not even have a model of language. All they have is a lot of detail connections, and that is it. No model and no understanding.
As to AGI, any predictions at this are pure hallucinations. We do not even have a theory how it could be done. The only thing known that is remotely like it (automated theorem proving) dies from combinatoric explosion before it can get to any real depth.
It's certainly not intelligence, but I think it's more than just detailed connections*. They are able to generate what sounds like reasoning, and come to appropriate conclusions. It's kind of a classic software emulation of a biological process, it does something astoundingly well, and others are a complete
I think their main limitations are that their view of the world is completely text based, so it's easy to gain misunderstandings of things you never see in person.
That and their context window is rigid, m
Re: (Score:1)
It's kind of a classic software emulation of a biological process
This nonsense needs to die. NNs are not similar in any meaningful way to biological systems. It's a bad analogy taken way too far.
They are able to generate what sounds like reasoning
It's important to remember that the appearance of reasoning is not the same as actual reasoning.
and come to appropriate conclusions
They don't draw conclusions, they generate text probabilistically, one token at a time, retaining no internal state in between. It is not possible for the system to "plan ahead" beyond the current token. Hell, the model doesn't even do that much, generating a list of next-token pro
Re: (Score:2)
Re:Bingo (Score:5, Interesting)
There is also the problem of expectation.. ML has been fantastically profitable, so anything that tries to challenge it, by VC logic, will have to be even more profitable on an even shorter time scale, which given the time involved in ramping this back up is just not going to happen. So it is going to have to simmer in unglamorous academia for a few decades at least.
Re: (Score:3)
You could be right, but that's not what I expect. I think the "coding assistants" will get good enough that the symbolic processing groups will quickly recover...possibly without anybody noticing that that's what they're doing. And even so, I don't think that symbolic AI + LLMs are a complete model. You need to factor an "interface with the physical world" into the mix. I think that will be added to the LLMs fairly quickly, as it is adapted to office management of various sorts. And there's enough know
Re: (Score:2)
Re: (Score:2)
Are they a necessary part? We have lots of examples of animals that do not possess complex language but are considered intelligent. We also have examples of humans who were not taught language and yet exhibited intelligence.
Language is certainly a quick way to train something with impressive capabilities, but it doesn't seem to be a necessary part of intelligence.
Re: (Score:2)
So my model still forecasts a basic AGI around 2035.
Oh gosh [xkcd.com]. So...when it's invented it will be awesome? I agree with that interpretation.
Do you have actual reasoning for your 2035 prediction? We're still lacking the core part of the algorithm.
Bullshit (Score:2)
These problems are in no way "unexpected" and to "overcome" them they will have to do better than about 70 years of AI research. They will not be able to do that.
What is actually going on is that these assholes are lying and misdirecting in order to keep the hype going a bit longer.
Time To Sell My AI Stock (Score:4, Funny)
Re: (Score:2)
The bubble is definitely on the verge of bursting. I do think it may keep going a bit longer with modern techniques of manipulation and audience metrics, but the house-of-cards they built could now collapse at any time.
Re:Time To Sell My AI Stock (Score:5, Interesting)
Re: (Score:2)
They're expensive and unreliable. The only thing keeping them around at the moment is the belief that they'll rapidly improve.
Re: (Score:2)
They're expensive and unreliable. The only thing keeping them around at the moment is the belief that they'll rapidly improve.
I'm paying 20USD/month for a ChatGPT Plus subscription, and it's worth every penny. Speculative investment isn't the only revenue model.
Re: (Score:2)
I'm paying 20USD/month for a ChatGPT Plus subscription, and it's worth every penny. Speculative investment isn't the only revenue model.
I think they mean creating the models is expensive and that companies are currently taking a loss on providing the service even at $20/month. This may not be sustainable. However it sounds like the cost of creating those models is coming down fast. So it may be sustainable after all.
(I'm also a subscriber (but to Google Gemini rather than ChatGPT) and think it's worth the subscription.)
Logical AI is the path forward (Score:2)
Re: (Score:2)
Bla, blubb. If we knew even remotely how to do that, we would have had a slow but capable version of it for decades. We do not.
Re: (Score:2)
"We do not."
We do. Look into how Cyc works, and the work of Doug Lenat. Look into how Palantir works. Just because you don't know how to do it, or you haven't seen it, it doesn't mean it hasn't been done or isn't a lot easier to do now that CPUs, ram, and other technologies exist which didn't exist back then. The same reason people thought neural nets couldn't scale, until the technology got to this point where it could be scaled u
There is no AI (Score:3)
There is no "AI". What we have is "EFPM" (Extremely Fast Pattern Matching".
Calling this technology "AI" is pure marketing bullshit. There is no intelligence involved whatsoever :)
Re: (Score:2)
It is a bit more like statistical graph traversal (i.e. sort-of Markov Networks), because pattern matching is a yes/no thing. But that is it. No global "understanding", no model of reality, no "intelligence", just blindly doing small steps. That is why an LLM cannot find out it is hallucinating. It does not have any way to check plausibility.
Re: (Score:2)
The term AI has always been about marketing. (See Pamela McCorduck's Machines Who Think for details) The time to object to it, however, was the Dartmouth conference. There were a few people who raised objections at the time, if that makes you feel better.
AI is a very broad term that covers quite a few things you'd object to calling AI, like linear regression and clustering algorithms. That you want to term to be restricted to science fiction doesn't matter. Like the term 'neural network', which I find
The history of AI (Score:2)
Try something new, get exciting preliminary results, speculate wildly about the future, run into problems, abandon the approach.
Rinse, repeat
"It's going to get so much better, fast!!" (Score:3)
I told my boss that 2024 will be much like 2023, the same stuff more or less, and I got penalized because I wasn't fully committed to the AI gold rush. There's not going to be a huge breakthrough in a year. We are already at the limit, and it's going to get worse from here as the main, natural dataset is flooded with crap. I'm assuming Orion is just modeled from a dataset they built from a swarm of freelancers writing solutions, which is not very useful for real, unsolved problems, just academic problems – you know, the type of stuff you're expected to solve yourself so you learn how to think. This stuff seems like a plague for education (granted, if someone is lazy during their adult education, that's mostly their fault, not AI vendors).
Re: (Score:2)
It _is_ a plague for educators from what my colleagues say. My best friend took a sabbatical indefinitely until the university figures out how to re-establish ethics and stop the blatant cheating. Cheating and complaining about getting bad grades repeatedly is a normal thing for certain student populations. I said "just fail them all" which he mostly was doing.
Re: (Score:2)
Re: (Score:2)
Text is at the limit. Still image generation is pretty much there too. But other fields are still improving. Video generation has improved by leaps and bounds this year. I also heard of someone taking the large-x-model approach to robotics. They're still building a data set of motion, so it may take them awhile, but they already had some decent results.
Re: (Score:2)
Zuck is pragmatic as ever (Score:2)
"Some CEOs, including Meta Platforms' Mark Zuckerberg, have said that in a worst-case scenario, there would still be a lot of room to build consumer and enterprise products on top of the current technology even if it doesn't improve."
tl;dr "I don't really care if we advance, I can still make a crapton of money off this stuff, even if it hallucinates."
Re: (Score:2)