How Google Finally Leapfrogged Rivals With New Gemini Rollout (msn.com) 38
An anonymous reader shares a report: With the release of its third version last week, Google's Gemini large language model surged past ChatGPT and other competitors to become the most capable AI chatbot, as determined by consensus industry-benchmark tests. [...] Aaron Levie, chief executive of the cloud content management company Box, got early access to Gemini 3 several days ahead of the launch. The company ran its own evaluations of the model over the weekend to see how well it could analyze large sets of complex documents. "At first we kind of had to squint and be like, 'OK, did we do something wrong in our eval?' because the jump was so big," he said. "But every time we tested it, it came out double-digit points ahead."
[...] Google has been scrambling to get an edge in the AI race since the launch of ChatGPT three years ago, which stoked fears among investors that the company's iconic search engine would lose significant traffic to chatbots. The company struggled for months to get traction. Chief Executive Sundar Pichai and other executives have since worked to overhaul the company's AI development strategy by breaking down internal silos, streamlining leadership and consolidating work on its models, employees say. Sergey Brin, one of Google's co-founders, resumed a day-to-day role at the company helping to oversee its AI-development efforts.
[...] Google has been scrambling to get an edge in the AI race since the launch of ChatGPT three years ago, which stoked fears among investors that the company's iconic search engine would lose significant traffic to chatbots. The company struggled for months to get traction. Chief Executive Sundar Pichai and other executives have since worked to overhaul the company's AI development strategy by breaking down internal silos, streamlining leadership and consolidating work on its models, employees say. Sergey Brin, one of Google's co-founders, resumed a day-to-day role at the company helping to oversee its AI-development efforts.
Like GPU benchmarks (Score:5, Insightful)
Eventually the models will be tuned to just perform well on the tests and perform like crap outside of the tests.
Re:Like GPU benchmarks (Score:5, Interesting)
Eventually? We are kind of already there. I recall some question on one of these going viral, attracting a lot of actual humans to write up why they felt the AIs struggled with it including answering in their writeups. So then their writeups made their way into the RAG inputs into LLMs and also into training material. The AIs suddenly got better at that question, what a surprise...
Just like most specific examples of LLM screwups get self-corrected in short order, automatically as the mocking ironically shapes the RAG component to avoid the specific behavior. Suddenly the LLMs got really good at counting the number of 'r's in strawberry, even as they couldn't actually count letters, but the internet now said how many rs were in strawberry just a whole bunch of times...
Re:Like GPU benchmarks (Score:5, Interesting)
What you're describing isn't an implementation of RAG, but supervised post-training like Directed Preference Optimization. In DPO, researchers compile sets of answers from LLMs to the same question with a human reviewer selecting which one is a better answer. This data is used to fine tune, which steers models towards the good answers and away from the bad ones, which is why it is such an effective means of changing a models' behavior to a certain style of answer.
It gets pushed way too far, which is why you can find tiny open-weights models that crush benchmarks, and tell you confidently that "there are 3 'r's' in strrawberrrry," because, as you point out, they can't count letters in token sequences.
Re: (Score:2)
and tell you confidently that "there are 3 'r's' in strrawberrrry," because, as you point out, they can't count letters in token sequences.
While there's certainly no guarantee for it to be- there's no reason such capability can't be generalized.
gpt-oss-120b:
Me:
How many times does the letter "v" appear in the following sentence:
```
Verily I say, it's very nice to invoke a vehement interaction with your visage
```
GPT:
Counting them:
1. V (in *Verily*)
2. v (in *very*)
3. v (in *invoke*)
4. v (in *vehement*)
5. v (in *visage*)
So the letter “v” appears **5 times** in the sentence (including the initial capital V).
*(If you count only lowercase“v”, there are 4 occurrences.)*
Re: (Score:2)
The fact that a model could not does not mean that LLMs cannot.
gpt-oss-120b:
Me:
How many times does the letter "z" appear in the following word: "zlkjxcvlzkjxlcvzzZZZzlskdfjasdzz"
GPT:
So:
- **Lowercase ‘z’: 7 occurrences**
- **Uppercase ‘Z’: 3 occurrences**
**Total (caseinsensitive) = 7+3 = 10**.
Have VW entered the market? (Score:4, Funny)
Just saying...
Re: (Score:2)
Seems to me that it already is. ChatGPT is significantly better at reading circuit diagrams (although it struggles with them too). Still, in this regard (and generally electronics) ChatGPT performs better than both Claude and Gemini.
Re: (Score:2)
What you describe is over-fitting.
The goal is generalization.
Models have become better and better at generalization, not more and more over-fitted.
New version of Rectum (Score:2)
New version of Rectum now produces shit at 3 times closer to shit consistency benchmarks than previous versions of competitors Anus and Colon. Given that we've already decided that you just need more shit constantly and forever and will shove it into every aspect of your life, this must make you very happy.
Good job (Score:5, Interesting)
I saw this news on LI a few days ago, so I headed over to try my 'stock test'. I have a programming question which isn't so easily found in online examples and whatnot, and on the face of it is easy, but actually it's a bit involved. I requires a two step solution (a sort of parse and then parse again type thing). I'd say it's maybe 100 lines of python to solve (at most). I didn't give it any examples of input/output, and my prompt was maybe one or two lines long at most - not a long essay spoon feeding it the implementation.
I asked ChatGPT and got the usual one-pass not-quite-there-yet answer. It's the sort of answer you'd expect a human junior to give you before you talked it over and explained the deficiencies of it.
I asked Gemini, and it gave me a working program, with a decent example input (with a proper two pass solution). The code had comments which actually explained what was going on, and the code was pretty nice (descriptive variable names and so on). I went on to ask it to make a relatively simple change, and again, same sort of response. I'd say it was close to "commit ready", if it were a task in a ticket I was working on or something. I'd probably do some more testing with a few more inputs (maybe get it to write some unit tests?) and assuming all was well, I'd commit it.
I realise this is just one test of billions of possible ones, but it's one that every AI I've tried it on has failed to answer properly. Since it's the first that actually did answer it, and honestly, answered it really well, I'd say they do really seem to have 'cracked it' somewhat, at least for Python programs. It probably doesn't solve every problem, and it's still prone to making stuff up, but it's definitely got something about it that's good. I am tempted to try and connect it up to my IDE to see what I can do with it, but haven't taken the plunge yet - it's the first AI I've felt is worth using.
What it knows about Lindsey Lohan I couldn't say though ;-)
Re: (Score:2)
So how long before it can start to rewrite its own code to "improve" itself?
I'm only half joking.
Re: (Score:2)
So how long before it can start to rewrite its own code to "improve" itself?
I'm only half joking.
Well, the LLMs don't really consist of "code" per se, but I think the AI labs are already using them to work on improving their own design. How far are they from being able to do this without human oversight and supervision? I have no idea.
Re: (Score:3)
"Well, the LLMs don't really consist of "code" per se"
Oh they do, a LOT of it, all the way from the high level python libraries such as TensorFlow, via sigmoid or relu activation functions down to the low level CUDA GPU libs. The only part that is pure data is the weighting of the neurons which themselves are code.
Re: (Score:3)
All of the stuff other than the weights is basically just scaffolding, though. Necessary, but the stuff that makes it "intelligent" is all in the weights.
Re: Good job (Score:2)
A database is useless without data too. Most computer programs are.
Re: (Score:2)
An untrained transformer is like a CPU. A trained transformer is like a CPU plus a program (the weights) that run on it.
You could say that the functioning of any computer program is mostly down to the CPU, but while somewhat true, that's a useless statement and fails to understand that it's the program where the logic is.
In the exact same way, saying that the functioning of a trained transformer (LLM) is down to the transformer, is somewhat true, but fails to capture what is really going on in exactly the s
Re: (Score:2)
The interesting part for AI to improve itself is coming up with better architectures and with better optimizers. The training part is just something that needs time and compute, but the next step in AI will probably involve an architecture solving some of the current drawbacks, like for example attention scaling badly for long context.
Re: (Score:2)
I just tried my standard "draw me an ASCII middle finger" and it flat-out refused to generate it for me! Grok still sucks at it but at least it tried. And it gave me the finger through emoji afterward. Gemini needs more work, I think.
Re: (Score:2)
I just tried my standard "draw me an ASCII middle finger" and it flat-out refused to generate it for me!
Yep, the guardrails are also improving.
Re: (Score:2)
At some point when I was bored and playing around with LLMs, I had chatGPT keep making ascii art over and over. I'd ask for something like a duck holding an umbrella. No matter what blobbly garbage it produced (and it was all blobby garbage), I just kept encouraging it and telling it to make more. Add details, refine, making it even duckier, etc. And it just kept taking the input blob and adding more blobs to it, like an infinite ascii blob spiral.
Re: (Score:2)
AI is still so bad at it that it makes for a great test. It's interesting too, because you can see that it sort of gets close, but it's still WAY off!
Re: (Score:3)
I just asked it a fairly simple (in my opinion) question: "What are the top 3 tier one parts suppliers in the North American automotive market, by revenue?"
It very confidently gave me 3 tier 1 suppliers for the 2022 fiscal year. The top, not surprisingly, was Magna, which is probably true. But it said the revenue was ~$18.9 billion. That doesn't seem to line up with any facts I can find online about Magna. Typical revenue is more like $10 billion per quarter, or $40 billion per year. I can't figure out
Re: Good job (Score:3)
Re: (Score:3)
Re: Good job (Score:2)
Re: (Score:2)
Too bad it can't, you know, go find me some facts and all
Tool-enabled LLMs can. This is old tech.
or at least tell me when it can't find any.
Can do that too.
It generates text that looks statistically like text it has seen.
This bullshit again.
It's not remotely true.
An LLM is trained to give responses that match existing text, this does not mean they "generate text that looks statistically like text it has seen."
Re: (Score:1)
Buy an ad.
Re: (Score:2)
Similar results here, Gemini 3 does an outstanding job at coding. Best I have seen.
Re: (Score:2)
Gemini failed like all others, falsely claiming it's impossible. I'd say it failed worse than any AI yet, as it offered an mostl
How much better is it *Really* ? (Score:1)
So how much better is Google's new product for real users ?
And are we seeing signs that LLMs are asymptotically approaching a maximum, a hard limit of the technology that throwing more bytes at them will not solve ?
Re: (Score:2)
Currently there is no asymptote yet. The improvements seem roughly linear and the model size/cost drops exponentially.
There is clearly a ceiling how much knowledge you can fit into a model of a certain size (so even when a 4B model beats GPT-3 in problem solving it won't beat it at knowledge questions), but if you instead go for "cleverness" there may be still room for improvement before hitting a ceiling.
Useless AI benchmarks (Score:4, Interesting)
Benchmarks are more marketing tools than accurate reflections of model capability. The only thing that matters is what users think.
Re: (Score:2)
Yeah, this. I prompted just a couple days ago for an example code gen of bit-bashing SD 4-bit SD mode and got garbage. Told it to use the specs from SD Association too.
If these LLMs can't see something to regurgitate then they're useless. There's no intelligence. They're just the new search engine is all.
Re: (Score:2)
That's not an insult- interacting with these things is a skill in itself.
Re: (Score:2)
You've missed a pretty big detail. 4-bit SD mode is not done as bit-bashing at all. Only 1-bit SPI mode has lots of examples of bit-bashing in the open.
All public native SD mode solutions revolve around a hardware controller handling all the data transfers, and uses a provided API wrapper. This closed situation is why I'm using it as a test of intelligence.
Since the LMM can't find any code base to reference, it therefore can't regurgitate any solution ... so responds accordingly. I suppose that's better
Who cares? (Score:1)
Dead company walking.
Maybe gemini will stop (Score:2)
making things up. I've searched for products and it has make up entire products complete with SKU's out of thin air.
Re: (Score:2)