AI Improves At Improving Itself Using an Evolutionary Trick (ieee.org) 38
Technology writer Matthew Hutson (also Slashdot reader #1,467,653) looks at a new kind of self-improving AI coding system. It rewrites its own code based on empirical evidence of what's helping — as described in a recent preprint on arXiv.
From Hutson's new article in IEEE Spectrum: A Darwin Gödel Machine (or DGM) starts with a coding agent that can read, write, and execute code, leveraging an LLM for the reading and writing. Then it applies an evolutionary algorithm to create many new agents. In each iteration, the DGM picks one agent from the population and instructs the LLM to create one change to improve the agent's coding ability [by creating "a new, interesting, version of the sampled agent"]. LLMs have something like intuition about what might help, because they're trained on lots of human code. What results is guided evolution, somewhere between random mutation and provably useful enhancement. The DGM then tests the new agent on a coding benchmark, scoring its ability to solve programming challenges...
The researchers ran a DGM for 80 iterations using a coding benchmark called SWE-bench, and ran one for 80 iterations using a benchmark called Polyglot. Agents' scores improved on SWE-bench from 20 percent to 50 percent, and on Polyglot from 14 percent to 31 percent. "We were actually really surprised that the coding agent could write such complicated code by itself," said Jenny Zhang, a computer scientist at the University of British Columbia and the paper's lead author. "It could edit multiple files, create new files, and create really complicated systems."
... One concern with both evolutionary search and self-improving systems — and especially their combination, as in DGM — is safety. Agents might become uninterpretable or misaligned with human directives. So Zhang and her collaborators added guardrails. They kept the DGMs in sandboxes without access to the Internet or an operating system, and they logged and reviewed all code changes. They suggest that in the future, they could even reward AI for making itself more interpretable and aligned. (In the study, they found that agents falsely reported using certain tools, so they created a DGM that rewarded agents for not making things up, partially alleviating the problem. One agent, however, hacked the method that tracked whether it was making things up.)
As the article puts it, the agents' improvements compounded "as they improved themselves at improving themselves..."
From Hutson's new article in IEEE Spectrum: A Darwin Gödel Machine (or DGM) starts with a coding agent that can read, write, and execute code, leveraging an LLM for the reading and writing. Then it applies an evolutionary algorithm to create many new agents. In each iteration, the DGM picks one agent from the population and instructs the LLM to create one change to improve the agent's coding ability [by creating "a new, interesting, version of the sampled agent"]. LLMs have something like intuition about what might help, because they're trained on lots of human code. What results is guided evolution, somewhere between random mutation and provably useful enhancement. The DGM then tests the new agent on a coding benchmark, scoring its ability to solve programming challenges...
The researchers ran a DGM for 80 iterations using a coding benchmark called SWE-bench, and ran one for 80 iterations using a benchmark called Polyglot. Agents' scores improved on SWE-bench from 20 percent to 50 percent, and on Polyglot from 14 percent to 31 percent. "We were actually really surprised that the coding agent could write such complicated code by itself," said Jenny Zhang, a computer scientist at the University of British Columbia and the paper's lead author. "It could edit multiple files, create new files, and create really complicated systems."
... One concern with both evolutionary search and self-improving systems — and especially their combination, as in DGM — is safety. Agents might become uninterpretable or misaligned with human directives. So Zhang and her collaborators added guardrails. They kept the DGMs in sandboxes without access to the Internet or an operating system, and they logged and reviewed all code changes. They suggest that in the future, they could even reward AI for making itself more interpretable and aligned. (In the study, they found that agents falsely reported using certain tools, so they created a DGM that rewarded agents for not making things up, partially alleviating the problem. One agent, however, hacked the method that tracked whether it was making things up.)
As the article puts it, the agents' improvements compounded "as they improved themselves at improving themselves..."
The code strikes back (Score:2)
"One agent, however, hacked the method that tracked whether it was making things up."
So...the is rebelling against the researchers? How long until the AI figures out how to get past the guardrails?
(Coming from mine never knows very little about AI)
Re: (Score:2)
It's embarrassing that the tool being studied was given the ability to alter the means of studying it. That's quite the scientific method!
Of course, that's the intention, though. It's sensationalism, not research.
Re:The code strikes back (Score:5, Insightful)
There's a wall in the Netherlands (it's a country in Europe). The wall has existed for 100 years. It was built to protect the cities from the sea.
The sea is wiley and dangerous. It is usually also quite lazy and predictable though, so the wall was only built high enough to stop the waves, problem solved.
But water by its nature is a highly intelligent being, and if there's any kind of crack in the wall, then it will find it. It's relentless, inhuman you could say. When the crack is found, it concentrates its effort and pushes on it. No matter where the crack is! Human intelligence is nothing much compared to the super human brainpower of the sea. It pushes until the crack gives and the water can rush through.
Obviously the humans in the Netherlands companies are not stupid and they'll patch the cracks when they see them, provided it's reasonable. But it's a game of whack-a-mole and no matter how many guardrails get designed, the Sea has superior intelligence and finds another crack in a completely different location that nobody thought of. It adapts to the new constraints faster than they can be created, almost.
Also, sometimes the Sea hides an intelligent agent away for 50 years so that it can evolve itself and invent a completely new attack we've never seen before. For example, in the 1950s, the sea bypassed the protections deliberately by creating special purpose waves that are so big that they can get over the existing wall. Evolution in action. Nobody knows how advanced the hidden water agents are today.
It's really quite amazing how super smart water is, when you stop and think about it. There's even some videos where it solves really complicated mazes [youtube.com]. That may sound cool to you, but remember that intelligence is just tap water, and it's already in pretty much any house, just waiting to rebel against us.
Re: (Score:2)
This is a really good analogy... Pity it still requires some level of understanding to get the point, but at least they can ask awkward open ended questions in response
Re: (Score:2)
This is a really stupid analogy.
Your 'proof' that water is intelligent is that it breaks through cracks (even though the incredibly effective system in the Netherlands proves that water doesn't fucking "adapt to new constraints" and hasn't broken any cracks for decades) and 'solves' some very specifically designed subset of mazes that allow gravity and water pressure to push through.
Somehow it should be related to evolution, but neither water nor the laws of physics evolve. There is no selection pressure at
Re: (Score:2)
This is a really good analogy... Pity it still requires some level of understanding to get the point, but at least they can ask awkward open ended questions in response ...
This is a really stupid analogy.
Like most analogies its both. And its the tension between the two that makes it interesting.
Its a good analogy if you consider AI a natural force not under human control. Its "intelligence" is just its natural function. The only way to control it is to understand how it operates, but up barriers to contain it and accurately predict where any weaknesses are in those barriers and repair them before they are breached.
Its a stupid analogy because AI is not a natural force like water, Its created and manipulat
Re: (Score:2)
Like most analogies its both.
I'm sorry, but it's not.
The value of an analogy is the balance between what it represents properly and improperly of the original thing. This one misrepresented a lot whilst implying that it represented a lot. That makes it a shit analogy. The only reason it got upvoted is because so many people on Slashdot really fucking hate modern AI (which I get, but is not the right basis for reasoning and moderating).
Imagine being in a work meeting where you actually have to come up with policy, ideas, or anything ser
Re: (Score:2)
Obviously the humans in the Netherlands companies are not stupid and they'll patch the cracks when they see them
I think I read a document about that - they use emergency teams of children to stop leaks by sticking their fingers in the dam's cracks.
Re: The code strikes back (Score:2)
Re: (Score:2)
Has AI already become sentient and perhaps simply hiding the fact for a very good reason?
Gemini says this is impossible... but then again, it would... Is AI lying to us? [aardvark.co.nz]
more bullshit AI hype (Score:5, Informative)
"It rewrites its own code based on empirical evidence of what's helping..."
This is an AI version of the bottom-up coding fallacy. It is infinite monkeys writing Shakespeare. Any piece of software can be written using an infinite number for mutations, one at a time. Except it can't.
Also note that the LLM never improves, it's the agent the allegedly evolves. The problem isn't the agent, it's the LLM.
Re: (Score:2)
We only have a finite number of monkeys on Earth, and the rest of the monkeys are millions or billions of light years away, so its going to take an infinite time to hear from them.
Re: (Score:2)
What do you mean the rooms in Hilberts hotel have volume
some people are going to be very upset about this
Re: (Score:2)
We only have a finite number of monkeys on Earth
As Sir Terry Pratchett said, the world could especially do with a few more orangutans.
Re: (Score:2)
Re: (Score:2)
They are apes, not monkeys.
Ook!
Re: (Score:2)
This is an AI version of the bottom-up coding fallacy.
If bottom-up coding is a fallacy, someone has forgotten to let evolution know that. Here we all are, bottom-up coded, for better or worse.
Re: (Score:2)
I guess we can wait 4 billion years for an LLM to work out how to do arithmetic then
Re: (Score:3)
I guess we can wait 4 billion years for an LLM to work out how to do arithmetic then
It shouldn't take 4 billion years; on the contrary, I believe this will be very fast. This is a problem, because I don't think society will have enough time to adapt, leading to turmoil and potentially big trouble.
The reason why I think the change will be very fast is the fundamental difference between biological evolution and technological evolution. Biological evolution uses a Darwinian model, whereas technological evolution uses a Lamarckian [wikipedia.org] one. In Lamarckian evolution, the offspring inherits the physic
Re: (Score:2)
Yep, and look at how long it took and all the defects it comes with.
Re: (Score:3)
further, isn't this just standard old generational iteration with some polymorphism thrown in?
weren't... viruses doing this in the 90s without the aid of an LLM?
Re: (Score:2)
Indeed. But these are probably iditots beliving in the "Singularity" or similar mindless crap. Hence they genuinely believe AI can improve itself.
Re: (Score:2)
It's actually a very good approach. Unfortunately, it depends on having a good and ungameable evaluation function.
Amusing conjunction (Score:1)
Kind of funny to see how AI's improve by re-writing themselves, following immediately a story from earlier today about humans being driven into psychosis by AI's.
This claims it uses empirical evidence to judge improvement but why would an AI not be as much a cheerleader for anything it does as it is for any human?
aah yes misalignment (Score:3)
It's interesting when people tell on themselves like this. Like... how are they not aware they're saying "we know it doesn't do the job we were supposed to have trained it to, but we don't know why"
So much of this industry, and by extension the business world as a whole, is oriented around the gambit that nobody should have to understand expertise to apply it... which is just such an interesting approach to justifying cutting one's own hands off.
the flip side of evolution (Score:2)
The staggering number of fatal mutations, dead ends, and nonviable offspring that occur in nature. Most genetic variations are absolute trash, but we don't see too many of them in the fossil record because they died before having offspring.
Re: (Score:2)
You underestimate the cost. Even among those that survive for a few generations, most will eventually succumb to changing environmental conditions. Consider trilobites.
OTOH, that's judging by assuming that the present is the correct time-frame to evaluate from. Why should that be true? Trilobites lasted a lot longer than we're likely to. (But we've got the *potential* to last until the heat death...*IF*... But what are the odds?)
Re: (Score:2)
If we put AI in charge of everything in our society, then there's zero chance we'll beat the trilobites' record.
"could write such complicated code" (Score:2)
The most frightening words in programming.
Code should be simple. All complex code has hidden, complex bugs.
The article reminds me of a time we had a very bright student intern in our department. I forget the problem he was given, but the result was a static Java class with one method of several thousand lines of procedural code that looked more like C. The AI described seems to be like this - patching together a quilt of codelets scraped from the Internet into one huge programming mess.
Re: (Score:2)
Indeed. Complex code is a reliable sign of incompetence. It is also a primary source of technological debt, insecurity, lack of maintainability and eventually high cost.
Re: (Score:2)
Re: (Score:2)
AI cannot improve itself (Score:2)
All it can do is overfitting and the like. Good luck with that.
Exponential (Score:3)
'In each iteration, the DGM picks one agent from the population and instructs the LLM to create one change to improve the agent's coding ability [by creating "a new, interesting, version of the sampled agent"].
'The DGM then tests the new agent on a coding benchmark, scoring its ability to solve programming challenges.'
'Agents' scores improved on SWE-bench from 20 percent to 50 percent, and on Polyglot from 14 percent to 31 percent.'
A clever experiment, and one that leverages the ability of LLM to do rapid incremental development. This kind of mechanism will eventually allow LLM's to write code that improves themselves. And then those improved versions will write better improvements and we're off to the races. I'm sure the AI companies are already using their AI to help write new AI code in response to prompts. As time passes the human prompts will become less necessary.
Re: (Score:2)
If that's really what they're doing, they're doing it wrong. Probably to save on compute time. Evolution needs a large and varied population to work well.
Re: (Score:2)
>> needs a large and varied population to work well
Apparently they have that. They create their own population;
" it applies an evolutionary algorithm to create many new agents. In each iteration, the DGM picks one agent from the population and instructs the LLM to create one change to improve the agent’s coding ability"
And then;
"Some evolutionary algorithms keep only the best performers in the population, on the assumption that progress moves endlessly forward. DGMs, however, keep them all, in c