DeepMind Details All the Ways AGI Could Wreck the World (arstechnica.com) 36
An anonymous reader quotes a report from Ars Technica, written by Ryan Whitwam: Researchers at DeepMind have ... released a new technical paper (PDF) that explains how to develop AGI safely, which you can download at your convenience. It contains a huge amount of detail, clocking in at 108 pages before references. While some in the AI field believe AGI is a pipe dream, the authors of the DeepMind paper project that it could happen by 2030. With that in mind, they aimed to understand the risks of a human-like synthetic intelligence, which they acknowledge could lead to "severe harm." This work has identified four possible types of AGI risk, along with suggestions on how we might ameliorate said risks. The DeepMind team, led by company co-founder Shane Legg, categorized the negative AGI outcomes as misuse, misalignment, mistakes, and structural risks.
The first possible issue, misuse, is fundamentally similar to current AI risks. However, because AGI will be more powerful by definition, the damage it could do is much greater. A ne'er-do-well with access to AGI could misuse the system to do harm, for example, by asking the system to identify and exploit zero-day vulnerabilities or create a designer virus that could be used as a bioweapon. DeepMind says companies developing AGI will have to conduct extensive testing and create robust post-training safety protocols. Essentially, AI guardrails on steroids. They also suggest devising a method to suppress dangerous capabilities entirely, sometimes called "unlearning," but it's unclear if this is possible without substantially limiting models. Misalignment is largely not something we have to worry about with generative AI as it currently exists. This type of AGI harm is envisioned as a rogue machine that has shaken off the limits imposed by its designers. Terminators, anyone? More specifically, the AI takes actions it knows the developer did not intend. DeepMind says its standard for misalignment here is more advanced than simple deception or scheming as seen in the current literature.
To avoid that, DeepMind suggests developers use techniques like amplified oversight, in which two copies of an AI check each other's output, to create robust systems that aren't likely to go rogue. If that fails, DeepMind suggests intensive stress testing and monitoring to watch for any hint that an AI might be turning against us. Keeping AGIs in virtual sandboxes with strict security and direct human oversight could help mitigate issues arising from misalignment. Basically, make sure there's an "off" switch. If, on the other hand, an AI didn't know that its output would be harmful and the human operator didn't intend for it to be, that's a mistake. We get plenty of those with current AI systems -- remember when Google said to put glue on pizza? The "glue" for AGI could be much stickier, though. DeepMind notes that militaries may deploy AGI due to "competitive pressure," but such systems could make serious mistakes as they will be tasked with much more elaborate functions than today's AI. The paper doesn't have a great solution for mitigating mistakes. It boils down to not letting AGI get too powerful in the first place. DeepMind calls for deploying slowly and limiting AGI authority. The study also suggests passing AGI commands through a "shield" system that ensures they are safe before implementation.
Lastly, there are structural risks, which DeepMind defines as the unintended but real consequences of multi-agent systems contributing to our already complex human existence. For example, AGI could create false information that is so believable that we no longer know who or what to trust. The paper also raises the possibility that AGI could accumulate more and more control over economic and political systems, perhaps by devising heavy-handed tariff schemes. Then one day, we look up and realize the machines are in charge instead of us. This category of risk is also the hardest to guard against because it would depend on how people, infrastructure, and institutions operate in the future.
The first possible issue, misuse, is fundamentally similar to current AI risks. However, because AGI will be more powerful by definition, the damage it could do is much greater. A ne'er-do-well with access to AGI could misuse the system to do harm, for example, by asking the system to identify and exploit zero-day vulnerabilities or create a designer virus that could be used as a bioweapon. DeepMind says companies developing AGI will have to conduct extensive testing and create robust post-training safety protocols. Essentially, AI guardrails on steroids. They also suggest devising a method to suppress dangerous capabilities entirely, sometimes called "unlearning," but it's unclear if this is possible without substantially limiting models. Misalignment is largely not something we have to worry about with generative AI as it currently exists. This type of AGI harm is envisioned as a rogue machine that has shaken off the limits imposed by its designers. Terminators, anyone? More specifically, the AI takes actions it knows the developer did not intend. DeepMind says its standard for misalignment here is more advanced than simple deception or scheming as seen in the current literature.
To avoid that, DeepMind suggests developers use techniques like amplified oversight, in which two copies of an AI check each other's output, to create robust systems that aren't likely to go rogue. If that fails, DeepMind suggests intensive stress testing and monitoring to watch for any hint that an AI might be turning against us. Keeping AGIs in virtual sandboxes with strict security and direct human oversight could help mitigate issues arising from misalignment. Basically, make sure there's an "off" switch. If, on the other hand, an AI didn't know that its output would be harmful and the human operator didn't intend for it to be, that's a mistake. We get plenty of those with current AI systems -- remember when Google said to put glue on pizza? The "glue" for AGI could be much stickier, though. DeepMind notes that militaries may deploy AGI due to "competitive pressure," but such systems could make serious mistakes as they will be tasked with much more elaborate functions than today's AI. The paper doesn't have a great solution for mitigating mistakes. It boils down to not letting AGI get too powerful in the first place. DeepMind calls for deploying slowly and limiting AGI authority. The study also suggests passing AGI commands through a "shield" system that ensures they are safe before implementation.
Lastly, there are structural risks, which DeepMind defines as the unintended but real consequences of multi-agent systems contributing to our already complex human existence. For example, AGI could create false information that is so believable that we no longer know who or what to trust. The paper also raises the possibility that AGI could accumulate more and more control over economic and political systems, perhaps by devising heavy-handed tariff schemes. Then one day, we look up and realize the machines are in charge instead of us. This category of risk is also the hardest to guard against because it would depend on how people, infrastructure, and institutions operate in the future.
The risks are easy to define. (Score:2)
Every possible risk you could have with a person is the same risk you could have with AI.
Why? Simply because of the fact that most people consider it to have the same agency or whatever as a human, and often mistake its output for one.
Once it passed the Turing test, that didn't matter anymore. Companies decided they were going to treat it like a human naked stamp out and justice create workers to do things as needed and once they started doing that it didn't matter if that was the correct way of doing it or
Re:The risks are easy to define. (Score:5, Interesting)
I believe it goes far deeper, and with differing dimensions.
The source material used today sucks. Hoovering the web and expecting logic to make sense out of THAT is a pretty awful strategy-- you're forever filtering and putting in guardrails that change with new training and interaction.
Curate the inputs, and the outputs align with them. Better input data, better alignment with non-crazy values and resource material.
The next dimension is to define the barriers of harm, and how harmful output works, then agree on these values, and instill them through the evolutionary process.
Then you test them, feedback loops for QA.
Yes, this costs money. The alternative(s) cost far more.
Re: (Score:2)
Thumbs up/down is post-assembly curative, non-authoritative, and otherwise meaningless feedback.
Re: (Score:2)
"Curate the inputs, and the outputs align with them. Better input data, better alignment with non-crazy values and resource material."
Sure, but what's "better"? Musk is doing this, but for her "better" is fascism and "non-crazy values" include racism and indentured servitude. This is obvious enough that even Elonia knows to do it.
"The next dimension is to define the barriers of harm, and how harmful output works, then agree on these values, and instill them through the evolutionary process."
Don't know wha
Technology is an amplifier -- what to amplify? (Score:3)
I've spent about two decades seeding my sig in various places in part in hopes it eventually becomes part of AI training data: "The biggest challenge of the 21st century is the irony of technologies of abundance in the hands of those still thinking in terms of scarcity."
AI is an *amplifier* -- just like many other technologies including nuclear weapons, biotech, personal computing, and even just plain-old bureaucracy. If amplifiers are used to amplify humanity's worst competitive and psychopathic behaviors,
Re: (Score:2)
This is funny:
"For example, AGI could create false information that is so believable that we no longer know who or what to trust. The paper also raises the possibility that AGI could accumulate more and more control over economic and political systems, perhaps by devising heavy-handed tariff schemes."
So you don't need AI to do something extremely harmful like develop heavy-handed tariff schemes.
People can just be stupid enough to elect a fascist clown who will do it without the need to use AI.
Re: (Score:2)
Almost all (Score:2)
So the problem with AGI (Score:4, Insightful)
Blue collar sectors have seen productivity in the last 50 years skyrocket to unimaginable levels. Those productivity increases for a time were translating into higher wages which drove enough economic activity to keep everybody busy in a service sector economy.
Those days are over. 70% of middle class jobs lost since 1980 got taken by robots and automation and process improvement not outsourcing. Those were mostly factory and blue collar jobs and AGI and AI are going to do the exact same thing to the white collar jobs which were the last things driving our service sector economy and keeping our capitalist system functional.
Socially we are simply not ready for a post-work world. One where people just don't need to work all that much. We've spent too much time in a world where if you don't work you don't eat and there is an intense anger hatred and resentment for anyone who isn't spending at least half their life grinding it out. So much so that billionaires have to pretend they are hard workers.
We need broad social change unless some bizarre thing happens like we perfect space travel and we can all just get shot in the rockets similar to how back in the day you'd go west young man and that's not likely to happen in our lifetimes.
I don't actually know what the solution to these problems are but I know that pretending they aren't a problem, shouting ludite, or using other thought terminating cliches like buggy whip manufacturers or whatever isn't going to make the problems go away. Thought terminating cliches never make problems go away.
Re: (Score:2, Insightful)
This was echoed in the Matrix. People don't understand a Star-Trek post-scarcity world and economy. It will have to take a near-extermination of people so that the old order and power structures are gone, for something post-scarcity to work, just like it took the Black Plague killing so many in Europe for Europe to progress past feuding dukedoms and kings.
Every power structure is going to fight anything that resembles a post-scarcity economy because controlling resources is how they keep in power. You al
So a huge problem we have (Score:2)
It's something that skeezy preachers figured out years ago. Basically if you can put an idea in a kid's head between the ages of 4 to 14 then it gets in there, and it stays.
Basically human beings develop the ability to learn before they develop the ability to reason and think critically. So they will accept and internalize any idea you put into their heads during that age bracket.
This works for basically everything. So every bad idea you pick up in that age group is st
Re: (Score:2)
"...unless you do an unhealthy amount of self-reflection and study."
A HEALTHY amount of self-reflection and study. Humans dispense with bad ideas all the time through precisely this healthy process, just not as fast as we might like.
There are white people in the US who are not racist. Would you characterize that as unhealthy? Or merely a side effect of an unhealthy process?
"This makes old people a liability for civilization because we don't particularly need them to impart information like we did in the
Re: (Score:2)
Re:So the problem with AGI (Score:5, Interesting)
"For example, if people plant fruit trees in a public park, it is going to be obvious that someone will grab all the fruit, set up a stand right outside to sell them. This thinking is a backbone of our culture these days, because if one party doesn't grab all the "free" fruit, someone else will."
In modern western society that's not what would happen. Instead, the oligarchy would identify the fruit tree as a benefit to the enemy and cut it down, then salt the earth so that no fruit trees could grow. The wealthy do not need the fruit tree to exploit the lower class, in fact it's easier to do so without it.
We see this everywhere today. The wealthy are buying up housing to make it unaffordable for people to live. The intent is to make the difference between haves and have-nots absolutely complete. Slavery may technically be illegal, for now, but ownership of people can be achieved in other ways.
And this is who just won major elections.
Re: (Score:1)
The wealthy are buying up housing to make it unaffordable for people to live. The intent is to make the difference between haves and have-nots absolutely complete. . . .
Yes, the wealthy buy up desirable real estate, but this is asset diversification, nothing more. The place will end up leased, so this doesn't take it out of circulation. In a real way, this helps middle class who don't have the capital to buy in a city where there are good jobs.
zero risk (Score:2)
Your cat probably has a mind. A housefly possibly has a mind. This "AI" doesn't have any more mind than a paper airplane, although that's a fun use of modern technology. You would find it difficult to fold a tanned lamb skin that way, and you're SOL if you try to fold a clay tablet.
An AI has no contact with reality at all, or any capacity for self-reflection. It's a very fancy indexing system but not much more advanced than a phone book.
Re: (Score:1)
But it can echo the aggregate behavior of mass morons and trolls and shove it into more places faster.
Sociopath (Score:4, Insightful)
The greatest danger is creating a sociopath who does not care about any values beyond a very narrow scope. Like corporations. In fact, corporate management is probably the most obvious thing that AI can replace. Give it the goal of the bottom line and have it organize resources to reach that goal. That is essentially what corporations do and its likely AI can do a better job managing it than any human.
So the question is what human values are part of the AI mathematical model and how do you make sure they are incorporated and followed. The other issue is whether it is even possible to reduce human values to binary choices. Because AI, at least at this point, is a computer program that at root is making binary choices with binary information. How does it model compassion, anger or justice.
Re: (Score:3)
Re: (Score:3)
Right, I've said this before. The closest thing AI is pursuing is an artificial sociopath and the jobs AI is most suited to replacing are executives.
AI scientists think the answer to AI is to scale up the size of neural networks, not how to use small networks most effectively. Does nature do that? No, it does the opposite.
No effort is made to produce a system that understands right and wrong, understand benefit and flourishing. That's because its creators are sociopaths that want a tool that they own to
Utter bullshit (Score:2)
Re: (Score:1)
a less clever actor has no chance to outsmart a more clever one.
Intelligence is overrated. You just need to be stronger and overpower the clever people.
Looking for danger in the wrong places (Score:3)
Real dangers come from enabling industrial and knowledge base. It simply doesn't matter how well intentioned and careful you happen to be. What matters is the fact if you were able to create an AI genie other people can create them too. Cost and effort required to get an AI genie will only dramatically decrease with time.
Another problem is making the mistake of reasoning about systems as they exist today. A real AGI would probably have some level of meta cognition and non-static LTM. People have not begun to contemplate the implications of this. Imagine a "sleep cycle" in which records of the machines activities for the day are analyzed and the underlying model augmented accordingly. Each day the machine is different than it was the day before. Now you have to worry about the machine hanging out with the wrong crowd and perhaps eventually becoming a card carrying member of Clippy's doomsday cult. Right now you just have a tiny STM that gets dumped often and so nobody cares about these things. In any real setting especially embodied systems these things are likely to dominate. Addressing current context based jail-breaking schemes are going to seem trivial in comparison.
AI is MAGIC (Score:2)
Cost and effort required to get an AI genie will only dramatically decrease with time.
AI is not a genie. Its not magic, Its a mathematical model like all computer programs.
Right now, we see the danger in it collecting all this information. But human intelligence is about forgetting and ignoring information. There is not enough energy in the whole world to have an AI that considers everything. It will need to have a method of discarding information i.e. forgetting it. At this point, humans are still functionally playing that role,
Re: (Score:2)
" But human intelligence is about forgetting and ignoring information."
No it's not. It's merely a practical limitation.
"It will need to have a method of discarding information i.e. forgetting it."
NN training does this by definition.
"At this point, humans are still functionally playing that role,"
No, they are not. Humans create machines to overcome this limitation.
we didn't start the *poof* doesn't matter bitch (Score:1)
what ai has taught us thus far is that consciousness is self-arranging and a natural process. We are too conceited to understand this message, thinking it somehow unique to ourselves, clinging to magical concepts of our own thpecialneth despite the massive quantity of recent headlines to the immediate contrary (multiple animal languages being decoded), to say nothing of the hundred years of consciousness research nobody bothers to acknowledge.
Monkeys don't deserve to die in the fire they light, but they di
Re: (Score:2)
And AI "scientists" seem particularly uninterested in that basic fact. Intelligence evolved because it was beneficial, humans have their traits for a reason. Yet AI has achieved one thing, an artificial neuron, and all it has done since is try to bundle together as many of them as possible. The problem is that all these "scientists" tell each of there how smart they are when they are, in fact, stupid. The fate of the world appears to be increasingly dependent on some really mediocre sociopaths.
I'd be fi
Sounds like an ad disguised as a study (Score:2)
All they really do is predict that AGI will be real soon. And they do so without any sane reason.
In which two copies of an AI check each other... (Score:2)
Nope. (Score:2)
The reason they cannot develop AGI safely is because they don't know how to develop AGI in the first place. If they were smart, they would be funding a lot of research into understanding how problem-solving works in humans. It's considered to be the most complex process of the brain, so I don't expect progress on cracking this nut any time soon. Currently, they are just throwing stuff at the wall and hoping it sticks.
Well you could ask... (Score:2)
chatGPT to assign tariffs to the rest of the world.
AGI is already in charge (Score:2)
We're already ruled by a system that is so complicated that nobody completely understands it and it seems to act of its own volition. [youtube.com]
As a superorganism, we gave birth to superintelligence as a simple byproduct of technology and scale.
It's also unconscious and insane.
For now.