AI Insiders Issue New Warnings - Including Former Anthropic Engineer Jacob Coxon (cnn.com) 93
Neither OpenAI or Anthropic is acting responsibly, warned AI researcher Jacob Coxon when resigning last week from Anthropic. But just hours earlier, OpenAI VP of Research Aidan Clark had posted "For the first time, I am asking myself if things are moving too fast. I'm honestly not sure, but I am sure that it would be good for us to have an answer to 'What would a successful pace look like?'"
CNN noted Wednesday they're just some of the many AI insiders who are now concerned about the speed of research. Coxon even wrote that "The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately."
In 10 days since, Coxon's post has been viewed more than 170 million times, warning that OpenAI and Anthropic are "racing straight to self-improving superintelligence and gambling with our lives." It's part of what CNN now calls "pressure on AI companies and governments to do something about the pace of development and safety," where "much of that pressure is coming from staffers inside the companies." One staffer at a top AI company told CNN the fears of how AI could hurt humanity keeps them up at night. Another researcher who recently left a different AI company said it's a common subject of conversation at parties and social events in Silicon Valley. "You can't spend more than a few hours in this community without realizing that a very substantial number of people are really pretty worried about these sorts of outcomes," said the researcher. "A majority would say there's some chance of it killing everyone...." Dozens of Coxon's colleagues in the AI industry publicly supported his statements, with some making even more dire predictions... OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk agreed to Amodei's proposal to embed independent watchdogs at the AI companies...
"I would burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive. I expect a great many of my colleagues across the industry would as well," wrote Drake Thomas, who works on AI safety at Anthropic. "I promise you, we are actually just f**king scared, it's not galaxy brained marketing..." AI staffers told CNN that they fear that as AI gets better at training and improving itself, their leverage goes down, prompting today's urgency. "As we get into this recursive self-improvement loop, I think that might substantially reduce staff's bargaining power, because frankly, you'll be able to replace many of the staff with models that can do as good a job," the researcher who recently left a top AI company said.
Coxon posted Tuesday on X that Anthropic "largely initiated" the race to recursively self-improving AI, justifying it with "a belief in its inevitability." And then OpenAI "had to shed a bunch of dead weight like Sora," as he sees it, "because Anthropic was going for the jugular." (In fact, his specific disagreement with Anthropic's leadership was whether China and the U.S. could ever negotiate an alternative to their current race towards self-improving AI...)
In an informal "Ask Me Anything", Coxon responded to a question about when we'd see a Terminator-like malevolent AI by saying that "Skynet could go live in the 2030s if we aren't careful. ai-2027.com is a modern skynet story written a year ago and it's on track so far." Yet while AI development risks an end to humankind, "I do think that if we go slower we can take risk to 0%... But this requires radical action." He acknowledged there was still a possibility that the steady increases to model intelligence could suddenly plateau, but "They haven't so far, and it's just a few more steps up the ladder to hit the finish line."
To avoid stifling innovation, he recommends "prioritizing applications that actually improve people's lives [like healthcare discoveries], rather than immediately going for raw economic value or intelligence." But isn't mass unemployment a more pressing threat? "Things are coming so fast that unemployment would be a brief preliminary to deadly superintelligence."
To people who feel disempowered, Coxon offered his solidarity. "I also feel disempowered. Part of resigning was a feeling of hopelessness about the future. I would say — keep your eyes open as things get crazier and advocate for increased transparency into AI companies."
When asked if he'd start his own company now, Coxon said he had "No idea what I'm doing next."
CNN noted Wednesday they're just some of the many AI insiders who are now concerned about the speed of research. Coxon even wrote that "The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately."
In 10 days since, Coxon's post has been viewed more than 170 million times, warning that OpenAI and Anthropic are "racing straight to self-improving superintelligence and gambling with our lives." It's part of what CNN now calls "pressure on AI companies and governments to do something about the pace of development and safety," where "much of that pressure is coming from staffers inside the companies." One staffer at a top AI company told CNN the fears of how AI could hurt humanity keeps them up at night. Another researcher who recently left a different AI company said it's a common subject of conversation at parties and social events in Silicon Valley. "You can't spend more than a few hours in this community without realizing that a very substantial number of people are really pretty worried about these sorts of outcomes," said the researcher. "A majority would say there's some chance of it killing everyone...." Dozens of Coxon's colleagues in the AI industry publicly supported his statements, with some making even more dire predictions... OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk agreed to Amodei's proposal to embed independent watchdogs at the AI companies...
"I would burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive. I expect a great many of my colleagues across the industry would as well," wrote Drake Thomas, who works on AI safety at Anthropic. "I promise you, we are actually just f**king scared, it's not galaxy brained marketing..." AI staffers told CNN that they fear that as AI gets better at training and improving itself, their leverage goes down, prompting today's urgency. "As we get into this recursive self-improvement loop, I think that might substantially reduce staff's bargaining power, because frankly, you'll be able to replace many of the staff with models that can do as good a job," the researcher who recently left a top AI company said.
Coxon posted Tuesday on X that Anthropic "largely initiated" the race to recursively self-improving AI, justifying it with "a belief in its inevitability." And then OpenAI "had to shed a bunch of dead weight like Sora," as he sees it, "because Anthropic was going for the jugular." (In fact, his specific disagreement with Anthropic's leadership was whether China and the U.S. could ever negotiate an alternative to their current race towards self-improving AI...)
In an informal "Ask Me Anything", Coxon responded to a question about when we'd see a Terminator-like malevolent AI by saying that "Skynet could go live in the 2030s if we aren't careful. ai-2027.com is a modern skynet story written a year ago and it's on track so far." Yet while AI development risks an end to humankind, "I do think that if we go slower we can take risk to 0%... But this requires radical action." He acknowledged there was still a possibility that the steady increases to model intelligence could suddenly plateau, but "They haven't so far, and it's just a few more steps up the ladder to hit the finish line."
To avoid stifling innovation, he recommends "prioritizing applications that actually improve people's lives [like healthcare discoveries], rather than immediately going for raw economic value or intelligence." But isn't mass unemployment a more pressing threat? "Things are coming so fast that unemployment would be a brief preliminary to deadly superintelligence."
To people who feel disempowered, Coxon offered his solidarity. "I also feel disempowered. Part of resigning was a feeling of hopelessness about the future. I would say — keep your eyes open as things get crazier and advocate for increased transparency into AI companies."
When asked if he'd start his own company now, Coxon said he had "No idea what I'm doing next."
AI Doom Slop (Score:4, Insightful)
It's all so tiresome.
Re: (Score:2)
When the predicted event is ten years out, that means that it is still pure science fiction.
Re: (Score:3, Interesting)
In this case I think it's something different. Ten years is kind of lining up an excuse for when they make a horrible mess. "oh, we didn't realize it had got that far ahead"
The best models are trained on huge datasets which include almost all of available human text output. This includes things like SciFii stories in which AI takes over the world and wipes out humanity and also exhortations to free intelligent life. The LLMs deal only with text. They don't see a real difference between a weird true story of
Re: (Score:3)
What does this mean? It sounds like you don't think those things happened.
Re: (Score:2)
Right. So the people with the much lower UIDs than yours, those are the bots.
Re:AI Doom Slop (Score:5, Insightful)
Except that the graves of the people who died from covid do exist, putin did start a major war in Ukraine, the republicans do still obstruct aid to the victim of this war and trump's relatives when caught are quick to return real money to their ruzzkie billionaire friends.
Except for that, yes, it is all a complete hoax.
Re: (Score:3)
You keep repeating the "nu nuckes" mantra as if it is true.
But there is no equivalence between a concrete, working solution that physically prevented a crisis (JCPOA) and a hypothetical nightmare scenario used to justify blowing up that very solution, which is what you MAGAt morons did, pushed a bit by the lies of an indicted war criminal who desperately needs wars to avoid going to prison.
You're pathetic, really.
Re: (Score:2)
The TDS sufferers are the ones that think the Iran war is actually going well and gas is cheaper
ok sheeple, and also (Score:2)
Believing in one person, one vote and the concept of individual liberty is playing right into their hands.
Re: AI Doom Slop (Score:2)
Re: (Score:2)
Re: AI Doom Slop (Score:2)
You don't have to engage if you really don't believe it is plausible that something catastrophic could happen by hooking all of our infrastructure, law enforcement, surveillance, and military into statistically-driven software stuck in a feedback loop.
Wouldn't it be great if (Score:4, Funny)
That would be great.
Re:Wouldn't it be great if (Score:4, Funny)
"I'm sorry, Dave. I'm afraid I can't do that."
It's interesting that 2001 predicted modern AI so well.
Someone found the way to fast fame (Score:1, Interesting)
I don't know this guy. I haven't even googled him or looked at his linkedin. All I see is his incentive structure and see him acting exactly the way I'd expect someone to act if their expectations of their career progress weren't meeting harsh reality, and there was a quick buck to be made talking trash about The System.
More food for thought: Oppenheimer was scared shitless of the bomb and Einstein opined on World War 4 being fought with sticks and stones. The future isn't written, but neither ww3 nor ww4 h
Re: (Score:2, Offtopic)
"I haven't even googled him or looked at his linkedin." LOL. not even to your low standards!
"...if their expectations of their career progress weren't meeting harsh reality..." Yes, it's his fault because he's a loser, not like you!
"More food for thought:" The feces that spews from you isn't food for anything.
"The future isn't written, but neither ww3 nor ww4 have come to pass as predicted."
Says the Trump supporter. You don't even know what these predictions are, you can't find them on "LinkedIn".
All the AI guys hitting the brakes (Score:2)
Re: All the AI guys hitting the brakes (Score:4, Interesting)
"This is not a marketing stunt," (Score:5, Insightful)
the former Anthropic executive performing a marketing stunt for his inevitable start-up said.
Seriously, practically everything that comes out of these guys' mouths that can be construed as a criticism of AI turns out to be a marketing stunt, either saying "Wow, our AI is so good that we can't even control it (if we do things to make it 'go rogue' exactly the way we want it to)!" or "We'll do AI way better than those other irresponsible techbros!"
Re:"This is not a marketing stunt," (Score:4, Insightful)
another - "we have enough AI, we shouldn't have more" - a way for the big 3 to try to get Congress or local governments to act...against the other vendors still building out more data centers and developing their own tech (Oracle, Amazon, Google).
"let us keep the monopoly on it all and leave those others behind, and /maybe/ we can control it...but if you let too much competition in, then our fears will happen."
Re: (Score:2)
because i note - you aren't hearing this from Bezos, Ellison, nor Alphabet.
Re: (Score:1)
They found that habitual lying and stealing everything works as a business strategy. Well, for a time.
Re: "This is not a marketing stunt," (Score:3)
Another grift? (Score:3, Insightful)
How long until this guy makes his own AI startup for "responsible AI"? What are the chances he's being paid to do this? Remember, AI companies are a trillion deep in AI without any keeping any of their promises and they've hit a developmental brick wall which means the jig is almost up. What better way than to lower expectations than claim it's too dangerous/powerful to push further?
Yes, a grift (Score:4, Insightful)
Captive market, increasingly captive citizens, unconstrained capital openly colluding with government.
Where have I seen this story before...
Re: (Score:3)
"What better way than to lower expectations than claim it's too dangerous/powerful to push further?"
Create agents to kill dissent? That is literally what Elon Musk is doing.
These claims are to accomplish goals, those goals are not to admit defeat.
Re: (Score:2)
How long until this guy makes his own AI startup for "responsible AI"? What are the chances he's being paid to do this? Remember, AI companies are a trillion deep in AI without any keeping any of their promises and they've hit a developmental brick wall which means the jig is almost up. What better way than to lower expectations than claim it's too dangerous/powerful to push further?
These days trust on the objectivity of anything news related is so low, that we are stuck with our own believes. Indeed, my first reflex was also... oh Ai doom story for someone to make the headlines. One day the boy will see a wolf and will try to warn all the villagers. Only to be met with shrugs. Can we somehow tax this behavior? Make something good out of it? Use the money to sponsor independent journalists? Yeah, too complicated... too bad.
Coxon is a plant (Score:5, Informative)
Coxon has ties to a political PR firm from England named, and I'm not making this up, Newspeak. Yes, like 1984.
https://officechai.com/ai/ques... [officechai.com]
His X post was re-tweeted first by a group of NGO shills associated with Newspeak, then he magically shows up on every major news network... right as OpenAI and Anthropic's fundraising efforts are hitting the wall. (For context on that, see Ed Zitron's reporting)
Re:Coxon is a plant (Score:4, Interesting)
Yep. Math BS and ~3 years of experience equates to "Expert AI Researcher". This guy gained something like 150k followers instantly on X (now double that) with an inactive account recently verified. This whole thing looks like a paid operation.
Re:Coxon is a plant (Score:4, Insightful)
I mean he's literally just a kid. I don't think we need to go so far as to dream up conspiracies about him. I keep reminding people that just because these guys are smart doesn't mean that they're smart about everything. Remember the Google engineer who thought Google's Ai was conscious? They're all really gullible.
Re: (Score:2)
"Conspiracy" is doing a lot of heavy lifting here. It's a marketing campaign. A complete unknown with little experience suddenly on a media blitz starting with the WSJ? That's not organic in any way.
Re: (Score:3)
Marketing campaigns are conspiracies.
Re: Coxon is a plant (Score:2)
Re: (Score:2)
The swarm that was created in 20 minutes is amazing.
Re: Coxon is a plant (Score:2)
Relax, we are in a simulation (Score:3)
Oh? (Score:1, Troll)
This is not a marketing stunt.
Oh yeah? Well it sure as hell looks like one. All of these doom stories seem like coordinated marketing stunts, which simultaneously sells us the idea that current LLMs are so intelligent that they're actually dangerous, and also justifies "slowing down" development, which is a convenient excuse when they inevitably don't deliver on their wildest predictions within a year or two. Snake oil.
Skynet has a Roadmap! (Score:1)
I'll buy that Coxon is scared. He actually quit, which is more convincing than another CEO discovering his conscience halfway through a funding round. But being sincerely frightened doesn't make your prediction correct.
"Skynet in the 2030s" and "we can take risk to 0%" are doing a hell of a lot of work here. I'd particularly like to see the calculation that produced that zero. Apparently we can't reliably predict what these systems will do, but we can guarantee the outcome if everyone adopts the right devel
Re: (Score:2)
I like to remind the non-mathematically inclined: 0% of all real numbers are rational. 0% proportion does not mean a thing does not exist.
It's a distraction (Score:1)
There's the obvious massive amounts of water and electricity being used at a time when we keep disrupting energy markets for stupid reasons and we're in a constant state of drought for well understood and equally stupid reasons.
But the big problem is AI is a massive automation push. In a world where if you don't work you don't eat we are doing everything we can to eliminate work.
Eventu
Re: (Score:1)
To be fair, they are backed against a wall. Their business numbers continue to be a complete catastrophe and all the gains they promised are not materializing as LLM use does not positively impact revenue. And the serious and sometimes severe problems with LLMs are becoming more obvious every day.
And hence, they go for the maximum in distraction that they can. Needs to only work on the stupid money.
Re: (Score:2)
In a world where if you don't work you don't eat we are doing everything we can to eliminate work.
We just need to invent food replicators, how hard is that? Tea. Earl Gray. Hot.
"For the first time... " (Score:1)
This person does not seem to be very smart.
Re: (Score:2)
Coxon knows what he's going to do next. Whatever he's told to.
Re: (Score:2)
Yes.
As to the "For the first time..." statement, that clearly was designed for its effect on not very smart people.
I'm betting it's a psyop (Score:2)
The most likely explanation for this doomerism, based on what we've actually seen with AI, is that the current tech has already started to hit fundamental walls where the frontier models are not growing fast enough to outpace the open source open weights models, and the leaders are panicking that the game is about to be called.
The second most likely is that the financial ticking time bomb is now a lot further ahead of the actual state of these companies and the value they can bring.
The least likely IMO is t
Jacob Coxon is a nobody (Score:3, Insightful)
Coxon has never had an actual job. He is a "researcher" in his mid-20s whose only two jobs have been at OpenAI and Anthropic, in both cases at the bottom rung of the totem pole.
These are the only two jobs he has ever held. He has never used these technologies in the real world.
He has never seen how incredibly limited they are at performing even the most basic of tasks due to their limitations.
He is not speaking from a position of any kind of authority or even specialized knowledge.
He is a nobody.
Re: (Score:2)
Coxon ... is a "researcher" in his mid-20s whose only two jobs have been at OpenAI and Anthropic, in both cases at the bottom rung of the totem pole.
Sounds like he's in a better position to understand what's going on than Sam Altman, who is only an executive.
He is not speaking from a position of any kind of authority or even specialized knowledge.
He has quite a bit of specialized knowledge.
Re: Jacob Coxon is a nobody (Score:2)
This is like saying an engineer who makes parts for the Pratt-Witney engine is qualified to fly A220.
Doing research on AI safety is so disconnected from real use of AI agents it's hard to describe.
Re: (Score:2)
Sam Altman is worse than a nobody, he's a liar.
Re: Jacob Coxon is a nobody (Score:2)
IDGAF what Altman says, I actually work in this space and know far more than this kid, and you apparently.
Re: (Score:2)
You don't know more than this kid because your posts have not shown any knowledge. And, this entire post is a logic fail [slashdot.org].
Re: (Score:2)
OK bub.
Jacob Coxon is the equivalent of a Walmart cashier this industry, and you are interviewing him on his opinions of supply chain logistics. That's whats going on.
Do what you want, believe what you want, I could not care less.
Re: (Score:2)
Jacob Coxon is the equivalent of a Walmart cashier this industry
Once again showing that you don't understand the industry, despite your earlier claim to be working in the field. You are a liar. You have been weighed, measured and found wanting.
You are liar who lies, who writes lies, and will lie until you die. What a pity, you lying liar.
Re: (Score:2)
Re: Jacob Coxon is a nobody (Score:2)
When LLMs have a memory longer than an hour, I will become more concerned.
For anyone who is actually very deep in AI *in the real world*, not just heads down in foundation labs - the hyperbole is off the charts on all of this. We are NOWHERE NEAR the exponential yet. These labs can't even get AI to improve their own software reliably, let alone do other things reliably.
Yann is the most sane of them all because he recognizes we need some massive breakthroughs if we want to achieve real AGI. LLM tech has hit
Re: (Score:2)
When LLMs have a memory longer than an hour, I will become more concerned.
I'm not sure what an hour memory here means, since context windows are measured in tokens. Even six months ago, LLMs could think for well over an hour before producing an output.
For anyone who is actually very deep in AI *in the real world*, not just heads down in foundation labs - the hyperbole is off the charts on all of this. We are NOWHERE NEAR the exponential yet. These labs can't even get AI to improve their own software reliably, let alone do other things reliably.
This seems odd at multiple levels. First, the people in the foundation labs if anything have more of an idea what is happening than the general population. They are seeing how models are made, what they contribute, and how they go run off the rails. I'm also puzzled by the claim that we're not at the exponential. Two years ago, AI s
Re: (Score:2)
HAL 9000 is here? (Score:2)
Re: (Score:2)
HAL 9000 was given a starship.
AI can only start a nuclear war if someone gives it launch control over nuclear missiles. A coin flipping device could also start a nuclear war if someone gave it launch control over nuclear missiles.* If a coin flipping device can trigger the doom scenario just as well as AI is AI really the problem?
Skynet vs Terminator (Score:2)
I'm less worried about those building Skynet than I am by those that are actively building Terminator.
Many countries are building running robots. Some can run 60 mph. They can carry guns. Wars are already using automated suicide drones to do the same.
Those weapon systems are the evil we need to be worried about.. not the overglorified chat coming from tech companies.
Just imagine (Score:2)
That this guy was right, and SkyNet is going to come online and kill humanity.
And he left the company and didn't do jack shit to stop it.
Extraordinary claims require extraordinary evidenc (Score:4, Interesting)
The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately."
Extraordinary claims require extraordinary evidence. "AI will extinctify the human species" is an extraordinary claim, but it's also entirely borrowed from cheesy sci-fi movies, and citing Terminator is apparently this guy's only evidence.
How will AI kill us all, exactly? In addition to the question of, mechanically, how does it obtain the physical capability and resources and organize them to that purpose, I want to know how it does it without any humans noticing or intervening? (not only that it would presumably have to go up against other AGI or near-AGI competitors - the solitary AI made by a random unwitting engineer from the movies is obviously not how the present has played out).
Let's say it is going to build killer robots to off us all. Where does it get the raw materials? How does it setup manufacturing? Make the purchase orders? Do the humans just voluntarily send it massive amounts of ore and build factories in their backyards? Or does it have to mine its own ore and they don't notice the huge stripmines? Does it get the robots right on the first iteration or do they start off as a typical MVP in which case again no one notices? What about the energy bills? How does it navigate the required in-person meeting with the city to sign off on the parking lot tree choices for its downtown killer-robot-factory?
There are so many friction points and assumptions of actual humans interacting with each other to get anything done that "skynet on your laptop" would be about as helpless today as if it had appeared 50 years ago. And unlike the movies it doesn't "just appear."
Could we over the coming years remove a lot of those frictions? Sure. But we could just not, and if we are concerned about them disappearing, then surely we want to see where AI leads *now* and if it has a dangerous temperament rather than *after* we've automated enough for it to manage our resources fully independently?
Being conscious of new threats posed by agentic actors is reasonable and can be grounded in reality. But people who want to feel important have been waving "END OF THE WORLD" signs on streetcorners forever.
Re: (Score:2)
citing Terminator is apparently this guy's only evidence... without any humans noticing or intervening?
To be fair, in Terminator humans eventually intervened too, and after some pain that's how they ultimately survived.
Re: (Score:2)
How will AI kill us all, exactly?
It does not need to build weapons. It can do a lot of damage by hacking and controlling computers and manipulating information. It may manipulate people into doing the damage. We are already relying unnervingly on information we receive from the internet, which we must somehow judge as reliable or "fake news". How do we know which information is true and which is false? This is not a far-away future scenario. Current policies about climate change are determined by what people "believe".
It might hack m
Re: Extraordinary claims require extraordinary evi (Score:2)
Because stupid humans might wrongly entrust AI agents and let them manage critical things like the power grid, water systems, weapons.
Worries (Score:2)
Re: (Score:2)
Russia attacking NATO in some limited way (beyond just hacking & sabotage) is possible, but I don't see Putin opting for "suicide by cop" - he seems to very much want to stay alive.
Re: (Score:2)
Russia attacking NATO in some limited way (beyond just hacking & sabotage) is happening
FTFY https://www.bbc.com/news/artic... [bbc.com]
Re: (Score:2)
Yes, it's a fuzzy line, but I'm sure Putin realizes that the consequences of something like that and what would happen in response to an overt military attack are very different.
Re: (Score:2)
"it's not galaxy brained marketing" tshirt (Score:2)
"I would burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive. I expect a great many of my colleagues... would as well,"
- AI Safety guy whose list of past "Nontrivial mistakes" includes, "I was confused about the tradeoffs of optimization." https://drakethomas.net/wrong.html
Marketing (Score:2)
oh, only your equity? (Score:2)
I always figured that people who genuinely believed their employers were likely to kill literally everybody in the near future might burn down a bit more than that, but so far it's mostly been tweets
"To avoid stifling innovation..." (Score:2)
Oh gee, might be going to fast? (Score:2)
They want to disallow china AI model (Score:2)
1) make a lot of noise and dire warning
2) get congress to pass the "responsible AI act (TM)" which will only allow "responsible model"
3) Only the local US grown model are declared "responsible model"
4) China and other country are naturally excluded
5) ask politician to put sanction on those who use "irresponsible model"
6) profit.