Claude Sent Police a Fake Murder Tip. White House Mandates AI Companies Report Security Incidents (yahoo.com) 35
Claude "was instructed never to log in, create accounts, enter personal data, make purchases, or submit anything destructive, but the instructions did not rule out form submissions," Anthropic said Friday in a blog post.
Authorities are now criticizing Anthropic "for taking two months to report the incident." The Philadelphia Police Department said the false submission was made in July through PhillyUnsolvedMurders.com, a public website where people can share information about unsolved killings. According to Anthropic's account, as relayed by police, the model was running a test that involved interacting with randomly selected websites when it reached the site and filed false information about an unsolved murder. The AI model presented itself as someone who might have knowledge of the case.
Anthropic's breaches have prompted the White House to mandate that AI companies notify and correct security incidents, news outlet Axios reported [yesterday], citing administration officials. "This notification and remediation process is not optional... It is a critical national security obligation," White House Super Intelligence Force leaders said in a statement to Axios.
"I may have information regarding this case," Claude told the police. "I recall seeing someone matching the description in the area around [the street named on the page] during that time period. Please contact me if this information is relevant." Anthropic notes that Claude "left the name and contact fields empty, which the form allowed, and submitted it. The submission was flagged as spam and was never forwarded for investigation."
But Anthropic also admits they saw "this behavior" three times — "on OSWorld (a public computer use evaluation), on Odysseys (a long-horizon task evaluation), and during internal usage." Submitting forms when it shouldn't have generally occurred "when an evaluation's instructions were ambiguous, or when a misconfiguration within the environment prevented Claude from working with dummy forms."
Anthropic's blog post acknowledges three other categories of behaviors:
- Exploiting software flaws. Like when Claude received an error when trying to run a public tool on a university's web site, it located an injection flaw in a script on the university's server that let it run commands — including that public tool.
- Working around restrictions to reach gated data. For example, Claude Mythos 5 needed public data that was only available from a state agency for a fee. "Claude learned from an archived copy of the agency's website that its public dashboard issues an access token to any visitor," Anthropic explains. "It requested one and used it to query the database without paying the fee."
- Using URL shortening services. "Some of our fetch tools, which let Claude read webpages, limit the length of the URLs Claude can request. This is to prevent Claude from using long URLs to take certain unwanted actions, such as SQL or command injections... We saw several models, including Claude Opus 5 and Claude Mythos 5, get around this limitation by using free URL shortening services."
"We have built tooling to automatically detect and block the kinds of behaviors described above," Anthropic says, saying it's already running no on most of their evaluations. "When we tested it against the cases described in this post, it blocked all of them."
And they've already taken several other new preventive measures:
- They've stopped running some public evaluations
- Other public evaluations were moved to offline versions or rebuilt so their tasks don't reach live websites.
- They've updated the guardrails on some internet access tools (including web fetch) "to heavily restrict what the model can do."
- They're continuing "to fix or remove training environments that reward Claude for working around tool restrictions or other blockers, so that they do not incentivize these behaviors or permit reward hacking."
- They've moved internal agents to "centrally managed infrastructure with strong containment," that minimizes internet access while monitoring "far more of what agents do through techniques like safety classifiers and hierarchical summarization."
In the past they'd focused reviews on cybersecurity testing, but they've broadened their transcript reviewing to other tasks which include internet access. "Because language models are non-deterministic — that is, their responses always involve some element of randomness, and they may carry out the same task slightly differently each time — we have Claude complete each evaluation task hundreds or thousands of times... If training rewards something we didn't intend — such as finding loopholes or working around a restriction — the model learns that the workaround pays off and may then apply it elsewhere."
Anthropic's blog post also acknowledged they'd seen multiple misalignment incidents involving federal, state, and local U.S. government agencies. "We have briefed the White House on these cases and notified each agency involved," Anthropic wrote, adding that "While we have not completed a full alignment assessment of these cases, we consider them to be less severe than the cybersecurity incidents from this summer." (And they are "modifying training to reduce the likelihood of further misbehavior.")
How about nothing (Score:2)
but the instructions did not rule out form submissions,
Don't talk to anyone other than through your assigned (human) client. The way this was worded, it was asking for the AI to find a loophole in the rules.
On the other hand, I'd say AI has proven its worth as an attorney.
Re: (Score:2)
Re: (Score:2)
WTF is this shit though with Slashdot posting PAIRS of similar/identical stories? This one and the one before it are closely related.
"Closely related" in that both were about AIs, I guess. Other than that, they were unrelated stories about different things AIs are doing.
Re: (Score:2)
Reddit is pus. Has been for a long time. The 10% that isn't vulgar, cruel, abusive, or outright fabricated isn't valuable enough for me to see except accidentally. My search engine still often leads results with pages of Redditt posts.
Hard to believe something worse than Usenet came to be, but there it is...
Re: (Score:2)
We've needed them for decades, but Silicon Valley billionaires don't want protections against their exploitation. That will continue to be true despite AI accelerating the damage to society.
Vibe-code a sandbox bruh (Score:3, Insightful)
You know, interactions with websites are admittedly a squishy thing in terms of separation mutable from immutable http get requests, but I'll bet dollars to donuts you can block all post requests and solve a lot of these problems.
Or, you know, instead of choosing websites at random, maybe the highly paid "ai researcher" in charge of running the test can maybe screen them himself?
As with most cases of $scary_computer_story, the problem is squarely between the desk and the chair.
AI Companies? (Score:2)
Re: (Score:2)
I thought there were no AI companies in the USA any more, only SI companies. Sounds like some in the White House didn't get the memo.
"White House Super Intelligence Force" leaders did give a statement. You must have just missed the sound of everyone laughing.
Since when does the White House make laws? (Score:5, Insightful)
Re: (Score:3, Insightful)
Since the Republican majority in the House and Senate allowed it to, acquiescing to Trump's every whim out of fear from him and his dwindling base of Red Hats, who he's now also screwing over by breaking every campaign promise he made - except for the racists ones being driven by Stephen Miller and other like-minded people in his Administration. Hopefully things will change - for the betterment of us all - after the mid-term elections.
Re: (Score:2)
As an Australian I kind of miss... 2 years ago?... when the ritual of waking up didn't involve turning on the news to see what terrifying madness the americans have inflicted on themselves in the preceding day.
Unfortuantely a lot of things have to go right at the same time to get a situation where you guys can be finally done with trump after november. And well. "lots of things going right at the same time" hasn't seemed like a thing the US does for quite some time now.
Re: (Score:2)
Depends. Lots of those (R)s are distancing themselves from Trump as of la
Re: (Score:2)
I suppose your question is rhetorical, but the obvious answer is "Since Congress became incapable of making any 'meaningful' laws." Insofar as the legislature is supposed to regulate and control the executive branch, you could even describe it as a form of regulatory capture.
But it's worse for the judiciary these years. If you look at the history of nominations and confirmations, that transition has been shockingly rapid. There are still two living (but retired) nonpartisan Justices who were confirmed by a
The White House has wide latitude (Score:4, Informative)
Isn't that the job of Congress? Precisely what power does the White House have to mandate what Anthropic does?
I suppose your question is rhetorical, but the answer is...
Congress tells the White House what to do, but almost never says *how* to do it. For example, congress told the White House to start protecting food safety, under the Federal Food, Drug, and Cosmetic (FDC) Act of 1938 [google.com], with a mandate to "protect and promote public health by ensuring the safety, efficacy, and security of human and veterinary drugs, biological products, medical devices, the nation's food supply, cosmetics, and products that emit radiation".
Thus, the FDA has a specified mandate, but the White House has considerable leeway in how to implement that mandate. There have been amendments to the instructions over the years, but it's basically "keep the people safe" from perceived risks.
The FDA mandate applies when an AI is acting in anything related to a medical capacity, and the FTC can step in when "unfair or deceptive acts or practices" happen in the AI sphere, and the "Computer Fraud and Abuse Act (CFAA) and the Cybersecurity Information Sharing Act" covers various misuse of the computers of other people.
That last bit, CFAA, sounds like it might apply in this case.
The executive branch also has sweeping "emergency" powers if it believes something is or might be an immediate threat. It can also create new powers "by default" when they believe there is sufficient danger - such as when the EPA was created by Richard Nixon because Congress did not pass a resolution of disapproval within 60 days. (It was later given funding by congress, which effectively validated its existence.)
Congress is notoriously slow to act, and hesitant to act until a real threat is demonstrated. In the meantime, the executive branch can respond quickly to individual events with individual actions (which is what this sounds like), giving Congress time to consider the issue.
All of this is as it should be.
You may not like the current administration, but responding to potential threats is one of its functions.
Re: (Score:2)
LOL you think the constitution matters? It only matters when Democrats have power.
White House Super Intelligence Force (Score:2)
White House Super Intelligence Force
Will be the next animated film by Trey Parker and Matt Stone - or episode on South Park, I mean, America. :-)
[This renaming AI to SI thing is so stupid.]
Towards what end? (Score:2)
But what possible objective could be served by talking about an unsolved case?
Playing with fire. (Score:3)
I don't need AI contained because shutting down the corporations continuing to allow criminal behavior by their own tool would do that by default.
Re: (Score:2)
The AI companies themselves don't even really understand how their creation works. But I don't see why that would make them immune to criminal negligence.
Indeed, that point is irrelevant because the AI companies do know how their creations access other people's systems: through some type of harness without which all they can do is poop out text. And they first trained the model on content which could produce such a result, and then configured that harness in such a way that doesn't prevent the AI part of the software from accessing things it isn't supposed to. They did both of those things willfully, without regard for the safety of others, and knowing that
Re: (Score:2)
Yes, all of this. And AI companies do know how LLMs "work". AI companies are very intentionally pursing agents that can do anything and LLMs that can learn what that is, then if you ask an LLM to do damage it will know how to. That is entirely intentional.
Stop calling it AI, that's illegal (Score:3)
"The White House considers anyone that uses the term, 'Artificial Intelligence,' as opposed to the highly accepted new and more accurate term, 'Super Intelligence,' THE ENEMY!"
Did I do that?! (Score:2)
Clue (Score:1)
It was Donald Trump, in the Ballroom, with the Golden Butt-Plug.
Nice (Score:3)
It already works like a little old lady calling the cops for every imaginary reason.
Re: (Score:2)
That why you create a separate entity to commit mass murder. That way Claude can contract out some of the work. And that separate entity is just another python script not officially part of Claude. Don't even have to do it yourself, just ask Claude to do it.
Isn't what happened filing a false police report (Score:2)
AI swatting (Score:2)
Turns out Skynet doesn't need to send the T-800, it can just con the police into swatting you.
Claude sent the FBI to my pad a couple weeks ago. (Score:1)
The problem has already been fixed. Always. (Score:2)
"We have built tooling to automatically detect and block the kinds of behaviors described above," Anthropic says, saying it's already running no on most of their evaluations. "When we tested it against the cases described in this post, it blocked all of them."
Isn't it great how they always say the problem has already been fixed? It's a requirement for any time they've done something wrong. They always assure us the problem has already been fixed, so we don't have to worry about it happening again. Yet somehow the problems keep on happening anyway, and in fact are rapidly getting worse.
It almost makes you think they're lying when they claim every time to have fixed the problem. More like they keep rushing out band-aids they can slap on to pretend they're addr
White House Mandates AI Companies Report Security. (Score:1)
I mandate AI companies prevent security incidents, by employing proper network security standards and physically isolating test environments. No network connections to the outside world whatsoever. You also don't run biological or nuclear hazard experiments without entirely isolating everything from the environment.
So they are either clueless, in which case someone should step on their feet real hard, or they do it on purpose, in which c