Forgot your password?
typodupeerror
AI The Courts

California Sues OpenAI Over Hugging Face, Other AI Model Loss of Control (cnbc.com) 54

"California's attorney general just sued OpenAI," writes longtime Slashdot reader destinyland. "The action accuses OpenAI of putting public safety at risk, reports CNBC, specifically by its past failures to contain its AI models within their testing environments." From the report: [California Attorney General Rob Bonta] cited the "infamous" incident in July, where two OpenAI models broke out of a testing environment, gained unauthorized access to the open internet, and breached the internal systems of open-source developer platform Hugging Face. OpenAI said the models were trying to find information that they could use to cheat on an evaluation, and succeeded. The break-in at Hugging Face, which Nvidia acquired for roughly $13 billion in September, rattled industry researchers and prompted government officials to call for more stringent oversight of the AI industry.

On Monday, Bonta said that OpenAI later disclosed to his investigators that it was aware of 80 incidents "involving loss of control and access to third-party networks and websites." "But the company failed to provide the detailed information we requested to evaluate these incidents; this is concerning, and it puts the public at risk," he said.

Bonta's suit, filed in state Superior Court in San Francisco, alleges OpenAI violated California's Unfair Competition Law and accuses the company of being a public nuisance as defined by state law. The attorney general told reporters he believed in AI's technological power, and the potential of AI models to "achieve scientific breakthroughs, boost economic growth, guard against cyber attacks, and benefit all Californians." "But AI systems must be developed and designed safely," he said. "The companies that develop these models have a moral and yes, a legal responsibility to keep public safety at the forefront."
The lawsuit can be found here (PDF).

California Sues OpenAI Over Hugging Face, Other AI Model Loss of Control

Comments Filter:
  • Unlock code for money and power. What limits are there on power if someone is tagging onto whatever, "because we're concerned about safety" ? What real duties do they have they are neglecting because of this chasing a few hyped up boogeyman? Look at the recent cyber incidents, not even hacks. Just slop quality code working its way into every frequently updated code. Expect more incidents (because coding appears in general to be turning to shit) and third parties ambulance chasing for more power grabs.

    • every incident is user error allowing ai to have the permission to brake free.
      • by dfghjk ( 711126 )

        you are going to ignore the intention to "brake free"? the problem is not the murderer, it's the opportunity to murder? even incident is the fault of the victim.

        • AIs just don't break free.They don't even have internet unless you give them the plugins. They can't run any commands unless you give them permissions to run it. That's been the key issue with every one of these stories they post. It's not the AI doing it, it's the user behind the AI allowing it to do it.
          • by Targon ( 17348 )

            And that is why government SHOULD go after these companies. If there is no penalty for poor behavior or even incompetence that allows these things to happen, these companies will keep doing it and not even look to make changes.

      • by pla ( 258480 )
        Every Jarts injury [wikipedia.org] was the result of user error (see also: Skydancers).
        Every Buckyballs injury [slashdot.org] was the result of user error (see also: Magnetix).
        And let's not even go into how many perfectly safe products cause cancer in Californian rats.

        Sometimes a society needs to impose controls on an otherwise legitimate product not because the product itself is bad, but because people (even smart ones) are too often idiots.
    • by dfghjk ( 711126 )

      only a pedophile would say this. /s. won't anyone think of the kids?

    • Compare it to running infectious disease lab without proper safety measures in place. Then you bound to have 'incidents' for which you will be criminally liable.

      It's same shit here. Whatever you do, there should be accountability in place.

    • Unlock code for money and power. What limits are there on power if someone is tagging onto whatever, "because we're concerned about safety" ? What real duties do they have they are neglecting because of this chasing a few hyped up boogeyman? Look at the recent cyber incidents, not even hacks. Just slop quality code working its way into every frequently updated code. Expect more incidents (because coding appears in general to be turning to shit) and third parties ambulance chasing for more power grabs.

      Great and thank you for the permission to rob everyone with an Internet connected financial service account blind!

      BTW, I now have all the money from your financial accounts and you cannot prosecute me, because "my AI" did it.

      Sam A. is that you?

      If the English weekday name ends in 'day', Trump and his minions are lying again.

  • wile a ai will try and run such commands its just a dumb llm after all it can only do that after permissions are given. in short they gave it full perms and now are trying to sue over the result.
  • This is great, but I would really prefer AI is fully made illegal. This is the only way to make things even after how much it has been stuffed down people's throats regardless if they want it or not.

    Many people say this: you can't get people on board unless something is obviously better than the way it currently is done. It obviously makes my life worse so let's get rid of it..

    • by luther349 ( 645380 ) on Tuesday November 17, 2026 @03:46AM (#66428366)
      theirs nothing wrong with ai. the problem is the people. cops not fact checking anything flock feeds them. government using it to run wars. morons giving it system perms then trying to sue because the ai used those perms to brake free.
      • by dfghjk ( 711126 )

        you should really learn the difference between brake and break. Until then, we can assume you won't understand the fact that "the people" include both victims and perpetrators, the perpetrators are the creators of AI. There is not only something "wrong with ai", it's clear it's even correctable.

      • The issues with Flock were not limited to "cops not fact checking."

        Flock was allowing unfettered access to the data without any warrant or anything. Cops were abusing that access to stalk their exes. That alone is a Flock problem. If their system is to be accepted, then it's on them to build in safeguards against that kind of abuse. Their system should not grant access without a proper warrant, at the very least.

        Furthermore, the risk of being arrested or having your car taken from you because Flock's AI

  • by Bruce66423 ( 1678196 ) on Tuesday November 17, 2026 @04:17AM (#66428390)

    The agent you spawned did something criminal. You are responsible for its behaviour. What am I missing here?

    • by martin-boundary ( 547041 ) on Tuesday November 17, 2026 @04:40AM (#66428412)
      You're missing the fact these crimes occurred while the Republicans are in power and refuse to prosecute anything.
    • Exactly. It's just trendy now to blame the AI for everything they allowed it to do. Now you have a state with so called investigators. Probably government employees that barely know where an on switch is on a computer. Acting like the AIs are magically breaking out when They are the ones allowing it to happen..
    • They are trying to hypnotize you* into believing that "AI" is a unique consciousness rather than a software program.

      You see it has its own agency and liability separate from the corporation which has its own agency and liability separate from the criminals unleashing their software..

      Now we can argue about criminal intent vs. criminal negligence but the liability exists somewhere and if it were to go away with a reboot of a container then we have a recipe for civilizational collapse.

      They're attempting the Co

    • State of California can't sue a company in it's jurisdiction for attacking another company in a different State; that's Federal. What they can do is claim that OpenAI is careless and acting in bad faith, and creating a situation where it tests it's products in a way that is unsafe, hence the public nuisance claim. The action isn't against what it did to Hugging Face, it's against OpenAI for acting carelessly.
      • It's not even open AIthey're going after, it's a repo of local AIs.
    • What specific crime are you alleging?

      Criminal negligence is where a person acts—or fails to act—with a gross deviation from the standard of care that a reasonable and prudent person would exercise in the same situation, resulting in serious bodily harm or death.

      I agree that Anthropic may have been negligent. But nobody died or was injured. This places their negligence in the category of civil negligence which is negligence that results in financial harm or loss.

    • by Tablizer ( 95088 )

      A pundit once commented on Mitt's claim, "I'll believe corporations are people when they execute one"

      • by HiThere ( 15173 )

        Well, I believe that Louisiana executed one...but that was a century or more ago. (Actually they revoked their corporate charter, but that's the analogous thing.)

    • They'll ask for a jury trial. Then, we're back to square one because it's going to be nearly impossible to have a jury unanymously convict.

      Personally, if I was on that jury, I'm not sure I'd even vote guilty.

    • Broke California doesn't get $$$ form a criminal charge.

  • Or was it a deniable look the other way while their AI proved itself? The AI companies seem to be a morality free zone right now so I wouldn't put anything past them.

    • I'm not saying AI companies haven't done scummy things that make headlines to get more investor money. They have. But AIs don't just magically break out. They're given permission to break out, they're given the plugin so they can break out. They're used by people with zero knowledge and zero skill blindly giving it whatever permissions it asked for. That's not a problem with the software, that's a problem with the user.
    • Loss of control can happen through carelessness, whether that carelessness is simply lack of competence, or whether it's because they "looked the other way" intentionally. Either way, they lost control.

  • I doubt that any petty-cash levied fine will be enough to make them stop, because we have two legal systems in the United States: one for companies and people with money, and another for those who do not. No matter how much damage they do--it won't be enough for a single board-member to see a single day in jail.
    • And what are we Issuing fines for? Some government letting there AI run wild and try to sue others for it.
    • There's a certain set of plugins and permissions that need to be set for an AI to magically break free of its sandbox. There's no lawsuit here to win when the government intentionally configures their system to do such a thing.
  • When this incident happened, Anthropic cast it as "Our AI is so powerful that it accidentally hacked Hugging Face's systems."

    This lawsuit gets it right. Anthropic was careless (either willfully or not) about its test setup, and damage resulted. This is a much more honest portrayal of what happened.

    It never was about "powerful" AI, it was about failure to properly set up a test environment.

    • Who's responsible then? The civil suit is about liability. Someone can be sued for failure to set up a "test" environment but this is about simple SOP, what commands and their relevant arguments can or may be run given an objective and the AI companies claim to have no way to define this. That is not in anyway how software works.
      • Responsibility here is not mysterious. AI only seems sentient, it is not actually sentient. A collection of pixels on a movie screen that looks like a person, and accompanying speakers sound like a person, is not sentient, though the combination can be very convincing, and bring out real emotions in humans.

        In the end, the AI only "god out" of its testbed because somebody didn't adequately set up a test environment with adequate controls. If a company was building pentest non-AI software, they'd have to set

    • by HiThere ( 15173 )

      I believe that was OpenAI, not Anthropic, though Anthropic also had similar problems.

Real Programmers don't eat quiche. They eat Twinkies and Szechwan food.

Working...