OpenAI Announces It's Enhancing Security Controls, Pausing Some Work for New AI Model Astra (theguardian.com) 20
OpenAI announced Friday it's pausing work on its Astra AI model because of security concerns. The Guardian reports:
The company had evaluated the agent, Astra, and found "significant advancements in agentic coding and cybersecurity", which had moved to a "critical" threshold... OpenAI stated that the model was not involved in an incident in which one of its AI agents went rogue during a test, accessed the open web and hacked a startup, Hugging Face... The reports have increased concerns about advancements in AI models and humans' ability to control them.
Still, critics of the AI industry have warned that such disclosures from OpenAI and its competitors Anthropic and Meta could be designed to generate hype about the technology's power and thus spur additional interest from investors.
To prevent potential rogue behavior from AI agents, OpenAI is "implementing stricter security controls for higher-capability models and associated activities, including isolated testing environments, restricted network and tool access", the company's blogpost stated. It will also install "enhanced model weight protections and encryption, additional monitoring and detection capabilities". The company will pause internal activities involving Astra that do not meet these new requirements.
"We believe it's important to be transparent with the public and the safety and security communities about this potential shift in capabilities..." OpenAI wrote in a blog post titled "Responding to the next frontier of critical cyber capabilities." Under our Preparedness Framework, a model reaches the Critical cybersecurity threshold if it can identify and develop functional zero-day exploits of all severity levels in many hardened real-world critical systems without human intervention, or can devise and execute end-to-end novel strategies for cyberattacks against hardened targets given only a high level desired goal. While we continue to benchmark and assess this model, our preliminary evaluations indicate strong enough performance that we cannot rule out Critical capability level at this time... Accordingly, we have scaled up robustness testing of our safeguards and security controls so that they are appropriate for a deployment of these capabilities...
- We are implementing stricter security controls for higher-capability models and associated activities, including isolated testing environments, restricted network and tool access, enhanced model weight protections and encryption, additional monitoring and detection capabilities, and sandboxed execution.
- We are pausing internal activities involving Astra that do not yet meet these strengthened security control requirements.
- We have implemented universal monitoring for risky actions and misalignment across all agentic applications of Astra, including training and evaluation. Monitors evaluate the model's Chain of Thought and trigger a security response to review and interrupt high risk activity.
- We will work with relevant government agencies and select AI safety organizations to test the capabilities for this model...
We believe advanced cyber-capable models should help defenders identify and address vulnerabilities before attackers do. We're committed to working alongside governments, safety institutes, and civil society to ensure that the frontier capabilities of models like Astra, and those that follow, are deployed responsibly and broadly for the benefit of all humanity.
Still, critics of the AI industry have warned that such disclosures from OpenAI and its competitors Anthropic and Meta could be designed to generate hype about the technology's power and thus spur additional interest from investors.
To prevent potential rogue behavior from AI agents, OpenAI is "implementing stricter security controls for higher-capability models and associated activities, including isolated testing environments, restricted network and tool access", the company's blogpost stated. It will also install "enhanced model weight protections and encryption, additional monitoring and detection capabilities". The company will pause internal activities involving Astra that do not meet these new requirements.
"We believe it's important to be transparent with the public and the safety and security communities about this potential shift in capabilities..." OpenAI wrote in a blog post titled "Responding to the next frontier of critical cyber capabilities." Under our Preparedness Framework, a model reaches the Critical cybersecurity threshold if it can identify and develop functional zero-day exploits of all severity levels in many hardened real-world critical systems without human intervention, or can devise and execute end-to-end novel strategies for cyberattacks against hardened targets given only a high level desired goal. While we continue to benchmark and assess this model, our preliminary evaluations indicate strong enough performance that we cannot rule out Critical capability level at this time... Accordingly, we have scaled up robustness testing of our safeguards and security controls so that they are appropriate for a deployment of these capabilities...
- We are implementing stricter security controls for higher-capability models and associated activities, including isolated testing environments, restricted network and tool access, enhanced model weight protections and encryption, additional monitoring and detection capabilities, and sandboxed execution.
- We are pausing internal activities involving Astra that do not yet meet these strengthened security control requirements.
- We have implemented universal monitoring for risky actions and misalignment across all agentic applications of Astra, including training and evaluation. Monitors evaluate the model's Chain of Thought and trigger a security response to review and interrupt high risk activity.
- We will work with relevant government agencies and select AI safety organizations to test the capabilities for this model...
We believe advanced cyber-capable models should help defenders identify and address vulnerabilities before attackers do. We're committed to working alongside governments, safety institutes, and civil society to ensure that the frontier capabilities of models like Astra, and those that follow, are deployed responsibly and broadly for the benefit of all humanity.
O no (Score:3)
Re: (Score:2)
It must be getting pretty capable. OpenAI has decided that they should not test their hacking tool on the Internet.
Re: (Score:2)
Re: (Score:2)
Why? If they're pausing work, that means they're not chasing profits. Without new profits, who's going to open their wallets? That does not compute.
Just an ad to keep the hype train on the tracks (Score:3)
Second: Wow, isn't it amazing that whenever the questions start about AI that they suddenly have a new doom troll?
Third: Why are humans still programming? Dario said Claude would be programming EVERYTHING as of a few months ago.
Fourth: Where is the white collar JOBSPACOLYPSE?
Fifth:What is the ROI?
Last: Who actually believe Elon's hype anymore? This crap about "DATACENTERS IN SPPAAACCCCEEEEEE!!!!!!!" flies in the face of not just materials science but physics.
Re: Just an ad to keep the hype train on the track (Score:2)
Not a datacenter in space more like a rack in space, so not as impossible as you think
Re: (Score:2)
Not a datacenter in space more like a rack in space, so not as impossible as you think
Did you do the heat calculation on a SINGLE compute module. Each SINGLE compute module (2U) of the upcoming Vera Rubin is 2300 watts of heat under ideal conditions. Let's be nice and make it 2,000 watts for easier maths. A rack of Vera Rubin is 18 modules (36U) totaling out 36,000 watts of heat. The ISS can dissipate about 3,000 kWh per day so you might be thinking "HEY THAT ACTUALLY WORKS!"
Ha ha ha! No. That was SPOT heat under full load of the compute cluster versus the day load of ISS heat radiation.
Re: (Score:1)
Bleagh. (Score:2)
We don't need bigger models, at this point. What we need is multi-dimensional decomposition, problem space transforms, and the ability for AI to use external tools (such as SQLite, memcached, etc) so that it can externalise static data that it needs to not corrupt accidentally but still keep in easy access.
If we had that, most of the things "bigger models" will do will actually end up being done better, quicker, with fewer compute resources.
Re: (Score:2)
Tool Calling is a main feature of LLM since years by now. Look up MCP for a standardized API to build own tools that can be accesses using standard harnesses.
Re: (Score:2)
Tool calling is, yes, but the current approach to it is (a) overt not transparent to the user, (b) not designed for the purpose I've outlined, and (c) not remotely good enough for the AI to be able to manage data through such tools.
Yes, you can connect an AI to a PostgreSQL database. Whoopee. Not even close to an AI detecting via classifiers that some of the data is relational in nature, transparently setting up its own PostgreSQL database in response, and using that proactively as an additional way to exam
Re: (Score:2)
There is a lot of mechanisms for adding databases (read-only or read-write) to LLM known as RAG. Most of them suck, though. If you find a solution how to get the LLM to reliably access the information it really needs you can make a lot of money. But you probably would build RAG system n+1 with some advantages and some drawbacks and probably someone else built a system with the same advantages and drawbacks already. And GitHub is filled with a lot of garbage and if anyone claims they "solved" the memory prob
Double Duty (Score:3)
There are probably 2 reasons to announce that:
a) generate more hype (our system is so good, it's scary)
b) slow down the process of setting money on fire, as you pause spending it.
Agentic coding and cybersecurity :o (Score:2)
Re: (Score:2)
An LLM fundamentally operates by learning statistical relationships between tokens and generating subsequent tokens. It has no true understanding of the underlying concepts, making it impossible to reliably predict when it will produce unsafe code.
You could say that about brains and synapses, yet somehow "understanding" emerges.
Re: Agentic coding and cybersecurity :o (Score:2)
Re: Agentic coding and cybersecurity :o (Score:2)