A New Era for Security? Anthropic's Claude Opus 4.6 Found 500 High-Severity Vulnerabilities (axios.com) 62
Axios reports:
Anthropic's latest AI model has found more than 500 previously unknown high-severity security flaws in open-source libraries with little to no prompting, the company shared first with Axios.
Why it matters: The advancement signals an inflection point for how AI tools can help cyber defenders, even as AI is also making attacks more dangerous...
Anthropic debuted Claude Opus 4.6, the latest version of its largest AI model, on Thursday. Before its debut, Anthropic's frontier red team tested Opus 4.6 in a sandboxed environment [including access to vulnerability analysis tools] to see how well it could find bugs in open-source code... Claude found more than 500 previously unknown zero-day vulnerabilities in open-source code using just its "out-of-the-box" capabilities, and each one was validated by either a member of Anthropic's team or an outside security researcher... According to a blog post, Claude uncovered a flaw in GhostScript, a popular utility that helps process PDF and PostScript files, that could cause it to crash. Claude also found buffer overflow flaws in OpenSC, a utility that processes smart card data, and CGIF, a tool that processes GIF files.
Logan Graham, head of Anthropic's frontier red team, told Axios they're considering new AI-powered tools to hunt vulnerabilities. "The models are extremely good at this, and we expect them to get much better still... I wouldn't be surprised if this was one of — or the main way — in which open-source software moving forward was secured."
Why it matters: The advancement signals an inflection point for how AI tools can help cyber defenders, even as AI is also making attacks more dangerous...
Anthropic debuted Claude Opus 4.6, the latest version of its largest AI model, on Thursday. Before its debut, Anthropic's frontier red team tested Opus 4.6 in a sandboxed environment [including access to vulnerability analysis tools] to see how well it could find bugs in open-source code... Claude found more than 500 previously unknown zero-day vulnerabilities in open-source code using just its "out-of-the-box" capabilities, and each one was validated by either a member of Anthropic's team or an outside security researcher... According to a blog post, Claude uncovered a flaw in GhostScript, a popular utility that helps process PDF and PostScript files, that could cause it to crash. Claude also found buffer overflow flaws in OpenSC, a utility that processes smart card data, and CGIF, a tool that processes GIF files.
Logan Graham, head of Anthropic's frontier red team, told Axios they're considering new AI-powered tools to hunt vulnerabilities. "The models are extremely good at this, and we expect them to get much better still... I wouldn't be surprised if this was one of — or the main way — in which open-source software moving forward was secured."
CVEs? (Score:5, Insightful)
So what are the 500 CVEs?
Re:CVEs? (Score:5, Funny)
Re: (Score:2)
Very likely. While LLM-type AI can create attack code from CVEs, it is total crap at finding CVE-level vulnerabilities. Also note that in actual CVEs, the rating gets verified, because the ones that find vulnerabilities have a rather strong tendency to strongly overstate things.
Re: (Score:2)
And because there are some clueless morons moderating stuff down they do not understand, let me burn some more karma. I have plenty.
Real vulnerabilities? (Score:4, Insightful)
Given various open source developers complaining about piles of AI slop vulnerability reports that aren't actually valid, it's the obvious question to ask.
And the other obvious question is whether it provided fixes.
Re: Real vulnerabilities? (Score:4, Interesting)
I fully believe 500 real vulnerabilities.
But did the noise of bullshit ones make those 500 less likely to be fixed is the real question.
Re:Real vulnerabilities? (Score:4, Interesting)
Two additional questions:
and each one was validated by either a member of Anthropic's team or an outside security researcher
1. What's the breakdown between the two - how many validated by each?
2. What was the previous relationship between the "outside security researcher" and Anthropic, if any?
Re:Real vulnerabilities? (Score:5, Informative)
and each one was validated by either a member of Anthropic's team or an outside security researcher
1. What's the breakdown between the two - how many validated by each?
2. What was the previous relationship between the "outside security researcher" and Anthropic, if any?
If you read the linked blog post [anthropic.com] in TFA, it's pretty clear that it was merely a matter of manpower and shouldn't be viewed as suspicious.
To ensure that Claude hadn’t hallucinated bugs (i.e., invented problems that don’t exist, a problem that increasingly is placing an undue burden on open source developers), we validated every bug extensively before reporting it. We focused on searching for memory corruption vulnerabilities, because they can be validated with relative ease. Unlike logic errors where the program remains functional, memory corruption vulnerabilities are easy to identify by monitoring the program for crashes and running tools like address sanitizers to catch non-crashing memory errors. But because not all inputs that cause a program to crash are high severity vulnerabilities, we then had Claude critique, de-duplicate, and re-prioritize the crashes that remain. Finally, for our initial round of findings, our own security researchers validated each vulnerability and wrote patches by hand. As the volume of findings grew, we brought in external (human) security researchers to help with validation and patch development. Our intent here was to meaningfully assist human maintainers in handling our reports, so the process optimized for reducing false positives. In parallel, we are accelerating our efforts to automate patch development to reliably remediate bugs as we find them.
Re:Real vulnerabilities? (Score:5, Funny)
If you read the linked blog post [anthropic.com] in TFA
Read the article? I get ragged on when I admit even reading the summary!
Re: (Score:2)
Re: (Score:2)
> If you read the linked blog post [anthropic.com] in TFA
That is not the Slashdot way.
Re:Real vulnerabilities? (Score:5, Interesting)
So basically they had Claude grep for "memcpy" and "strcpy", and then had humans actually review to see if those two functions were being called unsafely.
I'm only being partially sarcastic here. Having seen the slop examples that Daniel Stenberg (curl dev) has posted repeatedly, we won't know if Claude has done anything useful unless we at least see how much chaff was separated from the wheat by human review. If those 500 "high security" vulnerabilities (in Ghostscript? We're using Ghostscript in high security situations now? Are printer makers running it as root or something?) were whittled down from 100,000 initial reports, has Claude done anything useful that a basic human review wouldn't have achieved?
I also find it interesting they picked the low hanging fruit for this. This wasn't a list of software that undergoes security reviews that often. So I'd expect more buffer overflow type issues simply because there's no urgent call for those kinds of bugs to be fixed.
I'm... skeptical here. I think they intentionally chose software they knew wasn't already under audit to increase the numbers, and I think the fact important stats were left out of the press release, like how many non-issues Claude found, makes it likely an exceedingly high volume of Claude's initial results were slop.
Re: (Score:2)
If those 500 "high security" vulnerabilities (in Ghostscript? We're using Ghostscript in high security situations now? Are printer makers running it as root or something?)
If ghostscript has access to everything you're printing, then a hole in it could potentially be used to exfiltrate any of that information. Unless every gs process is run with process separation, which TBF is a thing that you could do, that's a real risk.
Re: (Score:2)
I would suggest that in most printers that use it the printer starts a new Ghostscript process for each print job. That would limit potential damage to, say, an embedded EPS, which is a format I'm practically dating myself by mentioning its existence.
Unless Ghostscript is being run as root I suspect pretty much any vulnerability is in practice useless.
Re: (Score:2)
Unless Ghostscript is being run as root I suspect pretty much any vulnerability is in practice useless.
If all of the gs processes are being run as the same user and not in containers then they are all vulnerable to one another.
Re: (Score:2)
If those 500 "high security" vulnerabilities (in Ghostscript? We're using Ghostscript in high security situations now? Are printer makers running it as root or something?)
That's "high severity," as in e.g. code execution. Ghostscript is a candidate for code execution by just displaying a PDF or printing a postscript file. Ghostscript as an external filter for PS, PDF etc. may still be part of the default installation of ImageMagick on many Linux distributions.
Re: (Score:2)
Postscript is Turing complete. We already know Ghostscript can run code!
But yeah, while technically someone might find some vulnerability, it's pretty limited in practice. Few people use Ghostscript to view files directly - PDF viewers often show it, but I believe they go through a basic PDF to PS transition first, limiting the attack surface. It is embedded in printers, and I believe it's part of some printing subsystems, but in both cases it's generally heavily sandboxed, in part because PS is TC. Most pe
Re: (Score:2)
Try reading more than the title. (Score:2)
Real vulnerabilities?
It says right in the summary, "each one was validated by either a member of Anthropic's team or an outside security researcher".
Re: (Score:3)
the cURL project just cancelled their bug bounty program because of AI slop garbage
Re: (Score:2)
But not because of AI being bad, but because of humans being bad at using AI.
It was not (insert your favorite AI here) that created the reports. It were humans who hoped to make a few dollar by copy&pasting code into ChatGPT. The misunderstanding is that some bounty hunters thought they can better use AI to find bugs than the project maintainers themselves. It's like when you think you're an author because you types "Write a story" into an LLM. An author can improve using an LLM, but to become and autho
Re: (Score:3)
Re: (Score:2)
The irrational believers will come up with any and all fake "explanations" why their belief is correct. They will never really think about it though. They will just go on mindlessly believing. The AI fools are just one instance of that problem.
Re: (Score:2)
I don't think you understood what I'm saying.
Re: (Score:3)
And on top of that, they may be "real" vulnerabilities that are prohibitively hard to exploit or that do not even come with a credible exploitation scenario. Also remember the mountain of slop the cURL people have been getting.
Without detailed documentation and independent analysis of each of these, the "500" claim is simply nonsense and probably qualifies as a lie by misdirection.
And finally we would need to how many vulnerabilities this thing did not find and how sensitive the process is to prompt variati
Re: (Score:2)
Given various open source developers complaining about piles of AI slop vulnerability reports that aren't actually valid, it's the obvious question to ask.
Very true. Also makes me wonder if there is a consistent definition of "slop" across AI systems. We should ask every AI to ask every other AI about that. Figure out who's on first and grab the popcorn.
And the other obvious question is whether it provided fixes.
Depending on how good it is at the findin', you might not want to read the output of the fixin'. You can choke to death on popcorn that way.
The even more obvious answer (Score:2)
The answer to your question is literally in the summary. Of course you will now suggest that they are motivated to confirm false bugs, because if we all know one thing, listening to all the "experts" here on Slashdot, AI can't possibly work!
A new era for aggressive ads on /. (Score:1)
An inflection point on the way to an unstoppable collapse into an irrelevance singularity.
Re: A new era for aggressive ads on /. (Score:2)
Re: (Score:2)
They clearly are getting more desperate. Their business numbers must be even worse than publicly known.
It's possible to imagine that... (Score:1)
...future AI tools will enable us to make bug free software. Not today, not soon, but it could be possible
As for the posted announcement, good progress, most likely exaggerated, but still progress
Re: (Score:2)
The only small problem will be the text in the labels.
Instead of "Ok" and "Cancel" you'll have random strings, say "Gker" and "Onfeellrp".
And they will change from version to version, along with the placement, colors and the other ui decorations.
But then it will all switch to voice.
Re: (Score:2)
Only if you are clueless. There is mathematical proof that LLMs cannot do that. See, for example, https://arxiv.org/pdf/2507.075... [arxiv.org]
Re: It's possible to imagine that... (Score:2)
Do you understand that paper?
Re: (Score:2)
Yes. Do you?
Re: It's possible to imagine that... (Score:2)
No I don't understand it. It gets technical quite quickly.
Does it say LLM can't solve complicated problems or does it say the attention thing can't?
Re: (Score:3)
Ok, let me do some gross simplification for you that should still capture most of the gist:
They say that there is a boundary on computational complexity in a query, above which an LLM will always (!) hallucinate (Theorem 1). This boundary is not really high and somewhere around O(n^3) in the input length, with an O(n^3) task to be solved in the input.
The thing is, code analysis and verification is an exponential (!) task in the size of the code to be verified (and hence the input), which is why there exist
Re: (Score:2)
> And that is why an LLM cannot and will never be able to reliably generate correct code, unless it is really simplistic code
Whilst I'm inclined to agree with you, I believe the hope is that scaling the LLM and its various input systems will push the level beyond "simplistic" up to "complex", and so "never" is unlikely.
FWIW, I think there's likely some capacity for this to be true. In my meagre experience, LLMs seem to lose the plot after a lot of context is added - that sounds like a "memory" sort of pr
Re: (Score:1)
It's pretty straight forward. An LLM inference algorithm performs less steps than is required for many tasks you can ask of it. The authors go into the details of why that's true. So if you ask an LLM to solve such a task, it will output something, but not the correct answer. But the authors' note:
"In this light, it is important to also note that while our work is about the limitations of individual LLMs, multiple LLMs working
together can obviously achieve higher abilities."
The flagship models and agents th
Re: (Score:2)
The specific limit does not apply to Opus 4.6, the principle is still valid, just possibly with somewhat changed parameters.
Re: (Score:1)
The principle is based on the inference algorithm used by LLMs. As far as I'm aware, models like Opus 4.6 do not have publicly available algorithms, but they seem to have variable runtime based on the task. As far as I can tell, they look more like iterative algorithms used in computational linear algebra: the LLM re-prompts, adjusting outputs until some stopping criteria is met. It's pretty well known, for example, that a model like o3 can arrive at better answers by "running longer." So usually the premiu
Re: (Score:2)
Bla, bla, bla. All I see here is some wishful thinking.
Re: (Score:2)
Re: (Score:3)
Possibly. I may even have read it back then. But I do not remember either.
The thing I remember from about the same time is that when IBM presented Watson, they never even called it AI when addressing professional audiences (I was at two events here, one was invitation only). They always called it only a "knowledge engine" and presented it essentially as a searcher, sorter, filter and prioritizer.
such bullshit (Score:1)
> with little to no prompting ... to see how well it could find bugs in open-source code
> tested Opus 4.6 in a sandboxed environment
Oh shit, Opus 4.6 spontaneously found a bunch of bugs, no prompting, just boom here they are!
Man, I can't wait for all of the pull requests to fix them. Surely, it will spontaneously fix them all too, with tests and documentation of course.
In just a few short months the world will be bug free. As long as we all start using the big subscription, and more, today.
How many false positives? (Score:3)
And who had to dig through those false positives to find the actual issues.
Further more, the article doesn't mention the security issues, nor the software it was found in. Just the number 500.
Re: (Score:2)
Further more, the article doesn't mention the security issues, nor the software it was found in. Just the number 500.
Indeed. That makes any verification of the claim impossible. In Science, such a statement is regarded as completely worthless and meaningless. In IT security, it is worse and regarded as basically a lie by misdirection. At least by competent IT security experts.
What a real test would show is how many vulnerabilities and which ones exactly were NOT found. And how much the set of vulnerabilities changes with different prompts. Because that is what attackers are going to do.
Re: (Score:2)
In security one rather has 100 false positives and 10 correct positives than 100 correct negatives and 10 false negatives.
If and only if you can validate the false positives fast enough. If it takes you more than a day to clear a day's worth of false positives you never have time to address the true positives and they become the equivalent of false negatives.
Re: (Score:2)
Yep. That is what happened with IDS consoles that were all the rage some time back: Hundreds of alerts each day, with probably some interesting ones in there. Bit nobody had the time or the money to look at them all. And hence these consoles became ignored, often after a few days, sometimes even faster.
While, in principle, you do not want to have false negatives, drowning in false positives will kill your analysis capability completely.
Low hanging fruit (Score:2, Interesting)
I don't know what to make of TFA. The problem I have is there is no information on the 500 provided. There are just three examples and it isn't clear to me which if any of the three constitute "high-severity vulnerabilities".
Ghostscript is where anyone would look for an easy source of bugs to find given it is older than dirt itself with relatively few eyes. We have tools like syzbot that managed to find several thousands of bugs in the Linux kernel alone. Just equivalent of grepping for strncpy et el is
legacy C code (Score:3)
The main fallacy here is to assume there was a lot of effort spend making old C code safe. Maybe they did some fuzzing, etc. and this is great. But if I were takes to make legacy C code safe, I would go over and replace every open-coded string or buffer manipulation by a safe string or buffer abstraction. Yes, nobody want to do this, because playing with AI or new languages is so much more fun.
Even the fact that an issue was fixed in one caller of the function, but nobody considered the other callers before, tells you that the least possible maintenance effort was spend on these issues before.
Re: (Score:2)
The main fallacy here is to assume there was a lot of effort spend making old C code safe. Maybe they did some fuzzing, etc. and this is great. But if I were takes to make legacy C code safe, I would go over and replace every open-coded string or buffer manipulation by a safe string or buffer abstraction. Yes, nobody want to do this, because playing with AI or new languages is so much more fun.
Even better: Tell the AI to go replace all of the potentially-unsafe constructs. LLMs are really good at that sort of refactoring.
And how many of those... (Score:2)
...are false positives or low hanging fruit? How many of those are "logging not enabled" audit findings? How many would be reproducible and not annoy thebdevs when reported?
Re: (Score:1)
Probably most of them. And then there is the question of how many it did miss. "Vulnerabilities found" is an amateur-level metrics. What counts is "vulnerabilities remaining".
All this shows is that the LLM may possibly be beneficial for attackers. If they just prompt different, they may find entirely different vulnerabilities. And the one type of code LLMs can write somewhat successfully is attack code, because it does not need to be reliable, secure or maintainable.
Hence while this is pushed as a "success"
No. The claim is a lie by misdirection. (Score:1)
What this actually shows is a benefit to attackers. To be a benefit to defenders, we need to know how many vulnerabilities it (or LLM-type AI in general) could have found but did not. Attacker can just try with variations and then get vulnerabilities this stunt did not find.
Let's face it: LLMs are really crap at generating secure code and at finding non-trivial vulnerabilities. Any claim to the contrary is simply a lie at this time.
Sounds like a diversion effort... (Score:2)
And closed source? (Score:5, Interesting)
What about non-OSS? (Score:2)
AI has lots of data about OSS. Source, working programs, guides, blogs, git/svn logs.
How would it work on proprietary code? Especially code that doesn't use lots of OSS (how much is left nowadays?)
Would it have enough data? This could be an argument against OSS being less secure.
Re: (Score:2)
Code is code. And security vulnerabilities are much the same everywhere.
The owners of proprietary code will probably insist that some sort of wall be put up between the acquisition of new training data and the code review. To keep their IP from leaking out.