AI Tools Give Dangerous Powers to Cyberattackers, Security Researchers Warn (msn.com) 21
"On a recent assignment to test defenses, Dave Brauchler of the cybersecurity company NCC Group tricked a client's AI program-writing assistant into executing programs that forked over the company's databases and code repositories," reports the Washington Post.
"We have never been this foolish with security," Brauchler said... Demonstrations at last month's Black Hat security conference in Las Vegas included other attention-getting means of exploiting artificial intelligence. In one, an imagined attacker sent documents by email with hidden instructions aimed at ChatGPT or competitors. If a user asked for a summary or one was made automatically, the program would execute the instructions, even finding digital passwords and sending them out of the network. A similar attack on Google's Gemini didn't even need an attachment, just an email with hidden directives. The AI summary falsely told the target an account had been compromised and that they should call the attacker's number, mimicking successful phishing scams.
The threats become more concerning with the rise of agentic AI, which empowers browsers and other tools to conduct transactions and make other decisions without human oversight. Already, security company Guardio has tricked the agentic Comet browser addition from Perplexity into buying a watch from a fake online store and to follow instructions from a fake banking email...
Advanced AI programs also are beginning to be used to find previously undiscovered security flaws, the so-called zero-days that hackers highly prize and exploit to gain entry into software that is configured correctly and fully updated with security patches. Seven teams of hackers that developed autonomous "cyber reasoning systems" for a contest held last month by the Pentagon's Defense Advanced Research Projects Agency were able to find a total of 18 zero-days in 54 million lines of open source code. They worked to patch those vulnerabilities, but officials said hackers around the world are developing similar efforts to locate and exploit them. Some longtime security defenders are predicting a once-in-a-lifetime, worldwide mad dash to use the technology to find new flaws and exploit them, leaving back doors in place that they can return to at leisure.
The real nightmare scenario is when these worlds collide, and an attacker's AI finds a way in and then starts communicating with the victim's AI, working in partnership — "having the bad guy AI collaborate with the good guy AI," as SentinelOne's [threat researcher Alex] Delamotte put it. "Next year," said Adam Meyers, senior vice president at CrowdStrike, "AI will be the new insider threat."
In August more than 1,000 people lost data to a modified Nx program (downloaded hundreds of thousands of times) that used pre-installed coding tools from Google/Anthropic/etc. According to the article, the malware "instructed those programs to root out" sensitive data (including passwords or cryptocurrency wallets) and send it back to the attacker. "The more autonomy and access to production environments such tools have, the more havoc they can wreak," the article points out — including this quote from SentinelOne threat researcher Alex Delamotte.
"It's kind of unfair that we're having AI pushed on us in every single product when it introduces new risks."
"We have never been this foolish with security," Brauchler said... Demonstrations at last month's Black Hat security conference in Las Vegas included other attention-getting means of exploiting artificial intelligence. In one, an imagined attacker sent documents by email with hidden instructions aimed at ChatGPT or competitors. If a user asked for a summary or one was made automatically, the program would execute the instructions, even finding digital passwords and sending them out of the network. A similar attack on Google's Gemini didn't even need an attachment, just an email with hidden directives. The AI summary falsely told the target an account had been compromised and that they should call the attacker's number, mimicking successful phishing scams.
The threats become more concerning with the rise of agentic AI, which empowers browsers and other tools to conduct transactions and make other decisions without human oversight. Already, security company Guardio has tricked the agentic Comet browser addition from Perplexity into buying a watch from a fake online store and to follow instructions from a fake banking email...
Advanced AI programs also are beginning to be used to find previously undiscovered security flaws, the so-called zero-days that hackers highly prize and exploit to gain entry into software that is configured correctly and fully updated with security patches. Seven teams of hackers that developed autonomous "cyber reasoning systems" for a contest held last month by the Pentagon's Defense Advanced Research Projects Agency were able to find a total of 18 zero-days in 54 million lines of open source code. They worked to patch those vulnerabilities, but officials said hackers around the world are developing similar efforts to locate and exploit them. Some longtime security defenders are predicting a once-in-a-lifetime, worldwide mad dash to use the technology to find new flaws and exploit them, leaving back doors in place that they can return to at leisure.
The real nightmare scenario is when these worlds collide, and an attacker's AI finds a way in and then starts communicating with the victim's AI, working in partnership — "having the bad guy AI collaborate with the good guy AI," as SentinelOne's [threat researcher Alex] Delamotte put it. "Next year," said Adam Meyers, senior vice president at CrowdStrike, "AI will be the new insider threat."
In August more than 1,000 people lost data to a modified Nx program (downloaded hundreds of thousands of times) that used pre-installed coding tools from Google/Anthropic/etc. According to the article, the malware "instructed those programs to root out" sensitive data (including passwords or cryptocurrency wallets) and send it back to the attacker. "The more autonomy and access to production environments such tools have, the more havoc they can wreak," the article points out — including this quote from SentinelOne threat researcher Alex Delamotte.
"It's kind of unfair that we're having AI pushed on us in every single product when it introduces new risks."
Ah yes... (Score:3)
Re: (Score:3)
Grammar and punctuation are not your enemies.
Re: Ah yes... (Score:2)
if you think its bad now (Score:2)
now its not "my refigerator is attacking my xbox" IoT style attack, its the yute down the street with a yagi
Closed source software (Score:2)
The other angle/question I had for some time: How much closed source software is really "closed", when one could try to brute-guess the prompt that was used to generate it?
How much security benefits closed source software would still have remaining?
P.S. AI-powered reverse-engineer (to automate all those tedious repetitive 99% of work) may be a fun topic too.
Re: (Score:2)
Unless the code is only running on your own machines that are isolat
Re: Closed source software (Score:2)
Every new technology enables good and evil (Score:2)
Every invention since the iron age has helped evildoers, and good people alike. AI is no different.
The good news is, the technology also gives "powers" to those who want to fight hackers.
Re: (Score:2)
The Hollywood trope of cyber-intrusion and penetration won by the person giving 'better' orders, will be true. Unfortunately, in reality, that would make every AI, a weapon for the enemy.
Awwww. (Score:2)
Even my manager has become dangerous (Score:3)
With his newly acquired AI powers, making him hallucinate that he can do anything.
"King of unfair"? (Score:3)
Why the timid language? Call it utterly dumb, greedy and immoral, because that is what this is. Just some assholes trying to get rich quick and damm the rest of humanity.
I like how the same people that think.. (Score:2)
Re: (Score:2)
I mean, it's a whole lot easier to break shit then build or fix shit. You can easily lookup CVEs to find exploits. It's not hard to scan for systems that are of the write version that contain the exploit. At that point, I'm quite sure AI could kludge together a tool or script that could gain you access to a system.
Shoot, there are already "security" tools that are freely available that do precisely what I just described. The only different between a security tool and an exploit tool is the motivation of the
screw that (Score:2)
China: oh yeah? Bzzzzzzzt!!!
Pay us ransom (Score:2)