Deepfake Fraud Taking Place On an Industrial Scale, Study Finds (theguardian.com) 53
Deepfake fraud has gone "industrial," an analysis published by AI experts has said. From a report: Tools to create tailored, even personalised, scams -- leveraging, for example, deepfake videos of Swedish journalists or the president of Cyprus -- are no longer niche, but inexpensive and easy to deploy at scale, said the analysis from the AI Incident Database.
It catalogued more than a dozen recent examples of "impersonation for profit," including a deepfake video of Western Australia's premier, Robert Cook, hawking an investment scheme, and deepfake doctors promoting skin creams. These examples are part of a trend in which scammers are using widely available AI tools to perpetuate increasingly targeted heists. Last year, a finance officer at a Singaporean multinational paid out nearly $500,000 to scammers during what he believed was a video call with company leadership. UK consumers are estimated to have lost $12.86bn to fraud in the nine months to November 2025.
"Capabilities have suddenly reached that level where fake content can be produced by pretty much anybody," said Simon Mylius, an MIT researcher who works on a project linked to the AI Incident Database. He calculates that "frauds, scams and targeted manipulation" have made up the largest proportion of incidents reported to the database in 11 of the past 12 months. He said: "It's become very accessible to a point where there is really effectively no barrier to entry."
It catalogued more than a dozen recent examples of "impersonation for profit," including a deepfake video of Western Australia's premier, Robert Cook, hawking an investment scheme, and deepfake doctors promoting skin creams. These examples are part of a trend in which scammers are using widely available AI tools to perpetuate increasingly targeted heists. Last year, a finance officer at a Singaporean multinational paid out nearly $500,000 to scammers during what he believed was a video call with company leadership. UK consumers are estimated to have lost $12.86bn to fraud in the nine months to November 2025.
"Capabilities have suddenly reached that level where fake content can be produced by pretty much anybody," said Simon Mylius, an MIT researcher who works on a project linked to the AI Incident Database. He calculates that "frauds, scams and targeted manipulation" have made up the largest proportion of incidents reported to the database in 11 of the past 12 months. He said: "It's become very accessible to a point where there is really effectively no barrier to entry."
Soon Donald will actually be correct: (Score:1)
everything everywhere will be rigged.
Re: (Score:3, Insightful)
And he'll get his cut.
Re: (Score:2)
Don't worry he has put Jared Kushner in charge of the anti deepfake club [youtube.com]
Re: (Score:2)
Prove it wasn’t rigged a fucking decade ago.
A decade ago was 2016. Just saying.
Re: (Score:2)
Most of us surely had no idea that a whole decade was 10 years ago.
Thank you for doing the math for us.
Re: (Score:2)
A decade ago / 10 years ago / the year 2016 ... was the year Donald Trump was first elected. Dragonslicer is right: that's not the flex the Biden-trashing AC thinks it is.
Never mind math. You should work on your knowledge of this discussion, as well as recent history.
Re: (Score:3, Informative)
> Prove it wasn't rigged a fucking decade ago.
I can't prove there are no unicorns on Pluto, that's not the way it works. Donny's team filed about 60 election fraud lawsuits, and lost 59 (snagged one legal technicality), half of them under GOP judges. MAGAs are bigly LOSERS in Court, where catchy but wrong lies don't work.
Re: (Score:1, Troll)
Nearly all those cases were dismissed without allowing any real fact finding to proceed. You are right they were dismissed on technicalities, standing issues, courts considering the issues moot because the elections were already certified.
What the entire thing proved is the democrats are actually pretty good at cheating and have spent decades carefully engineering the law and election procedures to make accountability nearly impossible.
Re: (Score:1)
Pick your top single best case, let's focus on one and deep-dive it!
Re: (Score:3)
Nearly all those cases were dismissed without allowing any real fact finding to proceed.
No. In each of those cases, Trump's lawyers were asked to provide evidence of their claims. In each case, Trump's lawyers declined because they were under oath and did not want to perjure themselves. Giuliani was eventually disbarred over this [factcheck.org].
Re: (Score:1)
If we were truly that good at cheating, by now we'd have single-payer healthcare, be taxing the rich, have a Dem SCOTUS, and would have cancelled the bloated F35 program.
Re: (Score:1, Flamebait)
They ruled on things like, "We won't take it up because it is already done" or, "Technically, you do not have standing." or any number of things other than, "Did they illegally cheat the election".
Hope that clarifies the legal situation. Also, Georgia recently admitted publicly that over 300,000 votes were improperly counted.
Re: (Score:2)
Citations please. For all of your claims.
Re: (Score:2)
Citations please. For all of your claims.
No. I will not cite every single case brought.
Here is Fulton County itself admitting to over 300,000 early votes not signed off yet submitted in violation of the law.
https://www.atlantanewsfirst.c... [atlantanewsfirst.com]
And here is a bunch of stuff they are just finding, my favorite being this
"Auditors assisting in the Risk Limiting Audit reported counting purported absentee ballots that had "never been creased or folded, as would be required for the ballot to be mailed to the voter and for the ballot to be returned in
Re: (Score:2)
No. I will not cite every single case brought.
I didn't ask you to. I asked you to cite evidence for your claims, not every single case.
Here is Fulton County itself admitting to over 300,000 early votes not signed off yet submitted in violation of the law.
That is not evidence of fraud. As this article [ms.now] points out, that's like accusing someone of tax fraud because they lost a receipt.
And here is a bunch of stuff they are just finding, my favorite being this[:] "Auditors assisting in the Risk Limiting Audit reported counting purported absentee ballots that had "never been creased or folded, as would be required for the ballot to be mailed to the voter and for the ballot to be returned in the sealed envelope requiring the voter’s signature for authentication.""
This is your favorite? Okay, whatever. I think the following paragraph from the article I linked addresses this point adequately.
Unfolded absentee ballots are not proof of fraud. As an affidavit witness (who worked for the state during the election) explained to the FBI, unfolded absentee ballots are common, because damaged ballots would at times be replaced with a clean, readable one. The supposed evidence of suspicious activity, as is the case with many of these allegations, comes from an amateur observer who likely misinterpreted what was happening.
So, unfolded absentee ballots are routine and not evidence of fraud, for the reason above.
That is a video showing what we all saw at the time when poll workers told everyone to leave then stayed behind to count votes without being watched.
The only source I can find for this is the one you supplied: something from
Re: (Score:2)
But the details of what Khan allegedly did are even more alarming than what's stated in this weekend's viral tweets. This case is a grab bag of every worst-case scenario that can come out of the combination of online voter registration, universal mail ballots, and ballot harvesting.
Here are the highlights (more detail can be found in the 2023 story, and all of it is taken from the press conference at the end of this piece):
During a raid of Khan's home in October 2020 on unrelated charges, officials observed a stash of 41 sealed and completed mail ballots for the 2020 presidential election. Due to CA's ballot harvesting laws the ballots were not seized, but investigators photographed the ballots and documented their findings.
In the fall of 2021, officials noticed 70 people were registered to vote at one address in Lodi, which they recognized as Khan's.
Sheriff's investigators determined that Khan had used the state's online voter registration system to re-register existing California voters from other districts to his address, and at least a few non-citizens living in foreign countries (including his brother in Pakistan) to vote using his address, email address, or phone number.
Investigators reviewed the ballot return envelopes from ballots cast in the 2020 general election, which had been maintained by the Registrar, and found that many of those tied to Khan's address all had the same handwriting on the outside.
In October 2022, officials found more than a dozen unopened ballots for the mid-term election, none of them addressed to Khan and many not sent to his address, at his home.
Officials also found Khan's nomination form for his city council candidacy, and determined that numerous signatures on it were forged.
Khan forced voters whose information he'd hijacked to vote for him and for Joe Biden, either by filling them out himself or threatening the voter.
When Khan heard that investigators were speaking to people he'd fraudulently registered to vote, he posted a video to TikTok to threaten and intimidate them.
Now, cover your ears and say "Nah, Nah, I can't hear you - no evidence! no evidence!"
Re: (Score:2)
Re: (Score:2)
None of those courts ruled on the facts of the case.
Yeah they did.
Re: (Score:2)
The only thing dumber than getting your history from an impersonator is getting whooshed by someone pretending to do just that.
FTFY
Is there anything AI doesn't make worse? (Score:2)
Anything?
Re: (Score:3)
Re: (Score:2)
Re:Is there anything AI doesn't make worse? (Score:5, Interesting)
Nah, its ruining chess too. Sites like chess.com its getting harder and harder to find a decent game where your playing a similar ranked human and not getting your shit kicked in by some cheater using stockfish.
This is how one loses faith in systems... (Score:5, Interesting)
There is a very important reason why in history, fraud was aggressively hunted down and destroyed. There comes a time when people will just stop trusting the system. Social media goes back to more decentralized items, perhaps just direct messaging or servers like Discord, or IRC channels. People stop using auction sites, and go back to word of mouth, or trusted supply lines with buyer guarantees. When finding a deal online becomes a gamble, we may even see a swing back to physical purchases, where a successor of Sears that may be a bit more expensive, but has a solid warranty, with some type of assurance that everything bought in the store will be useful and not junk.
If banks start becoming untrustworthy, people will take their cash and use their mattresses as ATMs. Same thing will happen with mail order, or other scamming vectors. For example, fewer and fewer people sell stuff on auction sites because someone will take the item, replace it with a lesser one, and then say they were cheated. Even if the sending of the package was filmed and signed, this rampant fraud can easily make it unprofitable for someone to work with auction sites. Maybe it might be good to use those old malls and start having flea markets again.
Re: (Score:2)
There is a very important reason why in history, fraud was aggressively hunted down and destroyed. There comes a time when people will just stop trusting the system
I'm not sure what time in history you are talking about.
Re: (Score:2)
There is a very important reason why in history, fraud was aggressively hunted down and destroyed. There comes a time when people will just stop trusting the system
I'm not sure what time in history you are talking about.
Rather recently... and that was only for the people who defrauded rich people.
You could rob all the grannies you wanted of their pensions, but if you dared to ply such craft on the rich you'd be hunted down like the vermin you are.
Re: (Score:2)
There is a very important reason why in history, fraud was aggressively hunted down and destroyed. There comes a time when people will just stop trusting the system. Social media goes back to more decentralized items, perhaps just direct messaging or servers like Discord, or IRC channels. People stop using auction sites, and go back to word of mouth, or trusted supply lines with buyer guarantees. When finding a deal online becomes a gamble, we may even see a swing back to physical purchases, where a successor of Sears that may be a bit more expensive, but has a solid warranty, with some type of assurance that everything bought in the store will be useful and not junk.
If banks start becoming untrustworthy, people will take their cash and use their mattresses as ATMs. Same thing will happen with mail order, or other scamming vectors. For example, fewer and fewer people sell stuff on auction sites because someone will take the item, replace it with a lesser one, and then say they were cheated. Even if the sending of the package was filmed and signed, this rampant fraud can easily make it unprofitable for someone to work with auction sites. Maybe it might be good to use those old malls and start having flea markets again.
First point of order, banks are already and always have been untrustworthy. There's a reason they're one of the most heavily regulated industries, they have a long history of being abusive. Banks, as any experience Civ player will tell you, are a necessity, you can't build a modern economy without banks and banking, simply put we need banks but that doesn't mean we should trust them in the slightest. One of the dumbest things I hear on a regular basis is "the bank will take care of me, they're on my side".
Re: (Score:2)
The only inoculation for this is critical thinking
Getting scammed once is a painful enough lesson to make the victim aware of such scams for next time. The hard part is for that first scam to be minor enough to recover from, but harsh enough to hurt. If you're growing up poor you have less capacity to recover, and if you're growing up rich you might have enough money that it doesn't hurt enough to learn a lesson. Or if you're really rich enough you can launder it as an "investment loss" or something.
Re: (Score:2)
There is a very important reason why in history, fraud was aggressively hunted down and destroyed. There comes a time when people will just stop trusting the system. Social media goes back to more decentralized items, perhaps just direct messaging or servers like Discord, or IRC channels. People stop using auction sites, and go back to word of mouth, or trusted supply lines with buyer guarantees. When finding a deal online becomes a gamble, we may even see a swing back to physical purchases, where a successor of Sears that may be a bit more expensive, but has a solid warranty, with some type of assurance that everything bought in the store will be useful and not junk.
If banks start becoming untrustworthy, people will take their cash and use their mattresses as ATMs. Same thing will happen with mail order, or other scamming vectors. For example, fewer and fewer people sell stuff on auction sites because someone will take the item, replace it with a lesser one, and then say they were cheated. Even if the sending of the package was filmed and signed, this rampant fraud can easily make it unprofitable for someone to work with auction sites. Maybe it might be good to use those old malls and start having flea markets again.
Thing is, this time around the big movers and shakers, the people with the money and power to make the big, sweeping decisions, have found a way to profit off of the fraud, by making fraud accessible to all through fraud as a service. I think the dictates of their one and only true God, Greed, will insist that the fraud continue until it becomes unprofitable or damages them in some way that makes the profit untenable. I don't see that happening anytime soon on our current trajectory.
Re: (Score:2)
I have been saying this for some time, now. The fake news/scams epidemic is only punctually about deceiving people. But the broader goal is to actually make people don't believe in anything anymore. So, only "the leader" can be "trusted".
Re: (Score:2)
If banks start becoming untrustworthy, people will take their cash and use their mattresses as ATMs.
This only works if all stores and utilities accept cash. I have no doubt this is why the banking cartel in Australia are trying to make it hard to use cash, by closing physical bank branches, charging for ATMs and giving sweetheart deals to merchants for being "cashless." It concentrates the power with the banks so people don't even have the choice to break away and just stuff cash in their mattress.
And platforms do nothing to stop it (Score:5, Insightful)
There is sufficient information in that data for a platform like Facebook, or YouTube to at least red flag accounts and get some humans to look at it. If that's too much effort for a platform, then make it easier for users to report scams. Even a toggle flag close to the ad which allows people to say "I think this may be a scam" instead of making them fill in a frigging form.
Re:And platforms do nothing to stop it (Score:5, Insightful)
Except platforms earn a significant part of their revenue from scams [reuters.com] (for FB, over 10%). So the incentive to curb scamming is not there...
Re: (Score:2)
If the 10% of the revenue generated on my platform from fraud started to pose any real risk to the 90% of the revenue generated by the legitimate users of my platform I think i'd be plenty motivated to deal with the fraudsters.
facebook and others will clean up the platforms the moment people stop tolerating the abuse. The open question is will people stop tolerating all the abuse?
Re: (Score:2)
I'm just going to give some pure speculation here, but I believe that the people who get scammed are not the same as the people who don't tolerate scamming, and therefore it is easy for social media's algorithms to only show scams to people who are receptive to it - and those people are, probably, happy to be abused (or if unhappy, they won't blame it on the system that plasters their feeds with them, but on the scammers themselves).
If this is real, then the 90% normal revenue is not at risk from the 10% sc
Re: (Score:2)
That this real problem isn't it, the platforms are not platforms they are co-conspirators. Few people blame the phone company if someone calls granny and pretends to be from her local bank branch, nobody blames the county library for putting up a public bulletin board where someone offers house cleaning and also helps themselves to the contents of your wife's jewelry box.
Meta, X, bluesky, tiktok, and friends all want be excused in the same way. Except we all know they are not innocent! They are not bulleti
Re: (Score:2)
Re: (Score:2)
It is quite possible if I did choose personalised ads, that I might be unlucky enough to be in a demographic
Taking place on an Industrial scale? (Score:2)
AI is great at dystopia, any utopia out there? (Score:2)
Amongst all this GenAI noise and nonsense we're missing the stories where it is being practically useful, and I don't mean generating wank for the marketing/scam department.
I use it for code, it's tripped me up a few times, it's like if slashdot was a parrot on my shoulder squawking, "pieces of code, pieces of code". I am lucky not to be in a high pressure delivery environment, I can see the advantages of vibe coding there perhaps for prototyping, but I suspect other things like established frameworks and
Re: AI is great at dystopia, any utopia out there? (Score:2)
That said, I have found it very useful in other areas. It's good for playing "find the thing" in long documents when a straight text search might not cover all key words I'm potentially looking for. It's also excellent for recipes, especially since recipe websites became hot garbage in the last 15 years.
Re: (Score:2)
I get what you're saying. I guess a good analogy would be atomic energy.
It's incredibly useful, not only for electricity generation, but also fields like medicine which greatly benefited from it. Unless someone decides to destroy humanity with it.
I think AI has immense potential. But if it comes at the cost of completely eroding human society, what's the point?