AI-Generated Voice Firm Clamps Down After 4chan Makes Celebrity Voices For Abuse (vice.com) 107
An anonymous reader quotes a report from Motherboard: It was only a matter of time before the wave of artificial intelligence-generated voice startups became a play thing of internet trolls. On Monday, ElevenLabs, founded by ex-Google and Palantir staffers, said it had found an "increasing number of voice cloning misuse cases" during its recently launched beta. ElevenLabs didn't point to any particular instances of abuse, but Motherboard found 4chan members appear to have used the product to generate voices that sound like Joe Rogan, Ben Sharpio, and Emma Watson to spew racist and other sorts of material. ElevenLabs said it is exploring more safeguards around its technology.
The clips uploaded to 4chan on Sunday are focused on celebrities. But given the high quality of the generated voices, and the apparent ease at which people created them, they highlight the looming risk of deepfake audio clips. In much the same way deepfake video started as a method for people to create non-consensual pornography of specific people before branching onto other use cases, the trajectory of deepfake audio is only just beginning. [...] The clips run the gamut from harmless, to violent, to transphobic, to homophobic, to racist. One 4chan post that included a wide spread of the clips also contained a link to the beta from ElevenLabs, suggesting ElevenLabs' software may have been used to create the voices.
On its website ElevenLabs offers both "speech synthesis" and "voice cloning." For the latter, ElevenLabs says it can generate a clone of someone's voice from a clean sample recording, over one minute in length. Users can quickly sign up to the service and start generating voices. ElevenLabs also offers "professional cloning," which it says can reproduce any accent. Target use cases include voicing newsletters, books, and videos, the company's website adds. [...] On Monday, shortly after the clips circulated on 4chan, ElevenLabs wrote on Twitter that "Crazy weekend -- thank you to everyone for trying out our Beta platform. While we see our tech being overwhelmingly applied to positive use, we also see an increasing number of voice cloning misuse cases." ElevenLabs added that while it can trace back any generated audio to a specific user, it was exploring more safeguards. These include requiring payment information or "full ID identification" in order to perform voice cloning, or manually verifying every voice cloning request.
The clips uploaded to 4chan on Sunday are focused on celebrities. But given the high quality of the generated voices, and the apparent ease at which people created them, they highlight the looming risk of deepfake audio clips. In much the same way deepfake video started as a method for people to create non-consensual pornography of specific people before branching onto other use cases, the trajectory of deepfake audio is only just beginning. [...] The clips run the gamut from harmless, to violent, to transphobic, to homophobic, to racist. One 4chan post that included a wide spread of the clips also contained a link to the beta from ElevenLabs, suggesting ElevenLabs' software may have been used to create the voices.
On its website ElevenLabs offers both "speech synthesis" and "voice cloning." For the latter, ElevenLabs says it can generate a clone of someone's voice from a clean sample recording, over one minute in length. Users can quickly sign up to the service and start generating voices. ElevenLabs also offers "professional cloning," which it says can reproduce any accent. Target use cases include voicing newsletters, books, and videos, the company's website adds. [...] On Monday, shortly after the clips circulated on 4chan, ElevenLabs wrote on Twitter that "Crazy weekend -- thank you to everyone for trying out our Beta platform. While we see our tech being overwhelmingly applied to positive use, we also see an increasing number of voice cloning misuse cases." ElevenLabs added that while it can trace back any generated audio to a specific user, it was exploring more safeguards. These include requiring payment information or "full ID identification" in order to perform voice cloning, or manually verifying every voice cloning request.
Who is yooh daddy, and wut does he do? (Score:5, Funny)
Re: (Score:2)
Soooo unexpected (Score:2)
> After 4chan Makes Celebrity Voices For Abuse
That's a 3 way redundancy I think
Re: (Score:3)
Weaponized Autism for the win.
Again.
Re: (Score:1)
Re: (Score:2)
On the bright side this is a win for privacy. Even if you say something off color you can cry deepfake.
Parody (Score:5, Insightful)
Parody can be problematic when it's not obvious (Score:5, Interesting)
Fox Business picked up the bear mascot story and ran it. I don't mean prime Time Tucker Carlson frothing Insanity I mean Fox Business.
Poe's law is now so powerful it sucked everyone on the right wing in and they can't tell fact from fiction anymore. Satire is dead when the other side is that absurd.
Re:Parody can be problematic when it's not obvious (Score:5, Insightful)
Gentle reminder than the other side believes a man is entitled to waddle into a ladies' room because he larps as a teenage girl with male pattern baldness
Actually, I believe that restrooms should be designed so every user has complete privacy and the issue is moot. See, I'm a gay man, ever since adolescence it always weirded me out that I was expected to use the bathroom of the sex I found myself attracted to. Weirded out meaning I felt like what I'm assuming is the same way it feels for a woman to know some dude might be sneaking a peek at her while she's using the bathroom.
I've never wanted to use the bathroom of the opposite sex, or thought that it would make sense to have bathrooms specifically for gay men/lesbian women. The most logical solution just seemed to be that bathrooms should simply be constructed so as not to make pissing and taking a shit some sort of bizarre communal experience and instead to have actual individual "rooms". You'd think drywall was made of gold or something.
Re: Parody can be problematic when it's not obviou (Score:3)
Yeah I have seen bathrooms like this in Europe and at nice places. Individual small rooms for everyone.
Re: Parody can be problematic when it's not obvio (Score:2)
But how big were they? You know regardless of size, the next homeless bum wont hesitate to shack up in there regardless of gender
Re: (Score:2)
Apparently in some European countries the government pays for homeless to have somewhere to live rather than abandoning them to die in the streets.
Probably raises taxes slightly, and probably some of those homeless people 'deserve' their death sentence, but the government cares for them anyway.
Always thought this was ridiculous (Score:2)
It's moral panic. Right Wing Woke.
Re: Parody can be problematic when it's not obviou (Score:2)
This doesn't make a whole lot of sense. Until two centuries ago public bathrooms were all unisex and there were no walls at all. You just sit on a hole, shit or piss, maybe have a conversation while you're at it, and nobody cares that they can see what's between your legs because they saw it all the damn time anyways.
Progressives don't like to hear this and will deny it because it goes against the narrative their power structure is built on, but what you're talking about is actually a symptom of the entire
Re: (Score:2)
You just sit on a hole, shit or piss, maybe have a conversation while you're at it, and nobody cares that they can see what's between your legs because they saw it all the damn time anyways.
Progressives don't like to hear this and will deny it because it goes against the narrative their power structure is built on
Bathroom laws
Ratio of dick to vag on wedding cake
Mommies per book
How many sticks you're allowed to bring to field hockey.
All progressive ideas supporting the power structure built on "The Way Things Used To Be", because they just can't stop thinking about what's between your legs.
https://www.psychologytoday.co... [psychologytoday.com]
Re: Parody can be problematic when it's not obvio (Score:2)
I'm not talking about "the way things used to be" but rather the fact that we're currently way better off than we have ever been at any point in history in every way you can imagine, including wealth. That last bit is what progressives will vehemently deny because it basically crushes the whole idea of class struggle that they want to sell to the public to get votes.
Re: (Score:3, Insightful)
Allow me to put forth a modest proposal that people failing to recognize satire isn't exactly a new thing.
Re: (Score:3)
The new thing is rather that it's impossible to produce satire without at least some people thinking you're some batshit insane loonie who is actually serious, because such batshit insane loonies do actually exist. Not just in padded walls but in mainstream news outlet opinion pieces.
Re:Parody can be problematic when it's not obvious (Score:4, Insightful)
That's not new. What's new is that your satire can reach absolute shedloads of idiots before even one intelligent person has a chance to comment on it.
Re: Parody can be problematic when it's not obviou (Score:1)
Re:Parody can be problematic when it's not obvious (Score:4, Funny)
Dude, just saying, the left wing is just as susceptible to this nonsense.
I personally see both sides frothing at the mouth of the exact same nonsense.
To me, all this proves is that political gas lighting is real, and that it works on the uneducated, the stressed, and the the bandwidth challenged.
So yea. Vote for Joe Trump 2025, or whatever -- same shit, imho.
(and yes, this is mildly farcical post meant to make you all step aside and think a bit, so "/s" ya dirty animals)
Re: (Score:2)
to be very clear "bandwidth challenged" means you're just so fucking busy with your own damn life that political crap doesn't get adequate time slice for you to do your own actual research, so you're effectively forced to take the word of the talking-head summarizers
Re: (Score:1)
I dunno, there does also seem to be some correlation between slow internet access and certain political leanings. My folks retired from Florida (yeah, how's that for ass backwards?) for a part of NC which up until very recently only had horribly slow 1Mbps DSL. They do now have Elon Musk's thing, which works well enough when it's not intermittently cutting out. My father has been a Republican for as long as I'm aware, and even he complains about the area he lives in being too far-right for his tastes.
Mea
Re: (Score:3)
Lol well played, but as I clarified, "bandwidth challenged" was a reference to personal free time.
I, for one, have noticed I'm increasingly infringed upon by my employer(s) at every fucking waking moment.
Now, they pay me well, and it's not awful to feel needed and wanted, but it does mean that I have less quiet time to reflect on literally anything that isn't directly about work.
For instance, I used to read 3-5 books of my own selection per year, usually while on vacation. I now read maybe 1-2, and it's dif
Re: (Score:2)
The problem here is that more often than not, the people who claim they do "their own actual research" ARE just taking the word of the talking-head summarizers.
Re: (Score:2)
So then their just bandwidth challenged, but desperately trying to "help" their neighbors nonetheless.
I won't say that's not destructive, because it clearly is, but they're honestly trying to bring good intentions, just with bad data. Again, no time to do real research to cull chaff.
Re:Parody can be problematic when it's not obvious (Score:4)
oof -- their vs they're -- I'm drunk, sorry grammar nazis, please spare my soul
Re: (Score:2)
You! I like you!
Seriously, that is probably the most solution oriented statement I have seen in a month on the internet.
Keep it up, the world desperately needs it.
Re: (Score:2)
Bandwidth challenged.
Damn, that fits so well. I'm gonna use that.
My way to protect myself, incidentally is not to just go with whatwver I hear but to just keep away from news and shrug and keep my mouth mostly shut when engaged in political topics by people around me.
Re: (Score:2)
It sounds like you, yourself, are bandwidth challenged, but have decided to address this problem by not expressing your feelings (as they are unfinished or lack information). Not to be derogatory, by any means. I too am bandwidth challenged, and I feel it impact my life every day. I do my best to get the best information available, but these days it's honestly very difficult to be prepared for casual political discussions amongst friends, let along for impromptu debates amongst newcomers (some of these folk
Re: (Score:2)
It sounds like you, yourself, are bandwidth challenged, but have decided to address this problem by not expressing your feelings (as they are unfinished or lack information). Not to be derogatory, by any means. I too am bandwidth challenged, and I feel it impact my life every day. I do my best to get the best information available, but these days it's honestly very difficult to be prepared for casual political discussions amongst friends, let along for impromptu debates amongst newcomers
dude, politics is actually pretty simple: the elite fucks over the working class, and will continue to do so until the working class is no longer needed. that's all the bandwidth you really need, any further detail on that is just circumstantial at best, mere gossip.
We're not really (Score:3)
What he found was that if he did right wing fake news it went viral and spread because people shared it. He made his money (millions) off the ad revenue from posting obvious and insane lies.
He was asked why he didn't do the same with with the left wing. Again, he's a businessman, no ideology whatsoever and he's not doing anything illegal so he just
Re:Parody can be problematic when it's not obvious (Score:4, Informative)
M&Ms did a joke about retiring the M&M's spokes candies because they were two divisive
(a) It's not a joke [nbcnews.com], and (b) the word is "too", not "two".
It would be ironic that you post illiterate lies and claim that the other side are the ones who can't tell reality from fiction -- if it were not so deeply ingrained in your shtick.
I use google's voice to text (Score:2)
Kinda proves my point about how right wing narratives spread. You guy's'll mod up anything.
Re: (Score:2)
Welcome to the new Slashdot. I've seen my comment scores fall hard in the last week or so in particular, so clearly something has changed with regard to who's getting mod points.
However, the comment was not Ad Hominem because it didn't say your comment couldn't be trusted because of something about you, which is what's required. Insults are not Ad Hominem unless an argument is constructed on them.
Re: (Score:2)
HAHA this place is dead why would anyone care about what you post?
Because it is an advertising vehicle, and anything that conflicts with the advertising message detracts from its value. If this place is so dead, why are there still advertisements here?
Re: (Score:2)
How did your lies get you a +5 comment except that leftists will mod up even wrong comments because they criticize conservatives?
It would be hilarious that you admitted to not knowing what logical fallacies were which when you falsely accused me of using one (pointing out your shitty behavior isn't a fallacy), but again -- it's something you do all the time, so it's just sad.
Likewise for how you made a lame excuse for using the wrong word, without addressing the fact that you were simply wrong on the facts.
Re: (Score:2)
M&Ms did a joke about retiring the M&M's spokes candies because they were two divisive
(a) It's not a joke [nbcnews.com], and (b) the word is "too", not "two".
It would be ironic that you post illiterate lies and claim that the other side are the ones who can't tell reality from fiction -- if it were not so deeply ingrained in your shtick.
Ahem. The reality is they are now called Ma&Ya's.
https://www.mms.com/en-us/ma-a... [mms.com]
Guys please try to keep up with this definitely not a joke, very serious rebranding, this is for real, and we don't want to hurt Entrope's feelings. Definitely do NOT tell her what irony is.
Re: (Score:2)
BOTH sides have elements that are that absurd. And both sides have court cases that uphold the absurd points of view. Consider the burglar who fell through a skylight he was breaking in through and sued the owner because he broke his leg...and won.
Poe's law is so powerful because in a large enough sample you can find people who will uphold ANY idiocy. They may be a small minority, but public debating rules ensure that the other side will give them wide publicity, and won't mention the more moderate point
Re: (Score:1)
Consider the burglar who fell through a skylight he was breaking in through and sued the owner because he broke his leg...and won.
Wow. Almost nothing in your version of that story is true. He was trespassing and not a burglar. He wasn't trying to break into the skylight. It was painted over and he fell through it, because he couldn't see it. He was accused of trying to steal a $35 flood light from the roof. He claims he was just moving it (snort). He sued the school district, because he fell 27 feet and was permanently disabled. He didn't win that law suit. It was settled out of court. So basically you got the gender and skyl
Re: (Score:2)
I think you may be talking about a different case. IIRC the case I'm thinking of involved a SF school district, and he fell through the skylight. I admit, though, that I don't know he was convicted of burglary, just that that's how the newspaper reported it. And if I recall the story correctly he was in his 20's, or possibly older. (I.e., he was not charged as a juvenile.)
That said, I never saw any follow-up. Perhaps the newspaper really distorted it sufficiently that it's the same as the case you're t
Re: (Score:1)
Re: (Score:3)
ElevenLabs is just milking this for the free publicity...
Don't confuse publicity with outrage avoidance. It's 2022. Every idiot who doesn't like something shouts it to their million twitter followers and Karen at home starts getting angry and vowing to sue when she sees it on her social media feed.
Some companies just don't want to deal with that shit, understandably so.
Re: (Score:2)
Could be defamation. Lying about someone being, say, antisemitic, could certainly harm their career or cause them to spend money publicly refuting the claim. Backing that lie up with a deepfake voice clip would just make it harder to undo the damage.
There's also a potential copyright/likeness rights issue with celebrity voices.
Re: (Score:2)
Parody is fine, but one needs to be able to tell that it *is* parody. This doesn't seem to really count. This is closer to slander or fraudulent misrepresentation.
Re: (Score:3)
That's not Parody.
They aren't using Elon Musk's voice (Score:2)
They gave up - there's nothing people can't imagine him saying.
Re: (Score:1)
They gave up - there's nothing people can't imagine him saying.
How about "maybe I'm not that smart after all".
Re: (Score:1)
He'd say it for attention.
Re:They aren't using Elon Musk's voice (Score:4, Funny)
"I'm redesigning the cybertruck to not look like a DeLorean rendered on a Nintendo 64."
Re: (Score:2)
Frontal area is the biggest factor and it has few protrusions, so it's not surprising if the aero is good. However, those big flat areas will be prone to oilcanning, and it's way more difficult to make a repair to a flat area look good than one to a curve because the curve tends to hide imperfections, and also because repairing a big flat without causing oilcanning is itself difficult. A bigger question than aero is whether the cybertruck can possibly pass US pedestrian crash safety requirements with a shar
Re: (Score:2)
"I'm redesigning the cybertruck to not look like a DeLorean rendered on a Nintendo 64."
I always thought it looked like its inspiration came from BattleZone [saymedia-content.com].
NO. FUN. ALLOWED (Score:5, Insightful)
Those awful, awful trolls made a celebrity voice say the n-word and laughed about it on their shitty board. This is a TRAVESTY, nay a TRAGEDY! It's basically like the holocaust all over again.
Re:NO. FUN. ALLOWED (Score:5, Interesting)
There really needs to be some legal clarification here. For example, deepfake porn. Is that allowed without the consent of the subject?
It's currently a legal grey area. Involuntary pornography, like leaking people's private photos or taking pictures of them with hidden cameras, is illegal in many places. On the other hand, magazines regularly publish photos of celebrities taken with zoom lenses pointed at private property, without consent.
AI artists can generate an image from a simple prompt like "draw a photorealistic nude of $celebrity". In itself not illegal, but could be if the image is distributed. That said, it's likely that commercial operators would want to block their service from being used to generate porn anyway.
It's not fun (Score:1)
Re: (Score:2)
There really needs to be some legal clarification here. For example, deepfake porn. Is that allowed without the consent of the subject?
In the USA it's almost certainly illegal at least to distribute and possibly even to create, because you have the right to control use of your likeness. In the UK it isn't, because you don't, unless someone is faking an endorsement or doing provable harm.
That said, it's likely that commercial operators would want to block their service from being used to generate porn anyway.
That ship has sailed, since Stable Diffusion exists and you can create your own embeddings, trainings, etc.
At some point. (Score:5, Insightful)
Everything will have to be in person to be trusted that it is actually being said by that person.
Re: (Score:3)
Video and audio will not be something that can be trusted. At this rate it won't be too long before they won't be usable as evidence in the court of law once the technology is refined.
Everything will have to be in person to be trusted that it is actually being said by that person.
or we have camera and other video equipment cryptographicly sign their output so as to have chain of evidence to show no tampering.
Re: (Score:1)
Re: (Score:2)
New way of DRM?
Nobody is proposing that the video be encrypted, or nonstandard, only signed. That's not DRM.
Re: (Score:1)
Re: (Score:2)
Sure, by compromising the chip that performs the function you could conceivably extract the keys and then sign whatever you wanted. All it really does is raise the bar to the point where only very wealthy actors can reasonably fake a video. That's not nothing, though.
Re: (Score:2)
Wouldn't help. People will just do what is needed to remove the signature entirely from the source files. If you can play it at all, it can be captured. You can't connect all parts of the chain without deeply invasive tools and the banishment of all the legacy stuff.
Re: (Score:2)
Wouldn't help. People will just do what is needed to remove the signature entirely from the source files.
That would be self-defeating, because without the signature the video stream would [obviously] be treated as untrusted.
Re: (Score:2)
Video and audio will not be something that can be trusted.
It hasn't been trustable since the advent of splicing magnetic tape. Like most things, computers are just making it a whole lot easier and more accessible to manipulate.
Re: (Score:3)
If that makes celebs stop to think before they run their mouth instead of spouting all sorts of nonsense for shock value and generating clicks, I can't really say I think it's a bad thing.
Re: (Score:2)
If that makes celebs stop to think before they run their mouth instead of spouting all sorts of nonsense for shock value and generating clicks, I can't really say I think it's a bad thing.
It won't, for two reasons. One, even without deepfake technology you can fake audio using things people actually said. Two, deepfake audio lets you create "recordings" of people saying things they never actually said.
Re: (Score:2)
Your "ends justify the means" approach breaks down once you consider that it doesn't end at the borders of celebrity. If a shady businessman wants out of a contract, he might show the other side in video or audio breaking an NDA, or some other term. If your ex is angry at you, deep faking you being openly and toxically racist might be on the agenda. Are you willing to take those terms just so celebrities behave more in line with you you think they ought to?
Re: (Score:1)
Re: (Score:2)
"Information wants to be free" is horseshit. Some people want information to be free. Shovel your freedom - we live in a society, and some freedoms have got to go.
Otherwise I agree with you. It is inevitable.
Re: (Score:2)
information wants to be free
Please don't anthropomorphise information, it really hates that.
Re: (Score:2)
That point was passed long ago.
Re: (Score:2)
Video and audio will not be something that can be trusted. At this rate it won't be too long before they won't be usable as evidence in the court of law once the technology is refined.
Everything will have to be in person to be trusted that it is actually being said by that person.
Came to say basically the same thing. It's already to the point that if I'm ever on a jury, I'd be skeptical of photos or recordings that don't have a chain of custody attached. (and even that doesn't really prove it was created from reality instead of by AI)
as opposed to their own podcasts (Score:3, Funny)
generate voices that sound like Joe Rogan, Ben Sharpio,... to spew racist and other sorts of material.
So what you made Joe Rogan and Ben Sharpio say racist thing, as opposed to the racist things they say on their own platforms? Oh No! Now there are recordings were they say offensive things without ad revenue how horrible.
How do they plan to do this? (Score:2)
If they maintain copies of, for example, James Earl Jones' voice recordings, to prevent 4chan from uploading clips of JEJs voice as their own, doesn't this mean that they need to have a license from JEJs estate? And the same for every other celebrity and voice actor? I can't imagine how much that would cost.
Re: (Score:1)
doesn't this mean that they need to have a license from JEJs estate?
You make it sound like he's dead.
Last time I checked, he's alive and kicking - so they would need to presumably get permission from the man himself, but it's also uncertain just how similar it has to be. You could make a voice that sounds kind of like him but subtlety different enough that it's not exactly the same in some way - where the dividing line is would most likely have to be decided in court.
well... (Score:3)
...what did they expect would happen when they made it available?
Re: (Score:2)
> ...what did they expect would happen when they made it available?
Shenanigans leading to free advertising?
Can it not recognize them (Score:3)
Re: (Score:2)
I would have thought there was enough clips of celebs speaking on the internet that the AI companies could compile a database of them and reject requests to generate speech from clips that sound too much like a celeb.
What makes you think AI companies are the gatekeepers here? Anyone(*) could write an AI to do the same. Companies are not the gatekeepers of technology, they are the ones who make it available to the uneducated masses (for a fee).
Having companies censor themselves will in no way solve the issue and in fact, might make it worse because then some people would be believe that the recording they are hearing is real because X company pledged to not allow such abuses.
(*) Anyone with some amount of experience. Thi
Ben Shapiro? (Score:3, Insightful)
The obvious protection (Score:2)
Would be to make an AI clone of Barbra Streisand's voice. That'll keep these things from spreading all over the Internet.
ElevenLabs ? (Score:2)
"My voice is my password." (Score:5, Interesting)
Re: (Score:2)
I'm sorry, I'll see myself out now.
Re: (Score:2)
I guess (Score:2)
...we can soon expect lawsuits about the 'look and feel' of voices.
People whose voice resembles a celebrity voice have to get their chords surgically removed.
Own a musket for home defence (Score:2)
Eminem's best freestyle IMHO:
https://www.youtube.com/watch?... [youtube.com]
Bonus points for the video creator using typically messed up AI generated images in the begining.
Cats already out of the bag (Score:1)