Catch up on stories from the past week (and beyond) at the Slashdot story archive

 


Forgot your password?
Close
typodupeerror
×
AI Software

AI-Generated Voice Firm Clamps Down After 4chan Makes Celebrity Voices For Abuse (vice.com) 107

Posted by BeauHD from the genie's-out-of-the-bottle dept.
An anonymous reader quotes a report from Motherboard: It was only a matter of time before the wave of artificial intelligence-generated voice startups became a play thing of internet trolls. On Monday, ElevenLabs, founded by ex-Google and Palantir staffers, said it had found an "increasing number of voice cloning misuse cases" during its recently launched beta. ElevenLabs didn't point to any particular instances of abuse, but Motherboard found 4chan members appear to have used the product to generate voices that sound like Joe Rogan, Ben Sharpio, and Emma Watson to spew racist and other sorts of material. ElevenLabs said it is exploring more safeguards around its technology.

The clips uploaded to 4chan on Sunday are focused on celebrities. But given the high quality of the generated voices, and the apparent ease at which people created them, they highlight the looming risk of deepfake audio clips. In much the same way deepfake video started as a method for people to create non-consensual pornography of specific people before branching onto other use cases, the trajectory of deepfake audio is only just beginning. [...] The clips run the gamut from harmless, to violent, to transphobic, to homophobic, to racist. One 4chan post that included a wide spread of the clips also contained a link to the beta from ElevenLabs, suggesting ElevenLabs' software may have been used to create the voices.

On its website ElevenLabs offers both "speech synthesis" and "voice cloning." For the latter, ElevenLabs says it can generate a clone of someone's voice from a clean sample recording, over one minute in length. Users can quickly sign up to the service and start generating voices. ElevenLabs also offers "professional cloning," which it says can reproduce any accent. Target use cases include voicing newsletters, books, and videos, the company's website adds. [...] On Monday, shortly after the clips circulated on 4chan, ElevenLabs wrote on Twitter that "Crazy weekend -- thank you to everyone for trying out our Beta platform. While we see our tech being overwhelmingly applied to positive use, we also see an increasing number of voice cloning misuse cases." ElevenLabs added that while it can trace back any generated audio to a specific user, it was exploring more safeguards. These include requiring payment information or "full ID identification" in order to perform voice cloning, or manually verifying every voice cloning request.
This discussion has been archived. No new comments can be posted.

AI-Generated Voice Firm Clamps Down After 4chan Makes Celebrity Voices For Abuse More

AI-Generated Voice Firm Clamps Down After 4chan Makes Celebrity Voices For Abuse

Comments Filter:

  • Who is yooh daddy, and wut does he do? (Score:5, Funny)

    by locater16 ( 2326718 ) on Monday January 30, 2023 @11:39PM (#63252741)
    Stop it! I'm a cop you idiot! It's not a tumah!

    • Re: (Score:2)

      by Gavino ( 560149 )
      Ah the good old days of a few tech support geeks sitting around a computer phoning up the "new guy" on loudspeaker as we play them the Arnie soundboard. They'd be holding up a Nokia 5110 to their ear thinking "wtf is this?!" before finally tweaking it's a prank due to the eventual sound of background laughter. Good times.

  • > After 4chan Makes Celebrity Voices For Abuse

    That's a 3 way redundancy I think

  • Parody (Score:5, Insightful)

    by TwistedGreen ( 80055 ) on Tuesday January 31, 2023 @12:03AM (#63252785)
    So what, these celebrities are fair game for parody. ElevenLabs is just milking this for the free publicity...

    • Parody can be problematic when it's not obvious (Score:5, Interesting)

      by rsilvergun ( 571051 ) on Tuesday January 31, 2023 @12:44AM (#63252833)
      And sometimes even what it is. M&Ms did a joke about retiring the M&M's spokes candies because they were two divisive and A&W ran with that that made a joke about their bear mascot being forced to wear pants by the wolk mob.

      Fox Business picked up the bear mascot story and ran it. I don't mean prime Time Tucker Carlson frothing Insanity I mean Fox Business.

      Poe's law is now so powerful it sucked everyone on the right wing in and they can't tell fact from fiction anymore. Satire is dead when the other side is that absurd.

      • Re: (Score:3, Insightful)

        by Powercntrl ( 458442 )

        Allow me to put forth a modest proposal that people failing to recognize satire isn't exactly a new thing.

      • Re:Parody can be problematic when it's not obvious (Score:4, Funny)

        by Arethan ( 223197 ) on Tuesday January 31, 2023 @01:05AM (#63252891) Journal

        Dude, just saying, the left wing is just as susceptible to this nonsense.
        I personally see both sides frothing at the mouth of the exact same nonsense.
        To me, all this proves is that political gas lighting is real, and that it works on the uneducated, the stressed, and the the bandwidth challenged.
        So yea. Vote for Joe Trump 2025, or whatever -- same shit, imho.

        (and yes, this is mildly farcical post meant to make you all step aside and think a bit, so "/s" ya dirty animals)

        • Re: (Score:2)

          by Arethan ( 223197 )

          to be very clear "bandwidth challenged" means you're just so fucking busy with your own damn life that political crap doesn't get adequate time slice for you to do your own actual research, so you're effectively forced to take the word of the talking-head summarizers

          • I dunno, there does also seem to be some correlation between slow internet access and certain political leanings. My folks retired from Florida (yeah, how's that for ass backwards?) for a part of NC which up until very recently only had horribly slow 1Mbps DSL. They do now have Elon Musk's thing, which works well enough when it's not intermittently cutting out. My father has been a Republican for as long as I'm aware, and even he complains about the area he lives in being too far-right for his tastes.

            Mea

            • Re: (Score:3)

              by Arethan ( 223197 )

              Lol well played, but as I clarified, "bandwidth challenged" was a reference to personal free time.

              I, for one, have noticed I'm increasingly infringed upon by my employer(s) at every fucking waking moment.
              Now, they pay me well, and it's not awful to feel needed and wanted, but it does mean that I have less quiet time to reflect on literally anything that isn't directly about work.

              For instance, I used to read 3-5 books of my own selection per year, usually while on vacation. I now read maybe 1-2, and it's dif

          • The problem here is that more often than not, the people who claim they do "their own actual research" ARE just taking the word of the talking-head summarizers.

          • Re: (Score:2)

            by Kokuyo ( 549451 )

            Bandwidth challenged.
            Damn, that fits so well. I'm gonna use that.

            My way to protect myself, incidentally is not to just go with whatwver I hear but to just keep away from news and shrug and keep my mouth mostly shut when engaged in political topics by people around me.

            • Re: (Score:2)

              by Arethan ( 223197 )

              It sounds like you, yourself, are bandwidth challenged, but have decided to address this problem by not expressing your feelings (as they are unfinished or lack information). Not to be derogatory, by any means. I too am bandwidth challenged, and I feel it impact my life every day. I do my best to get the best information available, but these days it's honestly very difficult to be prepared for casual political discussions amongst friends, let along for impromptu debates amongst newcomers (some of these folk

              • Re: (Score:2)

                by znrt ( 2424692 )

                It sounds like you, yourself, are bandwidth challenged, but have decided to address this problem by not expressing your feelings (as they are unfinished or lack information). Not to be derogatory, by any means. I too am bandwidth challenged, and I feel it impact my life every day. I do my best to get the best information available, but these days it's honestly very difficult to be prepared for casual political discussions amongst friends, let along for impromptu debates amongst newcomers

                dude, politics is actually pretty simple: the elite fucks over the working class, and will continue to do so until the working class is no longer needed. that's all the bandwidth you really need, any further detail on that is just circumstantial at best, mere gossip.

        • we found that out when fake news started. There was an interview with one of they guys who sells fake news. He wasn't a Republican, he was a businessman.

          What he found was that if he did right wing fake news it went viral and spread because people shared it. He made his money (millions) off the ad revenue from posting obvious and insane lies.

          He was asked why he didn't do the same with with the left wing. Again, he's a businessman, no ideology whatsoever and he's not doing anything illegal so he just

      • Re:Parody can be problematic when it's not obvious (Score:4, Informative)

        by Entrope ( 68843 ) on Tuesday January 31, 2023 @07:50AM (#63253303) Homepage

        M&Ms did a joke about retiring the M&M's spokes candies because they were two divisive

        (a) It's not a joke [nbcnews.com], and (b) the word is "too", not "two".

        It would be ironic that you post illiterate lies and claim that the other side are the ones who can't tell reality from fiction -- if it were not so deeply ingrained in your shtick.

        • and don't proof read /. comments. Sue me. But how is it your logical fallacy (Ad Hominem? I don't know my fallacies all that well and I can't be bothered learning their names) got you a +4 comment.

          Kinda proves my point about how right wing narratives spread. You guy's'll mod up anything.

          • Welcome to the new Slashdot. I've seen my comment scores fall hard in the last week or so in particular, so clearly something has changed with regard to who's getting mod points.

            However, the comment was not Ad Hominem because it didn't say your comment couldn't be trusted because of something about you, which is what's required. Insults are not Ad Hominem unless an argument is constructed on them.

          • Re: (Score:2)

            by Entrope ( 68843 )

            How did your lies get you a +5 comment except that leftists will mod up even wrong comments because they criticize conservatives?

            It would be hilarious that you admitted to not knowing what logical fallacies were which when you falsely accused me of using one (pointing out your shitty behavior isn't a fallacy), but again -- it's something you do all the time, so it's just sad.

            Likewise for how you made a lame excuse for using the wrong word, without addressing the fact that you were simply wrong on the facts.

        • M&Ms did a joke about retiring the M&M's spokes candies because they were two divisive

          (a) It's not a joke [nbcnews.com], and (b) the word is "too", not "two".

          It would be ironic that you post illiterate lies and claim that the other side are the ones who can't tell reality from fiction -- if it were not so deeply ingrained in your shtick.

          Ahem. The reality is they are now called Ma&Ya's.
          https://www.mms.com/en-us/ma-a... [mms.com]

          Guys please try to keep up with this definitely not a joke, very serious rebranding, this is for real, and we don't want to hurt Entrope's feelings. Definitely do NOT tell her what irony is.

      • Re: (Score:2)

        by HiThere ( 15173 )

        BOTH sides have elements that are that absurd. And both sides have court cases that uphold the absurd points of view. Consider the burglar who fell through a skylight he was breaking in through and sued the owner because he broke his leg...and won.

        Poe's law is so powerful because in a large enough sample you can find people who will uphold ANY idiocy. They may be a small minority, but public debating rules ensure that the other side will give them wide publicity, and won't mention the more moderate point

        • Re: (Score:1)

          by Anonymous Coward

          Consider the burglar who fell through a skylight he was breaking in through and sued the owner because he broke his leg...and won.

          Wow. Almost nothing in your version of that story is true. He was trespassing and not a burglar. He wasn't trying to break into the skylight. It was painted over and he fell through it, because he couldn't see it. He was accused of trying to steal a $35 flood light from the roof. He claims he was just moving it (snort). He sued the school district, because he fell 27 feet and was permanently disabled. He didn't win that law suit. It was settled out of court. So basically you got the gender and skyl

          • Re: (Score:2)

            by HiThere ( 15173 )

            I think you may be talking about a different case. IIRC the case I'm thinking of involved a SF school district, and he fell through the skylight. I admit, though, that I don't know he was convicted of burglary, just that that's how the newspaper reported it. And if I recall the story correctly he was in his 20's, or possibly older. (I.e., he was not charged as a juvenile.)

            That said, I never saw any follow-up. Perhaps the newspaper really distorted it sufficiently that it's the same as the case you're t

      • Says the little faggot who wants to bleed in a silicone cup. GTFOH.

    • ElevenLabs is just milking this for the free publicity...

      Don't confuse publicity with outrage avoidance. It's 2022. Every idiot who doesn't like something shouts it to their million twitter followers and Karen at home starts getting angry and vowing to sue when she sees it on her social media feed.

      Some companies just don't want to deal with that shit, understandably so.

    • Re: (Score:2)

      by AmiMoJo ( 196126 )

      Could be defamation. Lying about someone being, say, antisemitic, could certainly harm their career or cause them to spend money publicly refuting the claim. Backing that lie up with a deepfake voice clip would just make it harder to undo the damage.

      There's also a potential copyright/likeness rights issue with celebrity voices.

    • Re: (Score:2)

      by HiThere ( 15173 )

      Parody is fine, but one needs to be able to tell that it *is* parody. This doesn't seem to really count. This is closer to slander or fraudulent misrepresentation.

    • That's not Parody.

  • They gave up - there's nothing people can't imagine him saying.

    • Re: (Score:1)

      by Anonymous Coward

      They gave up - there's nothing people can't imagine him saying.

      How about "maybe I'm not that smart after all".

    • Re:They aren't using Elon Musk's voice (Score:4, Funny)

      by Powercntrl ( 458442 ) on Tuesday January 31, 2023 @01:06AM (#63252893) Homepage

      "I'm redesigning the cybertruck to not look like a DeLorean rendered on a Nintendo 64."

      • "I'm redesigning the cybertruck to not look like a DeLorean rendered on a Nintendo 64."

        I always thought it looked like its inspiration came from BattleZone [saymedia-content.com].

  • NO. FUN. ALLOWED (Score:5, Insightful)

    by yuriklastalov ( 4536597 ) on Tuesday January 31, 2023 @12:25AM (#63252809)

    Those awful, awful trolls made a celebrity voice say the n-word and laughed about it on their shitty board. This is a TRAVESTY, nay a TRAGEDY! It's basically like the holocaust all over again.

    • Re:NO. FUN. ALLOWED (Score:5, Interesting)

      by AmiMoJo ( 196126 ) on Tuesday January 31, 2023 @06:55AM (#63253225) Homepage Journal

      There really needs to be some legal clarification here. For example, deepfake porn. Is that allowed without the consent of the subject?

      It's currently a legal grey area. Involuntary pornography, like leaking people's private photos or taking pictures of them with hidden cameras, is illegal in many places. On the other hand, magazines regularly publish photos of celebrities taken with zoom lenses pointed at private property, without consent.

      AI artists can generate an image from a simple prompt like "draw a photorealistic nude of $celebrity". In itself not illegal, but could be if the image is distributed. That said, it's likely that commercial operators would want to block their service from being used to generate porn anyway.

      • +1 informative

      • There really needs to be some legal clarification here. For example, deepfake porn. Is that allowed without the consent of the subject?

        In the USA it's almost certainly illegal at least to distribute and possibly even to create, because you have the right to control use of your likeness. In the UK it isn't, because you don't, unless someone is faking an endorsement or doing provable harm.

        That said, it's likely that commercial operators would want to block their service from being used to generate porn anyway.

        That ship has sailed, since Stable Diffusion exists and you can create your own embeddings, trainings, etc.

  • At some point. (Score:5, Insightful)

    by Fly Swatter ( 30498 ) on Tuesday January 31, 2023 @12:53AM (#63252861) Homepage
    Video and audio will not be something that can be trusted. At this rate it won't be too long before they won't be usable as evidence in the court of law once the technology is refined.

    Everything will have to be in person to be trusted that it is actually being said by that person.

    • Video and audio will not be something that can be trusted. At this rate it won't be too long before they won't be usable as evidence in the court of law once the technology is refined.

      Everything will have to be in person to be trusted that it is actually being said by that person.

      or we have camera and other video equipment cryptographicly sign their output so as to have chain of evidence to show no tampering.

      • New way of DRM? defectivebydesign.org

        • New way of DRM?

          Nobody is proposing that the video be encrypted, or nonstandard, only signed. That's not DRM.

          • Understood (but it can be faked too)

            • Sure, by compromising the chip that performs the function you could conceivably extract the keys and then sign whatever you wanted. All it really does is raise the bar to the point where only very wealthy actors can reasonably fake a video. That's not nothing, though.

      • Wouldn't help. People will just do what is needed to remove the signature entirely from the source files. If you can play it at all, it can be captured. You can't connect all parts of the chain without deeply invasive tools and the banishment of all the legacy stuff.

        • Wouldn't help. People will just do what is needed to remove the signature entirely from the source files.

          That would be self-defeating, because without the signature the video stream would [obviously] be treated as untrusted.

    • Video and audio will not be something that can be trusted.

      It hasn't been trustable since the advent of splicing magnetic tape. Like most things, computers are just making it a whole lot easier and more accessible to manipulate.

    • If that makes celebs stop to think before they run their mouth instead of spouting all sorts of nonsense for shock value and generating clicks, I can't really say I think it's a bad thing.

      • If that makes celebs stop to think before they run their mouth instead of spouting all sorts of nonsense for shock value and generating clicks, I can't really say I think it's a bad thing.

        It won't, for two reasons. One, even without deepfake technology you can fake audio using things people actually said. Two, deepfake audio lets you create "recordings" of people saying things they never actually said.

      • Your "ends justify the means" approach breaks down once you consider that it doesn't end at the borders of celebrity. If a shady businessman wants out of a contract, he might show the other side in video or audio breaking an NDA, or some other term. If your ex is angry at you, deep faking you being openly and toxically racist might be on the agenda. Are you willing to take those terms just so celebrities behave more in line with you you think they ought to?

        • No, I'm willing to take those terms because because this is inevitable, because information wants to be free, because I don't think this should be legislated, because I don't think laws against it can be enforced, and because FREEDOM, bitch.

          • "Information wants to be free" is horseshit. Some people want information to be free. Shovel your freedom - we live in a society, and some freedoms have got to go.

            Otherwise I agree with you. It is inevitable.

          • information wants to be free

            Please don't anthropomorphise information, it really hates that.

    • That point was passed long ago.

    • Re: (Score:2)

      by eth1 ( 94901 )

      Video and audio will not be something that can be trusted. At this rate it won't be too long before they won't be usable as evidence in the court of law once the technology is refined.

      Everything will have to be in person to be trusted that it is actually being said by that person.

      Came to say basically the same thing. It's already to the point that if I'm ever on a jury, I'd be skeptical of photos or recordings that don't have a chain of custody attached. (and even that doesn't really prove it was created from reality instead of by AI)

  • as opposed to their own podcasts (Score:3, Funny)

    by lister king of smeg ( 2481612 ) on Tuesday January 31, 2023 @12:55AM (#63252865)

    generate voices that sound like Joe Rogan, Ben Sharpio,... to spew racist and other sorts of material.

    So what you made Joe Rogan and Ben Sharpio say racist thing, as opposed to the racist things they say on their own platforms? Oh No! Now there are recordings were they say offensive things without ad revenue how horrible.

  • If they maintain copies of, for example, James Earl Jones' voice recordings, to prevent 4chan from uploading clips of JEJs voice as their own, doesn't this mean that they need to have a license from JEJs estate? And the same for every other celebrity and voice actor? I can't imagine how much that would cost.

    • Re: (Score:1)

      by vivian ( 156520 )

      doesn't this mean that they need to have a license from JEJs estate?

      You make it sound like he's dead.
      Last time I checked, he's alive and kicking - so they would need to presumably get permission from the man himself, but it's also uncertain just how similar it has to be. You could make a voice that sounds kind of like him but subtlety different enough that it's not exactly the same in some way - where the dividing line is would most likely have to be decided in court.

  • well... (Score:3)

    by boundary ( 1226600 ) on Tuesday January 31, 2023 @01:04AM (#63252883)

    ...what did they expect would happen when they made it available?

  • Can it not recognize them (Score:3)

    by bugs2squash ( 1132591 ) on Tuesday January 31, 2023 @01:20AM (#63252917)
    I would have thought there was enough clips of celebs speaking on the internet that the AI companies could compile a database of them and reject requests to generate speech from clips that sound too much like a celeb.

    • I would have thought there was enough clips of celebs speaking on the internet that the AI companies could compile a database of them and reject requests to generate speech from clips that sound too much like a celeb.

      What makes you think AI companies are the gatekeepers here? Anyone(*) could write an AI to do the same. Companies are not the gatekeepers of technology, they are the ones who make it available to the uneducated masses (for a fee).

      Having companies censor themselves will in no way solve the issue and in fact, might make it worse because then some people would be believe that the recording they are hearing is real because X company pledged to not allow such abuses.

      (*) Anyone with some amount of experience. Thi

  • Ben Shapiro? (Score:3, Insightful)

    by algaeman ( 600564 ) on Tuesday January 31, 2023 @01:32AM (#63252935)
    Can't you just give him a script with racist rants to read for a $25 cameo?

  • Would be to make an AI clone of Barbra Streisand's voice. That'll keep these things from spreading all over the Internet.

  • Did they name the company after this skit [youtube.com]?

  • "My voice is my password." (Score:5, Interesting)

    by VeryFluffyBunny ( 5037285 ) on Tuesday January 31, 2023 @05:18AM (#63253099)
    We better disable phone-banking now or at least find a more secure way to identify ourselves over the phone.

  • ...we can soon expect lawsuits about the 'look and feel' of voices.

    People whose voice resembles a celebrity voice have to get their chords surgically removed.

  • Eminem's best freestyle IMHO:

    https://www.youtube.com/watch?... [youtube.com]

    Bonus points for the video creator using typically messed up AI generated images in the begining.

  • ElevenLabs might be able to block 4chan users from using their particular voice generator but the cat's already out of the bag. There are open source voice generators like tortoise-tts that create output that's just as good as what ElevenLabs puts out and anybody with a reasonably decent GPU or even just a google colab account can feed about a minute's worth of a celebrity's voice into it and generate a surprisingly realistic copy. I used just five 10-second clips of Joe Rogan and was able to produce this o [youtu.be]

Slashdot Top Deals

All seems condemned in the long run to approximate a state akin to Gaussian noise. -- James Martin

Close