AI Firms Say They Can't Respect Copyright. But A Nonprofit's Researchers Just Built a Copyright-Respecting Dataset (msn.com) 63
Is copyrighted material a requirement for training AI? asks the Washington Post. That's what top AI companies are arguing, and "Few AI developers have tried the more ethical route — until now.
"A group of more than two dozen AI researchers have found that they could build a massive eight-terabyte dataset using only text that was openly licensed or in public domain. They tested the dataset quality by using it to train a 7 billion parameter language model, which performed about as well as comparable industry efforts, such as Llama 2-7B, which Meta released in 2023." A paper published Thursday detailing their effort also reveals that the process was painstaking, arduous and impossible to fully automate. The group built an AI model that is significantly smaller than the latest offered by OpenAI's ChatGPT or Google's Gemini, but their findings appear to represent the biggest, most transparent and rigorous effort yet to demonstrate a different way of building popular AI tools....
As it turns out, the task involves a lot of humans. That's because of the technical challenges of data not being formatted in a way that's machine readable, as well as the legal challenges of figuring out what license applies to which website, a daunting prospect when the industry is rife with improperly licensed data. "This isn't a thing where you can just scale up the resources that you have available" like access to more computer chips and a fancy web scraper, said Stella Biderman [executive director of the nonprofit research institute Eleuther AI]. "We use automated tools, but all of our stuff was manually annotated at the end of the day and checked by people. And that's just really hard."
Still, the group managed to unearth new datasets that can be used ethically. Those include a set of 130,000 English language books in the Library of Congress, which is nearly double the size of the popular-books dataset Project Gutenberg. The group's initiative also builds on recent efforts to develop more ethical, but still useful, datasets, such as FineWeb from Hugging Face, the open-source repository for machine learning... Still, Biderman remained skeptical that this approach could find enough content online to match the size of today's state-of-the-art models... Biderman said she didn't expect companies such as OpenAI and Anthropic to start adopting the same laborious process, but she hoped it would encourage them to at least rewind back to 2021 or 2022, when AI companies still shared a few sentences of information about what their models were trained on.
"Even partial transparency has a huge amount of social value and a moderate amount of scientific value," she said.
"A group of more than two dozen AI researchers have found that they could build a massive eight-terabyte dataset using only text that was openly licensed or in public domain. They tested the dataset quality by using it to train a 7 billion parameter language model, which performed about as well as comparable industry efforts, such as Llama 2-7B, which Meta released in 2023." A paper published Thursday detailing their effort also reveals that the process was painstaking, arduous and impossible to fully automate. The group built an AI model that is significantly smaller than the latest offered by OpenAI's ChatGPT or Google's Gemini, but their findings appear to represent the biggest, most transparent and rigorous effort yet to demonstrate a different way of building popular AI tools....
As it turns out, the task involves a lot of humans. That's because of the technical challenges of data not being formatted in a way that's machine readable, as well as the legal challenges of figuring out what license applies to which website, a daunting prospect when the industry is rife with improperly licensed data. "This isn't a thing where you can just scale up the resources that you have available" like access to more computer chips and a fancy web scraper, said Stella Biderman [executive director of the nonprofit research institute Eleuther AI]. "We use automated tools, but all of our stuff was manually annotated at the end of the day and checked by people. And that's just really hard."
Still, the group managed to unearth new datasets that can be used ethically. Those include a set of 130,000 English language books in the Library of Congress, which is nearly double the size of the popular-books dataset Project Gutenberg. The group's initiative also builds on recent efforts to develop more ethical, but still useful, datasets, such as FineWeb from Hugging Face, the open-source repository for machine learning... Still, Biderman remained skeptical that this approach could find enough content online to match the size of today's state-of-the-art models... Biderman said she didn't expect companies such as OpenAI and Anthropic to start adopting the same laborious process, but she hoped it would encourage them to at least rewind back to 2021 or 2022, when AI companies still shared a few sentences of information about what their models were trained on.
"Even partial transparency has a huge amount of social value and a moderate amount of scientific value," she said.
"Respecting copyright" != "Ethically" (Score:5, Insightful)
Re: (Score:1)
I will respect copyright when it respects me.
Word.
Re: (Score:1)
Re: (Score:3, Informative)
Let's just run with the "AI training" misconception.
There's a document. It was created by an author. The author has the exclusive right to copy the document in its entirety onto his own website (copy=1,violation=0). Your browser knocks on the door. It asks for the document. The author's website copies the document over the network onto your browser's process memory (copy=2,violation=0). That's fine, because the author's HTTP server initiated and the author intended to authorize the copy.
Now you copy th
Re: (Score:2)
Let's just run with the "AI training" misconception.
There's a document. It was created by an author. The author has the exclusive right to copy the document in its entirety onto his own website (copy=1,violation=0). Your browser knocks on the door. It asks for the document. The author's website copies the document over the network onto your browser's process memory (copy=2,violation=0). That's fine, because the author's HTTP server initiated and the author intended to authorize the copy.
After that thing get murky, several copies are made but to what purpose ?
Web browser copies it into its cache on disk (used,eg, if you do a page refresh avoid downloading again over the Internet). Is this a legal copy ? This is a standard browser thing. Other similar copies might be made, eq by squid (a caching and forwarding HTTP web proxy). I will ignore these copies as no one seems to be upset about them. (Actually that is not entirely true.)
You read the document, another copy is made that resides in you
Re: (Score:2)
There are major problems with copyright. Like the absurdly long terms that mean a century after a work is written, the author's descendants may still be collecting royalties on it. Or DMCA style laws that abuse copyright for unrelated purposes, like saying you can't repair your own possessions because it would violate a copyright. It's absurd and it needs to be fixed.
But the AI companies don't care about that. They aren't on your side. They aren't fighting those things. The only thing they care about
Re: (Score:2)
Um... (Score:5, Insightful)
AI Firms Say They Can't Respect Copyright
Pretty sure it's not really up to them, legally.
A group of more than two dozen AI researchers have found that they could build a massive eight-terabyte dataset using only text that was openly licensed or in public domain.
So it's really more like "won't" than "can't" ...
Re:Um... (Score:5, Insightful)
AI Firms Say They Can't Respect Copyright
Pretty sure it's not really up to them, legally.
And their response: "Who cares about legality - that's for the courts to settle, after you spend money you don't have suing us."
Re: (Score:3, Insightful)
In a fair and just world, you would be right. In this world, however, the super-rich are beholden to a different set of rules than the rest of us, and something like AI is just too interesting to allow pesky laws to get in the way (especially laws that are, by and large, only protecting copyrights held by the not-so-rich).
Re: (Score:2)
People like to pretend the "rules" are clear. Just because AI billionaires are criminals does not mean the commit the copyright violations alleged.
Re: (Score:2)
"Pretty sure it's not really up to them, legally."
Nor do they say it.
"So it's really more like "won't" than "can't" ..."
They don't say that either.
There is no reason to tell lies, Ai companies are scumbags, doesn't mean we have to be.
They've got hundreds of billions of dollars (Score:2)
Re: (Score:2)
Re: (Score:2)
If you can't do what you want legally, then you don't do it.
Just joking, lol. Obviously, if you can't do it legally, you just have to do it illegally. There's no other choice. It's the capitalist way!
Correction (Score:5, Insightful)
AI firms won't pay to respect copyright
On they one hand, I can only hope this leads to revisiting the insanity of copyright law.
On the other, fuck them for double dealing with regards to what ownership actually means ("I'm alright, Jack.").
Training does Respect Copyright (Score:2)
AI firms won't pay to respect copyright
They do not need to pay. Copyright, as the name says, is the right to copy and distributute something. So long as you purchase a legal copy you are allowed to use it as you wish provided you do not distribute copies.
If I buy a book the copyright holder cannot tell me that I'm only allowed to read 5 pages a day, or that I can't use it to balance a table, prop open a door or even burn it. Similarly, they can't tell me that I'm not allowed to use it to train a machine learning algorithm provided that the a
Re: (Score:2)
Uh-huh.
Take something like music. There are specific licenses for specific uses. We've already have a legal framework with regards to sampling. Imagine my dismay how none of these people spoke up then, but now cost of sampling and the morass of licensing is an issue.
But tell me, is any of the software copyrighted?
Oh...
Re: (Score:2)
There are specific licenses for specific uses.
Yes but only around two things: public performance and copying/distrubution and arguably public performance is a form of distribution.
Re: (Score:2)
Yeah but Meta torrented the entirety of Z-Library, reportedly.
They won't even pay for one copy, even setting aside the issue of trained networks being a derivative work.
Re: (Score:1)
Re: (Score:1)
They're lying. (Score:5, Insightful)
They say that because they're fucking liars.
They only want to serve their real customers, which is their (potentially future) investors/shareholders - they don't give a shit about anyone else, including those that ever produced the content their models have been trained on (the models which wouldn't have any use without that content exiting to begin with).
Re: (Score:3)
they don't give a shit about anyone else,
So they're like everyone who makes excuses for why they steal music, videos, and software?
Re: (Score:2)
Except in the AI case, it's not clear it's not fair use. People now accusing AI training of criminality are worse.
Re: (Score:2)
"They say that because they're fucking liars."
They don't say it, that's a just a troll that you are excited to believe. But, yes they are fucking liars.
Copyrights redefined (Score:2)
Re: (Score:1)
Very simple, really.
Medicine is inherently useful.
The mouse is only useful for making money.
Re: Copyrights redefined (Score:1)
Re: (Score:2)
No, that's not what happened because once something enters the Public Domain, nobody owns it any more, and anybody who wants is free to use it however they want. That's why Disney sticks to stories in the Public Domain so that they don't have to pay royalties.
An AI without the training data .. (Score:3)
Re: (Score:2)
"Marketing AI without compensating data creators is, in essence, intellectual property theft."
It is not, marketing is marketing.
And it remains to be seen if training is IP theft, so far the focus has been on copying and storing data, not training with it. They are doing that because it's not clear that training isn't fair use.
Re: (Score:2)
Obviously. And a criminal business model should not only get you shut down. It should get you sent to prison.
Maybe an adversarial approach (Score:2)
Re: (Score:2)
That is true, but a "copyright classifier" contains what? Given the term used, it appears you are suggesting another LLM with imperfect memorization. How do you think that solves any problem?
The hard part is in the doing, what are suggesting is obvious.
Re: (Score:2)
... "you" ...
Also, it should be mentioned that humans are notorious for inadvertently making these kinds of copyright violations themselves, not necessarily with text because their recall isn't that good, but with music it happens frequently. If you think a detector applied to output is going to solve problems, intuition says you will be disappointed.
But you are right, it's not clear there is copyright violation during training but there certainly is during inferencing. Problem is, it is not AT ALL clear
Re: (Score:3)
Re: Maybe an adversarial approach (Score:2)
Re: (Score:2)
In school, kids do not copy the works (mostly... ;-) they just read them (or not... ;-)
You can buy a book. You can do whatever you like with that book physically. It's yours. You cannot copy that book. For example, you can't photocopy the pages and collect those photocopies into a new book. That would be violating the copyright. You also can't memorize the book, then recite it verbatim into a recording device, making your own "audio book" or write out the book in longhand, etc.
You can certainly read the
Re: Maybe an adversarial approach (Score:2)
Re: Maybe an adversarial approach (Score:2)
Then don't exist. (Score:3)
If your business is incapable of existing without breaking the law then the obvious answer is that your business should not exist. How is this even a question? In the past EVERY company that flaunted copyright has been bankrupted but now with companies doing it en masse it's suddenly OK?
I'm calling bullshit on all of these companies. If you want to reform copyright then do it like all the other businesses have, buy a congressman because you aren't special.
Pointless debate (Score:2)
The laws will be rewritten to suit the needs of AI because they suit the needs of your ruling class.
And human beings won't do away with their ruling class because they like to pretend that all the chaos and misery in the world is under control.
Re: (Score:2)
the ability for AI to replace white collar workers is worth trillions.
Ah, yes, does not look good on that front. More like single digit percentages in efficiency gains. But more stress on the workers, so these may be negative gains in effect. Overall, AI is, again, an abject failure that delivers a miniscule amount of what its proponents claim.
Protection... (Score:2)
If you cant respect the law (Score:3)
Then you cannot operate, full stop. Shut them all down until they can obey the laws.
Re: (Score:2)
That is far too friendly. Shut them down, impund their fortunes and imprison the perpetrators.
AI Firms Say They Can't Respect Copyright (Score:4, Insightful)
AI Firms Say They Can't Respect Copyright
Then your business model is illegal. Shut it down.
Re: (Score:2)
Indeed. Criminals usually claim they are not criminals and they had no choice and it really is somebody else's fault.
Copyrights != Fairness (Score:3)
There seems to be widespread misunderstanding between notions of copyright and fairness. The two are in no way synonymous.
Imagine you spend a huge amount of time and money surfacing new knowledge nobody knew before. You spill the beans in a book and sell it. Someone else comes along, reads your book and blabs what you learned to the world for free or in a much cheaper book of their own.
Imagine you painstakingly compile a phone book of numbers that would be useful to a certain niche audience. Someone takes the book, OCRs all the numbers into a computer database and gives it all away for free.
Neither of these are copyright issues, you can have the opinion they are unfair or should not be allowed yet nonetheless not a copyright concern.
Copyright holders should be careful what they wish for because an AI trained on a known dataset that can be shown to be ignorant of a copyright holders work is an affirmative defense against claims of derivative works when someone publishes the output of the AI. Before this such a defense is absurdly difficult because the author would have had to prove a negative.
well too bad... (Score:3)
I expect at least some ads (Score:2)
I expect at least some ads: "You wouldn't train your AI with a CAR".
Seems future AI is going to be ... (Score:1)
... a respectless asshole. Terminators incoming! Hope the OSS approach gets wind. We need AI with a conscience.
How about AI vs Disney ? (Score:2)
Disney is ferocious in its protection of its copyright. What happens if you ask an AI about Mickey Mouse, what does it say ? How did that AI learn about MM but by reading copyrighted material or viewing copyrighted movies ?
Has Disney said anything about AI companies using its copyrighted material ?
Not Just About Copyright (Score:2)
Even if you have little respect for the notion of copyright, at the very least you must understand the bad precedents being set up when companies can just swipe any and all data available on the web for commercial purposes. This is not going to stop at struggling artists and hobbyist programmers; the plan is to effectively kill the efficacy of opt-in and out efforts for data privacy. They want to take every piece of data online, either publicly or privately for data training with no respect for compensation
BIG lobbyists stealing our rights (Score:1)
These copyright maximalists start by ASSUMING that Copyright MUST exist and ignoring the social contract where we allow content creators some license in return for their contribution to society. Today's "amazing" copyrights that last nearly a CENTURY after the creator has died do nothing to provide an incentive to creators... only something to be monetized by these TERRORIST LEECHES who produced NOTHING but got government-granted rights to the NOTHING they produced so they can prevent true innovators (oh a