Cannibal AIs Could Risk Digital 'Mad Cow Disease' Without Fresh Data (sciencealert.com) 74
A new article in ScienceAlert describes new research into the dangers of "heavily processed sources of digital nourishment" for generative AI:
A new study by researchers from Rice University and Stanford University in the US offers evidence that when AI engines are trained on synthetic, machine-made input rather than text and images made by actual people, the quality of their output starts to suffer.
The researchers are calling this effect Model Autophagy Disorder (MAD). The AI effectively consumes itself, which means there are parallels for mad cow disease — a neurological disorder in cows that are fed the infected remains of other cattle. Without fresh, real-world data, content produced by AI declines in its level of quality, in its level of diversity, or both, the study shows. It's a warning about a future of AI slop from these models.
"Our theoretical and empirical analyses have enabled us to extrapolate what might happen as generative models become ubiquitous and train future models in self-consuming loops," says computer engineer Richard Baraniuk, from Rice University. "Some ramifications are clear: without enough fresh real data, future generative models are doomed to MADness."
The article notes that "faces began to look more and more like each other when fresh, human-generated training data wasn't involved. In tests using handwritten numbers, the numbers gradually became indecipherable.
"Where real data was used but in a fixed way without new data being added, the quality of the output was still degraded, merely taking a little longer to break down. It appears that freshness is crucial."
Thanks to long-time Slashdot reader schwit1 for sharing the news.
The researchers are calling this effect Model Autophagy Disorder (MAD). The AI effectively consumes itself, which means there are parallels for mad cow disease — a neurological disorder in cows that are fed the infected remains of other cattle. Without fresh, real-world data, content produced by AI declines in its level of quality, in its level of diversity, or both, the study shows. It's a warning about a future of AI slop from these models.
"Our theoretical and empirical analyses have enabled us to extrapolate what might happen as generative models become ubiquitous and train future models in self-consuming loops," says computer engineer Richard Baraniuk, from Rice University. "Some ramifications are clear: without enough fresh real data, future generative models are doomed to MADness."
The article notes that "faces began to look more and more like each other when fresh, human-generated training data wasn't involved. In tests using handwritten numbers, the numbers gradually became indecipherable.
"Where real data was used but in a fixed way without new data being added, the quality of the output was still degraded, merely taking a little longer to break down. It appears that freshness is crucial."
Thanks to long-time Slashdot reader schwit1 for sharing the news.
One thing to say about this AI fad (Score:3)
It is interesting.
Re:One thing to say about this AI fad (Score:5, Interesting)
Some aspects of it are. For example, this data problem was pretty clear from the start, yet people chose to ignore it and that it is basically unsolvable.
Re: (Score:2)
Ohhhh, the humans won't likely run out of data.
That's what they do.
Re:One thing to say about this AI fad (Score:5, Interesting)
Ohhhh, the humans won't likely run out of data.
That's what they do.
AI models have already run out of data. There is no secret about this - it is widely discussed. The creation of the public Internet about 30 years ago allowed a substantial fraction of the language output of the entire human race to become available for crawling, not just the new material generated since its creation, but also texts representing a substantial fraction of all published communication across human history.
And then in the span of only a year or two these LLMs have snarfed up all that text at once to create vast parrot bots that still fail when you wander into low probability of discussion topics -- the long tail of human communication.
This was a one-time only trick that cannot be repeated. Additional data to train on will only trickle in by comparison and is now subject to contamination for GenAI garbage.
Re: (Score:2)
Perhaps then, the next approach in the quest for AI is to prioritize quality over quantity?
--
Third class pterodactyl in a society already extinct
Re:One thing to say about this AI fad (Score:4, Insightful)
Perhaps then, the next approach in the quest for AI is to prioritize quality over quantity?
The whole business plan for AI is to sell their services as a cost-savings measure. Once you throw human curation of data into the mix - which is basically what would be required - any savings go out the window.
And that's even before we start discussing whether the AI Bullshitting (what the AI folks have rebranded as "hallucinations") problem is even solvable.
Re: (Score:2)
Well, the "hallucinations" are clearly solvable, but only at the price of decreasing the originality. The clear reason is that LLMs aren't grounded in "reality" so they can't sanity check against it. Restrict them to a well-characterized domain and they become a LOT more reliable. They're still guessing, e.g., just how that molecule will fold, but there can be feedback that acts as a corrective, and even without that they make guesses about as good as a human expert in the field (who also doesn't get tim
Re:One thing to say about this AI fad (Score:4, Insightful)
Well, the "hallucinations" are clearly solvable, but only at the price of decreasing the originality.
Hallucinations cannot be fixed with LLMs. You cannot "ground" LLMs in reality, as they have no reasoning ability. Seriously get a clue about the state-of-the-art before coming in with amateur ideas.
Re: (Score:2)
Not trolling here. I am fairly certain there is no agreement on what "quality" in data is.
I think the AI people have stumbled upon philosophy.
I have a friend who fancies himself a philospher.. studied it in university... and he'll talk for as long as you'll listen... and never really get to a point.
Let the arguing begin.
Re: (Score:3)
> I have a friend who fancies himself a philosopher.. studied it in university... and he'll talk for as long as you'll listen... and never really get to a point.
Modern philosopher = Intellectual Masturbation. /s
It is fine to ponder and reflect from time to time but at the end of the day stuff still needs to done.
i.e. Religion is applied internal philosophy, and Politics is applied external philosophy.
Re: (Score:2)
They generally consider themselves to be "the keepers of the flame".. the flame here is usually "language" or "meaning".
because... only philosophers can discern meaning because all arguments come down to the definition of words, and axioms, that which is self evident and cannot be questioned.
These modern day philosophers usually live in a world of self importance because they have "mastered" logic (with words) and think tha
Re: (Score:2)
Forget it. Training an LLM with quality data would make it a LM. I think you have no idea abut the amount of data involved and how much pre-filtering is already being done to reduce the complete garbage in the input data.
Re: (Score:2)
Re: (Score:2)
I agree on that. And I think the whole LLM idea is deeply flawed for general applications. Specialized LLMs may not work well in the end, but general LLMs never will.
Re: (Score:2)
Yes.
Now create an AI that can detect "quality".
Sorry, but I'm not holding my breath on that one!
Re: (Score:3)
To boot, the AI models have less data to choose from now, especially that art sites have shut their doors both technically and legally to AI feeding site scrapers. The data available now is getting less and less, and of course, one can have a scraper ignore copyrights (then do a system of third parties as plausible deniability), but this only can make things worse, especially when sites start using image poisoning algorithms or actively showing different images to AI bots than they do to people.
With less d
Re: (Score:2)
This was a one-time only trick that cannot be repeated. Additional data to train on will only trickle in by comparison and is now subject to contamination for GenAI garbage.
Yep. And that is where we are now. General LLMs have already peaked. Essentially a straw-fire. Great to get rich quick, but that is mostly it.
Re: (Score:2)
The logic for this is baffling. We have humans that can 'train' on an minute subset of the data LLMs are trained on, something called going to school/college/university, yet somehow the entirety of that dataset is 'too little data' for LLMs ever to reach a human level of intelligence. Sure, there are areas where having had a life of sensory data from the real world matters a lot for understanding, but there are also many areas where it doesn't matter one flying fuck (math, anyone?).
This study is also a terr
Re: (Score:2)
"yet somehow the entirety of that dataset is 'too little data' for LLMs ever to reach a human level of intelligence."
Yes. They aren't intelligent at all, they are LLMs. They probabilistically produce tokens - parts of words - that's not intelligence.
Re: (Score:2)
Model drift isn't an AI problem. If you fit any model to it's own output it will drift.
An age-old favourite way to do that is to do something like linear regression, use it to interpolate missing data, and then calculate test statistics based on the "complete" data set.
Re: One thing to say about this AI fad (Score:2)
Re: (Score:2)
I'm not sure that's simpler, but sure, that's also an example.
We used to make medical imaging atlases by aligning lots of scans, computing the average, then aligning the scans to the average, computing a new average, and repeating. The model would drift off into all sorts of interesting distortions if you didn't constrain it by, for example, requiring that the average transform be zero.
Alpha Go Zero exploits it by having iterations of their models play each other and selecting ones that drift in the directi
Producing quality content for the models here (Score:2)
Seems as though the gotcha interview of a politician, business leader, general, etc.you can always find the ones that get a response to a question which cannot be put back into a 10 second sound bite.
Expect people to, earthworms trap CO2 to regulate climate, interject significant facts in their written works to help expand knowledge.
Apparenty there's a science study about worm (Score:2)
New study suggests earthworms sequester more CO2 than they release
https://phys.org/news/2013-10-... [phys.org]
Re: (Score:2)
For example, this data problem was pretty clear from the start
Indeed. I even described the problem here in an off-hand way before the paper [arxiv.org] that coined the term "model collapse".
I'm willing to bet that quite a few people were annoyed that they didn't think to publish something and coin a new term. (Think of the citations!) It was just such an obvious problem that it wasn't something you thought all that much about.
I never would have thought to publish a paper with my own competing term (model autophagy disorder). Which is exactly what the paper linked in the paper
Re: (Score:2)
The verbiage is anyway.
Re: (Score:2)
If information is being destroyed by a machine feeding on its own output, where is it going?
:-)
Re: (Score:2)
Hey, stop it. You're engaging in Model Autophagy Disorder by reading the output of other humans. Worse, you're producing your own output which will be read by even more humans!
MADness!
Re: (Score:2)
And with that, Zarathud--the large feathery pterodactyl from Alpha Centuri--was enlightened.
:-)
--
(With apologies to Douglas Adams)
Wouldn't coprafagic be a better name than canibal (Score:1)
Re: (Score:2)
>It is interesting.
And at last we'll know how Denny Crane got mad cow disease!
Or is this because the AI digested too many episodes?
Re: (Score:2)
They're inbreeding and producing retarded offspring.
So they're from West Virginia [the-sun.com]?
Dan Simmons' Cruciform (Score:1)
For anyone who has read Dan Simmons' Hyperion, specifically the Priest's tale of encountering the Cruciform, then this is completely predictable
Re: Dan Simmons' Cruciform (Score:2)
Not my first thing to think of in this regard but I approve of it nonetheless.
Re: (Score:2)
Thanks for reminding me to check. I couldn't figure out what you were referring to since I had read a number of his books but somehow missed that one. Should be able to start on it in a couple of days...
On the AI topic, I've been reading lots of nonfiction and feeling more ignorant all the time. This story only caught my eye because of personal links to the schools... But the funniest fictional version of AI that I remember might be When Harlie was One by David Gerrold. (Same author as "The Trouble with T
Re: (Score:2)
Why? While we don't know how human intelligence works, we can say with absolute certainty that it isn't a simple statistical model.
Re: (Score:2)
Not at all.
LLM's aren't intelligent at all and they have no agency.
Besides, if you give humans garbage data they will perform just as poorly as AIs fed garbage data.
Re: Let me know when things get worse (Score:2)
However for reasons to prevent NWS images the newer official AI models aren't trained well on unclothed human bodies and that's one contributing factor when it comes to wonky bodyshapes and fused limbs.
No shit Sherlock :o (Score:2)
This is not Artificial Intelligence but a LLM using the Web as source. As such it cannot make real inference or create new knowledge. “The Shadow that bred them can only mock, it cannot make: not real things of its own”.
--
Government’s secret counter-disinformation policies revealed [archive.is]
Re: (Score:2)
Re: (Score:2)
False.
True.
Oh no, recursive Schrödinger's posting...
Re: (Score:2)
Quick, close your eyes.
Re: (Score:2)
Sorry but LLMs generate tokens based on probability.
That's it.
They don't generalize.
They don't create.
They don't think.
Goverment Oversight and Regulation (Score:2)
bad faith BS` (Score:2)
Doesn't even pass the sniff test.
"...without enough fresh real data, future generative models are doomed to MADness."
What does "fresh" have to do with anything? This is a transparent attempt to demonize those who want to be paid for training data.
You know, if synthetic training data doesn't turn out to be valuable, then it won't be used. Existing, "real data" does not get stale.
Re: (Score:3)
You should see a otolaryngologist, your sense of smell is badly defective. "Fresh" means new data that has not been ingested by the model yet, and since the early iterations of GenAI have already hoovered up all the data they could access through the Internet getting more is a serious problem right now. And "fresh" also means data that has not been debased by incorporating GenAI output, which is increasingly hard to filter out as it generated in vast amounts very easily. This paper is only formalizing a pro
Garbage in, garbage out (Score:3)
The researchers are calling this effect Model Autophagy Disorder (MAD).
Isn't the proper name for this "garbage in, garbage out"?
Re: (Score:2)
"Isn't the proper name for this "garbage in, garbage out"?"
No, because the quality of the source material is irrelevant to whether you will get garbage out eventually.
Re: (Score:2)
No. They are specifically talking about getting garbage out even when you produce a quality input, and iteratively it gets worse.
Met a phor (Score:2)
Editors never met a phor they didn't like. Overload. What the f are they talking about?
What about me (Score:2)
Can I get dumber by consuming machine generated content?
Re: (Score:2)
you have to include counting humans as machines, but yes.
Re: (Score:2)
Clarification: "Can" is a very impresice word. It may happen, but it may not necessarily happen. Even though it seems most humans get dumber with age.
Had to be gonzo, didn't it. (Score:2)
Models that work by consuming human input and then mimicking and mixing it to produce new output, start to produce random garbage when they're fed with their own output instead. Who'd have thought.
It was pretty clear from the outset that whatever AI models would come up with would go downhill the more such AI output is already out in the wild and so gets picked up by the crawlers sucking in content for training.
The only MAD idea here is to sensationalize it into some kind of 'disease'.
Cannibals... (Score:2)
Er (Score:2)
Humans might get pretty batty too, if you isolated them and just fed them nonsense.
Are we afraid that all the human content is going to vaporize, because something something?
Re: (Score:2)
No, but training on the same data over and over with different models isn't very sexy.
Starting to sound like raising children (Score:3)
Name every conflict: jews/arabs, russians/ukrainians, Beatles/Stones, Pineapple on pizza.
It always comes down to preferences, indoctrination, and what makes ME money.
So, you HAVE to tell the child which god to worship, you HAVE to stop your child from putting their finger into the electrical socket, you HAVE to tell them right from wrong. It's the same with these models. YOU HAVE to put your finger on the scale at some point and bias the model.
Looking forward to the AI bust, probably next year.
Re: (Score:2)
Pineapple on pizza is so 80's. Personally I don't like my fruit in heated main courses. In a salad maybe, or at most a diced regular apple in a hot pot for some acidity.
Oh noes my profits - that means we need a (Score:2)
They could pay the artists and writers (Score:1)