Major AI Conference Flooded With Peer Reviews Written Fully By AI (nature.com) 34
An analysis of submissions to next year's International Conference on Learning Representations has found that roughly one in five peer reviews were fully generated by AI, a discovery that came after researchers including Carnegie Mellon's Graham Neubig grew suspicious of feedback on their manuscripts that seemed unusually verbose and requested non-standard statistical analyses.
Neubig posted on X offering a reward for anyone who could scan the conference's submissions for AI-generated text, and Max Spero, CEO of detection tool developer Pangram Labs, responded the next day. Pangram screened all 19,490 studies and 75,800 peer reviews submitted to ICLR 2026, finding that 21% of reviews were fully AI-generated and more than half showed signs of AI use. The conference had permitted AI tools for polishing text but prohibited falsified content. Each reviewer was assigned five papers to review in two weeks on average -- a load that senior programme chair Bharath Hariharan described as "much higher than what has been done in the past."
Neubig posted on X offering a reward for anyone who could scan the conference's submissions for AI-generated text, and Max Spero, CEO of detection tool developer Pangram Labs, responded the next day. Pangram screened all 19,490 studies and 75,800 peer reviews submitted to ICLR 2026, finding that 21% of reviews were fully AI-generated and more than half showed signs of AI use. The conference had permitted AI tools for polishing text but prohibited falsified content. Each reviewer was assigned five papers to review in two weeks on average -- a load that senior programme chair Bharath Hariharan described as "much higher than what has been done in the past."
Yo Dawg (Score:5, Funny)
I heard you like AI, so I put an AI in your AI so you can AI while you AI.
8th grade (Score:3)
I wonder if telling your AI to write at an 8th grade level would help fool detectors?
Re: (Score:2)
8th grade is the last year of standard Reading class taught in American schools. It's basic literacy, nothing to be ashamed of.
Re: (Score:2)
"..until you realize the illiterate idiots"
You mean people that can't read at an 8th grade level? WFT, are you even going on about? It appears you are one of them.
AI detectors remain garbage. (Score:5, Interesting)
At one point last week I pasted the first ~300 words or so of the King James Bible into an AI detector. It told me that over half of it was AI generated.
And seriously, considering some of the god-awful stuff passing peer review in "respectable" journals these days, like a paper in AIP Advances [aip.org] that claims God is a scalar field becoming a featured article [phys.org], or a paper in Nature [nature.com] whose Figure 1 is an unusually-crappy AI image [springernature.com] talking about "Runctitiononal Features", "Medical Fymblal", "1 Tol Line storee", etc... at the very least, getting a second opinion from an AI before approving a paper would be wise.
Re: (Score:3)
I pasted the first ~300 words or so of the King James Bible into an AI detector. It told me that over half of it was AI generated.
Probably generated by Abraham Intelligence!
King James ]Re:AI detectors remain garbage.] (Score:2)
At one point last week I pasted the first ~300 words or so of the King James Bible into an AI detector. It told me that over half of it was AI generated.
Sure. It reads very stilted; no actual human would write text like that these days.
Re: (Score:1)
At one point last week I pasted the first ~300 words or so of the King James Bible into an AI detector. It told me that over half of it was AI generated.
Sure. It reads very stilted; no actual human would write text like that these days.
Obviously! It was written by God!
Re: (Score:2, Troll)
And clearly God (who as we know, is a scalar field [phys.org]) is an AI. That's why there's so much "slop" in the Bible - factual errors, contradictions, different versions of the same text that heavily contradict each other, etc etc. It all makes so much more sense now!
Re: (Score:2)
And yet, your imaginary friend still isn't real. ;)
Re: (Score:2)
Re: (Score:2)
They clearly didn't even use a proper image generator - that's clearly the old crappy ChatGPT-builtin image generator. It's not like it's a useful figure with a few errors - the entire thing is sheer nonsense - the more you look at it, the worse it gets. And this is Figure 1 in a *paper in Nature*. Just insane.
This problem will decrease with time (here are [bsky.app] two infographics [bsky.app] from Gemini 3 I made just by pasting in an entire very long thread on Bluesky and asking for infographics, with only a few minor bits
Re: (Score:2)
Re:AI detectors remain garbage. (Score:4, Insightful)
Seems like we are going to have AI that is smarter than people pretty soon even if the AI tech never improves.
Re: (Score:3)
or a paper in Nature
That's not a paper in Nature, it's a paper in Scientific Reports, a much lower tier journal published by the same company.
Hells bells that is godawful though, and inexcusable from a publishing point of view.
SciRep is a very mixed bag. It's where you go when you've exhausted the higher impact factor journals. There are some really good papers in there, but also bad ones and with the vast deluge (even pre AI slop) of papers, getting good peer reviewers for mid to lower tier journals has be
Re: (Score:2)
The hallmarks of AI are good prose
ROTFL you don't know good prose.
Not surprised about peer review (Score:5, Interesting)
I am not at all surprised about AI peer review.
Peer review is part of the fundamental basis verifying the integrity of the scientific enterprise, but it is done anonymously, gets you no credit, nobody knows whether you do a good job or a bad one, and is basically a time sink with little reward except a vague feeling that you did something useful. I personally do NOT use LLM models (for peer review or anything else), but I absolutely can see how it would be very tempting to do so, a tremendous time saver with no down side.
Re: (Score:1)
Re: (Score:3)
Peer review is part of the fundamental basis verifying the integrity of the scientific enterprise, but it is done anonymously, gets you no credit, nobody knows whether you do a good job or a bad one, and is basically a time sink with little reward except a vague feeling that you did something useful.
There is a lot of truth in what you said. However, reviewers (well at least primary reviewers) get to be listed as program committee members. And PC members generally get more consideration as future PC and general chairs. As a PC chair, I would always pass along info to the next PC chairs about which reviewers were slackers in terms of writing short or useless reviews, failing to submit reviews on time, using an inordinately high number of secondary reviewers, or scoring papers significantly differently th
Re: (Score:2)
Nah. The Program Committee members are the ones who pick the peer reviewers.
To be fair, though, usually reviewers are hard to motivate, and some are late or just drup out of sight, so as the program-committee area head you probably end up having to review the papers you can't find reviewers for.
Re: (Score:2)
Nah. The Program Committee members are the ones who pick the peer reviewers.
To be fair, though, usually reviewers are hard to motivate, and some are late or just drup out of sight, so as the program-committee area head you probably end up having to review the papers you can't find reviewers for.
Different conferences do it differently. I'm in computer architecture, and for conferences in that field, the PC members are the primary reviewers because the assumption is that PC members are the top experts in the field. Anyone they ask to do a review is a secondary reviewer. Ostensibly the reason for finding a secondary reviewer is that you know someone is an expert that would be better than you for that specific paper. However, most of the time, secondary reviewers are chosen because the primary rev
scammers do scam (Score:2)
news at 11
They are ... (Score:4, Funny)
Well, I am on Slashdot, so ...
I would simply boot all of them. (Score:2)
Re: (Score:2)
The papers were not the problem. The peer reviews were AI generated.
Re: (Score:2)
199 manuscripts (1%) were found to be fully AI-generated; 61% of submissions were mostly human-written; but 9% contained more than 50% AI-generated text
If they don't do something this year next year it will go exponential.
Nip it in the bud now and simply kick out any use of AI or their little problem will become as bad as the current internet.
Re: (Score:2)
That's why we have peer review. It's unlikely that anyone whose realistic career depends on it will use 50% AI generated text, too much risk of a ruined career forever. But often there are little limits who is allowed to submit. If every unmotivated student with an university e-mail address can send a paper without looking suspicous (there is usually not much verification of submitters) there will be a few for who it doesn't matter if they won't be taken serious academically afterward.
Who would say (Score:2)
What an Amazing Irony.
Karma (Score:2)
Maybe they should introduce meta-moderation, where authors and institutions get Karma assigned for the quality of their paper and review. Low Karma could then result in more oversight, more reviews, fewer allowed submissions, or a 1-year ban.