25% of Incoming Pull Requests May Be AI Slop, Finds Study From Linux Foundation Research

"59% of maintainers report growing contribution volume since the rise of GenAI," according to a new study published by Linux Foundation Research. But 25% of their incoming pull requests "are estimated to be low-quality, also known as 'AI slop'." And on large projects, 69% see growing contributions — and estimate that 31% are low-quality/AI slop.

So what do these results show us? "Generative AI is fundamentally changing how people participate in open source," says the head of Open Source Ecosystem Development at Huawei's R&D subsidiary Futurewei, which helped support the study. Specifically GenAI is "lowering barriers to contribution" but also "creating new challenges" for maintainers and communities. Their recommendation? "Build the governance, transparency and trust mechanisms that allow AI-assisted contributions to strengthen, but not overwhelm, the open source ecosystem."

"Open source projects need to design for a growing AI-generated contributor base, not simply react to it," said Hilary Carter, SVP of Research at the Linux Foundation. "That means setting clear expectations for GenAI, putting responsibility for testing and disclosure on contributors and giving maintainers better tools to manage incoming pull requests. If we get those fundamentals right, AI can expand participation without overburdening and increasing costs for maintainers."

More from the blog It's FOSS: LF Research is the Linux Foundation's research arm. It was set up in 2021 and publishes studies on how open source is built and used, drawing on surveys and project data. Its latest report, Navigating Open Source Software Development in the Generative AI Era, surveyed 234 contributors and maintainers on how generative AI is affecting the way open source gets made...

64% say reviews now take longer. Burnout is their top concern at 63%, narrowly ahead of contribution quality at 60%.

You will also find that policies have not caught up to the AI-assisted coding and vulnerability reporting trend. Of the 119 respondents who answered the question, 35% said their project has a formal policy permitting AI use. Another 32% have none and no plans for one, 23% plan to write one, and 10% ban AI contributions outright....

Most maintainers (71%) say AI makes it easier for newcomers to start contributing.
Other interesting findings:

• "Respondents estimate average time reductions across all six OSS development activities measured, including proposal discussions (19%) and code authoring (52%). For high-complexity tasks, however, 21% of respondents report taking more time when using GenAI."

• "71% of maintainers agree GenAI reduces barriers to entry for OSS, but 53% of maintainers disagree that GenAI has improved community trust."

• Developers are reinvesting the time they save back into open source: 66% report using productivity gains to address technical debt, 63% to develop new features and 58% for learning and upskilling.

• Continued investment in human mentorship remains important: 46% of maintainers say GenAI increases the need for mentorship of new contributors, compared with 32% who say it decreases that need.

OpenSSF and LF AI & Data also helped support the study.

nonsensical

[timeOday]

"may be AI slop" is a nonsensical statement, because if it's slop it doesn't matter who wrote it. Whether AI did the coding is not really relevant, although it is a more objective claim.

Re:

[evanh]

AI is slop. Sloppy AI automation lowers the barrier for entry. Any idiot can throw some slop now.

Re:nonsensical

[martin-boundary]

AI slop is a completely different kind of slop from bad human programming. You can't mix up AI slop and bad human programming into one single category. That's an oversimplification which hinders intelligent discussion.

Re:

[fleeped]

The slop term is on point, as it also communicates the industrious volume which is produced, and this has some nasty side-effects, on human time. And if you want to use AI to combat slop, congratulations, you're now enabling this problem twice as much.

Re:

[allo]

That's true, but if you are interested in the number of PR that are AI slop, you want to count AI slop and not just slop. What you do with that number and if you next count the human slop is another question.

Programming with AI for dummies

[thesjaakspoiler]

You could be a well respected programmer like the rest of us!

Re: Programming with AI for dummies

[firewrought]

Is that the motivation for all the AI slop hitting open source projects? People trying to pick up cheap credibility and resume points? I'm really curious why it's such a problem...

Re:

[toxonix]

It could be? I am not sure. I get enough computer usage as a programmer for work that I don't do it in my free time.
I would consider AI slop or meat proxy code anything the user just let Claude etc do without reading or reviewing the code for 100 line comments, useless historical facts about the development cycle of the change nobody wants to know ("we abandon that approach because x and y, and went with a and b instead")

I don't look much at people's GitHub stats. Maybe before everyone started using AI, but

Re:

[allo]

I contributed to the Linux kernel!
What did you do?
I fixed a typo in the documentation.

When you can't be bothered to to write something..

[BitterEpic]

When you can't be bothered to to write something...why should anyone be bothered to read it? We have some idiots doing that right now with AI here on Slashdot.

Re:

[jezwel]

I didn't even read the comment I'm responding too!

More seriously though, if AI is pushing sloppy contributions and overwhelming repo owners, pushback will probably be AI agents that review the code and dump it to NULL.
Maybe it would be better for reviewers that find an issue to send in their prompt and toolchain details so owners can reproduce and fix as desired.

Re:

[toxonix]

It would be ideal of you could charge the PR creator's accounts for the tokens needed to do the automated PR loop.
I mean wouldn't it? It would.

Re:

[toxonix]

s/of/if/

Use AI to deal with pull requests

[TheMiddleRoad]

It's the obvious solution. Triage the requests. Toss the garbage. Prioritize what came through. Convert it to easily readable English. Then you make decisions.

Re:

[printman]

There is overhead to this.

Over in OpenPrinting we are seeing about 75% of the reported security vulnerabilities falling into the "AI slop" category and around 40% of the regular bug reports. It takes a LOT of time to sift through the slop, to the point that I'm spending half of each day just triaging bugs and not getting new development work done. Definitely reaching the point at which I may just stop working on OpenPrinting projects entirely...

On top of that, assuming we have a valid security vulnerability

Re:

[TheMiddleRoad]

Please define AI slop in this case, then. Automated bots running around doing automated fake bug fixing? Some guy who ran into a bug while using the project and used his local model or claude to file the bug? Serious question here. I modify some open source projects for my own use, and I'm hesitant to send stuff upstream because I'm not sure what's acceptable.

Re:

[printman]

AI slop in this case is bug reports entirely written by (and given the timing between reports from the same user) submitted by an AI tool/agent. Often the reports rely on test configurations that don’t or can’t exist in the real world or on theoretical range or overflow conditions that the tool isn’t smart enough to realize can’t happen or that it doesn’t matter if they do happen.

I have no problem with reporters that use an AI tool to find a bug or write up a quick PoC that rep

Re:

[TheMiddleRoad]

Thanks for replying. I'm using and modifying a small project made by a few people and a large project made by many people. For the large project, I've made major improvements, but I'm hesitant to share them upstream. For the smaller project, I'm fixing significant bugs, but Claude and DS both reported one issue that they wanted reported upstream. I resisted. Eventually, I realized that they were completely focused on latency and not throughput when throughput is what matters far more, so I smacked them

Re:

[printman]

It's always OK to reach out and explain what you're doing and that you want to contribute (back). If the project has a mailing list/channel/whatever for discussing things, get on it and introduce yourself!

Most projects don't have enough developers so if you can work well with others, have a good grasp of the technical details, and aren't making huge architectural changes, then more than likely they will welcome your contributions. Just be polite and explain what you want to contribute. Start a conversation

Re: Use AI to deal with pull requests

[devilops]

will you pay for the tokens?

Licensing barrier

[LainTouko]

"If this contains the output of any LLM, provide a full list of the licenses of everything in its training material so we can check compatibility with our license."

Re:

[AmiMoJo]

It's a legally unclear situation. Most human developers do not provide a list of all the open source code they may have read or been influenced by. On the other hand, because these LLMs are offered as a commercial service, there might be issues with everything they generate.

My guess (IANAL) would be that any claim would be against the company that made the LLM, unless the code was copied verbatim from somewhere else, in which case it's no different to a human doing the same thing.

Re:

[LainTouko]

That can't be how it works in general. Suppose A trains a LLM on a load of GPL code, and then B generates some code from that LLM and distributes the compiled version. If B also makes the source code available, says it's distributed under the GPL and includes a copy of the GPL to tell users about their rights, this seems all above board. If B keeps their source code secret, it can hardly be A who is to blame for the (certainly moral, maybe legal) ensuing GPL violation.

Re:

[AmiMoJo]

The original violation would be making a derivative product, the LLM, that was not also GPL licensed.

Re:

[karmawarrior]

This would, if someone would do it, resolve a hell of a lot of issues that I think are serious in the current model. Alas big-AI seems reluctant to actually partition training sets like that, and instead are focusing on lobbying to try to get copyright laws to ignore the licenses of the content they're training on. And honestly, that strategy doomed because not every jurisdiction in the world is going to go along with "Please let us ignore copyrights because we have lots of money" the same way the US seems

Re:

[LainTouko]

If an LLM learns skills from reading GPL code, and then uses those skills to write code for me, I have not copied the GPL code and therefore I am not required to release the code under GPL.

If LLMs could do this, we would have a very different conversation. However in the real world, what they do is populate a massive data structure tallying what next words it's seen following previous word combinations and how often. Hence "large language model". The output of an LLM trained entirely on GPLed code will be

Dipshits, it's still a bug if AI finds it

[EmagGeek]

This anti-AI bullshit has got to stop. If an AI finds a real bug in your program, sing hallelujah and move on with life. Computers will always be able to execute algorithms faster than humans, and with greater depth and accuracy. That's the whole point of their existence - to do stuff that is too complex for humans to do, and to do it quickly.

Re:Dipshits, it's still a bug if AI finds it

[karmawarrior]

If you're overloaded with slop bug reports - that is, bullshit bug reports that aren't a bug in your program but are the results of genAI users simply believing the output of Claude and shoving it into a ticket so they can pretend they contributed - you might have a very different view on this.

Slop is slop. If genAI was producing high quality output that's usable we wouldn't call it that.