AI Coding Agents Generate More Code, But Not More Software (arstechnica.com) 66
A study of more than 700 software firms found that AI coding agents significantly increase the amount of code developers produce, but not the amount of software companies actually ship. Ars Technica reports: A recent study (PDF) of actual coding practices across hundreds of firms finds that human code review forms a significant "bottleneck" for the overall efficiency of AI coding tools, resulting in "little evidence that firms increase software output or reduce employment" by using them. Any efficiency increased during the actual coding phase, the study authors find, is "absorbed by downstream constraints in the production process"; as "the code review process significantly increases in length, pull requests are more likely to require revisions, and reviewers leave more comments."
[...] In terms of raw code being produced, the results are clear and stark. The introduction of AI coding agents at a firm leads to a 30 percent increase in total lines of code generated, a 20 percent rise in the number of total commits, and a 23 percent increase in pull requests on average, the researchers write. But all that extra code doesn't translate directly into improved software output on the firm level. On the contrary, the resolution rate for Issues and Epics (i.e. wholesale software features) tracked by tools like Jira did not change in a statistically significant way after AI tools were introduced (the researchers also found no "compositional shift" in the size or complexity of those Jira-tracked issues across the AI introduction).
The reason for that discrepancy can be found directly in the code review process, which takes markedly longer on average after the introduction of AI coding agents. Overall, the average "review process" time between a pull request getting submitted and it being merged into the codebase balloons 49 percent on average after AI agents are introduced. That effect can be seen in more granular data, too, with "the share of pull requests with changes requested nearly doubl[ing], and the number of comments per pull request increas[ing] by 35%" following the AI agent shift, the researchers write.
In response to this change, the researchers found a 14 percent increase in the share of workers performing code reviews after AI agents' introduction. They also write that they "cannot attribute significant employment changes to AI" after looking at total active workers across Jellyfish and cross-referencing with LinkedIn data at those firms. While AI could also theoretically help with this review process, the researchers found that, so far, that impact has been marginal. Although 80 percent of measured firms used some form of AI code review by March 2026, AI agents were only responsible for 23.3 percent of all review comments and 10.8 percent of all pull requests, suggesting humans were still responsible for the vast majority of this work.
[...] In terms of raw code being produced, the results are clear and stark. The introduction of AI coding agents at a firm leads to a 30 percent increase in total lines of code generated, a 20 percent rise in the number of total commits, and a 23 percent increase in pull requests on average, the researchers write. But all that extra code doesn't translate directly into improved software output on the firm level. On the contrary, the resolution rate for Issues and Epics (i.e. wholesale software features) tracked by tools like Jira did not change in a statistically significant way after AI tools were introduced (the researchers also found no "compositional shift" in the size or complexity of those Jira-tracked issues across the AI introduction).
The reason for that discrepancy can be found directly in the code review process, which takes markedly longer on average after the introduction of AI coding agents. Overall, the average "review process" time between a pull request getting submitted and it being merged into the codebase balloons 49 percent on average after AI agents are introduced. That effect can be seen in more granular data, too, with "the share of pull requests with changes requested nearly doubl[ing], and the number of comments per pull request increas[ing] by 35%" following the AI agent shift, the researchers write.
In response to this change, the researchers found a 14 percent increase in the share of workers performing code reviews after AI agents' introduction. They also write that they "cannot attribute significant employment changes to AI" after looking at total active workers across Jellyfish and cross-referencing with LinkedIn data at those firms. While AI could also theoretically help with this review process, the researchers found that, so far, that impact has been marginal. Although 80 percent of measured firms used some form of AI code review by March 2026, AI agents were only responsible for 23.3 percent of all review comments and 10.8 percent of all pull requests, suggesting humans were still responsible for the vast majority of this work.
Shocked (Score:3)
Turns out it takes longer to review code that no-one understands.
The solution, obviously, is to have the AI review the code too so humans don't need to understand it.
Re: (Score:2)
Human written code is also frequently not understandable by anyone but the author, and even that is in question.
Re:Shocked (Score:4, Insightful)
Working on the code every day myself is what makes it possible for me to review code that people contribute.
If I use AI to write it all, then I don't internalize the code anymore, and I slowly lose the ability to review it.
Been nice knowing you all, the industry is going to hit a brick wall in part because of reliance on AI in the very stages that should include humans.
Re: (Score:2)
That is the way we're heading. So don't build your life around relying on software working because nothing is going to be reliable in future.
One of our problems is that we can't make major changes to the software because it invalidates over a decade of use in the field and massive amounts of testing required to ensure it doesn't kill people. Sooner or later that will be replaced with AI testing, so the code will be developed, tested and installed without a human ever checking whether the AI did it right or
next up (Score:3)
The industry will be optimizing AI to produce hard to understand complicated one-liners.
Re: (Score:3)
I usually have to convince Claude to stop writing 5+ line comments in the code. Too much exposition, too much boiler plate, and it often writes a complex generic solution instead of a concise specific solution.
Re: (Score:2)
There is a happy middle somewhere, AI has not found it yet.
Re: (Score:2)
I can get it to iterate and improve. So I believe it is possible with a better model and better prompts.
For writing "business logic", AI is already capable enough. AI is cranking out satisfactory code for the garbage requirements that coders have to deal with in the corporate sphere.
For elegant art that coding as a hobby demands, the AI doesn't satisfy. It isn't satisfying to use, and the results are not satisfying to read. That said, I made a bunch of shitty little games with AI.
What actually impressed me
Re: (Score:2)
I find it has a surprising amount of industry domain knowledge. It really helps with data analysis. Someone recently sent me a file they thought I could use, but we didn't know where it came from or how it was generated. Claude found that it was similar to an industry standard file (IATA SSIM data) and it seemed to know all about airline industry standards. Also naval architecture. It knows all about the standards in the ship building industry as far as what software is used the most, workflows, libraries o
Re: (Score:2)
Get the interfaces pretty. The internal code can be fixed later if needed.
Re: (Score:2)
Context engineering solves most of these problems. Claude behaves much differently for me than for my colleagues due to the differences in system prompt.
The problem I have is the peer review process. Getting merge approval is just as slow as it was before Claude started generating all this code. I can have agents run rounds and rounds of code reviews, but somebody who's not me ultimately has to click the approve button before I can merge anything. I haven't had to manually resolve any merge conflicts in a l
Doing it properly works but is NOT FUN (Score:5, Interesting)
The solution is to require developers to fully understand the code they submit, and check that understanding
When you take the time to understand the AI-written code ( often using the AI to explain its own code ), there is no problem and you can proceed significantly faster than without AI.
The problem is that that work to gain understanding is boring.
While I often had to do it in the days before AI, I did not become a programmer to spend all my time understanding code I did not write myself.
I think *that* is the serious problem for our industry.
Re: (Score:2)
I wouldn't bother testing the submitter. We need to be more willing to reject submissions without feeling the need to explain ourselves in depth.
If they're using AI, then it should be trivial for them to rewrite it. So nothing of value is lost by dropping bloated PRs.
Re: (Score:2)
True, however I have used AI to help me fix bugs in code I didn't understand. But I didn't let it make blind PRs for me. I ha
Re: (Score:2)
Most of the time we trust the developers so as reviewers we just skim through the process, but that's harder to do with AI.
compactness (Score:2)
From the little I see, AI code tends to not worry about compactness or elegance, just getting the job done. Human code that tries to be compact by excessive generalization and indirection is much harder to debug, even if it has fewer lines of code for the same job.
Re: (Score:2)
AI code tends to not worry about compactness or elegance
This was definitely true in the beginning, but the latest generation models are putting out some pretty amazing results.
Re: (Score:3)
I have seen claude, qwen, and codex churn out some very nice code. I also see a lot of it as being written for not reason.
The spec might say ARGV[1] will be a dotted decimal ipv4 address. The next thing you know you will get a bunch code to see if it might be a hostname instead and try to resolve, check if was passed as single integer, make sure the user did not enter it in hex..
It might all work too! It is none the less 10s of extra lines to handle every possible way an addresses when what was called for
Re: (Score:2)
That's why you have tests to ensure they don't.
Otherwise you pass it a hostname instead of an IP address, the low-level code happily accepts it in code paths that have never been tested and before you know it something explodes.
Re: (Score:2)
which is why I suggested a human might have implemented a regex, test and quick exit with an error code.
Detect and Abend is usually a lot safe than guessing that 'grep' when you were expecting something like 198.51.100.45 means you should try to resolve grep.example.com and run with whatever address happens to comeback.
However in tool designed to be used by end users the try to figure-out what they intended and do it approach often is right. All depends on the stakes, but my example was one intended to be s
Complexity is cancer (Score:3)
AI tools easily generate more complexity.
We need better tools to understand, manage and reduce complexity, not tools that allow us to create more.
Re: (Score:2)
Yes, very much so. But that would be good engineering and good engineering is rare in the software space.
Re: (Score:2)
We need better tools to understand, manage and reduce complexity, not tools that allow us to create more.
That is begging the question. What I mean is, a lot of people want more complexity, because they want things to take longer, because they make more money. Middle managers want programmers to be busy but go slowly so they can justify hiring more people. There was an interview with Marc Andreesen about that [zerobugsan...faster.net], and Bjarne Stroustrup earlier.
Bottom line: faster isn't better.
Re: (Score:2)
Re: (Score:2)
Tests are as important as the code they test. It's just as essential that they be correct. If you're going to have AI write your tests, you need to review them, understand them, and verify that they really test what you intended. Otherwise you're likely to get a lot of useless tests that report success even when the code is wrong.
I'll sometimes intentionally break a piece of code just so I can see the test fail. I need to prove to myself that the test really does fail if the thing it's testing is wrong.
Re: (Score:2)
Unit tests are generally a mechanical translation of the requirements into software that verifies each requirement has been met.
Those are functional tests, not unit tests (unless your specification was so detailed it was written in UML).
Re: (Score:2)
Though already CI/CD pipelines were choking on the unit tests. So they aren't *free*.
Had an LLM generated request that was pretty straightforward, and the LLM generated 272 unit tests for a very simple function. Can't really afford that.
Also, despite the 272 unit tests, when I tried to try it out in the way a human would, it immediately failed. All the mocking the AI generated never actually properly imitated the real world. However, the submitter was *quite* proud of having hundreds of unit tests, desp
Bad study (Score:2)
I've got a black swan for ya!
I just shipped software that AI wrote [practically] entirely that I would not have shipped otherwise. I don't think I'm the only one.
Re: (Score:2)
But how do you plan to maintain this software? That's the real effort of software engineering.
Re: (Score:2)
The general plan appears to be to tell the AI to rewrite it from scratch after a while.
Re: (Score:2)
But how do you plan to maintain this software? That's the real effort of software engineering.
I'm not sure how to answer that question. My flip answer is "the usual way." But nothing about software is the usual anymore.
100% of the code that .. makes up this project is code that I did not write. But I absolutely told AI what I wanted and in many cases *how* I wanted it. And any code that had "interesting" design elements is code that I reviewed and in some cases told AI to change. Where interesting refers to meaningful architectural decisions.
At this point it is a very light project - somewhat l
Re: (Score:2)
Re: (Score:2)
Perfectly reasonable study, because small-scale development is not the same. I have to understand the code of 10+ programmers on a regular basis. Our game engine is involved. The only way for me to learn the code is to use it and step through it.
If you're writing apps yourself, AI is great. It's a teammate that follows your instructions and implements the things that you want. You've already got a mental model of what you need, and it gives that to you. Of course it works for you.
Big codebases are just diff
Re: (Score:2)
My day job is a (saas) app that's about 20 years old and has been worked on continuously by a team that has varied in size from about 5 to now about 15. It is complicated and connects to several external services. There are parts of the app that are too hairy for AI to reason about. But we have found, recently, that AI is surprisingly good at explaining how some rather complicated (but well written) parts of the app work.
The 2-3 really hairy parts of the app are written in an arbitrarily complicated way
Another reason for discrepency.. (Score:3)
Like always, more code != more valuable software behavior.
So you have the AI doing overtly silly things. Someone asked to adjust layout to fix a gap in a webpage. Eventually they submitted their pull request, hundreds and hundreds of lines of CSS. Often the exact same rule repeated 5 times, which I'm guessing is how many rounds the operator went with the system before the gap actually closed. The lines were *mostly* just '!important'ing already existing rules, which had no impact, but hey, worth a shot right? Ultimately I discarded the merge request and made the one-line css tweak that was actually relevant.
Then you have the AI doing... well arguably slightly silly things. Like protecting against an invalid value redundantly throughout a call stack. Arguably defensible, as maybe one day the deep function will be called from somewhere else and passing through the error might not present it with the most informative context.
Finally the AI doing... well things the human should have been doing. With the advent of a lot of code scanning, this is less likely, but things like calling functions in C and charging ahead without ever checking the status of the call. functions that, practically speaking, will likely never fail, however you really *should* be being more defensive about the possibility.
The 'human review as a bottleneck' isn't really the angle I would fixate on. Completely unfettered AI usage also stalls out in many situations. If the AI can't get there on its own trivially, then it's going to leave an almost-correct looking mess that isn't quite there but is very hard to get across the finish line. If the AI can get it, then it's probably not 'software being shipped', because while perhaps slightly better than a generic off-the-shelf existing procedure, probably not worth the trouble to 'ship' a solution that largely overlaps with a lot of generic stuff.
AI levels things out (Score:2)
Just a reminder: (Score:3)
Re: (Score:2)
Hehehe, stupid is as stupid does. That is why real mathematicians verify a proofs before they build on the result.
Re: (Score:2)
Re: (Score:2)
We have an AI review all code changes for security holes because it's been programmed to be very good at finding ways to break security that make perfect sense but I would never have thought of. But it's purely advisory, I'd never let it have the final say on whether a change is safe.
Re: (Score:2)
I had students try that out in some thesis work. Results were that LLMs find some types of vulnerabilities reasonably reliably and others not at all or only rarely.
Skill problem (Score:2)
If you're not shipping code with AI faster than ever, it's 100% a skill and/or organizational problem.
In fact, it might be the organizations themselves which are the problem. Plenty of solo devs are doing amazing work with AI.
Re: (Score:2)
"it's 100% a skill and/or organizational problem", I agree. In my experience the recent models generate very good code complete with unit tests and thorough documentation. If you've got more new code than you know what to do with then maybe you need to get your programmers to focus more on the reviews. Make it a Jira ticket, that tends to get people's attention.
Re: (Score:2)
Nonsense. The real problem on display here is that you have no clue how to produce quality code.
Re: (Score:2)
Solo development is a different job. The tasks you give to AI are based on what you need, and your own mental model of they code they need to spit out.
Large-scale development is different, because people are different. I've worked on 5 or 6 different game engines over the years, and Unreal is the current one that I'm on. Unreal has its quirks. The rest of our game engine has historical baggage that I need to account for. The interactions between systems are more organic, because not everything can be planne
NO SHIT (Score:2)
Re: (Score:2)
Glazing machines
They're called "enrobers." (humor)
So using LLMs makes coding more expensive... (Score:2)
Because you just have to pay the same number of coders even if more are code reviewers not, but you pay for the LLM on top of that as well. I am sooo not surprised. And, apparently, reviewing LLM code is mind-numbing and far more unpleasant than reviewing human code. Hence there is another really bad problem in there.
Just have Claude do the code reviews! (Score:2)
What could possibly go wrong?
I joke, but this is *literally* what my bosses were telling me I should have my teams do, to speed up AI coding.
Re: (Score:2)
Re: (Score:2)
Yes, indeed I have used multiple models to review pull requests, including Claude and Copilot. I do find the reviews to be generally decent. Generally, the comments suggest improvements to the code structure or readability. Claude does *not* know if the code does what it is *intended* to do.
Re: Just have Claude do the code reviews! (Score:2)
Re: (Score:2)
Insults only make your argument weaker.
There are different levels of verification. QA testing only goes so far. It doesn't test that the underlying code is architected correctly. Unit tests help, but again, major flaws can often hide in plain sight.
AI still makes very dumb mistakes. it does this because it takes the input it is given, and assumes the input is complete and correct. It then runs with those instructions, even when there are critical requirements that a human forgot to provide. AI code reviews
Re: Just have Claude do the code reviews! (Score:2)
Re: (Score:2)
Clearly, you have no actual point to make, or you would make it.
Re: Just have Claude do the code reviews! (Score:2)
I think it all depends (Score:2)
Like most questions, it depends on what kind of code you're writing, and how thoroughly it must be reviewed.
If you're writing code that is used by thousands of customers, yeah, I can totally see how review would be a bottleneck.
But if you're writing code that's used by a single team or department at your own company, AI probably does actually speed up this kind of work. An outage or bug in this type of software is often not a serious disruption.
Software companies have to change (Score:2)
Companies not moving faster will be outcompeted.
Open source .. will win.
Badly changes the job description (Score:2)
One of the big problems with this is that it effectively changes the job description of programmers to managers.
You're not just responsible for your own code, you're responsible for telling someone/something else what to do, and then reviewing their work, and then integrating that work and making sure it's correct, because if it gets checked in and it's wrong, you're on the hook for it. But you didn't even write it, so it's harder to understand. Most programmers I know are better at understanding the code t