305 lines
17 KiB
HTML
305 lines
17 KiB
HTML
<html lang="en">
|
||
<!-- WARNING: Automatically generated file. Do not modify. Modify the blog directory entries instead. -->
|
||
<head>
|
||
<!-- vim: foldmethod=indent:spell:sw=2:smarttab
|
||
-->
|
||
<meta charSet="utf-8"/>
|
||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">
|
||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js"></script>
|
||
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script>
|
||
<script async src="https://cse.google.com/cse.js?cx=001456668591254139637:te2uquaxidc"></script>
|
||
<script async defer crossorigin="anonymous" src="https://connect.facebook.net/en_GB/sdk.js#xfbml=1&version=v11.0" nonce="61tQy9ee"></script>
|
||
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.7.1/css/all.min.css">
|
||
<title>blog dds: 2026-05-23 — Why reviewing AI-generated code is devilishly hard</title>
|
||
<meta charset="utf-8">
|
||
<meta name="Author" content="Diomidis Spinellis">
|
||
<link rel="stylesheet" href="https://www.spinellis.gr/blog/style.css" type="text/css" />
|
||
<link href="../../a/pandoc-syntax-2.css" rel="stylesheet" type="text/css" />
|
||
|
||
|
||
<link rel="ToC" href="https://www.spinellis.gr/blog/contents.html" />
|
||
<link rev="Subdocument" href="https://www.spinellis.gr/blog" />
|
||
<link rel="me" href="https://twitter.com/CoolSWEng" />
|
||
<link rel="previous" href="https://www.spinellis.gr/blog/20260413" /><link rel="next" href="https://www.spinellis.gr/blog/20260702" />
|
||
<meta name="Generator" content="blog.pl 3c6ad07 2025-12-24 23:55:57 +0200">
|
||
|
||
<link rel="shortcut icon" href="https://www.spinellis.gr/favicon.ico" />
|
||
</head>
|
||
<body>
|
||
|
||
<div class="container">
|
||
<div class="row"> <!-- Logo row -->
|
||
<div class='col-sm-12 dds-bg'>
|
||
<span class="global-title-spinellis"><a href="https://www.spinellis.gr/blog">Diomidis Spinellis blog</a></span>
|
||
<img src="https://www.spinellis.gr/images/dds-logo.png" class="pull-right dds-logo-size" alt="dds logo" />
|
||
</div>
|
||
</div>
|
||
<hr class='dds-hr' />
|
||
|
||
<div class="row"> <!-- Title row -->
|
||
<div class="col-sm-9 blogtext">
|
||
<h1>Why reviewing AI-generated code is devilishly hard</h1>
|
||
</div>
|
||
|
||
<div class="col-sm-2">
|
||
<a href="https://www.spinellis.gr"><i class="fa fa-home fa-lg"></i></a>
|
||
<a href="https://bsky.app/profile/CoolSWEng.bsky.social"><i class="fa-brands fa-bluesky fa-lg" style="color: #1185FE"></i></a>
|
||
<a href="https://mastodon.acm.org/"><i class="fa-brands fa-mastodon fa-lg" style="color: #595aff"></i></a>
|
||
<a href="https://www.facebook.com/diomidis.spinellis"><i class="fa-brands fa-facebook fa-lg" style="color: #3b5998"></i></a>
|
||
<a href="https://github.com/dspinellis/"><i class="fa-brands fa-github fa-lg" style="color: #444444"></i></a>
|
||
<a href="https://www.youtube.com/user/dspinellis/"><i class="fa-brands fa-youtube fa-lg" style="color: #ff0000"></i></a>
|
||
<a href="https://www.linkedin.com/in/dspinellis"><i class="fa-brands fa-linkedin fa-lg" style="color: #007bb6"></i></a>
|
||
</div>
|
||
|
||
<div class="col-sm-1"> <!-- Search button -->
|
||
<div class="btn-group pull-right">
|
||
<button class="btn btn-secondary btn-sm" data-toggle="modal" data-target="#modalSearch">
|
||
<span class="glyphicon glyphicon-search"></span>
|
||
<span class="hidden-sm hidden-md hidden-lg">Search</span>
|
||
</button>
|
||
</div>
|
||
<div class="pull-right"> </div>
|
||
|
||
</div>
|
||
</div> <!-- Title row -->
|
||
|
||
<!-- Modal search window -->
|
||
<div id="modalSearch" class="modal fade" role="dialog">
|
||
<div class="modal-dialog">
|
||
<!-- Modal content-->
|
||
<div class="modal-content">
|
||
<div class="modal-header">
|
||
<h4 class="modal-title">Search the blog</h4>
|
||
</div>
|
||
<div class="modal-body">
|
||
<!-- See https://cse.google.com/cse/ -->
|
||
<div class='gcse-search'></div>
|
||
</div>
|
||
<div class="modal-footer">
|
||
<button type="button" class="btn btn-default" data-dismiss="modal">Close</button>
|
||
</div>
|
||
</div> <!-- modal content -->
|
||
</div>
|
||
</div>
|
||
|
||
<div class="row"> <!-- Content / Contents row -->
|
||
<div class="col-sm-9 blogtext"> <!-- Left content -->
|
||
<p>Here’s the thing: when working on code with GenAI assistance
|
||
(from a chat-bot, through IDE auto-completion, or, increasingly,
|
||
with an AI agent)
|
||
you need a better understanding of the system than when working without.
|
||
Cognitive psychology and the workings of large language models (LLMs)
|
||
give us four clues on why this happens.</p>
|
||
<p>When working without AI assistance on a non-trivial task
|
||
and on code you don’t know,
|
||
you first need to comprehend it in order to perform your task.
|
||
Otherwise you’re hacking (in the sense of performing undisciplined changes),
|
||
not programming, and most likely you won’t go anywhere (fast).
|
||
This is an objective built-in control gate of the human-only
|
||
software development process:
|
||
if you don’t understand the code, you can’t contribute to it and you you fail.</p>
|
||
<p>When working with AI assistance and you have to review an AI-generated change
|
||
that passed continuous integration, the control gate is missing:
|
||
there’s no objective mechanism to determine whether you <em>truly</em> understand
|
||
the code or not.
|
||
This means that you may accept an AI-generated change in the mistaken belief
|
||
that you understand it, when in fact you don’t.</p>
|
||
<p>The required type of thinking, <em>metacognition</em>,
|
||
involves not only understanding the code but also assessing your understanding.
|
||
In brief, it involves two abilities.</p>
|
||
<ul>
|
||
<li><strong>Metacognitive monitoring</strong>:
|
||
assessing what you know
|
||
(e.g. a specific data structure or design pattern used in the code),
|
||
how well you understand the change,
|
||
confidence in your review comments,
|
||
and detecting confusion.</li>
|
||
<li><strong>Metacognitive judgment</strong>: specific evaluative acts within monitoring,
|
||
such as
|
||
<ul>
|
||
<li><em>judgment of learning</em>
|
||
(I’ll remember this change when we discuss our new architecture);</li>
|
||
<li><em>feeling of knowing</em> (I’ll understand this code when I see it again);</li>
|
||
<li><em>confidence judgment</em> (I’m 95% sure this code is correct); and</li>
|
||
<li><em>ease-of-processing judgments</em>
|
||
(this change feels easy, so I’ll probably understand it).</li>
|
||
</ul></li>
|
||
</ul>
|
||
<p>Unfortunately, a couple of influential psychological studies have shown that
|
||
people are often poor at accurately evaluating their own knowledge
|
||
or performance.
|
||
Most famously,
|
||
the <a href="https://doi.org/10.1037//0022-3514.77.6.1121">Dunning-Kruger effect</a>
|
||
states that people with low competence tend to overestimate their competence
|
||
because the skills needed to perform a task are also needed to evaluate
|
||
performance.
|
||
The implication of this is that junior programmers are more at risk
|
||
from accepting faulty AI-generated code.</p>
|
||
<p>In addition, through the
|
||
<a href="https://doi.org/10.1207/s15516709cog2605_1">illusion of explanatory depth</a>
|
||
people think they understand mechanisms with far greater precision,
|
||
coherence, and depth than they really do.
|
||
Moreover, this gap is strongest for explanatory knowledge
|
||
than many other kinds of knowledge,
|
||
such as that for facts, procedures, or narratives.
|
||
In the original study, its authors tested how well students could explain
|
||
the working of devices such as a sewing machine, a can opener,
|
||
a self-winding watch, a nuclear power plant, or a photocopier.
|
||
This could well apply to explanatory knowledge of code:
|
||
algorithms, data structures, designs, interactions, and architecture.
|
||
Importantly, in programming, explanatory knowledge,
|
||
which allows reasoning across the software development lifecycle,
|
||
is a lot more important than
|
||
facts (can be established with tools),
|
||
procedures (should be automated),
|
||
or narratives (are rarely embedded into code).</p>
|
||
<p>Finally, the fluency of LLMs makes faulty code appear more trustworthy
|
||
than it deserves and also feeds another cognitive trait.
|
||
Consider the diverse ways in which a programmer can err:
|
||
slips, lapses, mistakes, knowledge-based reasoning failure, rule-based
|
||
misapplication, cognitive overload, confirmation bias, availability bias,
|
||
anchoring, overconfidence, abstraction mismatch, inattentional blindness,
|
||
plan-composition failure, or specification ambiguity.
|
||
For most, it is probable that a reviewer (especially a more experienced one)
|
||
may be able to detect the resulting fault by thinking differently.
|
||
In contrast, AI-generated code is written so as to look plausibly correct,
|
||
even when it’s faulty.
|
||
(This is how due to how LLMs work: they generate a series
|
||
of the next most probable tokens.)
|
||
This makes it more difficult for a human reviewer to detect a fault in the code.</p>
|
||
<p>The plausibility of LLM code gets compounded by a specific trait in the
|
||
<a href="https://doi.org/10.1518/001872097778543886">complex relationship of humans and automation</a>,
|
||
<em>automation bias</em>:
|
||
the well-documented tendency of people to place unwarranted trust in
|
||
automated systems, reducing their own independent verification effort.
|
||
<a href="https://doi.org/10.1006/ijhc.1999.0252">A study of automation across diverse critical domains</a>
|
||
has shown that when automation provides recommendations, people
|
||
are more likely to accept incorrect suggestions (<em>errors of commission</em>,
|
||
say a duplicated routine)
|
||
and less likely to detect problems that the automation failed to address
|
||
(<em>errors of omission</em>, e.g. a missing handler or test).</p>
|
||
<p>In short,
|
||
when reviewing AI-generated code,
|
||
remember that we humans are at a severe disadvantage
|
||
and try to be even more vigilant.</p>
|
||
|
||
<p />
|
||
<!-- COMMENTS -->
|
||
<span><a class="btn btn-primary" href="https://www.spinellis.gr/cgi-bin/comment.pl?date=20260523#comments">Comments</a></span>
|
||
|
||
<!-- Bluesky post -->
|
||
|
||
<a class="btn btn-primary"
|
||
href="https://bsky.app/intent/compose?text=Why%20reviewing%20AI-generated%20code%20is%20devilishly%20hard%20https%3A%2F%2Fwww.spinellis.gr%2Fblog%2F20260523%20via%20%40CoolSWEng.bsky.social" target="_blank" style="background-color: #1185FE; color: white;"><i class="fa-brands fa-bluesky" style="margin-right: 6px;"></i>Post</a>
|
||
|
||
<!-- Mastodon post -->
|
||
<a class="btn btn-primary" href="https://toot.kytta.dev/?text=Why%20reviewing%20AI-generated%20code%20is%20devilishly%20hard%20https%3A%2F%2Fwww.spinellis.gr%2Fblog%2F20260523%20via%20%40CoolSWEng%40mastodon.acm.org" target="_blank" style="background-color: #595aff"><i class="fa-brands fa-mastodon" style="margin-right: 6px;"></i> Toot!</a>
|
||
<!-- Twitter button -->
|
||
<span class="btn"><a href="https://twitter.com/share?ref_src=twsrc%5Etfw" class="twitter-share-button" data-show-count="false" data-size="large">Tweet</a><script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></span>
|
||
|
||
<!-- Facebook button -->
|
||
<span class="btn fb-share-button" data-size="large" data-href="https://www.spinellis.gr/blog/20260523/" data-layout="button" data-size="small"><a target="_blank" href="https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.spinellis.gr%2Fblog%2F20260523%2F&src=sdkpreparse" class="fb-xfbml-parse-ignore">Share</a></span>
|
||
<p />
|
||
|
||
</div> <!-- Left content -->
|
||
|
||
<div class="col-sm-3"> <!-- Right content -->
|
||
|
||
<div class="panel panel-default">
|
||
<div class="panel-heading">Navigation</div>
|
||
<div class="panel-body">
|
||
|
||
<a href="https://www.spinellis.gr/blog/contents.html">blog contents</a> <br />
|
||
<a href="https://www.spinellis.gr/blog/index.html">dds blog</a> <br />
|
||
<a href="https://www.spinellis.gr/">dds home</a> <br />
|
||
<a href="https://www.spinellis.gr/cgi-bin/comment.pl?date=20260523#comments">comments</a><br />
|
||
<a href="https://www.spinellis.gr/blog/20260413/index.html">« Empirical software research in the age of AI</a> <br />
|
||
<a href="https://www.spinellis.gr/blog/20260702/index.html">» Documenting AI-generated code commits
|
||
</a> <br />
|
||
</div> <!-- panel body -->
|
||
</div> <!-- panel -->
|
||
|
||
<div class="panel panel-default">
|
||
<div class="panel-heading">Tagged as</div>
|
||
<div class="panel-body">
|
||
<a href="https://www.spinellis.gr/blog/AI.html" rel="tag">AI</a><br /> <a href="https://www.spinellis.gr/blog/Programming.html" rel="tag">Programming</a><br /> <a href="https://www.spinellis.gr/blog/Software engineering.html" rel="tag">Software engineering</a><br /> </div> <!-- panel body -->
|
||
</div> <!-- panel -->
|
||
|
||
<div class="panel panel-default">
|
||
<div class="panel-heading">Become a Unix command line wizard</div>
|
||
<div class="panel-body">
|
||
<a href="https://www.spinellis.gr/unix?source=blog-banner"><img src="/unix/blog-banner.png" class="img-responsive img-rounded" border="0" alt="edX MOOC on Unix Tools: Data, Software, and Production Engineering" /></a>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="panel panel-default">
|
||
<div class="panel-heading">Debug like a master</div>
|
||
<div class="panel-body">
|
||
<a href="https://www.spinellis.gr/debugging/"><img src="/debugging/img/book/cover.jpg" class="img-responsive" alt="Book cover of Effective Debugging" /></a>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="panel panel-default">
|
||
<div class="panel-heading">Compute with style</div>
|
||
<div class="panel-body">
|
||
<a href="https://www.spinellis.gr/computingstyle"><img src="/computingstyle/img/book/cover.jpg" class="img-responsive img-rounded" alt="Book cover of The Elements of Computing Style" /></a>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="panel panel-default">
|
||
<div class="panel-heading">Syndication</div>
|
||
<div class="panel-body">
|
||
|
||
This blog is also available as an RSS feed: <a href="https://www.spinellis.gr/blog/dds-blog-rss.xml"><i class="fas fa-rss"></i></a><p />
|
||
</div> <!-- panel body -->
|
||
</div> <!-- panel -->
|
||
|
||
<div class="panel panel-default">
|
||
<div class="panel-heading">Recent posts</div>
|
||
<div class="panel-body">
|
||
<a href="https://www.spinellis.gr/blog/20260702/index.html">Documenting AI-generated code commits
|
||
</a> (2026-07-02)<br />
|
||
<a href="https://www.spinellis.gr/blog/20260523/index.html">Why reviewing AI-generated code is devilishly hard</a> (2026-05-23)<br />
|
||
<a href="https://www.spinellis.gr/blog/20260413/index.html">Empirical software research in the age of AI</a> (2026-04-13)<br />
|
||
<a href="https://www.spinellis.gr/blog/20260302/index.html">Vibe coding toward the incident horizon</a> (2026-03-02)<br />
|
||
<a href="https://www.spinellis.gr/blog/20251223/index.html">An initial analysis of the discovered Unix V4 tape</a> (2025-12-23)<br />
|
||
<a href="https://www.spinellis.gr/blog/20250926/index.html">Why I choose email over messaging</a> (2025-09-26)<br />
|
||
<a href="https://www.spinellis.gr/blog/20250626/index.html">Is it legal to use copyrighted works to train LLMs?</a> (2025-06-26)<br />
|
||
<a href="https://www.spinellis.gr/blog/20250520/index.html">I’m removing the BSD advertising clause</a> (2025-05-20)<br />
|
||
<a href="https://www.spinellis.gr/blog/20250411/index.html">The perils of GenAI student submissions</a> (2025-04-11)<br />
|
||
<a href="https://www.spinellis.gr/blog/20241015/index.html">Unix make vs Apache Airflow</a> (2024-10-15)<br />
|
||
</div> <!-- panel body -->
|
||
</div> <!-- panel -->
|
||
|
||
</div> <!-- Right content -->
|
||
</div>
|
||
|
||
<div class='row'> <!-- HR -->
|
||
<div class='span12'>
|
||
<hr />
|
||
</div>
|
||
</div>
|
||
<div class='row'> <!-- Modification time -->
|
||
<div class='col-sm-10'>
|
||
<p class="small">
|
||
Last modified: Saturday, May 23, 2026 9:53 pm
|
||
</p>
|
||
</div>
|
||
</div>
|
||
<div class='row'> <!-- License BEGIN -->
|
||
|
||
<div class='col-sm-2'>
|
||
<!--Creative Commons License logo-->
|
||
<a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/"><img alt="Creative Commons Licence BY NC" style="border-width:0;height:5ex" src="/a/by-nc.eu.png" /></a>
|
||
</div>
|
||
<div class='col-sm-10'>
|
||
<p class="small">
|
||
Unless otherwise expressly stated, all original material on this page created by Diomidis Spinellis is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">Creative Commons Attribution-NonCommercial 4.0 International License</a>.
|
||
</p>
|
||
<br />
|
||
</div>
|
||
|
||
</div> <!-- License END -->
|
||
</div> <!-- container -->
|
||
</body>
|
||
</html>
|