Files
nexus/sreweekly/articles/119/07-how-to-get-a-core-dump-for-a-segfault-on-linux.html
2026-09-12 17:23:01 +08:00

388 lines
21 KiB
HTML

<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta charset="utf-8">
<title>How to get a core dump for a segfault on Linux</title>
<meta name="author" content="Julia Evans">
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
<meta name="description" content="How to get a core dump for a segfault on Linux">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:title" content='How to get a core dump for a segfault on Linux'>
<meta property="og:type" content="website" />
<meta property="og:url" content="https://jvns.ca/blog/2018/04/28/debugging-a-segfault-on-linux/" />
<meta property="og:site_name" content="Julia Evans" />
<link rel="canonical" href="https://jvns.ca/blog/2018/04/28/debugging-a-segfault-on-linux/">
<link href="/favicon.ico" rel="icon">
<link href="/stylesheets/screen.css" rel="preload" type="text/css" as="style">
<link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
<link href="/stylesheets/print.css" media="print" rel="stylesheet" type="text/css">
<link href="/atom.xml" rel="alternate" title="Julia Evans" type="application/atom+xml">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.css" integrity="sha384-vKruj+a13U8yHIkAyGgK1J3ArTLzrFGBbBc0tDp4ad/EyewESeXE/Iv67Aj8gKZ0" crossorigin="anonymous">
<script defer data-domain="jvns.ca" src="https://plausible.io/js/script.js"></script>
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.js" integrity="sha384-PwRUT/YqbnEjkZO0zZxNqcxACrXe+j766U2amXcgMg5457rve2Y7I6ZJSm2A0mS4" crossorigin="anonymous"></script>
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/contrib/auto-render.min.js" integrity="sha384-+VBxd3r6XgURycqtZ117nYw44OOcIax56Z4dCRWbxyPt0Koah1uHoK0o4+/RRE05" crossorigin="anonymous" onload="renderMathInElement(document.body);"></script>
<script defer type="text/javascript">
window.heap=window.heap||[],heap.load=function(e,t){window.heap.appid=e,window.heap.config=t=t||{};var r=document.createElement("script");r.type="text/javascript",r.async=!0,r.src="https://cdn.heapanalytics.com/js/heap-"+e+".js";var a=document.getElementsByTagName("script")[0];a.parentNode.insertBefore(r,a);for(var n=function(e){return function(){heap.push([e].concat(Array.prototype.slice.call(arguments,0)))}},p=["addEventProperties","addUserProperties","clearEventProperties","identify","resetIdentity","removeEventProperty","setEventProperties","track","unsetEventProperty"],o=0;o<p.length;o++)heap[p[o]]=n(p[o])};
heap.load("2242143965");
</script>
</head>
<body>
<div id="skiptocontent">
<a href="#main">Skip to main content</a>
</div>
<div id="wrap">
<header role="banner">
<hgroup>
<h1><a href="/">Julia Evans</a></h1>
</hgroup>
<ul class="header-links">
<li><a href="/about">About</a></li>
<li><a href="/talks">Talks</a></li>
<li><a href="/projects/">Projects</a></li>
<li><a rel="me" href="https://social.jvns.ca/@b0rk">Mastodon</a></li>
<li><a href="https://bsky.app/profile/b0rk.jvns.ca">Bluesky</a></li>
<li><a href="https://github.com/jvns">Github</a></li>
</ul>
</header>
<nav role="navigation" class="header-nav"><ul class="main-navigation">
<li><a href="/categories/favorite/">Favorites</a></li>
<li><a href="/til/">TIL</a></li>
<li><a href="https://wizardzines.com">Zines</a></li>
<li class="subscription" data-subscription="rss"><a href="/atom.xml" rel="subscribe-rss" title="subscribe via RSS">RSS</a></li>
</ul>
</nav>
<div id="main">
<div id="content">
<div>
<article class="hentry" role="article">
<header>
<h1 class="entry-title">How to get a core dump for a segfault on Linux</h1>
<div class="post-tags">
</div>
<p class="meta sans">
<time class="date" datetime="2018-04-28T13:18:30" pubdate data-updated="true">
April 28, 2018
</time>
</p>
</header>
<main>
<p>This week at work I spent all week trying to debug a segfault. I&rsquo;d never done this before, and some
of the basic things involved (get a core dump! find the line number that segfaulted!) took me a long
time to figure out. So here&rsquo;s a blog post explaining how to do those things!</p>
<p>At the end of this blog post, you should know how to go from &ldquo;oh no my program
is segfaulting and I have no idea what is happening&rdquo; to &ldquo;well I know what its
stack / line number was when it segfaulted, at least!&rdquo;.</p>
<h3 id="what-s-a-segfault" class="post-heading">
<a href="#what-s-a-segfault">
what&rsquo;s a segfault?
</a>
</h3>
<p>A &ldquo;segmentation fault&rdquo; is when your program tries to access memory that it&rsquo;s not allowed to access,
or tries to .
This can be caused by:</p>
<ul>
<li>trying to dereference a null pointer (you&rsquo;re not allowed to access the memory address <code>0</code>)</li>
<li>trying to dereference some other pointer that isn&rsquo;t in your memory</li>
<li>a C++ vtable pointer that got corrupted and is pointing to the wrong place, which causes the
program to try to execute some memory that isn&rsquo;t executable</li>
<li>some other things that I don&rsquo;t understand, like I think misaligned memory accesses can also
segfault</li>
</ul>
<p>This &ldquo;C++ vtable pointer&rdquo; thing is what was happening to my segfaulting program. I might explain that
in a future blog post because I didn&rsquo;t know any C++ at the beginning of this week and this vtable
lookup thing was a new way for a program to segfault that I didn&rsquo;t know about.</p>
<p>But! This blog post isn&rsquo;t about C++ bugs. Let&rsquo;s talk about the basics, like, how do we even get a
core dump?</p>
<h3 id="step-1-run-valgrind" class="post-heading">
<a href="#step-1-run-valgrind">
step 1: run valgrind
</a>
</h3>
<p>I found the easiest way to figure out why my program is segfaulting was to use valgrind: I ran</p>
<pre><code>valgrind -v your-program
</code></pre>
<p>and this gave me a stack trace of what happened. Neat!</p>
<p>But I also wanted to do a more in-depth investigation and find out more than just what
valgrind was telling me! So I wanted to get a core dump and explore it.</p>
<h3 id="how-to-get-a-core-dump" class="post-heading">
<a href="#how-to-get-a-core-dump">
How to get a core dump
</a>
</h3>
<p>A <strong>core dump</strong> is a copy of your program&rsquo;s memory, and it&rsquo;s useful when you&rsquo;re trying to debug what
went wrong with your problematic program.</p>
<p>When your program segfaults, the Linux kernel will sometimes write a core dump to disk. When I
originally tried to get a core dump, I was pretty frustrated for a long time because &ndash; Linux wasn&rsquo;t
writing a core dump!! Where was my core dump????</p>
<p>Here&rsquo;s what I ended up doing:</p>
<ol>
<li>Run <code>ulimit -c unlimited</code> before starting my program</li>
<li>Run <code>sudo sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t</code></li>
</ol>
<h3 id="ulimit-set-the-max-size-of-a-core-dump" class="post-heading">
<a href="#ulimit-set-the-max-size-of-a-core-dump">
ulimit: set the max size of a core dump
</a>
</h3>
<p><code>ulimit -c</code> sets the <strong>maximum size of a core dump</strong>. It&rsquo;s often set to 0, which means that the
kernel won&rsquo;t write core dumps at all. It&rsquo;s in kilobytes. ulimits are <strong>per process</strong> &ndash; you can see
a process&rsquo;s limits by running <code>cat /proc/PID/limit</code></p>
<p>For example these are the limits for a random Firefox process on my system:</p>
<pre><code>$ cat /proc/6309/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 30571 30571 processes
Max open files 1024 1048576 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 30571 30571 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
</code></pre>
<p>The kernel uses the <strong>soft limit</strong> (in this case, &ldquo;max core file size = 0&rdquo;) when deciding how big of
a core file to write. You can increase the soft limit up to the hard limit using the <code>ulimit</code> shell
builtin (<code>ulimit -c unlimited</code>!)</p>
<h3 id="kernel-core-pattern-where-core-dumps-are-written" class="post-heading">
<a href="#kernel-core-pattern-where-core-dumps-are-written">
kernel.core_pattern: where core dumps are written
</a>
</h3>
<p><code>kernel.core_pattern</code> is a kernel parameter or a &ldquo;sysctl setting&rdquo; that controls where the Linux
kernel writes core dumps to disk.</p>
<p>Kernel parameters are a way to set <strong>global</strong> settings on your system. You can get a list of every
kernel parameter by running <code>sysctl -a</code>, or use <code>sysctl kernel.core_pattern</code> to look at the
<code>kernel.core_pattern</code> setting specifically.</p>
<p>So <code>sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t</code> will write core dumps to <code>/tmp/core-&lt;a bunch of stuff identifying the process&gt;</code></p>
<p>If you want to know more about what these <code>%e</code>, <code>%p</code> parameters read, see <a href="http://man7.org/linux/man-pages/man5/core.5.html">man core</a>.</p>
<p>It&rsquo;s important to know that <code>kernel.core_pattern</code> is a global settings &ndash; it&rsquo;s good to be a little
careful about changing it because it&rsquo;s possible that other systems depend on it being set a certain
way.</p>
<h3 id="kernel-core-pattern-ubuntu" class="post-heading">
<a href="#kernel-core-pattern-ubuntu">
kernel.core_pattern &amp; Ubuntu
</a>
</h3>
<p>By default on Ubuntu systems, this is what <code>kernel.core_pattern</code> is set to</p>
<pre><code>$ sysctl kernel.core_pattern
kernel.core_pattern = |/usr/share/apport/apport %p %s %c %d %P
</code></pre>
<p>This caused me a lot of confusion (what is this apport thing and what is it doing with my core
dumps??) so here&rsquo;s what I learned about this:</p>
<ul>
<li>Ubuntu uses a system called &ldquo;apport&rdquo; to report crashes in apt packages</li>
<li>Setting <code>kernel.core_pattern=|/usr/share/apport/apport %p %s %c %d %P</code> means that core dumps will
be piped to <code>apport</code></li>
<li>apport has logs in /var/log/apport.log</li>
<li>apport by default will ignore crashes from binaries that aren&rsquo;t part of an Ubuntu packages</li>
</ul>
<p>I ended up just overriding this Apport business and setting <code>kernel.core_pattern</code> to <code>sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t</code> because I was on a dev machine, I didn&rsquo;t care whether Apport was working on not, and I didn&rsquo;t feel like trying to convince Apport to give me my core dumps.</p>
<h3 id="so-you-have-a-core-dump-now-what" class="post-heading">
<a href="#so-you-have-a-core-dump-now-what">
So you have a core dump. Now what?
</a>
</h3>
<p>Okay, now we know about ulimits and <code>kernel.core_pattern</code> and you have actually have a core dump
file on disk in <code>/tmp</code>. Amazing! Now what??? We still don&rsquo;t know why the program segfaulted!</p>
<p>The next step is to open the core file with <code>gdb</code> and get a backtrace.</p>
<h3 id="getting-a-backtrace-from-gdb" class="post-heading">
<a href="#getting-a-backtrace-from-gdb">
Getting a backtrace from gdb
</a>
</h3>
<p>You can open a core file with gdb like this:</p>
<pre><code>$ gdb -c my_core_file
</code></pre>
<p>or maybe</p>
<pre><code>$ gdb executable -c my_core_file
</code></pre>
<p>Next, we want to know what the stack was when the program crashed. Running <code>bt</code> at
the gdb prompt will give you a backtrace. In my case gdb hadn&rsquo;t loaded symbols for the binary, so it
was just like <code>??????</code>. Luckily, loading symbols fixed it.</p>
<p>Here&rsquo;s how to load debugging symbols.</p>
<pre><code>symbol-file /path/to/my/binary
sharedlibrary
</code></pre>
<p>This loads symbols from the binary and from any shared libraries the binary uses. Once I did that,
gdb gave me a beautiful stack trace with line numbers when I ran <code>bt</code>!!!</p>
<p>If you want this to work, the binary should be compiled with debugging symbols. Having line numbers
in your stack traces is extremely helpful when trying to figure out why a program crashed :)</p>
<h3 id="look-at-the-stack-for-every-thread" class="post-heading">
<a href="#look-at-the-stack-for-every-thread">
look at the stack for every thread
</a>
</h3>
<p>Here&rsquo;s how to get the stack for every thread in gdb!</p>
<pre><code>thread apply all bt full
</code></pre>
<h3 id="gdb-core-dumps-amazing" class="post-heading">
<a href="#gdb-core-dumps-amazing">
gdb + core dumps = amazing
</a>
</h3>
<p>If you have a core dump &amp; debugging symbols and gdb, you are in an amazing situation!! You can go up
and down the call stack, print out variables, and poke around in memory to see what happened. It&rsquo;s
the best.</p>
<p>If you are still working on being a gdb wizard, you can also just print out the stack trace with
<code>bt</code> and that&rsquo;s okay :)</p>
<h3 id="asan" class="post-heading">
<a href="#asan">
ASAN
</a>
</h3>
<p>Another path to figuring out your segfault is to do one compile the program with AddressSanitizer
(&ldquo;ASAN&rdquo;) (<code>$CC -fsanitize=address</code>) and run it. I&rsquo;m not going to discuss that in this post because
this is already pretty long and anyway in my case the segfault disappeared with ASAN turned on for
some reason, possibly because the ASAN build used a different memory allocator (system malloc
instead of tcmalloc).</p>
<p>I might write about ASAN more in the future if I ever get it to work :)</p>
<h3 id="getting-a-stack-trace-from-a-core-dump-is-pretty-approachable" class="post-heading">
<a href="#getting-a-stack-trace-from-a-core-dump-is-pretty-approachable">
getting a stack trace from a core dump is pretty approachable!
</a>
</h3>
<p>This blog post sounds like a lot and I was pretty confused when I was doing it but really there
aren&rsquo;t all that many steps to getting a stack trace out of a segfaulting program:</p>
<ol>
<li>try valgrind</li>
</ol>
<p>if that doesn&rsquo;t work, or if you want to have a core dump to investigate:</p>
<ol>
<li>make sure the binary is compiled with debugging symbols</li>
<li>set <code>ulimit</code> and <code>kernel.core_pattern</code> correctly</li>
<li>run the program</li>
<li>open your core dump with <code>gdb</code>, load the symbols, and run <code>bt</code></li>
<li>try to figure out what happened!!</li>
</ol>
<p>I was able using gdb to figure out that there was a C++ vtable entry that is pointing to some
corrupt memory, which was somewhat helpful and helped me feel like I understood C++ a bit better.
Maybe we&rsquo;ll talk more about how to use gdb to figure things out another day!</p>
</main>
<footer>
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; display: inline;}
#mc_embed_signup {
display: inline;
}
#mc_embed_signup input.button {
background: #ff5e00;
display: inline;
color: white;
padding: 6px 12px;
}
</style>
<div class="sharing">
<style>
.form-inline {
display:flex; flex-flow: row wrap; justify-content: center;
}
.form-inline input, .form-inline span {
padding: 10px;
}
.form-inline input {
display:inline;
max-width:30%;
margin: 0 10px 0 0;
background-color: #fff;
border: 1px solid #ddd;
border-radius: 5px;
padding: 10px;
}
button {
background-color: #f50;
box-shadow: none;
border: 0;
border-radius: 5px;
color: white;
padding: 5px 10px;
}
@media (max-width: 800px) {
.form-inline input {
margin: 10px 0;
max-width:100% !important;
}
.form-inline {
flex-direction: column;
align-items: stretch;
}
}
</style>
<div align="center">
<form class="form-inline" action="https://app.convertkit.com/forms/1052396/subscriptions" method="post" data-uid="8884355abb" data-format="inline" data-version="5">
<span> Want a weekly digest of this blog?</span>
<input name="email_address" type="text" placeholder="Email address" />
<button type="submit" data-element="submit">Subscribe</button>
</form>
</div>
</div>
<p class="meta">
<a class="basic-alignment left" href="https://jvns.ca/blog/2018/04/16/new-perf-zine/" title="Previous Post: New zine: Profiling &amp; tracing with perf!!">New zine: Profiling &amp; tracing with perf!!</a>
<a class="basic-alignment right" href="https://jvns.ca/blog/2018/05/11/batch-editing-files-with-ed/" title="Next Post: Batch editing files with ed">Batch editing files with ed</a>
</p>
</footer>
</article>
</div>
</div>
</div>
<nav role="navigation" class="footer-nav"> <a href="/">Archives</a>
</nav>
<footer role="contentinfo"><span class="credit">&copy; Julia Evans. </span>
<span>If you like this, you may like <a href="https://web.archive.org/web/20181228051203/http://www.uliaea.ca/">Ulia Ea</a> or, more seriously, this list of <a href="https://jvns.ca/blogroll">blogs I love</a> or some <a href="https://jvns.ca/bookshelf">books I've read</a>. <br>
<p class="rc-scout__text"><i class="rc-scout__logo"></i>
You might also like the <a class="rc-scout__link" href="https://www.recurse.com/scout/click?t=546ea46360584b522270b8c3e5d830f8">Recurse Center</a>, my very favorite programming community <a href="/categories/hackerschool/">(my posts about it)</a></p>
</span>
<style class="rc-scout__style" type="text/css">.rc-scout{display:block;padding:0;border:0;margin:0;}.rc-scout__text{display:block;padding:0;border:0;margin:0;height:100%;font-size:100%;}.rc-scout__logo{display:inline-block;padding:0;border:0;margin:0;width:0.85em;height:0.85em;background:no-repeat center url('data:image/svg+xml;utf8,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2012%2015%22%3E%3Crect%20x%3D%220%22%20y%3D%220%22%20width%3D%2212%22%20height%3D%2210%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%221%22%20width%3D%2210%22%20height%3D%228%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%222%22%20width%3D%228%22%20height%3D%226%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%229%22%20width%3D%224%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%2211%22%20width%3D%2210%22%20height%3D%224%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%220%22%20y%3D%2212%22%20width%3D%2212%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%225%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%227%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%228%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%229%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3C%2Fsvg%3E');}.rc-scout__link:link,.rc-scout__link:visited{color:#61ae24;text-decoration:underline;}.rc-scout__link:hover,.rc-scout__link:active{color:#4e8b1d;}</style>
</footer>
<script type="text/rocketscript">
(function(){
var twitterWidgets = document.createElement('script');
twitterWidgets.type = 'text/javascript';
twitterWidgets.async = true;
twitterWidgets.src = 'http://platform.twitter.com/widgets.js';
document.getElementsByTagName('head')[0].appendChild(twitterWidgets);
})();
</script>
</div>
</body>
</html>