282 lines
14 KiB
HTML
282 lines
14 KiB
HTML
<!DOCTYPE html>
|
|
|
|
|
|
<html class="no-js" lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<title>Finding out if/why a server is dropping packets</title>
|
|
<meta name="author" content="Julia Evans">
|
|
<meta name="HandheldFriendly" content="True">
|
|
<meta name="MobileOptimized" content="320">
|
|
<meta name="description" content="Finding out if/why a server is dropping packets">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
|
|
<meta property="og:title" content='Finding out if/why a server is dropping packets'>
|
|
<meta property="og:type" content="website" />
|
|
<meta property="og:url" content="https://jvns.ca/blog/2017/09/05/finding-out-where-packets-are-being-dropped/" />
|
|
<meta property="og:site_name" content="Julia Evans" />
|
|
|
|
<link rel="canonical" href="https://jvns.ca/blog/2017/09/05/finding-out-where-packets-are-being-dropped/">
|
|
<link href="/favicon.ico" rel="icon">
|
|
|
|
<link href="/stylesheets/screen.css" rel="preload" type="text/css" as="style">
|
|
|
|
<link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
|
|
<link href="/stylesheets/print.css" media="print" rel="stylesheet" type="text/css">
|
|
|
|
|
|
<link href="/atom.xml" rel="alternate" title="Julia Evans" type="application/atom+xml">
|
|
|
|
|
|
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.css" integrity="sha384-vKruj+a13U8yHIkAyGgK1J3ArTLzrFGBbBc0tDp4ad/EyewESeXE/Iv67Aj8gKZ0" crossorigin="anonymous">
|
|
<script defer data-domain="jvns.ca" src="https://plausible.io/js/script.js"></script>
|
|
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.js" integrity="sha384-PwRUT/YqbnEjkZO0zZxNqcxACrXe+j766U2amXcgMg5457rve2Y7I6ZJSm2A0mS4" crossorigin="anonymous"></script>
|
|
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/contrib/auto-render.min.js" integrity="sha384-+VBxd3r6XgURycqtZ117nYw44OOcIax56Z4dCRWbxyPt0Koah1uHoK0o4+/RRE05" crossorigin="anonymous" onload="renderMathInElement(document.body);"></script>
|
|
|
|
<script defer type="text/javascript">
|
|
window.heap=window.heap||[],heap.load=function(e,t){window.heap.appid=e,window.heap.config=t=t||{};var r=document.createElement("script");r.type="text/javascript",r.async=!0,r.src="https://cdn.heapanalytics.com/js/heap-"+e+".js";var a=document.getElementsByTagName("script")[0];a.parentNode.insertBefore(r,a);for(var n=function(e){return function(){heap.push([e].concat(Array.prototype.slice.call(arguments,0)))}},p=["addEventProperties","addUserProperties","clearEventProperties","identify","resetIdentity","removeEventProperty","setEventProperties","track","unsetEventProperty"],o=0;o<p.length;o++)heap[p[o]]=n(p[o])};
|
|
heap.load("2242143965");
|
|
</script>
|
|
</head>
|
|
<body>
|
|
<div id="skiptocontent">
|
|
<a href="#main">Skip to main content</a>
|
|
</div>
|
|
<div id="wrap">
|
|
<header role="banner">
|
|
<hgroup>
|
|
<h1><a href="/">Julia Evans</a></h1>
|
|
</hgroup>
|
|
<ul class="header-links">
|
|
<li><a href="/about">About</a></li>
|
|
<li><a href="/talks">Talks</a></li>
|
|
<li><a href="/projects/">Projects</a></li>
|
|
<li><a rel="me" href="https://social.jvns.ca/@b0rk">Mastodon</a></li>
|
|
<li><a href="https://bsky.app/profile/b0rk.jvns.ca">Bluesky</a></li>
|
|
<li><a href="https://github.com/jvns">Github</a></li>
|
|
</ul>
|
|
</header>
|
|
<nav role="navigation" class="header-nav"><ul class="main-navigation">
|
|
<li><a href="/categories/favorite/">Favorites</a></li>
|
|
<li><a href="/til/">TIL</a></li>
|
|
<li><a href="https://wizardzines.com">Zines</a></li>
|
|
<li class="subscription" data-subscription="rss"><a href="/atom.xml" rel="subscribe-rss" title="subscribe via RSS">RSS</a></li>
|
|
</ul>
|
|
</nav>
|
|
<div id="main">
|
|
<div id="content">
|
|
|
|
|
|
<div>
|
|
<article class="hentry" role="article">
|
|
<header>
|
|
<h1 class="entry-title">Finding out if/why a server is dropping packets</h1>
|
|
|
|
<div class="post-tags">
|
|
|
|
</div>
|
|
<p class="meta sans">
|
|
<time class="date" datetime="2017-09-05T23:39:28" pubdate data-updated="true">
|
|
|
|
September 5, 2017
|
|
|
|
</time>
|
|
</p>
|
|
</header>
|
|
<main>
|
|
<p>When packets are being dropped on a computer, they’re being dropped for
|
|
a <em>reason</em>. How do you find out whether/why packets are being dropped?</p>
|
|
<p>Here’s the situations we want to understand:</p>
|
|
<ol>
|
|
<li>a packet enters the network stack of your computer (<code>RX</code>) (say on
|
|
port 8000). It gets dropped before the application listening on port
|
|
8000 receives it.</li>
|
|
<li>you send a packet (<code>TX</code>). Before it makes it out of your computer, it
|
|
gets dropped.</li>
|
|
</ol>
|
|
<p>I’m not interested here in “packets are being dropped somewhere else on the
|
|
internet, let’s diagnose it with traceroute / by counting TCP retransmits”
|
|
(though that’s important too!)</p>
|
|
<h3 id="how-do-you-even-know-if-packets-are-being-dropped" class="post-heading">
|
|
<a href="#how-do-you-even-know-if-packets-are-being-dropped">
|
|
how do you even know if packets are being dropped?
|
|
</a>
|
|
</h3>
|
|
<p>I asked on Twitter and got the very useful answer “look at <code>netstat -i</code>!”
|
|
Here’s what that looks like on my laptop:</p>
|
|
<pre><code>bork@kiwi~> sudo netstat -i
|
|
Kernel Interface table
|
|
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
|
|
docker0 1500 0 0 0 0 0 0 0 0 0 BMU
|
|
enp0s25 1500 0 1235101 0 242 0 745760 0 0 0 BMRU
|
|
lo 65536 0 21558 0 0 0 21558 0 0 0 LRU
|
|
nlmon0 3776 0 551262 0 0 0 0 0 0 0 ORU
|
|
</code></pre>
|
|
<p>It looks like there are some received (<code>RX</code>) packets being lost on <code>enp0s25</code>
|
|
(my wireless card). No <code>TX</code> packets lost though!</p>
|
|
<p>Someone also told me that running <code>ethtool -S</code> is useful but my ethtool doesn’t have a <code>-S</code> option.</p>
|
|
<h3 id="how-do-you-know-why-packets-are-being-dropped" class="post-heading">
|
|
<a href="#how-do-you-know-why-packets-are-being-dropped">
|
|
how do you know <strong>why</strong> packets are being dropped
|
|
</a>
|
|
</h3>
|
|
<p>I was googling and I found this cool tool called <code>dropwatch</code>. There isn’t any Ubuntu package for it but it’s on github: <a href="https://github.com/pavel-odintsov/drop_watch">https://github.com/pavel-odintsov/drop_watch</a>.</p>
|
|
<p>Here are the instructions that worked for me to compile it:</p>
|
|
<pre><code>sudo apt-get install -y libnl-3-dev libnl-genl-3-dev binutils-dev libreadline6-dev
|
|
git clone https://github.com/pavel-odintsov/drop_watch
|
|
cd drop_watch/src
|
|
vim Makefile # comment out the -Werror argument to gcc
|
|
make
|
|
</code></pre>
|
|
<p>And here’s the output! It tells me at which kernel function I’m losing packets. Cool!</p>
|
|
<pre><code>sudo ./dropwatch -l kas
|
|
Initalizing kallsyms db
|
|
dropwatch> start
|
|
Enabling monitoring...
|
|
Kernel monitoring activated.
|
|
Issue Ctrl-C to stop monitoring
|
|
|
|
1 drops at tcp_v4_do_rcv+cd (0xffffffff81799bad)
|
|
10 drops at tcp_v4_rcv+80 (0xffffffff8179a620)
|
|
1 drops at sk_stream_kill_queues+57 (0xffffffff81729ca7)
|
|
4 drops at unix_release_sock+20e (0xffffffff817dc94e)
|
|
1 drops at igmp_rcv+e1 (0xffffffff817b4c41)
|
|
1 drops at igmp_rcv+e1 (0xffffffff817b4c41)
|
|
</code></pre>
|
|
<h3 id="monitoring-dropped-packets-with-perf" class="post-heading">
|
|
<a href="#monitoring-dropped-packets-with-perf">
|
|
monitoring dropped packets with perf
|
|
</a>
|
|
</h3>
|
|
<p>Here’s another cool way to debug what’s happening!</p>
|
|
<p><a href="https://twitter.com/tgraf__">thomas graf</a> told me that you can monitor the
|
|
<code>kfree_skb</code> event using perf, and that will tell you when packets are being
|
|
dropped (where in the kernel stack it happened):</p>
|
|
<pre><code>sudo perf record -g -a -e skb:kfree_skb
|
|
sudo perf script
|
|
</code></pre>
|
|
<h3 id="advanced-reading" class="post-heading">
|
|
<a href="#advanced-reading">
|
|
advanced reading
|
|
</a>
|
|
</h3>
|
|
<p>There’s also these two cool articles:</p>
|
|
<ul>
|
|
<li><a href="https://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/">Monitoring and Tuning the Linux Networking Stack: Receiving Data</a></li>
|
|
<li><a href="https://blog.packagecloud.io/eng/2017/02/06/monitoring-tuning-linux-networking-stack-sending-data/">Monitoring and Tuning the Linux Networking Stack: Sending Data</a></li>
|
|
</ul>
|
|
<p>I still haven’t read them in full but they are extremely detailed and cool.</p>
|
|
<h3 id="tell-me-if-you-know-more" class="post-heading">
|
|
<a href="#tell-me-if-you-know-more">
|
|
tell me if you know more!
|
|
</a>
|
|
</h3>
|
|
<p>I still haven’t ever used these tools to seriously debug a packet loss problem
|
|
yet, just wanted to write this down so that I have the notes if I do want to in
|
|
the future!</p>
|
|
<p>If you have better tips for debugging whether/why packets are being dropped on
|
|
a computer, let me know!</p>
|
|
|
|
</main>
|
|
|
|
<footer>
|
|
|
|
<style type="text/css">
|
|
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; display: inline;}
|
|
#mc_embed_signup {
|
|
display: inline;
|
|
}
|
|
#mc_embed_signup input.button {
|
|
background: #ff5e00;
|
|
display: inline;
|
|
color: white;
|
|
padding: 6px 12px;
|
|
}
|
|
</style>
|
|
<div class="sharing">
|
|
|
|
<style>
|
|
.form-inline {
|
|
display:flex; flex-flow: row wrap; justify-content: center;
|
|
}
|
|
.form-inline input, .form-inline span {
|
|
padding: 10px;
|
|
}
|
|
.form-inline input {
|
|
display:inline;
|
|
max-width:30%;
|
|
margin: 0 10px 0 0;
|
|
background-color: #fff;
|
|
border: 1px solid #ddd;
|
|
border-radius: 5px;
|
|
padding: 10px;
|
|
}
|
|
button {
|
|
background-color: #f50;
|
|
box-shadow: none;
|
|
border: 0;
|
|
border-radius: 5px;
|
|
color: white;
|
|
padding: 5px 10px;
|
|
}
|
|
@media (max-width: 800px) {
|
|
.form-inline input {
|
|
margin: 10px 0;
|
|
max-width:100% !important;
|
|
}
|
|
.form-inline {
|
|
flex-direction: column;
|
|
align-items: stretch;
|
|
}
|
|
}
|
|
</style>
|
|
|
|
<div align="center">
|
|
<form class="form-inline" action="https://app.convertkit.com/forms/1052396/subscriptions" method="post" data-uid="8884355abb" data-format="inline" data-version="5">
|
|
<span> Want a weekly digest of this blog?</span>
|
|
<input name="email_address" type="text" placeholder="Email address" />
|
|
<button type="submit" data-element="submit">Subscribe</button>
|
|
</form>
|
|
</div>
|
|
|
|
|
|
</div>
|
|
|
|
<p class="meta">
|
|
|
|
<a class="basic-alignment left" href="https://jvns.ca/blog/2017/09/03/debugging-netlink-requests/" title="Previous Post: Debugging netlink requests">Debugging netlink requests</a>
|
|
|
|
|
|
<a class="basic-alignment right" href="https://jvns.ca/blog/2017/09/09/data-structure--the-treap-/" title="Next Post: Data structure: the treap!">Data structure: the treap!</a>
|
|
|
|
</p>
|
|
</footer>
|
|
|
|
</article>
|
|
</div>
|
|
|
|
</div>
|
|
</div>
|
|
<nav role="navigation" class="footer-nav"> <a href="/">Archives</a>
|
|
</nav>
|
|
<footer role="contentinfo"><span class="credit">© Julia Evans. </span>
|
|
<span>If you like this, you may like <a href="https://web.archive.org/web/20181228051203/http://www.uliaea.ca/">Ulia Ea</a> or, more seriously, this list of <a href="https://jvns.ca/blogroll">blogs I love</a> or some <a href="https://jvns.ca/bookshelf">books I've read</a>. <br>
|
|
<p class="rc-scout__text"><i class="rc-scout__logo"></i>
|
|
You might also like the <a class="rc-scout__link" href="https://www.recurse.com/scout/click?t=546ea46360584b522270b8c3e5d830f8">Recurse Center</a>, my very favorite programming community <a href="/categories/hackerschool/">(my posts about it)</a></p>
|
|
</span>
|
|
<style class="rc-scout__style" type="text/css">.rc-scout{display:block;padding:0;border:0;margin:0;}.rc-scout__text{display:block;padding:0;border:0;margin:0;height:100%;font-size:100%;}.rc-scout__logo{display:inline-block;padding:0;border:0;margin:0;width:0.85em;height:0.85em;background:no-repeat center url('data:image/svg+xml;utf8,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2012%2015%22%3E%3Crect%20x%3D%220%22%20y%3D%220%22%20width%3D%2212%22%20height%3D%2210%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%221%22%20width%3D%2210%22%20height%3D%228%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%222%22%20width%3D%228%22%20height%3D%226%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%223%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%225%22%20width%3D%222%22%20height%3D%221%22%20fill%3D%22%2361ae24%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%229%22%20width%3D%224%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%221%22%20y%3D%2211%22%20width%3D%2210%22%20height%3D%224%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%220%22%20y%3D%2212%22%20width%3D%2212%22%20height%3D%223%22%20fill%3D%22%23000%22%3E%3C%2Frect%3E%3Crect%20x%3D%222%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%223%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%224%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%225%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%226%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%227%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%228%22%20y%3D%2213%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3Crect%20x%3D%229%22%20y%3D%2212%22%20width%3D%221%22%20height%3D%221%22%20fill%3D%22%23fff%22%3E%3C%2Frect%3E%3C%2Fsvg%3E');}.rc-scout__link:link,.rc-scout__link:visited{color:#61ae24;text-decoration:underline;}.rc-scout__link:hover,.rc-scout__link:active{color:#4e8b1d;}</style>
|
|
</footer>
|
|
<script type="text/rocketscript">
|
|
(function(){
|
|
var twitterWidgets = document.createElement('script');
|
|
twitterWidgets.type = 'text/javascript';
|
|
twitterWidgets.async = true;
|
|
twitterWidgets.src = 'http://platform.twitter.com/widgets.js';
|
|
document.getElementsByTagName('head')[0].appendChild(twitterWidgets);
|
|
})();
|
|
</script>
|
|
</div>
|
|
</body>
|
|
</html>
|
|
|