# Finding out if/why a server is dropping packets

- **期号**: SRE Weekly Issue #88（2017-09-10）
- **作者**: —
- **链接**: https://jvns.ca/blog/2017/09/05/finding-out-where-packets-are-being-dropped/

## 简介

Julia Evans just blew my mind (once again). In this article, among other things, she links to a tool that tells you which function in the kernel dropped a packet. I’ve been wishing for such a tool for years!

## 正文

# Finding out if/why a server is dropping packets

When packets are being dropped on a computer, they’re being dropped for
a *reason*. How do you find out whether/why packets are being dropped?

Here’s the situations we want to understand:

1. a packet enters the network stack of your computer (`RX` ) (say on
port 8000). It gets dropped before the application listening on port
8000 receives it.
2. you send a packet (`TX` ). Before it makes it out of your computer, it
gets dropped.

I’m not interested here in “packets are being dropped somewhere else on the internet, let’s diagnose it with traceroute / by counting TCP retransmits” (though that’s important too!)

### 
    
      [how do you even know if packets are being dropped?](https://jvns.ca#how-do-you-even-know-if-packets-are-being-dropped)
    

I asked on Twitter and got the very useful answer “look at `netstat -i`!”
Here’s what that looks like on my laptop:

```
bork@kiwi~> sudo netstat -i
Kernel Interface table
Iface       MTU Met   RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
docker0    1500 0         0      0      0 0             0      0      0      0 BMU
enp0s25    1500 0   1235101      0    242 0        745760      0      0      0 BMRU
lo        65536 0     21558      0      0 0         21558      0      0      0 LRU
nlmon0     3776 0    551262      0      0 0             0      0      0      0 ORU
```
It looks like there are some received (`RX`) packets being lost on `enp0s25`
(my wireless card). No `TX` packets lost though!

Someone also told me that running `ethtool -S` is useful but my ethtool doesn’t have a `-S` option.

### 
    
      [how do you know **why** packets are being dropped](https://jvns.ca#how-do-you-know-why-packets-are-being-dropped)
    

**why**packets are being dropped

I was googling and I found this cool tool called `dropwatch`. There isn’t any Ubuntu package for it but it’s on github: [https://github.com/pavel-odintsov/drop_watch](https://github.com/pavel-odintsov/drop_watch).

Here are the instructions that worked for me to compile it:

```
sudo apt-get install -y libnl-3-dev libnl-genl-3-dev binutils-dev libreadline6-dev
git clone https://github.com/pavel-odintsov/drop_watch
cd drop_watch/src
vim Makefile # comment out the -Werror argument to gcc
make
```
And here’s the output! It tells me at which kernel function I’m losing packets. Cool!

```
sudo ./dropwatch -l kas
Initalizing kallsyms db
dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
1 drops at tcp_v4_do_rcv+cd (0xffffffff81799bad)
10 drops at tcp_v4_rcv+80 (0xffffffff8179a620)
1 drops at sk_stream_kill_queues+57 (0xffffffff81729ca7)
4 drops at unix_release_sock+20e (0xffffffff817dc94e)
1 drops at igmp_rcv+e1 (0xffffffff817b4c41)
1 drops at igmp_rcv+e1 (0xffffffff817b4c41)
```
### 
    
      [monitoring dropped packets with perf](https://jvns.ca#monitoring-dropped-packets-with-perf)
    

Here’s another cool way to debug what’s happening!

[thomas graf](https://twitter.com/tgraf__) told me that you can monitor the
`kfree_skb` event using perf, and that will tell you when packets are being
dropped (where in the kernel stack it happened):

```
sudo perf record -g -a -e skb:kfree_skb
sudo perf script
```
### 
    
      [advanced reading](https://jvns.ca#advanced-reading)
    

There’s also these two cool articles:

- [Monitoring and Tuning the Linux Networking Stack: Receiving Data](https://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/)
- [Monitoring and Tuning the Linux Networking Stack: Sending Data](https://blog.packagecloud.io/eng/2017/02/06/monitoring-tuning-linux-networking-stack-sending-data/)

I still haven’t read them in full but they are extremely detailed and cool.

### 
    
      [tell me if you know more!](https://jvns.ca#tell-me-if-you-know-more)
    

I still haven’t ever used these tools to seriously debug a packet loss problem yet, just wanted to write this down so that I have the notes if I do want to in the future!

If you have better tips for debugging whether/why packets are being dropped on a computer, let me know!