# The Case of the Connection Timeout
- **期号**: SRE Weekly Issue #266(2021-04-18)
- **作者**: Julia Evans
- **链接**: https://mysteries.wizardzines.com/connection-timeout.html
## 简介
This is fun! Try your hand at troubleshooting a connection issue in this game-ified role-play scenario.
BONUS CONTENT: Read about the author’s motivations, design decisions, and plans here.
## 正文
Your browser lacks required capabilities. Please upgrade it or switch to another to continue.
Loading…
<
Adding "<
< you've found new information!!
<
Hello! This is a mystery where your goal is to solve a debugging problem! You'll collect clues, interpret evidence, and ultimately solve the Case of the Connection Timeout!
In the sidebar, you'll notice a "What you know" list. As you collect more clues, the list will update with what you've learned.
Click "Start" to get started.
You need to access a service on a weird port, so you type in
machine.corp.yourcompany.com:1234 in your browser. You're very
sure that you have the right address and port number.
It says that it's loading for about 60 seconds (so long!!!), and then you get this error message:
You run:
$ curl machine.corp.yourcompany.com:1234
After waiting for 2 minutes, you finally get this error message:
curl: (28) Failed to connect to machine.corp.yourcompany.com port 1234: Connection timed out
Interesting, that's the same message as before.
<You run: sudo tcpdump -i any port 1234
This is the output you see:
$ sudo tcpdump -i any port 1234Here's the tcpdump output you see:
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
17:33:01.563903 lo In IP 192.168.1.121.38002 > 12.3.2.5.1234: Flags [S], seq 596753197, win 65495, options [mss 65495,sackOK,TS val 4247063724 ecr 0,nop,wscale 7], length 0<
17:33:02.582918 lo In IP 192.168.1.121.38002 > 12.3.2.5.1234: Flags [S], seq 596753197, win 65495, options [mss 65495,sackOK,TS val 4247064743 ecr 0,nop,wscale 7], length 0
17:33:04.662878 lo In IP 192.168.1.121.38002 > 12.3.2.5.1234: Flags [S], seq 596753197, win 65495, options [mss 65495,sackOK,TS val 4247066823 ecr 0,nop,wscale 7], length 0
17:33:08.719754 lo In IP 192.168.1.121.38002 > 12.3.2.5.1234: Flags [S], seq 596753197, win 65495, options [mss 65495,sackOK,TS val 4247070880 ecr 0,nop,wscale 7], length 0
<
These packets are all SYN packets. A SYN packet is the first packet sent in any TCP connection. The way you can tell they're SYN packets is from the Flags [S] in this tcpdump line:
17:33:01.563903 lo In IP 192.168.1.121.38002 > 12.3.2.5.1234: Flags [S], seq 596753197, win 65495
So now we know that there are a bunch of SYN packets being sent, but not any other packets.
<The weird thing about this is that usually there's just 1 SYN packet sent at the beginning of each TCP connection, but here there are 4, being sent over and over again.
<SYN packets being retried usually happens because something (usually a firewall) is blocking the SYN packets from being sent, so there's no reply.
<But what could it be?
Your server is hosted in AWS. A couple of words you remember hearing related to firewalls are "security groups" and "iptables".
You have SSH access to the machine, so you run
ssh machine.corp.yourcompany.com
And you're in. What do you want to do next?
You go to the AWS console, and look at the security groups. You find the security groups rule definition for that instance, and it looks like this. The port number you were using was 1234.
Do you think this is the cause of your problem?
[[Yes->AWS 2]] [[No->AWS 2]]These rules are saying that only traffic on ports 22, 80, and 443 is allowed. So your request on port 1234 isn't allowed! It looks like AWS is the problem!
<You double check with your security team to make sure it's safe to allow traffic on that port, and you add a new rule to allow traffic on port 1234.
It works! Everything is perfect.
Connection timeouts aren't always caused by firewall problems, but it's a very
common cause! Anytime I see a very very long wait followed by a connection
timeout, I check if I'm having a firewall problem by using tcpdump
to look for the pattern of SYN packets being retried over and over and over and
over again.
To see if this problem is caused by the iptables firewall, you need to print out all iptables rules.
You look up how to print out all iptables rules, and you see that you can do it
by running sudo iptables-save.
Here's what you see:
$ sudo iptables-save<
# Generated by iptables-save v1.8.7 on Wed Apr 14 17:57:11 2021
*filter
:INPUT ACCEPT [33501:125953140]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [23862:4282122]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Wed Apr 14 17:57:11 2021
# Generated by iptables-save v1.8.7 on Wed Apr 14 17:57:11 2021
*nat
:PREROUTING ACCEPT [17006:1104893]
:INPUT ACCEPT [16958:1088669]
:OUTPUT ACCEPT [145378:14161750]
:POSTROUTING ACCEPT [144916:14119410]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Wed Apr 14 17:57:11 2021
Do you want to keep investigating iptables, or go back?
That was a lot of iptables rules. How can we figure out if they're what's causing our problem? <Here's what we're going to do. iptables has a way to check exactly which individual iptables rules are being triggered called counters, so we're going to ask iptables!
$ sudo iptables-save -c
# Generated by iptables-save v1.8.7 on Thu Apr 15 12:04:21 2021
*filter
:INPUT ACCEPT [452:2223662]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [373:80464]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
[0:0] -A FORWARD -j DOCKER-USER
[0:0] -A FORWARD -j DOCKER-ISOLATION-STAGE-1
[0:0] -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
[0:0] -A FORWARD -o docker0 -j DOCKER
[0:0] -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
[0:0] -A FORWARD -i docker0 -o docker0 -j ACCEPT
[0:0] -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
[0:0] -A DOCKER-ISOLATION-STAGE-1 -j RETURN
[0:0] -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
[0:0] -A DOCKER-ISOLATION-STAGE-2 -j RETURN
[0:0] -A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Apr 15 12:04:21 2021
# Generated by iptables-save v1.8.7 on Thu Apr 15 12:04:21 2021
*nat
:PREROUTING ACCEPT [17826:1154724]
:INPUT ACCEPT [17776:1137824]
:OUTPUT ACCEPT [151426:14790277]
:POSTROUTING ACCEPT [150964:14747937]
:DOCKER - [0:0]
[21:1566] -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
[119:5540] -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
[8400:613089] -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
[0:0] -A DOCKER -i docker0 -j RETURN
COMMIT
What do you think this means? Have any of these rules ever been used?
[[Yes->Iptables 4]] [[No->Iptables 4]]Some of these rules have been used! iptables' output is pretty confusing at first, so let's look at this line from the output and break it down if you haven't looked at it before:
[21:1566] -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
The [21:1566] is the count of how many times that rule has been triggered.
But we still don't know how we could tell if these iptables rules are actually causing our problem. iptables isn't just used for blocking traffic, it's used for lots of things.
<Here's what we're going to do. The idea is that if we reset all of iptables' counters, it'll be easier to tell if they've been triggered by something we did, or by something unrelated.
Let's get started on that:
[[Reset the counters->Iptables 6]]You look up how to reset iptables' counters, and you find this article. It says to runsudo iptables -Z.
So you do that, and then you look at the counters again.
$ sudo iptables -ZWait... nothing has changed! Some of the counters still haven't been cleared. What's going on? <
$ sudo iptables-save -c # Generated by iptables-save v1.8.7 on Thu Apr 15 12:08:38 2021
*filter
:INPUT ACCEPT [452:2223662]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [373:80464]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
[0:0] -A FORWARD -j DOCKER-USER
[0:0] -A FORWARD -j DOCKER-ISOLATION-STAGE-1
[0:0] -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
[0:0] -A FORWARD -o docker0 -j DOCKER
[0:0] -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
[0:0] -A FORWARD -i docker0 -o docker0 -j ACCEPT
[0:0] -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
[0:0] -A DOCKER-ISOLATION-STAGE-1 -j RETURN
[0:0] -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
[0:0] -A DOCKER-ISOLATION-STAGE-2 -j RETURN
[0:0] -A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Apr 15 12:04:21 2021
# Generated by iptables-save v1.8.7 on Thu Apr 15 12:04:21 2021
*nat
:PREROUTING ACCEPT [17826:1154724]
:INPUT ACCEPT [17776:1137824]
:OUTPUT ACCEPT [151426:14790277]
:POSTROUTING ACCEPT [150964:14747937]
:DOCKER - [0:0]
[21:1566] -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
[119:5540] -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
[8400:613089] -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
[0:0] -A DOCKER -i docker0 -j RETURN
COMMIT
What's going on is that iptables has multiple "tables", and sudo iptables
-Z actually only clears the counters for the default table (the filter
table). This one isn't obvious at all -- I've gotten tripped up by this many times and I always forget about it.
You look at the output again, and see that the counters that aren't being cleared are in the nat table
sudo iptables -t nat -Z to clear the nat table.
$ sudo iptables -t nat -ZGreat. Everything is cleared. Now let's make a request to the service and see if it triggers any of those counters. [[Make a request->Iptables 8]]You go to
$ sudo iptables-save -c
# Generated by iptables-save v1.8.7 on Thu Apr 15 12:26:28 2021
*filter
:INPUT ACCEPT [10509:49406485]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [8539:1872558]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
[0:0] -A FORWARD -j DOCKER-USER
[0:0] -A FORWARD -j DOCKER-ISOLATION-STAGE-1
[0:0] -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
[0:0] -A FORWARD -o docker0 -j DOCKER
[0:0] -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
[0:0] -A FORWARD -i docker0 -o docker0 -j ACCEPT
[0:0] -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
[0:0] -A DOCKER-ISOLATION-STAGE-1 -j RETURN
[0:0] -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
[0:0] -A DOCKER-ISOLATION-STAGE-2 -j RETURN
[0:0] -A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Apr 15 12:26:28 2021
# Generated by iptables-save v1.8.7 on Thu Apr 15 12:26:28 2021
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
[0:0] -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
[0:0] -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
[0:0] -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
[0:0] -A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Thu Apr 15 12:26:28 2021
machine.corp.yourcompany.com:1234 in your browser (it fails again, just like last time), and then you go back to your SSH session. You look at the counters one last time.
Here's what you see.
$ sudo iptables-save -c # Generated by iptables-save v1.8.7 on Thu Apr 15 12:30:42 2021What does this mean? Is the problem caused by iptables? [[Yes->Iptables final]] [[No->Iptables final]]
*filter
:INPUT ACCEPT [12550:58251661]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [10155:2222169]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
[0:0] -A FORWARD -j DOCKER-USER
[0:0] -A FORWARD -j DOCKER-ISOLATION-STAGE-1
[0:0] -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
[0:0] -A FORWARD -o docker0 -j DOCKER
[0:0] -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
[0:0] -A FORWARD -i docker0 -o docker0 -j ACCEPT
[0:0] -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
[0:0] -A DOCKER-ISOLATION-STAGE-1 -j RETURN
[0:0] -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
[0:0] -A DOCKER-ISOLATION-STAGE-2 -j RETURN
[0:0] -A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Apr 15 12:30:42 2021
# Generated by iptables-save v1.8.7 on Thu Apr 15 12:30:42 2021
*nat
:PREROUTING ACCEPT [13:741]
:INPUT ACCEPT [13:741]
:OUTPUT ACCEPT [73:8565]
:POSTROUTING ACCEPT [73:8565]
:DOCKER - [0:0]
[0:0] -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
[0:0] -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
[0:0] -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
[0:0] -A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Thu Apr 15 12:30:42 2021
All of those counters are still zero, which means that none of the rules were triggered. So it looks like the problem wasn't iptables at all. It took a lot of work just to figure that out, but that's what debugging's like sometimes!
<This procedure of clearing the counters, making the failed request, and then checking the counters again can be really helpful when debugging firewall issues and I've used it to find problems with my iptables rules in the past.
Onto the next potential culprit...
You run:
$ curl localhost:1234/status
{"status": "ok"}
You get a reply instantly and everything looks good.
<