278 lines
14 KiB
HTML
278 lines
14 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en">
|
||
<head><title>TCP connection repair [LWN.net]</title>
|
||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||
<meta property="og:title" content="TCP connection repair">
|
||
<meta property="og:site_name" content="LWN.net">
|
||
<meta property="og:type" content="article">
|
||
<meta property="og:description" content="Migrating a running container from one physical host to another is a tricky job on a number of [...]">
|
||
<meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8">
|
||
<META NAME="robots" CONTENT="noai, noimageai">
|
||
<link rel="icon" href="https://static.lwn.net/images/favicon.png"
|
||
type="image/png">
|
||
<link rel="alternate" type="application/rss+xml" title="LWN.net headlines" href="https://lwn.net/headlines/rss">
|
||
<link rel="stylesheet" href="/CSS/lwn-v1">
|
||
<style>div.MakeALink { display: none }</style>
|
||
|
||
|
||
</head>
|
||
<body>
|
||
<a name="t"></a>
|
||
<div id="menu"><a href="/" aria-label="Return to the front page">
|
||
<img src="https://static.lwn.net/images/logo/barepenguin-70.webp" class="logo"
|
||
border="0" alt="LWN.net Logo">
|
||
<span class="logo">LWN<br>.net</span>
|
||
<span class="logobl">News from the source</span></a>
|
||
<a href="/" aria-label="Return to the front page">
|
||
<img src="https://static.lwn.net/images/lcorner-ss.png" class="sslogo"
|
||
border="0" alt="LWN"></a><div class="navmenu-container">
|
||
<ul class="navmenu">
|
||
<li><a class="navmenu" href="#t"><b>Content</b></a><ul><li><a href="/current/">Weekly Edition</a></li><li><a href="/Archives/">Archives</a></li><li><a href="/Search/">Search</a></li><li><a href="/Kernel/">Kernel</a></li><li><a href="/Security/">Security</a></li><li><a href="/Calendar/">Events calendar</a></li><li><a href="/Comments/unread">Unread comments</a></li><li><hr></li><li><a href="/op/FAQ.lwn">LWN FAQ</a></li><li><a href="/op/AuthorGuide.lwn">Write for us</a></li></ul></li>
|
||
<li><a class="navmenu" href="#t"><b>Edition</b></a><ul><li><a href="/Articles/494926/">Return to the Kernel page</a></li></ul></li>
|
||
</ul></div>
|
||
</div> <!-- menu -->
|
||
<div class="topnav-spacer"></div>
|
||
<div class="top-banner not-print"><ins data-revive-zoneid="12072"
|
||
data-revive-id="fa8c6c9da7f33852f7097c4a94da1070"></ins>
|
||
<script async src="//servedby.aqua-adserver.com/asyncjs.php"></script></div>
|
||
<div class="topnav-container">
|
||
<div class="not-handset"><form action="https://lwn.net/Login/" method="post" name="loginform"
|
||
class="loginform">
|
||
<label><b>User:</b> <input type="text" name="uname" value="" size="8" id="uc" /></label>
|
||
<label><b>Password:</b> <input type="password" name="pword" size="8" id="pc" /></label> <input type="hidden" name="target" value="/Articles/495304/" /> <input type="submit" name="submit" value="Log in" /></form> |
|
||
<form action="https://lwn.net/subscribe/" method="post" class="loginform">
|
||
<input type="submit" name="submit" value="Subscribe" />
|
||
</form> |
|
||
<form action="https://lwn.net/Login/newaccount" method="post" class="loginform">
|
||
<input type="submit" name="submit" value="Register" />
|
||
</form>
|
||
</div>
|
||
<div class="handset-only">
|
||
<a href="/Login/"><b>Log in</b></a> /
|
||
<a href="/subscribe/"><b>Subscribe</b></a> /
|
||
<a href="/Login/newaccount"><b>Register</b></a>
|
||
</div>
|
||
</div><div class="maincolumn flexcol">
|
||
<div class="middlecolumn">
|
||
<div class="PageHeadline">
|
||
<h1>TCP connection repair</h1>
|
||
</div>
|
||
<div class="ArticleText"><main>
|
||
<blockquote class="ad">
|
||
<b>Ready to give LWN a try?</b>
|
||
<p>
|
||
With a subscription to LWN, you can stay current with what is happening in the Linux and free-software community and take advantage of subscriber-only site features. We are pleased to offer you <b><a href="https://lwn.net/Promo/nst-trial1/claim">a free trial subscription</a></b>, no credit card required, so that you can see for yourself. Please, join us!
|
||
</blockquote>
|
||
<div class="FeatureByline">
|
||
By <b>Jonathan Corbet</b><br>May 1, 2012</br>
|
||
</div>
|
||
Migrating a running container from one physical host to another is a tricky
|
||
job on a number of levels. Things get even harder if, as is likely, the
|
||
container has active network connections to processes outside of that
|
||
container. It is natural to want those connections to follow the container
|
||
to its new host, preferably without the remote end even noticing that
|
||
something has changed, but the Linux networking stack was not written with
|
||
this kind of move in mind. Even so, it appears that transparent relocation
|
||
of network connections, in the form of Pavel Emelyanov's <a
|
||
href="/Articles/493983/">TCP connection repair patches</a>, will be
|
||
supported in the 3.5 kernel.
|
||
<p>
|
||
The first step in moving a TCP connection is to gather all of the
|
||
information possible about its current state. Much of that information is
|
||
available from user space now; by digging around in <tt>/proc</tt> and
|
||
<tt>/sys</tt>, one can determine the address and port of the remote end,
|
||
the sizes of the send and receive queues, TCP sequence numbers, and a
|
||
number of parameters
|
||
negotiated between the two end points. There are still a few things that
|
||
user space will need to obtain, though, before it can finish the job; that
|
||
requires some additional support from the kernel.
|
||
<p>
|
||
With Pavel's patch, that support is available to suitably privileged
|
||
processes.
|
||
To dig into the internals of an active network connection, user space must
|
||
put the associated socket into a new "repair mode." That is done with the
|
||
<tt>setsockopt()</tt> system call, using the new <tt>TCP_REPAIR</tt>
|
||
option. Changing a process's repair mode status requires the
|
||
<tt>CAP_NET_ADMIN</tt> capability; the socket must also either be closed or
|
||
in the "established" state. Once the socket is in repair mode, it can be
|
||
manipulated in a number of ways.
|
||
<p>
|
||
One of those is to read the contents of the send and receive queues. The
|
||
send queue contains data that has not yet been successfully transmitted to
|
||
the remote end; that data needs to move with the connection so it can be
|
||
transmitted from the new location. The receive queue, instead, contains
|
||
data received from the remote end that has not yet been consumed by the
|
||
application being moved; that data, too, should move so it will be waiting
|
||
on the new host when the application gets around to reading it. Obtaining
|
||
the contents of these queues is done with a two-step sequence:
|
||
(1) call <tt>setsockopt(TCP_REPAIR_QUEUE)</tt> with either
|
||
<tt>TCP_RECV_QUEUE</tt> or <tt>TCP_SEND_QUEUE</tt>, then (2) call
|
||
<tt>recvmesg()</tt> to read the contents of the selected queue.
|
||
<p>
|
||
It turns out there is only one other important piece of information that
|
||
cannot already be obtained from user space: the maximum value of the MSS
|
||
(maximum segment size) negotiated between the two endpoints at connection
|
||
setup time. To make this value available, Pavel's patch changes the
|
||
semantics of the <tt>TCP_MAXSEG</tt> socket option (for
|
||
<tt>getsockopt()</tt>) when the connection is
|
||
in repair mode: it returns the maximal "clamp" MSS value rather than the
|
||
currently active value.
|
||
<p>
|
||
Finally, if a connection is closed while it is in the repair mode, it is
|
||
simply deleted with no notification to the remote end. No FIN or RST
|
||
packets will be sent, so the remote side will have no idea that things have
|
||
changed.
|
||
<p>
|
||
Then there is the matter of establishing the connection on the new host.
|
||
That is done by creating a new socket and putting it immediately into the
|
||
repair mode. The socket can then be bound to the proper port number; a
|
||
number of the usual checks for port numbers are suspended when the socket
|
||
is in repair mode. The <tt>TCP_REPAIR_QUEUE</tt> <tt>setsockopt()</tt>
|
||
call comes into play again, but this time <tt>sendmsg()</tt> is used to
|
||
restore the contents of the send and receive queues.
|
||
<p>
|
||
Another important task is to restore the send and receive sequence numbers.
|
||
These numbers are normally generated randomly when the connection is
|
||
established, but that cannot be done when a connection is being moved.
|
||
These numbers can be set with yet another call to <tt>setsockopt()</tt>,
|
||
this time with the <tt>TCP_QUEUE_SEQ</tt> option. This operation applies
|
||
to whichever queue was previously selected with <tt>TCP_REPAIR_QUEUE</tt>,
|
||
so the refilling of a queue's content and the setting of its sequence
|
||
number are best done at the same time.
|
||
<p>
|
||
A few negotiated parameters also need to be restored so that the two ends
|
||
will remain in agreement with each other; these include the MSS clamp
|
||
described above, along with the active maximum segment size, the window
|
||
size, and whether the selective acknowledgment and timestamp features can
|
||
be used. One last <tt>setsockopt()</tt> option,
|
||
<tt>TCP_REPAIR_OPTIONS</tt>, has been added to make it possible to set
|
||
these parameters from user space.
|
||
<p>
|
||
Once the socket has been restored to a state approximating that which
|
||
existed on the old host, it's time to put it into operation. When
|
||
<tt>connect()</tt> is called on a socket in repair mode, much of the
|
||
current setup and negotiation code is shorted out; instead, the connection
|
||
goes directly to the "established" state without any communication from the
|
||
remote end. As a final step, when the socket is taken out of the repair
|
||
mode, a window probe is sent to restart traffic
|
||
between the two ends; at that point, the socket can resume normal operation
|
||
on the new host.
|
||
<p>
|
||
These patches have been through a few revisions over a number of months;
|
||
with version 4, networking maintainer David Miller <a
|
||
href="/Articles/495318/">accepted</a> them into net-next. From there,
|
||
those changes will almost certainly hit the mainline during the 3.5 merge
|
||
window. The TCP connection repair patches do not represent a complete
|
||
solution to the problem of checkpointing and restoring containers, but they
|
||
are an important step in that direction.<br clear="all"><table class="IndexEntries">
|
||
<tr><th colspan=2>Index entries for this article</th></tr>
|
||
<tr><td><a href="/Kernel/Index">Kernel</a></td><td><a href="/Kernel/Index#Checkpointing">Checkpointing</a></td></tr>
|
||
<tr><td><a href="/Kernel/Index">Kernel</a></td><td><a href="/Kernel/Index#Containers">Containers</a></td></tr>
|
||
<tr><td><a href="/Kernel/Index">Kernel</a></td><td><a href="/Kernel/Index#Networking">Networking</a></td></tr>
|
||
</table><br clear="all">
|
||
<hr width="60%%" align="left">
|
||
<form action="/Login/" method="post">
|
||
<input type="hidden" name="target" value="/Articles/495304/" />
|
||
<input type="submit" name="login" value="Log in" /> to post comments
|
||
<p>
|
||
<p><a name="Comments"></a>
|
||
<a name="CommAnchor496856"></a>
|
||
<details class="CommentBox" open>
|
||
<summary><h3 class="CommentTitle">TCP connection repair</h3>
|
||
<div class="AnnLine">
|
||
<p class="CommentPoster"> Posted May 11, 2012 10:19 UTC (Fri)
|
||
by <b>kolyshkin</b> (guest, #34342)
|
||
[<a href="/Articles/496856/">Link</a>]
|
||
</p>
|
||
|
||
</div>
|
||
</summary>
|
||
<div class="FormattedComment">
|
||
I'd like to add that this work is done for CRIU project, <a href="http://criu.org/">http://criu.org/</a>, and there you can find the userspace code to migrate TCP connections.<br>
|
||
<p>
|
||
Ultimately, CRIU should replace in-kernel checkpoint-restore functionality that we have in OpenVZ.<br>
|
||
</div>
|
||
|
||
|
||
<div class="CommentReplyButton">
|
||
<form action="/Articles/496856/comment" method="post">
|
||
<input type="submit" value="Reply to this comment">
|
||
</form>
|
||
</div>
|
||
|
||
<p>
|
||
|
||
</details>
|
||
<a name="CommAnchor786191"></a>
|
||
<details class="CommentBox" open>
|
||
<summary><h3 class="CommentTitle">TCP connection repair</h3>
|
||
<div class="AnnLine">
|
||
<p class="CommentPoster"> Posted Apr 18, 2019 2:32 UTC (Thu)
|
||
by <b>weishuangjian</b> (guest, #131509)
|
||
[<a href="/Articles/786191/">Link</a>] (1 responses)
|
||
</p>
|
||
|
||
</div>
|
||
</summary>
|
||
<div class="FormattedComment">
|
||
This article is wonderful and detailed. But what I want to know is, when dumping and restoring TCP connections, what happens if the remote discovers that the current process is not responding? Anyone answer me ：）?<br>
|
||
</div>
|
||
|
||
|
||
<div class="CommentReplyButton">
|
||
<form action="/Articles/786191/comment" method="post">
|
||
<input type="submit" value="Reply to this comment">
|
||
</form>
|
||
</div>
|
||
|
||
<p>
|
||
|
||
<a name="CommAnchor815650"></a>
|
||
<details class="CommentBox" open>
|
||
<summary><h3 class="CommentTitle">TCP connection repair</h3>
|
||
<div class="AnnLine">
|
||
<p class="CommentPoster"> Posted Mar 21, 2020 1:38 UTC (Sat)
|
||
by <b>felart18</b> (guest, #137871)
|
||
[<a href="/Articles/815650/">Link</a>]
|
||
</p>
|
||
|
||
</div>
|
||
</summary>
|
||
<div class="FormattedComment">
|
||
TCP should resend the packet if it's not acknowledged by the peer.<br>
|
||
</div>
|
||
|
||
|
||
<div class="CommentReplyButton">
|
||
<form action="/Articles/815650/comment" method="post">
|
||
<input type="submit" value="Reply to this comment">
|
||
</form>
|
||
</div>
|
||
|
||
<p>
|
||
|
||
</details>
|
||
</details>
|
||
|
||
</main></div> <!-- ArticleText -->
|
||
</div> <!-- middlecolumn -->
|
||
<div class="rightcol not-print">
|
||
<!-- no side ad served -->
|
||
</div>
|
||
</div> <!-- maincolumn -->
|
||
|
||
<br clear="all">
|
||
<center>
|
||
<P>
|
||
<span class="ReallySmall">
|
||
Copyright © 2012, Eklektix, Inc.<BR>
|
||
This article may be redistributed under the terms of the
|
||
<a href="http://creativecommons.org/licenses/by-sa/4.0/">Creative
|
||
Commons CC BY-SA 4.0</a> license<br>
|
||
Comments and public postings are copyrighted by their creators.<br>
|
||
Linux is a registered trademark of Linus Torvalds<br>
|
||
</span>
|
||
</center>
|
||
|
||
</body></html>
|