381 lines
17 KiB
HTML
381 lines
17 KiB
HTML
<!DOCTYPE html>
|
|
<html>
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
|
|
<title>What is Observability</title>
|
|
<meta name="viewport" content="width=device-width">
|
|
<meta name="description" content="What is Observability">
|
|
<meta name="keywords" content="observability,performance,blog">
|
|
|
|
<!-- syntax highlighting CSS -->
|
|
<link rel="stylesheet" href="/blog/css/syntax.css">
|
|
|
|
<!-- Custom CSS -->
|
|
<link rel="stylesheet" href="/blog/css/main.css">
|
|
|
|
<!-- Google tag (gtag.js) -->
|
|
<script async src="https://www.googletagmanager.com/gtag/js?id=G-SYD11KXST1"></script>
|
|
<script>
|
|
window.dataLayer = window.dataLayer || [];
|
|
function gtag(){dataLayer.push(arguments);}
|
|
gtag('js', new Date());
|
|
|
|
gtag('config', 'G-SYD11KXST1');
|
|
</script>
|
|
|
|
</head>
|
|
<body>
|
|
|
|
<div class="page">
|
|
<div class="nav">
|
|
<p class="navhdr">Brendan's site:</p>
|
|
<a href="/overview.html">Start Here</a><br>
|
|
<a href="/index.html">Homepage</a><br>
|
|
<a href="/blog/index.html">Blog</a><br>
|
|
<!-- <a href="/sitemap.html">Full Site Map</a><br> -->
|
|
<a href="/systems-performance-2nd-edition-book.html">Sys Perf book</a><br>
|
|
<a href="/bpf-performance-tools-book.html">BPF Perf book</a><br>
|
|
<a href="/linuxperf.html">Linux Perf</a><br>
|
|
<a href="/ebpf.html">eBPF Tools</a><br>
|
|
<a href="/perf.html">perf Examples</a><br>
|
|
<a href="/methodology.html">Perf Methods</a><br>
|
|
<a href="/usemethod.html">USE Method</a><br>
|
|
<a href="/tsamethod.html">TSA Method</a><br>
|
|
<a href="/offcpuanalysis.html">Off-CPU Analysis</a><br>
|
|
<a href="/activebenchmarking.html">Active Bench.</a><br>
|
|
<a href="/wss.html">WSS Estimation</a><br>
|
|
<a href="/flamegraphs.html">Flame Graphs</a><br>
|
|
<a href="/flamescope.html">Flame Scope</a><br>
|
|
<a href="/heatmaps.html">Heat Maps</a><br>
|
|
<a href="/frequencytrails.html">Frequency Trails</a><br>
|
|
<a href="/colonygraphs.html">Colony Graphs</a><br>
|
|
<a href="/dtrace.html">DTrace Tools</a><br>
|
|
<a href="/dtracetoolkit.html">DTraceToolkit</a><br>
|
|
<a href="/dtkshdemos.html">DtkshDemos</a><br>
|
|
<a href="/guessinggame.html">Guessing Game</a><br>
|
|
<a href="/specials.html">Specials</a><br>
|
|
<a href="/books.html">Books</a><br>
|
|
<a href="/sites.html">Other Sites</a><br>
|
|
|
|
</div>
|
|
|
|
<div class="recent">
|
|
<!-- (this is for the blog) recent books: -->
|
|
<!-- <center><a href="https://informit.com/sale/booksgiving"><img border=0 width=180 src="/Images/booksgiving2021.jpg"></center><br><b>Book sale until Dec 1, 2021: 55% off for 2 or more</b></a><br><br> -->
|
|
<center><a href="/systems-performance-2nd-edition-book.html"><img src="/Images/sysperf2nd_bookcover_360.jpg" width=180></a><br><font size=-2><i><a href="/systems-performance-2nd-edition-book.html">Systems Performance 2nd Ed.</a></i></font></center><br><br>
|
|
|
|
<center><a href="/bpf-performance-tools-book.html"><img src="/Images/bpfperftools_bookcover_360.jpg" width=180></a><br><font size=-2><i><a href="/bpf-performance-tools-book.html">BPF Performance Tools book</a></i></font></center>
|
|
<!--
|
|
<br><center><a href="https://www.portal.reinvent.awsevents.com/connect/search.ww?#loadSearch-searchPhrase=OPN303&searchType=session&tc=0&sortBy=abbreviationSort&p="><img src="/Images/Speaker/reInvent2019_200.jpg" width=180" border=0></a><br><font size=-2><i>I'm speaking at <a href="https://www.portal.reinvent.awsevents.com/connect/search.ww?#loadSearch-searchPhrase=OPN303&searchType=session&tc=0&sortBy=abbreviationSort&p=">AWS re:Invent 2019</a></i></font></center>
|
|
-->
|
|
<br>
|
|
Recent posts:<br>
|
|
<ul style="padding-left:18px">
|
|
|
|
<li>07 Feb 2026 »<br>
|
|
<a href="/blog/2026-02-07/why-i-joined-openai.html">
|
|
Why I joined OpenAI</a></li>
|
|
|
|
<li>05 Dec 2025 »<br>
|
|
<a href="/blog/2025-12-05/leaving-intel.html">
|
|
Leaving Intel</a></li>
|
|
|
|
<li>28 Nov 2025 »<br>
|
|
<a href="/blog/2025-11-28/ai-virtual-brendans.html">
|
|
On "AI Brendans" or "Virtual Brendans"</a></li>
|
|
|
|
<li>22 Nov 2025 »<br>
|
|
<a href="/blog/2025-11-22/intel-is-listening.html">
|
|
Intel is listening, don't waste your shot</a></li>
|
|
|
|
<li>17 Nov 2025 »<br>
|
|
<a href="/blog/2025-11-17/third-stage-engineering.html">
|
|
Third Stage Engineering</a></li>
|
|
|
|
<li>04 Aug 2025 »<br>
|
|
<a href="/blog/2025-08-04/when-to-hire-a-computer-performance-engineering-team-2025-part1.html">
|
|
When to Hire a Computer Performance Engineering Team (2025) part 1 of 2</a></li>
|
|
|
|
<li>22 May 2025 »<br>
|
|
<a href="/blog/2025-05-22/3-years-of-extremely-remote-work.html">
|
|
3 Years of Extremely Remote Work</a></li>
|
|
|
|
<li>01 May 2025 »<br>
|
|
<a href="/blog/2025-05-01/doom-gpu-flame-graphs.html">
|
|
Doom GPU Flame Graphs</a></li>
|
|
|
|
<li>29 Oct 2024 »<br>
|
|
<a href="/blog/2024-10-29/ai-flame-graphs.html">
|
|
AI Flame Graphs</a></li>
|
|
|
|
<li>22 Jul 2024 »<br>
|
|
<a href="/blog/2024-07-22/no-more-blue-fridays.html">
|
|
No More Blue Fridays</a></li>
|
|
|
|
<li>24 Mar 2024 »<br>
|
|
<a href="/blog/2024-03-24/linux-crisis-tools.html">
|
|
Linux Crisis Tools</a></li>
|
|
|
|
<li>17 Mar 2024 »<br>
|
|
<a href="/blog/2024-03-17/the-return-of-the-frame-pointers.html">
|
|
The Return of the Frame Pointers</a></li>
|
|
|
|
<li>10 Mar 2024 »<br>
|
|
<a href="/blog/2024-03-10/ebpf-documentary.html">
|
|
eBPF Documentary</a></li>
|
|
|
|
<li>28 Apr 2023 »<br>
|
|
<a href="/blog/2023-04-28/ebpf-security-issues.html">
|
|
eBPF Observability Tools Are Not Security Tools</a></li>
|
|
|
|
<li>01 Mar 2023 »<br>
|
|
<a href="/blog/2023-03-01/computer-performance-future-2022.html">
|
|
USENIX SREcon APAC 2022: Computing Performance: What's on the Horizon</a></li>
|
|
|
|
<li>17 Feb 2023 »<br>
|
|
<a href="/blog/2023-02-17/srecon-apac-2023.html">
|
|
USENIX SREcon APAC 2023: CFP</a></li>
|
|
|
|
<li>02 May 2022 »<br>
|
|
<a href="/blog/2022-05-02/brendan-at-intel.html">
|
|
Brendan@Intel.com</a></li>
|
|
|
|
<li>15 Apr 2022 »<br>
|
|
<a href="/blog/2022-04-15/netflix-farewell-1.html">
|
|
Netflix End of Series 1</a></li>
|
|
|
|
<li>09 Apr 2022 »<br>
|
|
<a href="/blog/2022-04-09/tensorflow-library-performance.html">
|
|
TensorFlow Library Performance</a></li>
|
|
|
|
<li>19 Mar 2022 »<br>
|
|
<a href="/blog/2022-03-19/why-dont-you-use.html">
|
|
Why Don't You Use ...</a></li>
|
|
|
|
</ul>
|
|
<a href="/blog/index.html">Blog index</a><br>
|
|
<a href="/blog/about.html">About</a><br>
|
|
<a href="/blog/rss.xml">RSS</a><br>
|
|
<!--
|
|
<br><center><a href="https://www.usenix.org/conference/lisa18"><img src="https://www.usenix.org/sites/default/files/lisa18_banner_join-me.png" width=180></a><br><font size=-2><i>I am program co-chair for LISA 2018</i></font></center>
|
|
-->
|
|
</div>
|
|
|
|
<div class="site">
|
|
<div class="header">
|
|
<h1 class="title"><a href="/blog/index.html">Brendan Gregg's Blog</a></h1>
|
|
<a class="extra" href="/blog/index.html">home</a>
|
|
</div>
|
|
|
|
<h2 class="big">What is Observability</h2>
|
|
<p class="meta">23 May 2021</p>
|
|
|
|
<div class="post">
|
|
<p>It's a made-up computer word that my word processor decorates with a wiggly red you-can't-spell line. At least it did until I clicked "Add to Dictionary" (it got too annoying as I was writing a book on computer observability).
|
|
Some people abbreviate it as o11y.</p>
|
|
|
|
<blockquote>Observability: The <i>ability</i> to <i>observe</i>.</blockquote>
|
|
|
|
<p>Observe-ability. Observability.</p>
|
|
|
|
<p>In computer engineering we use it to describe the tools, data sources, and methods for understanding (observing!) how a technology is operating. We don't use the <em>real</em> word "observable" since that implies the wrong thing. Imagine "observable metrics": Are there metrics that <em>aren't</em> observable?</p>
|
|
|
|
<p>Using observability in sentences:</p>
|
|
|
|
<ul>
|
|
<li>What observability tools are installed? (Means: What tools exist that only read state?)</li>
|
|
<li>What observability does that database have? (Means: What metrics and logs does it have?)</li>
|
|
<li>How do you do observability? (Means: What products do you use for metrics, tracing, etc.?)</li>
|
|
<li>Let me try some observability first. (Means: Let me look at the system without changing it.)</li>
|
|
</ul>
|
|
|
|
<p>Wait, aren't all performance tools observability tools? No. <em>Experimental</em> tools change the state of the system to understand it. For example, benchmarks.</p>
|
|
|
|
<p>As an analogy, a car's dashboard is a collection of observability tools that let you understand how the car is operating (speed, rpm, temperature). A car's 0-60 mph time is an <em>experiment</em>.</p>
|
|
|
|
<p>When I was a performance consultant I'd show up to random companies who wanted me to fix their computer performance issues. If they trusted me with a login to their production servers, I could help them a lot quicker. To get that trust I knew which tools looked but didn't touch: Which were observability tools and which were experimental tools. "I'll start with observability tools only" is something I'd say at the start of every engagement.</p>
|
|
|
|
<p>Note that observability tools aren't completely harmless: Their execution consumes resources, usually negligible, but in some cases it's enough to perturb the target of study. This is the "observer effect."</p>
|
|
|
|
<p>Another use of the term observability is as a reminder to switch between tool types, and not to get stuck on one. A colleague (Roch Bourbonnais from memory) once told me:</p>
|
|
|
|
<blockquote>"You have two hands. Observability and experimentation."</blockquote>
|
|
|
|
<p>It stuck with me as it also makes the point that when you're only using one type to solve a performance problem <strong>you're working one-handed</strong>.</p>
|
|
|
|
</div>
|
|
|
|
|
|
|
|
<br><hr>
|
|
<script type="text/javascript">
|
|
var disqus_shortname = 'brendangregg';
|
|
|
|
function loadshowdisqus(id) {
|
|
// from disqus:
|
|
var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
|
|
dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
|
|
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
|
|
// mine:
|
|
var c = document.getElementById(id); c.style.display = (c.style.display == "none" ? "block" : "none");
|
|
}
|
|
</script>
|
|
<div id="comments_button" onclick='loadshowdisqus("comments_content")'><p style="color:blue"><u>Click here for Disqus comments (ad supported).</u></p></div>
|
|
<div id="comments_content" style="display:none">
|
|
<p><font size=-1><i>You are welcome to comment here, but I've been meaning to switch comment systems one day and I don't know yet if I can preserve existing comments (I'll try to find a way).</i></font></p>
|
|
<div id="disqus_thread"></div>
|
|
<noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>
|
|
<a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a>
|
|
</div>
|
|
|
|
|
|
<div class="recentmobile">
|
|
<hr><center>Site Navigation</center><br>
|
|
<!-- (this is for /index.html etc) recent books: -->
|
|
<!-- <center><a href="https://informit.com/sale/booksgiving"><img border=0 width=180 src="/Images/booksgiving2021.jpg"></center><br><b>Book sale until Dec 1, 2021: 55% off for 2 or more</b></a><br><br> -->
|
|
<center><a href="/systems-performance-2nd-edition-book.html"><img src="/Images/sysperf2nd_bookcover_360.jpg" width=180></a><br><font size=-2><i><a href="/systems-performance-2nd-edition-book.html">Systems Performance 2nd Ed.</a></i></font></center><br><br>
|
|
|
|
<center><a href="/bpf-performance-tools-book.html"><img src="/Images/bpfperftools_bookcover_360.jpg" width=180></a><br><font size=-2><i><a href="/bpf-performance-tools-book.html">BPF Performance Tools book</a></i></font></center>
|
|
<!--
|
|
<br><center><a href="https://www.portal.reinvent.awsevents.com/connect/search.ww?#loadSearch-searchPhrase=OPN303&searchType=session&tc=0&sortBy=abbreviationSort&p="><img src="/Images/Speaker/reInvent2019_200.jpg" width=180" border=0></a><br><font size=-2><i>I'm speaking at <a href="https://www.portal.reinvent.awsevents.com/connect/search.ww?#loadSearch-searchPhrase=OPN303&searchType=session&tc=0&sortBy=abbreviationSort&p=">AWS re:Invent 2019</a></i></font></center>
|
|
-->
|
|
<br>
|
|
|
|
Recent posts:<br>
|
|
<ul style="padding-left:18px">
|
|
|
|
<li class="recent">07 Feb 2026 »<br>
|
|
<a href="/blog/2026-02-07/why-i-joined-openai.html">
|
|
Why I joined OpenAI</a></li>
|
|
|
|
<li class="recent">05 Dec 2025 »<br>
|
|
<a href="/blog/2025-12-05/leaving-intel.html">
|
|
Leaving Intel</a></li>
|
|
|
|
<li class="recent">28 Nov 2025 »<br>
|
|
<a href="/blog/2025-11-28/ai-virtual-brendans.html">
|
|
On "AI Brendans" or "Virtual Brendans"</a></li>
|
|
|
|
<li class="recent">22 Nov 2025 »<br>
|
|
<a href="/blog/2025-11-22/intel-is-listening.html">
|
|
Intel is listening, don't waste your shot</a></li>
|
|
|
|
<li class="recent">17 Nov 2025 »<br>
|
|
<a href="/blog/2025-11-17/third-stage-engineering.html">
|
|
Third Stage Engineering</a></li>
|
|
|
|
<li class="recent">04 Aug 2025 »<br>
|
|
<a href="/blog/2025-08-04/when-to-hire-a-computer-performance-engineering-team-2025-part1.html">
|
|
When to Hire a Computer Performance Engineering Team (2025) part 1 of 2</a></li>
|
|
|
|
<li class="recent">22 May 2025 »<br>
|
|
<a href="/blog/2025-05-22/3-years-of-extremely-remote-work.html">
|
|
3 Years of Extremely Remote Work</a></li>
|
|
|
|
<li class="recent">01 May 2025 »<br>
|
|
<a href="/blog/2025-05-01/doom-gpu-flame-graphs.html">
|
|
Doom GPU Flame Graphs</a></li>
|
|
|
|
<li class="recent">29 Oct 2024 »<br>
|
|
<a href="/blog/2024-10-29/ai-flame-graphs.html">
|
|
AI Flame Graphs</a></li>
|
|
|
|
<li class="recent">22 Jul 2024 »<br>
|
|
<a href="/blog/2024-07-22/no-more-blue-fridays.html">
|
|
No More Blue Fridays</a></li>
|
|
|
|
<li class="recent">24 Mar 2024 »<br>
|
|
<a href="/blog/2024-03-24/linux-crisis-tools.html">
|
|
Linux Crisis Tools</a></li>
|
|
|
|
<li class="recent">17 Mar 2024 »<br>
|
|
<a href="/blog/2024-03-17/the-return-of-the-frame-pointers.html">
|
|
The Return of the Frame Pointers</a></li>
|
|
|
|
<li class="recent">10 Mar 2024 »<br>
|
|
<a href="/blog/2024-03-10/ebpf-documentary.html">
|
|
eBPF Documentary</a></li>
|
|
|
|
<li class="recent">28 Apr 2023 »<br>
|
|
<a href="/blog/2023-04-28/ebpf-security-issues.html">
|
|
eBPF Observability Tools Are Not Security Tools</a></li>
|
|
|
|
<li class="recent">01 Mar 2023 »<br>
|
|
<a href="/blog/2023-03-01/computer-performance-future-2022.html">
|
|
USENIX SREcon APAC 2022: Computing Performance: What's on the Horizon</a></li>
|
|
|
|
<li class="recent">17 Feb 2023 »<br>
|
|
<a href="/blog/2023-02-17/srecon-apac-2023.html">
|
|
USENIX SREcon APAC 2023: CFP</a></li>
|
|
|
|
<li class="recent">02 May 2022 »<br>
|
|
<a href="/blog/2022-05-02/brendan-at-intel.html">
|
|
Brendan@Intel.com</a></li>
|
|
|
|
<li class="recent">15 Apr 2022 »<br>
|
|
<a href="/blog/2022-04-15/netflix-farewell-1.html">
|
|
Netflix End of Series 1</a></li>
|
|
|
|
<li class="recent">09 Apr 2022 »<br>
|
|
<a href="/blog/2022-04-09/tensorflow-library-performance.html">
|
|
TensorFlow Library Performance</a></li>
|
|
|
|
<li class="recent">19 Mar 2022 »<br>
|
|
<a href="/blog/2022-03-19/why-dont-you-use.html">
|
|
Why Don't You Use ...</a></li>
|
|
|
|
</ul>
|
|
<a href="/blog/index.html">Blog index</a><br>
|
|
<a href="/blog/about.html">About</a><br>
|
|
<a href="/blog/rss.xml">RSS</a><br>
|
|
<!-- also edit _layouts/default.html -->
|
|
<!--
|
|
<br><a href="https://www.usenix.org/conference/lisa18"><img src="https://www.usenix.org/sites/default/files/lisa18_banner_join-me.png" width=180></a><br><font size=-2><i>I am program co-chair for LISA 2018</i></font>
|
|
-->
|
|
|
|
<hr>
|
|
</div>
|
|
<div class="navmobile">
|
|
<p class="navhdr">Brendan's site:</p>
|
|
<a href="/overview.html">Start Here</a><br>
|
|
<a href="/index.html">Homepage</a><br>
|
|
<a href="/blog/index.html">Blog</a><br>
|
|
<!-- <a href="/sitemap.html">Full Site Map</a><br> -->
|
|
<a href="/systems-performance-2nd-edition-book.html">Sys Perf book</a><br>
|
|
<a href="/bpf-performance-tools-book.html">BPF Perf book</a><br>
|
|
<a href="/linuxperf.html">Linux Perf</a><br>
|
|
<a href="/ebpf.html">eBPF Tools</a><br>
|
|
<a href="/perf.html">perf Examples</a><br>
|
|
<a href="/methodology.html">Perf Methods</a><br>
|
|
<a href="/usemethod.html">USE Method</a><br>
|
|
<a href="/tsamethod.html">TSA Method</a><br>
|
|
<a href="/offcpuanalysis.html">Off-CPU Analysis</a><br>
|
|
<a href="/activebenchmarking.html">Active Bench.</a><br>
|
|
<a href="/wss.html">WSS Estimation</a><br>
|
|
<a href="/flamegraphs.html">Flame Graphs</a><br>
|
|
<a href="/flamescope.html">Flame Scope</a><br>
|
|
<a href="/heatmaps.html">Heat Maps</a><br>
|
|
<a href="/frequencytrails.html">Frequency Trails</a><br>
|
|
<a href="/colonygraphs.html">Colony Graphs</a><br>
|
|
<a href="/dtrace.html">DTrace Tools</a><br>
|
|
<a href="/dtracetoolkit.html">DTraceToolkit</a><br>
|
|
<a href="/dtkshdemos.html">DtkshDemos</a><br>
|
|
<a href="/guessinggame.html">Guessing Game</a><br>
|
|
<a href="/specials.html">Specials</a><br>
|
|
<a href="/books.html">Books</a><br>
|
|
<a href="/sites.html">Other Sites</a><br>
|
|
|
|
</div>
|
|
|
|
<div class="footer">
|
|
<div class="contact">
|
|
Copyright 2025 Brendan Gregg.<br><a href="/blog/about.html">About this blog</a>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
</body>
|
|
</html>
|