793 lines
22 KiB
HTML
793 lines
22 KiB
HTML
<!DOCTYPE html>
|
||
<html>
|
||
<head>
|
||
<meta charset="utf-8" />
|
||
<meta http-equiv="X-UA-Compatible" content="IE=edge"><title>Studying an Incident</title><link rel="icon" type="image/png" href=/favicon-16x16.png /><meta name="viewport" content="width=device-width, initial-scale=1">
|
||
<meta name="description" content="It is not often you get an opportunity to study an incident to illustrate a few lessons. A recent incident that I describe below teaches…">
|
||
|
||
|
||
<meta itemprop="name" content="Studying an Incident">
|
||
<meta itemprop="description" content="It is not often you get an opportunity to study an incident to illustrate a few lessons. A recent incident that I describe below teaches…">
|
||
<meta itemprop="datePublished" content="2019-12-30T15:36:18+00:00">
|
||
<meta itemprop="dateModified" content="2019-12-30T15:36:18+00:00">
|
||
<meta itemprop="wordCount" content="1357">
|
||
<meta itemprop="image" content="https://www.subbu.org/subbu.jpg"><meta property="og:url" content="https://www.subbu.org/essays/2019/studying-an-incident/">
|
||
<meta property="og:site_name" content="Writing is clarifying">
|
||
<meta property="og:title" content="Studying an Incident">
|
||
<meta property="og:description" content="It is not often you get an opportunity to study an incident to illustrate a few lessons. A recent incident that I describe below teaches…">
|
||
<meta property="og:locale" content="en">
|
||
<meta property="og:type" content="article">
|
||
<meta property="og:image" content="https://www.subbu.org/subbu.jpg">
|
||
<meta property="article:published_time" content="2019-12-30T15:36:18Z"><meta property="article:modified_time" content="2019-12-30T15:36:18Z"><link href='https://fonts.googleapis.com/css?family=Playfair+Display:700' rel='stylesheet' type='text/css'>
|
||
<link rel="stylesheet" type="text/css" media="screen" href="https://www.subbu.org/css/normalize.css" />
|
||
<link rel="stylesheet" type="text/css" media="screen" href="https://www.subbu.org/css/main.css" />
|
||
<link rel="stylesheet" type="text/css" href="https://www.subbu.org/css/custom.css" />
|
||
|
||
|
||
<link id="dark-scheme" rel="stylesheet" type="text/css" href="https://www.subbu.org/css/dark.css" />
|
||
|
||
<script src="https://www.subbu.org/js/feather.min.js"></script>
|
||
|
||
<script src="https://www.subbu.org/js/main.js"></script>
|
||
<script>
|
||
window.goatcounter = {
|
||
path: function(p) { return location.pathname; }
|
||
}
|
||
</script>
|
||
<script data-goatcounter="https://www.subbu.org/gc/count"
|
||
async src="https://www.subbu.org/gc/count.js"></script>
|
||
<noscript><img src="https://www.subbu.org/gc/count?p=%2fessays%2f2019%2fstudying-an-incident%2f"></noscript>
|
||
</head>
|
||
|
||
<body>
|
||
<div class="container wrapper">
|
||
<div class="header">
|
||
|
||
<div class="avatar">
|
||
<a href="https://www.subbu.org/">
|
||
<img src="/subbu.jpg" alt="Writing is clarifying" />
|
||
</a>
|
||
</div>
|
||
|
||
<h1 class="site-title"><a href="https://www.subbu.org/">Writing is clarifying</a></h1>
|
||
<div class="site-description"><p>Writing on leadership and technology</p><nav class="nav social">
|
||
<ul class="flat"><li><a href="https://www.linkedin.com/in/subbu/" title="LinkedIn"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M16 8a6 6 0 0 1 6 6v7h-4v-7a2 2 0 0 0-2-2 2 2 0 0 0-2 2v7h-4v-7a6 6 0 0 1 6-6z"></path><rect x="2" y="9" width="4" height="12"></rect><circle cx="4" cy="4" r="2"></circle></svg></a></li><li><a href="https://bsky.app/profile/sallamar.bsky.social" title="Bluesky"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 568 501" fill="currentColor"><path d="M123.121 33.664C188.241 82.553 258.281 181.68 284 234.873c25.719-53.192 95.759-152.32 160.879-201.21C491.866-1.611 568-28.906 568 57.947c0 17.346-9.945 145.713-15.778 166.555-20.275 72.453-94.155 90.933-159.875 79.748C507.222 323.8 536.444 388.56 473.333 453.32c-119.86 122.992-172.272-30.859-185.702-70.281-2.462-7.227-3.614-10.608-3.631-7.733-.017-2.875-1.169.506-3.631 7.733-13.43 39.422-65.842 193.273-185.702 70.281-63.111-64.76-33.89-129.52 80.986-149.071-65.72 11.185-139.6-7.295-159.875-79.748C10.945 203.659 1 75.291 1 57.946 1-28.906 76.135-1.612 123.121 33.664z"></path></svg></a></li><li><a href="/index.xml" title="RSS"><i data-feather="rss"></i></a></li><li><a href="#" class="scheme-toggle" id="scheme-toggle"></a></li></ul>
|
||
</nav>
|
||
</div>
|
||
|
||
<nav class="nav">
|
||
<ul class="flat">
|
||
|
||
<li>
|
||
<a href="/">Home</a>
|
||
</li>
|
||
|
||
<li>
|
||
<a href="/coaching/">Coaching</a>
|
||
</li>
|
||
|
||
<li>
|
||
<a href="/essays">Writing</a>
|
||
</li>
|
||
|
||
<li>
|
||
<a href="/series/mountain">The Hold</a>
|
||
</li>
|
||
|
||
<li>
|
||
<a href="/subscribe">Subscribe</a>
|
||
</li>
|
||
|
||
<li>
|
||
<a href="/about">About</a>
|
||
</li>
|
||
|
||
</ul>
|
||
</nav>
|
||
</div>
|
||
|
||
|
||
<div class="post article">
|
||
<div class="post-header">
|
||
<div class="matter">
|
||
<h1 class="title">Studying an Incident</h1>
|
||
<p class="article-date"> Monday, December 30, 2019 </p>
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="series-note home-coaching-note">
|
||
<p><strong>New:</strong> <a href="/coaching/">one-on-one coaching</a> for tech professionals working through hard leadership problems—whether you manage a team or the whitespace between them.</p>
|
||
</div>
|
||
|
||
|
||
|
||
<div class="markdown">
|
||
<p>It is not often you get an opportunity to study an incident to illustrate a few lessons. A recent incident that I describe below teaches three key lessons:</p>
|
||
<ul>
|
||
<li>There are multiple perspectives on what happened and how to improve. The more complex the system is, the more perspectives you’re likely to discover.</li>
|
||
<li>Asking for what went well and how things worked, instead of just asking about what went wrong, opens possibilities for improvements that you would otherwise miss.</li>
|
||
<li>Resilience is what people do, and being resilient involves likely doing things you’ve not done before.</li>
|
||
</ul>
|
||
<p>Below is an approximate representation of the incident timeline with certain notable events. I’ve omitted a few partially explored parallel paths.</p>
|
||
<p><img src="/img/1__x5v1JAi9__8kgU94ZNOMqFQ.png" alt=""></p>
|
||
<h3 id="multiple-perspectives">Multiple Perspectives<a class="heading-anchor" href="#multiple-perspectives" aria-label="Link to this section"><svg xmlns="http://www.w3.org/2000/svg" width="0.7em" height="0.7em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.8" stroke-linecap="round" stroke-linejoin="round"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/></svg></a></h3>
|
||
<p>First, notice that there are at least four separate perspectives of this incident.</p>
|
||
<p>Team A that operates the shared compute cluster:</p>
|
||
<blockquote>
|
||
<p>“We should have checked before deleting the cluster … We should avoid manual deletes like this … We should create smaller clusters to reduce the blast radius …We should chaos test this …”</p>
|
||
</blockquote>
|
||
<p>Team B that owns the apps in the critical path:</p>
|
||
<blockquote>
|
||
<p>“They (team A) did what? How could they? … Can we bring up these apps quickly in the second region? … Who knows the steps? … Who do we need to verify?”</p>
|
||
</blockquote>
|
||
<p>Team C (external) that runs the throttled AWS services</p>
|
||
<blockquote>
|
||
<p>“They (team A) did what? How could they? … How do we (recover from the throttle)? … What is the risk to other consumers of those APIs? …”</p>
|
||
</blockquote>
|
||
<p>Team D that is orchestrating the events on the incident bridge:</p>
|
||
<blockquote>
|
||
<p>“How is the rest of the site doing? What else went wrong? … Can we shift traffic to the other region? Why not? Have we tested this before? … Do we have a list of apps running on that cluster? … Who do we need to call?”</p>
|
||
</blockquote>
|
||
<p>Such perspectives show that the narrative of the postmortem report can vary based on who is writing the postmortem. Collecting all these perspectives, reconstructing the incident, and identifying potential corrective actions, may take separate interviews with each group. A single live large gathering of all the parties involved in a post-incident review meeting may not uncover all these perspectives. A typical operational review with a senior titled person at the head of the table will undoubtedly miss most of these perspectives, and the discussion will most likely focus on what that person feels everyone should do.</p>
|
||
<h3 id="what-went-wrong-vs-what-went-well-andhow">What Went Wrong vs. What Went Well and How<a class="heading-anchor" href="#what-went-wrong-vs-what-went-well-andhow" aria-label="Link to this section"><svg xmlns="http://www.w3.org/2000/svg" width="0.7em" height="0.7em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.8" stroke-linecap="round" stroke-linejoin="round"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/></svg></a></h3>
|
||
<p>A common practice of conducting incident postmortems to ask “what went wrong,” then a series of whys to find what that happened, and then identify future corrective actions to prevent such an incident from reoccurring.</p>
|
||
<p>Such an approach, in this example, will lead to the first event in the incident timeline, which is when an operator made a change to delete a compute cluster. Subsequent probing would discover that that the operator made an incorrect assumption, that the operator did not make sufficient attempts to verify that assumption, and that the change was not peer-reviewed.</p>
|
||
<p>Consequently, you would identify potential corrective actions like the following:</p>
|
||
<ul>
|
||
<li>Automating all deletes to include validation checks</li>
|
||
<li>Peer-review of all manual changes</li>
|
||
<li>Avoid creating large clusters to reduce the blast radius</li>
|
||
<li>Chaos testing cluster rehydration</li>
|
||
</ul>
|
||
<p>However, asking for “what went well and how” would uncover a different set of events and potential corrective actions.</p>
|
||
<p>Recall from the timeline that the deleted cluster was hosting 100s of applications, most of which were customer critical. But why was impact not broader than noticed? What protected the rest? This line of questioning would uncover the following:</p>
|
||
<ul>
|
||
<li>Most of those apps are redundantly deployed in a second region on a similar compute cluster in an active-active configuration.</li>
|
||
<li>The traffic management layer automatically routed the traffic to those apps after the apps hosted on the deleted cluster became unavailable.</li>
|
||
<li>The compute cluster in the second region scaled up automatically to support additional traffic.</li>
|
||
<li>However, a few apps, including the ones found to be in the critical path of the impacted customer segment, were not redundantly deployed in the second region.</li>
|
||
<li>The deployment system and the post-deployment configuration and validation checks were time-consuming, which prevented the quick deployment of those apps in the second region.</li>
|
||
</ul>
|
||
<p>In other words, a certain amount of robustness was built into the architecture, which helped limit the impact.</p>
|
||
<p>You would then come up with a different set of action items from those identified in the “what went wrong” approach.</p>
|
||
<ul>
|
||
<li>Ensure that all critical apps are deployed in at least two regions. Q: How do we know what is critical?</li>
|
||
<li>Identify all critical apps. Q: How do we keep it up to date?</li>
|
||
<li>Periodically test automated failover between regions.</li>
|
||
</ul>
|
||
<p>This example shows that probing for both “what went wrong” and “what went well and how” are essential. Each contributes to improving your understanding of the complexity of the system, how things work and identify potential corrective actions. Extending this line of thinking to all the perspectives listed in the previous section further enriches this understanding.</p>
|
||
<h3 id="resilience-is-what-peopledo">Resilience is What People do<a class="heading-anchor" href="#resilience-is-what-peopledo" aria-label="Link to this section"><svg xmlns="http://www.w3.org/2000/svg" width="0.7em" height="0.7em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.8" stroke-linecap="round" stroke-linejoin="round"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/></svg></a></h3>
|
||
<p>Finally, this incident also illustrates that resilience is what people do. This incident involved opening up minds to alternative explanations, conducting parallel streams of analyses, adapting to new information as it emerged, and attempting things not done before. Each major component involved in this incident behaved as it was supposed to. Regardless, the system as a whole faced a catastrophe, and it took human coordination, quick thinking, and ingenuity to recover from the disaster. In fact, what is typical between incidents is how people come together to restore the system, with the rest being unique to each incident.</p>
|
||
<p>As John Allspaw says:</p>
|
||
<p>Though such a distinction between robustness and resilience seems nuanced and pedantic, understanding and appreciating the difference can lead to vastly different thinking, investments, and outcomes. For example, not recognizing this difference may limit you to only focus on software solutions like below.</p>
|
||
<ul>
|
||
<li>Improving observability, so you’re quick to know and learn about the dynamic behavior of the system.</li>
|
||
<li>Building or adopting closed-loop automation systems — These are solutions that follow an act-observe-correct pattern in a closed-loop to automatically remediate components of a system when those are observed to be drifting from their desired state.</li>
|
||
<li>Adopting cloud-managed services for their availability and more predictable behaviors, to replace do-it-yourself solutions.</li>
|
||
<li>Reducing blast radius to contain faults and to minimize coupling with techniques like circuit breakers and fall-backs</li>
|
||
<li>Traffic shifting and shedding for quick recovery</li>
|
||
<li>Chaos testing the system by subjecting it to various failure hypotheses</li>
|
||
</ul>
|
||
<p>These are all essential investments. But such steps alone will not let the overall system, which includes people, the tools they use, and the rituals they follow, build the capacity to adapt to changing conditions during a catastrophe and to learn from those conditions. You need to go beyond to include the following to build resilience.</p>
|
||
<ul>
|
||
<li>Understand how humans interact with the system and the assumptions they make when operating the system.</li>
|
||
<li>Avoid language that prevents dialog and discovery during and after incidents.</li>
|
||
<li>Move away from root cause hunting to understanding how the system works, improve your mental models for the system, and use that understanding to invest for the future.</li>
|
||
<li>Ensure that people with titles are also learning. Most in such positions likely have dated understanding of how to build and operate complex production systems. However, since such people set the tone of the culture for their teams, it is vital for them to also learn in this process.</li>
|
||
</ul>
|
||
<h3 id="in-summary">In Summary<a class="heading-anchor" href="#in-summary" aria-label="Link to this section"><svg xmlns="http://www.w3.org/2000/svg" width="0.7em" height="0.7em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.8" stroke-linecap="round" stroke-linejoin="round"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/></svg></a></h3>
|
||
<p>For any company generating value from its production systems, complexity is a moat. Despite our attempts to shuffle the complexity through automated layers of abstractions, complexity is the natural state of real-world systems. Each layer we add and each change we make change our assumptions about how the system is supposed to work, and how it is actually working. Incidents provide an excellent opportunity to validate those assumptions and discover ways to improve. The practice of continuous learning from incidents must be an integral part of your operations-culture to build resiliency.</p>
|
||
<p>Thanks to <a href="https://medium.com/@willie.wheeler">Willie Wheeler</a> for reviewing an earlier draft of this post.</p>
|
||
|
||
</div>
|
||
|
||
<div class="subscribe-cta">
|
||
<p><em>If you enjoyed this essay, consider subscribing for future essays. I write about
|
||
technology and leadership. I will never share your email with anyone. You can unsubscribe
|
||
at any time.</em></p>
|
||
<form
|
||
name="newsletter"
|
||
method="POST"
|
||
action="/api/subscribe"
|
||
>
|
||
<input class="hidden" name="gotcha" />
|
||
<input type="email" name="email" id="bd-email" placeholder="Your email" required />
|
||
<div class="cf-turnstile" data-sitekey="0x4AAAAAACqsW1JNiZtH5Pc7"></div>
|
||
<input type="submit" value="Subscribe" />
|
||
</form>
|
||
<script src="https://challenges.cloudflare.com/turnstile/v0/api.js" async defer></script>
|
||
</div>
|
||
|
||
|
||
<nav class="post-nav">
|
||
<a class="post-nav-prev" href="https://www.subbu.org/essays/2019/retinitis-pigmentosa/">← Retinitis Pigmentosa</a>
|
||
<a class="post-nav-next" href="https://www.subbu.org/essays/2019/lessons-from-2019/">Lessons from 2019 →</a>
|
||
</nav>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<h2 class="related-title">See Also</h2>
|
||
|
||
|
||
<h3 class="related-subtitle">Related</h3>
|
||
<ul class="related-posts">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<li>
|
||
<a href="https://www.subbu.org/essays/2019/why-learn-from-incidents/">Why Learn from Incidents</a> <small>(96% match, Dec 17, 2019)</small>
|
||
</li>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<li>
|
||
<a href="https://www.subbu.org/essays/2017/fault-domains-and-the-vegas-rule/">Fault Domains and the Vegas Rule</a> <small>(94% match, Feb 17, 2017)</small>
|
||
</li>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<li>
|
||
<a href="https://www.subbu.org/essays/2019/if-only-production-incidents-could-speak/">If Only Production Incidents Could Speak</a> <small>(94% match, Jul 18, 2019)</small>
|
||
</li>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<li>
|
||
<a href="https://www.subbu.org/essays/2019/incidents-trends-from-the-trenches/">Incidents — Trends from the Trenches</a> <small>(92% match, Feb 26, 2019)</small>
|
||
</li>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<li>
|
||
<a href="https://www.subbu.org/essays/2019/forming-failure-hypothesis/">Forming Failure Hypothesis</a> <small>(92% match, Sep 27, 2019)</small>
|
||
</li>
|
||
|
||
</ul>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<h3 class="related-subtitle">More to Read</h3>
|
||
<ul class="related-posts">
|
||
|
||
<li>
|
||
<a href="/essays/2026/the-elephant-in-the-brownfield/">The Elephant in the Brownfield</a> <small>(Aug 21, 2026)</small>
|
||
</li>
|
||
|
||
<li>
|
||
<a href="/essays/2026/two-bars/">Two Bars, Spreading Apart</a> <small>(Jul 28, 2026)</small>
|
||
</li>
|
||
|
||
<li>
|
||
<a href="/essays/2026/a-lesson-from-the-feedlot/">A Lesson From the Feedlot</a> <small>(Jun 3, 2026)</small>
|
||
</li>
|
||
|
||
<li>
|
||
<a href="/essays/2026/a-lesson-from-the-cockpit/">A Lesson From the Cockpit</a> <small>(May 8, 2026)</small>
|
||
</li>
|
||
|
||
<li>
|
||
<a href="/essays/2026/the-hold/">The Hold</a> <small>(Apr 22, 2026)</small>
|
||
</li>
|
||
|
||
</ul>
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
</div>
|
||
|
||
<div class="footer wrapper">
|
||
<nav class="nav">
|
||
<div>2026 © Subbu Allamaraju </div>
|
||
</nav>
|
||
</div><script>feather.replace()</script>
|
||
<!-- Cloudflare Pages Analytics --><script defer src='https://static.cloudflareinsights.com/beacon.min.js' data-cf-beacon='{"token": "a90d59c0629844c39a1d6db2cb98e81a"}'></script><!-- Cloudflare Pages Analytics --><script>(function(){function c(){var b=a.contentDocument||(a.contentWindow&&a.contentWindow.document);if(b){var d=b.createElement('script');d.innerHTML="window.__CF$cv$params={r:'a39abd0f6c3ee196',t:'MTc4OTE3MjM0NA=='};var a=document.createElement('script');a.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js';document.getElementsByTagName('head')[0].appendChild(a);";b.getElementsByTagName('head')[0].appendChild(d)}}if(document.body){var a=document.createElement('iframe');a.height=1;a.width=1;a.style.position='absolute';a.style.top=0;a.style.left=0;a.style.border='none';a.style.visibility='hidden';document.body.appendChild(a);if('loading'!==document.readyState)c();else if(window.addEventListener)document.addEventListener('DOMContentLoaded',c);else{var e=document.onreadystatechange||function(){};document.onreadystatechange=function(b){e(b);'loading'!==document.readyState&&(document.onreadystatechange=e,c())}}}})();</script></body>
|
||
</html>
|