50 lines
8.0 KiB
HTML
50 lines
8.0 KiB
HTML
<!doctype html><html lang=en><head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><meta charset=utf-8><title>QA Instability Implies Production Instability - michaelnygard.com</title>
|
|
<meta content='QA Instability Implies Production Instability - michaelnygard.com' property='title'><meta content='QA Instability Implies Production Instability - michaelnygard.com' property='og:title'><meta property="og:description" content="Many companies that have trouble delivering software on time exhibit a common pathology. Developers working on the next release are frequently interrupted for production support issues with the current release. These interrupts never appear in project schedules but can take up half of the developers' hours. When you include the cost of task-switching, this means less than half of their available time is spent on the new feature work. Invariably, when I see a lot of developer effort in production support I also find an unreliable QA environment."><meta property="og:type" content="article"><meta property="og:url" content="https://michaelnygard.com/blog/2016/07/qa-instability-implies-production-instability/"><meta property="article:published_time" content="2016-07-14T10:06:11-05:00"><meta property="article:modified_time" content="2016-07-14T10:06:11-05:00"><meta property="article:section" content="blog"><meta name=generator content="Hugo 0.123.7"><link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:400,600" rel=stylesheet><style type=text/css>:root{--main-color:#8C6056;--secondary-color:#AFD5AA;--logo-text-color:#fff;--body-text-color:#3d3d3d;--heading-text-color:#383838;--background-color:#fff}</style><link href=/css/tachyons.min.css rel=stylesheet><link href=/css/styles.css rel=stylesheet><link rel=icon href=/favicon.ico type=image/x-icon><script id=MathJax-script async src=https://cdn.jsdelivr.net/npm/mathjax@4/tex-mml-chtml.js></script><script>MathJax={tex:{displayMath:[["\\[","\\]"],["$$","$$"]],inlineMath:[["\\(","\\)"]]},loader:{load:["ui/safe"]}}</script></head><body class=global-font><nav class="justify-between border-box pa3 pl3-l pr2-l mt1 mt0-ns" id=navbar><div class=flex><a class="f4 fw6 ttu no-underline dim bg-main-color pv1 ph2 br2" id=site-title href=/ title=Home>michaelnygard.com</a></div></nav><main class="center mv4 content-width ph3"><div class="f3 fw6 heading-color heading-font post-title">QA Instability Implies Production Instability</div><p class="silver f6 mt1 mb4 post-meta">Posted on <time>14 Jul 2016</time></p><div class="lh-copy post-content"><p>Many companies that have trouble delivering software on time exhibit a
|
|
common pathology. Developers working on the next release are
|
|
frequently interrupted for production support issues with the current
|
|
release. These interrupts never appear in project schedules but can
|
|
take up half of the developers' hours. When you include the cost of
|
|
task-switching, this means less than half of their available time is
|
|
spent on the new feature work.</p><p>Invariably, when I see a lot of developer effort in production support
|
|
I also find an unreliable QA environment. It is both unreliable in
|
|
that it is frequently not available for testing, and unreliable in the
|
|
sense that the system's behavior in QA is not a good predictor of its
|
|
behavior in production.</p><p>QA environments are often configured differently than production. Not
|
|
just in the usual sense of consuming a QA version of external
|
|
services, but also in more basic ways. Server topology may be
|
|
different. Memory settings, capacity ratios, and the presence of
|
|
network components can all vary. QA often has a much simpler traffic
|
|
routing scheme than production, particularly when a CDN is involved.</p><p>The other major source of QA unavailability has to do with data
|
|
refreshes. QA environments either run with a miniscule, curated test
|
|
data set, or they use some form of backward migration from production
|
|
data. Each backward migration can be very disruptive, leading to one
|
|
or more days where QA is not available.</p><p>Disruption arises when testers have to do manual data setup in order
|
|
to test new features. These setups get overwritten with the next
|
|
refresh. Sometimes, production data must be cleansed or scrubbed of
|
|
PII before use in QA. This cleansing process often introduces its own
|
|
data quality problems. The backward migration process must also be
|
|
kept up to date so it can propagate data back into the schema for the
|
|
next release. This requires copying data and schema into QA, then
|
|
forward-migrating the schema according to the new release.</p><p>When many teams contend to get into a QA environment, that contention
|
|
can result in lost time as well. Time is lost in delays when one team
|
|
cannot move their code into QA during another team's test. It is also
|
|
lost when one team overwrites test data that a different team had set
|
|
up. And it can be lost when one team's code has bugs that prevent
|
|
other teams from proceeding with their tests. Suppose one team
|
|
works on login and registration, while another team works on friend
|
|
requests. Clearly, the friend requests team cannot do their testing
|
|
when login is broken. This last issue also applies across service
|
|
boundaries: a service consumer may not be able to test because the QA
|
|
version of their service provider is broken.</p><p>Finally, problems in QA simply take a lower priority than problems in
|
|
production. Thus, the operations team may be fully consumed with
|
|
current production issues, leaving the QA environment broken for
|
|
extended periods. In a vicious feedback loop, this makes it likely
|
|
that the next release will also create production problems.</p><p>My recommendations are these:</p><ul class=org-ul><li>Give priority to well-functioning test environments.</li><li>Virtualize your test environments, so you can avoid inter-team
|
|
dependencies on a QA environment.</li><li>Automate the backward data propagation, and make it part of spinning
|
|
up a QA environment. When you must scrub PII, automate that process
|
|
so that every QA environment can draw from a snapshot of cleansed
|
|
data without impinging on the production DBAs.</li><li>If your QA stays unavailable because there are too many production
|
|
issues, recognize that this is a self-sustaining pattern. You can
|
|
temporarily redirect a "SWAT" team to fix QA and it will pay
|
|
dividends for all future releases.</li></ul></div></main><div class="pagination tc tr-l db fixed-l bottom-2-l right-2-l mb3 mb0-l"><a id=scroll-to-top class="f6 o-0 link br2 ph2 pv1 mb1 bg-main-color pointer" onclick=topFunction() style="color:#fff;visibility:hidden;display:none;transition:opacity .5s,visibility .5s" title="back to top">back to top</a><br><p class="mb0 mt2"><a href=https://michaelnygard.com/blog/2016/07/wittgenstein-and-design/>prev post</a>
|
|
<a href=https://michaelnygard.com/blog/2016/07/remember-dat/>next post</a></p></div><footer class="content-width mt0 mt5-l mb4 f6 center ph3 gray tc tl-l"><hr class="dn db-l ml0-l gray w3"><br>Powered by <a href=https://gohugo.io/ target=_blank class="link gray dim">Hugo</a>, based on the <a href=https://github.com/lingxz/er target=_blank class="link gray dim">Er</a> theme.<br>Copyright (c) 2002 - 2026 Michael T. Nygard</footer><script type=text/javascript>var prevScrollpos=window.pageYOffset;window.onscroll=function(){var e=window.pageYOffset;document.getElementById("tag-cloud")!==null&&(prevScrollpos>e?(document.getElementById("tag-cloud").style.visibility="visible",document.getElementById("tag-cloud").style.opacity="1"):(document.getElementById("tag-cloud").style.visibility="hidden",document.getElementById("tag-cloud").style.opacity="0")),document.body.scrollTop>1e3||document.documentElement.scrollTop>1e3?(document.getElementById("scroll-to-top").style.display="inline",document.getElementById("scroll-to-top").style.visibility="visible",document.getElementById("scroll-to-top").style.opacity="1"):(document.getElementById("scroll-to-top").style.visibility="hidden",document.getElementById("scroll-to-top").style.opacity="0"),prevScrollpos=e};function topFunction(){document.body.scrollTop=0,document.documentElement.scrollTop=0}</script></body></html> |