SRE weekly 所有文章
This commit is contained in:
583
sreweekly/articles/339/02-how-complex-systems-fail.html
Normal file
583
sreweekly/articles/339/02-how-complex-systems-fail.html
Normal file
@@ -0,0 +1,583 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
|
||||
<meta
|
||||
name="viewport"
|
||||
content="width=device-width, initial-scale=1.0 maximum-scale=5.0"
|
||||
/>
|
||||
<meta
|
||||
http-equiv="Content-Security-Policy"
|
||||
content="default-src 'self'; style-src 'sha256-xS9q5srXP4ACNWPJWocv6Gm+PoVzDgBhIU2TxjRMyL8=';"
|
||||
/>
|
||||
|
||||
<link rel="canonical" href="https://how.complexsystems.fail/" />
|
||||
|
||||
<title>How Complex Systems Fail</title>
|
||||
<style>
|
||||
@media (prefers-color-scheme: light) {
|
||||
:root {
|
||||
--text-color: #000;
|
||||
--footnote-color: #eee;
|
||||
}
|
||||
}
|
||||
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root {
|
||||
--text-color: #fff;
|
||||
--background-color: #252525;
|
||||
--link-color: rgb(125, 175, 255);
|
||||
--hover-color: rgb(60, 110, 200);
|
||||
--footnote-color: #454545;
|
||||
}
|
||||
|
||||
body {
|
||||
background-color: var(--background-color);
|
||||
}
|
||||
|
||||
a {
|
||||
color: var(--link-color);
|
||||
}
|
||||
|
||||
a:hover {
|
||||
color: var(--hover-color);
|
||||
}
|
||||
}
|
||||
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont,
|
||||
"Segoe UI", "Roboto", "Oxygen", "Ubuntu", "Cantarell",
|
||||
"Fira Sans", "Droid Sans", "Helvetica Neue",
|
||||
sans-serif;
|
||||
color: var(--text-color);
|
||||
font-size: 20px;
|
||||
margin: 0 auto;
|
||||
max-width: 700px;
|
||||
padding: 10px;
|
||||
line-height: 25px;
|
||||
}
|
||||
|
||||
header {
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
header h1 {
|
||||
font-size: 24px;
|
||||
font-weight: bolder;
|
||||
}
|
||||
|
||||
header h1 p {
|
||||
font-size: 16px;
|
||||
font-weight: lighter;
|
||||
text-align: justify;
|
||||
}
|
||||
|
||||
ol {
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
ol li {
|
||||
margin-left: 10px;
|
||||
padding-bottom: 10px;
|
||||
}
|
||||
|
||||
ol li h2 {
|
||||
font-size: 20px;
|
||||
text-align: initial;
|
||||
}
|
||||
|
||||
ol li h2 a {
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
ol li h2 a:hover {
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
p[id^="footnote-"] {
|
||||
font-size: 0.9em;
|
||||
padding: 5px;
|
||||
background-color: var(--footnote-color);
|
||||
}
|
||||
|
||||
h2 {
|
||||
font-size: 16px;
|
||||
}
|
||||
|
||||
ul {
|
||||
list-style-type: none;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
ul li {
|
||||
margin-bottom: 18px;
|
||||
margin-left: 2em;
|
||||
text-indent: -2em;
|
||||
}
|
||||
|
||||
footer {
|
||||
border-top: 1px solid var(--text-color);
|
||||
font-size: 16px;
|
||||
}
|
||||
|
||||
@media (max-width: 700px) {
|
||||
ol {
|
||||
padding-left: 1em;
|
||||
padding-right: 1em;
|
||||
}
|
||||
|
||||
ol li {
|
||||
text-align: justify;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<header>
|
||||
<h1>
|
||||
How Complex Systems Fail
|
||||
<p>
|
||||
<em>
|
||||
(Being a Short Treatise on the Nature of Failure; How Failure is
|
||||
Evaluated; How Failure is Attributed to Proximate Cause; and the
|
||||
Resulting New Understanding of Patient Safety)
|
||||
</em>
|
||||
</p>
|
||||
</h1>
|
||||
<div>
|
||||
Richard I. Cook, MD <br />
|
||||
Cognitive Technologies Labratory <br />
|
||||
University of Chicago
|
||||
</div>
|
||||
</header>
|
||||
<ol>
|
||||
<li id="1">
|
||||
<h2 id="Complex-systems-are-intrinsically-hazardous-systems">
|
||||
<a href="#1">Complex systems are intrinsically hazardous systems.</a>
|
||||
</h2>
|
||||
<p>
|
||||
All of the interesting systems (e.g. transportation, healthcare, power
|
||||
generation) are inherently and unavoidably hazardous by the own
|
||||
nature. The frequency of hazard exposure can sometimes be changed but
|
||||
the processes involved in the system are themselves intrinsically and
|
||||
irreducibly hazardous. It is the presence of these hazards that drives
|
||||
the creation of defenses against hazard that characterize these
|
||||
systems.
|
||||
</p>
|
||||
</li>
|
||||
<li id="2">
|
||||
<h2
|
||||
id="Complex-systems-are-heavily-and-successfully-defended-against-failure"
|
||||
>
|
||||
<a href="#2"
|
||||
>Complex systems are heavily and successfully defended against
|
||||
failure</a
|
||||
>
|
||||
</h2>
|
||||
<p>
|
||||
The high consequences of failure lead over time to the construction of
|
||||
multiple layers of defense against failure. These defenses include
|
||||
obvious technical components (e.g. backup systems, ‘safety’ features
|
||||
of equipment) and human components (e.g. training, knowledge) but also
|
||||
a variety of organizational, institutional, and regulatory defenses
|
||||
(e.g. policies and procedures, certification, work rules, team
|
||||
training). The effect of these measures is to provide a series of
|
||||
shields that normally divert operations away from accidents.
|
||||
</p>
|
||||
</li>
|
||||
<li id="3">
|
||||
<h2
|
||||
id="Catastrophe-requires-multiple-failures-single-point-failures-are-not-enough"
|
||||
>
|
||||
<a href="#3"
|
||||
>Catastrophe requires multiple failures – single point failures are
|
||||
not enough.</a
|
||||
>
|
||||
</h2>
|
||||
<p>
|
||||
The array of defenses works. System operations are generally
|
||||
successful. Overt catastrophic failure occurs when small, apparently
|
||||
innocuous failures join to create opportunity for a systemic accident.
|
||||
Each of these small failures is necessary to cause catastrophe but
|
||||
only the combination is sufficient to permit failure. Put another way,
|
||||
there are many more failure opportunities than overt system accidents.
|
||||
Most initial failure trajectories are blocked by designed system
|
||||
safety components. Trajectories that reach the operational level are
|
||||
mostly blocked, usually by practitioners.
|
||||
</p>
|
||||
</li>
|
||||
<li id="4">
|
||||
<h2
|
||||
id="Complex-systems-contain-changing-mixtures-of-failures-latent-within-them"
|
||||
>
|
||||
<a href="#4"
|
||||
>Complex systems contain changing mixtures of failures latent within
|
||||
them.</a
|
||||
>
|
||||
</h2>
|
||||
<p>
|
||||
The complexity of these systems makes it impossible for them to run
|
||||
without multiple flaws being present. Because these are individually
|
||||
insufficient to cause failure they are regarded as minor factors
|
||||
during operations. Eradication of all latent failures is limited
|
||||
primarily by economic cost but also because it is difficult before the
|
||||
fact to see how such failures might contribute to an accident. The
|
||||
failures change constantly because of changing technology, work
|
||||
organization, and efforts to eradicate failures.
|
||||
</p>
|
||||
</li>
|
||||
<li id="5">
|
||||
<h2 id="Complex-systems-run-in-degraded-mode">
|
||||
<a href="#5">Complex systems run in degraded mode.</a>
|
||||
</h2>
|
||||
<p>
|
||||
A corollary to the preceding point is that complex systems run as
|
||||
broken systems. The system continues to function because it contains
|
||||
so many redundancies and because people can make it function, despite
|
||||
the presence of many flaws. After accident reviews nearly always note
|
||||
that the system has a history of prior ‘proto-accidents’ that nearly
|
||||
generated catastrophe. Arguments that these degraded conditions should
|
||||
have been recognized before the overt accident are usually predicated
|
||||
on naïve notions of system performance. System operations are dynamic,
|
||||
with components (organizational, human, technical) failing and being
|
||||
replaced continuously.
|
||||
</p>
|
||||
</li>
|
||||
<li id="6">
|
||||
<h2 id="Catastrophe-is-always-just-around-the-corner">
|
||||
<a href="#6">Catastrophe is always just around the corner.</a>
|
||||
</h2>
|
||||
<p>
|
||||
Complex systems possess potential for catastrophic failure. Human
|
||||
practitioners are nearly always in close physical and temporal
|
||||
proximity to these potential failures – disaster can occur at any time
|
||||
and in nearly any place. The potential for catastrophic outcome is a
|
||||
hallmark of complex systems. It is impossible to eliminate the
|
||||
potential for such catastrophic failure; the potential for such
|
||||
failure is always present by the system’s own nature.
|
||||
</p>
|
||||
</li>
|
||||
<li id="7">
|
||||
<h2
|
||||
id="Post-accident-attribution-to-a-root-cause-is-fundamentally-wrong"
|
||||
>
|
||||
<a href="#7"
|
||||
>Post-accident attribution to a ‘root cause’ is fundamentally
|
||||
wrong.</a
|
||||
>
|
||||
</h2>
|
||||
<p>
|
||||
Because overt failure requires multiple faults, there is no isolated
|
||||
‘cause’ of an accident. There are multiple contributors to accidents.
|
||||
Each of these is necessarily insufficient in itself to create an
|
||||
accident. Only jointly are these causes sufficient to create an
|
||||
accident. Indeed, it is the linking of these causes together that
|
||||
creates the circumstances required for the accident. Thus, no
|
||||
isolation of the ‘root cause’ of an accident is possible. The
|
||||
evaluations based on such reasoning as ‘root cause’ do not reflect a
|
||||
technical understanding of the nature of failure but rather the
|
||||
social, cultural need to blame specific, localized forces or events
|
||||
for outcomes. <a href="#footnote-1"><sup>1</sup></a>
|
||||
</p>
|
||||
|
||||
<p id="footnote-1">
|
||||
<a href="#7"><sup>1</sup></a> Anthropological field research provides
|
||||
the clearest demonstration of the social construction of the notion of
|
||||
‘cause’ (cf. Goldman L (1993), The Culture of Coincidence: accident
|
||||
and absolute liability in Huli, New York: Clarendon Press; and also
|
||||
Tasca L (1990), The Social Construction of Human Error, Unpublished
|
||||
doctoral dissertation, Department of Sociology, State University of
|
||||
New York at Stonybrook)
|
||||
</p>
|
||||
</li>
|
||||
<li id="8">
|
||||
<h2
|
||||
id="Hindsight-biases-post-accident-assessments-of-human-performance"
|
||||
>
|
||||
<a href="#8"
|
||||
>Hindsight biases post-accident assessments of human performance.</a
|
||||
>
|
||||
</h2>
|
||||
<p>
|
||||
Knowledge of the outcome makes it seem that events leading to the
|
||||
outcome should have appeared more salient to practitioners at the time
|
||||
than was actually the case. This means that
|
||||
<em>ex post facto</em> accident analysis of human performance is
|
||||
inaccurate. The outcome knowledge poisons the ability of
|
||||
after-accident observers to recreate the view of practitioners before
|
||||
the accident of those same factors. It seems that practitioners
|
||||
“should have known” that the factors would “inevitably” lead to an
|
||||
accident. <a href="#footnote-2"><sup>2</sup></a>
|
||||
<em>
|
||||
Hindsight bias remains the primary obstacle to accident
|
||||
investigation, especially when expert human performance is involved.
|
||||
</em>
|
||||
</p>
|
||||
<p id="footnote-2">
|
||||
<a href="#8"><sup>2</sup></a> This is not a feature of medical
|
||||
judgements or technical ones, but rather of all human cognition about
|
||||
past events and their causes.
|
||||
</p>
|
||||
</li>
|
||||
<li id="9">
|
||||
<h2
|
||||
id="Human-operators-have-dual-roles-as-producers-and-as-defenders-against-failure"
|
||||
>
|
||||
<a href="#9"
|
||||
>Human operators have dual roles: as producers & as defenders
|
||||
against failure.</a
|
||||
>
|
||||
</h2>
|
||||
<p>
|
||||
The system practitioners operate the system in order to produce its
|
||||
desired product and also work to forestall accidents. This dynamic
|
||||
quality of system operation, the balancing of demands for production
|
||||
against the possibility of incipient failure is unavoidable. Outsiders
|
||||
rarely acknowledge the duality of this role. In non-accident filled
|
||||
times, the production role is emphasized. After accidents, the defense
|
||||
against failure role is emphasized. At either time, the outsider’s
|
||||
view misapprehends the operator’s constant, simultaneous engagement
|
||||
with both roles.
|
||||
</p>
|
||||
</li>
|
||||
<li id="10">
|
||||
<h2 id="All-practitioner-actions-are-gambles">
|
||||
<a href="#10">All practitioner actions are gambles.</a>
|
||||
</h2>
|
||||
<p>
|
||||
After accidents, the overt failure often appears to have been
|
||||
inevitable and the practitioner’s actions as blunders or deliberate
|
||||
willful disregard of certain impending failure. But all practitioner
|
||||
actions are actually gambles, that is, acts that take place in the
|
||||
face of uncertain outcomes. The degree of uncertainty may change from
|
||||
moment to moment. That practitioner actions are gambles appears clear
|
||||
after accidents; in general, <em>post hoc</em> analysis regards these
|
||||
gambles as poor ones. But the converse: that successful outcomes are
|
||||
also the result of gambles; is not widely appreciated.
|
||||
</p>
|
||||
</li>
|
||||
<li id="11">
|
||||
<h2 id="Actions-at-the-sharp-end-resolve-all-ambiguity">
|
||||
<a href="#11">Actions at the sharp end resolve all ambiguity.</a>
|
||||
</h2>
|
||||
<p>
|
||||
Organizations are ambiguous, often intentionally, about the
|
||||
relationship between production targets, efficient use of resources,
|
||||
economy and costs of operations, and acceptable risks of low and high
|
||||
consequence accidents. All ambiguity is resolved by actions of
|
||||
practitioners at the sharp end of the system. After an accident,
|
||||
practitioner actions may be regarded as ‘errors’ or ‘violations’ but
|
||||
these evaluations are heavily biased by hindsight and ignore the other
|
||||
driving forces, especially production pressure.
|
||||
</p>
|
||||
</li>
|
||||
<li id="12">
|
||||
<h2
|
||||
id="Human-practitioners-are-the-adaptable-element-of-complex-systems"
|
||||
>
|
||||
<a href="#12"
|
||||
>Human practitioners are the adaptable element of complex
|
||||
systems.</a
|
||||
>
|
||||
</h2>
|
||||
<p>
|
||||
Practitioners and first line management actively adapt the system to
|
||||
maximize production and minimize accidents. These adaptations often
|
||||
occur on a moment by moment basis. Some of these adaptations include:
|
||||
(1) Restructuring the system in order to reduce exposure of vulnerable
|
||||
parts to failure. (2) Concentrating critical resources in areas of
|
||||
expected high demand. (3) Providing pathways for retreat or recovery
|
||||
from expected and unexpected faults. (4) Establishing means for early
|
||||
detection of changed system performance in order to allow graceful
|
||||
cutbacks in production or other means of increasing resiliency.
|
||||
</p>
|
||||
</li>
|
||||
<li id="13">
|
||||
<h2 id="Human-expertise-in-complex-systems-is-constantly-changing">
|
||||
<a href="#13"
|
||||
>Human expertise in complex systems is constantly changing</a
|
||||
>
|
||||
</h2>
|
||||
<p>
|
||||
Complex systems require substantial human expertise in their operation
|
||||
and management. This expertise changes in character as technology
|
||||
changes but it also changes because of the need to replace experts who
|
||||
leave. In every case, training and refinement of skill and expertise
|
||||
is one part of the function of the system itself. At any moment,
|
||||
therefore, a given complex system will contain practitioners and
|
||||
trainees with varying degrees of expertise. Critical issues related to
|
||||
expertise arise from (1) the need to use scarce expertise as a
|
||||
resource for the most difficult or demanding production needs and (2)
|
||||
the need to develop expertise for future use.
|
||||
</p>
|
||||
</li>
|
||||
<li id="14">
|
||||
<h2 id="Change-introduces-new-forms-of-failure">
|
||||
<a href="#14">Change introduces new forms of failure.</a>
|
||||
</h2>
|
||||
<p>
|
||||
The low rate of overt accidents in reliable systems may encourage
|
||||
changes, especially the use of new technology, to decrease the number
|
||||
of low consequence but high frequency failures. These changes maybe
|
||||
actually create opportunities for new, low frequency but high
|
||||
consequence failures. When new technologies are used to eliminate well
|
||||
understood system failures or to gain high precision performance they
|
||||
often introduce new pathways to large scale, catastrophic failures.
|
||||
Not uncommonly, these new, rare catastrophes have even greater impact
|
||||
than those eliminated by the new technology. These new forms of
|
||||
failure are difficult to see before the fact; attention is paid mostly
|
||||
to the putative beneficial characteristics of the changes. Because
|
||||
these new, high consequence accidents occur at a low rate, multiple
|
||||
system changes may occur before an accident, making it hard to see the
|
||||
contribution of technology to the failure.
|
||||
</p>
|
||||
</li>
|
||||
<li id="15">
|
||||
<h2
|
||||
id="Views-of-cause-limit-the-effectiveness-of-defenses-against-future-events"
|
||||
>
|
||||
<a href="#15"
|
||||
>Views of ‘cause’ limit the effectiveness of defenses against future
|
||||
events.</a
|
||||
>
|
||||
</h2>
|
||||
<p>
|
||||
Post-accident remedies for “human error” are usually predicated on
|
||||
obstructing activities that can “cause” accidents. These
|
||||
end-of-the-chain measures do little to reduce the likelihood of
|
||||
further accidents. In fact that likelihood of an identical accident is
|
||||
already extraordinarily low because the pattern of latent failures
|
||||
changes constantly. Instead of increasing safety, post-accident
|
||||
remedies usually increase the coupling and complexity of the system.
|
||||
This increases the potential number of latent failures and also makes
|
||||
the detection and blocking of accident trajectories more difficult.
|
||||
</p>
|
||||
</li>
|
||||
<li id="16">
|
||||
<h2
|
||||
id="Safety-is-a-characteristic-of-systems-and-not-of-their-components"
|
||||
>
|
||||
<a href="#16"
|
||||
>Safety is a characteristic of systems and not of their
|
||||
components</a
|
||||
>
|
||||
</h2>
|
||||
<p>
|
||||
Safety is an emergent property of systems; it does not reside in a
|
||||
person, device or department of an organization or system. Safety
|
||||
cannot be purchased or manufactured; it is not a feature that is
|
||||
separate from the other components of the system. This means that
|
||||
safety cannot be manipulated like a feedstock or raw material. The
|
||||
state of safety in any system is always dynamic; continuous systemic
|
||||
change insures that hazard and its management are constantly changing.
|
||||
</p>
|
||||
</li>
|
||||
<li id="17">
|
||||
<h2 id="People-continuously-create-safety">
|
||||
<a href="#17">People continuously create safety.</a>
|
||||
</h2>
|
||||
<p>
|
||||
Failure free operations are the result of activities of people who
|
||||
work to keep the system within the boundaries of tolerable
|
||||
performance. These activities are, for the most part, part of normal
|
||||
operations and superficially straightforward. But because system
|
||||
operations are never trouble free, human practitioner adaptations to
|
||||
changing conditions actually create safety from moment to moment.
|
||||
These adaptations often amount to just the selection of a
|
||||
well-rehearsed routine from a store of available responses; sometimes,
|
||||
however, the adaptations are novel combinations or de novo creations
|
||||
of new approaches.
|
||||
</p>
|
||||
</li>
|
||||
<li id="18">
|
||||
<h2 id="Failure-free-operations-require-experience-with-failure">
|
||||
<a href="#18"
|
||||
>Failure free operations require experience with failure.</a
|
||||
>
|
||||
</h2>
|
||||
<p>
|
||||
Recognizing hazard and successfully manipulating system operations to
|
||||
remain inside the tolerable performance boundaries requires intimate
|
||||
contact with failure. More robust system performance is likely to
|
||||
arise in systems where operators can discern the “edge of the
|
||||
envelope”. This is where system performance begins to deteriorate,
|
||||
becomes difficult to predict, or cannot be readily recovered. In
|
||||
intrinsically hazardous systems, operators are expected to encounter
|
||||
and appreciate hazards in ways that lead to overall performance that
|
||||
is desirable. Improved safety depends on providing operators with
|
||||
calibrated views of the hazards. It also depends on providing
|
||||
calibration about how their actions move system performance towards or
|
||||
away from the edge of the envelope.
|
||||
</p>
|
||||
</li>
|
||||
</ol>
|
||||
<div>
|
||||
<h2>Other Materials</h2>
|
||||
<ul>
|
||||
<li>
|
||||
<a href="citations/Gaps.pdf">
|
||||
Cook, Render, Woods (2000). Gaps in the continuity of care and
|
||||
progress on patient safety. <em>British Medical Journal</em> 320:
|
||||
791-4.
|
||||
</a>
|
||||
</li>
|
||||
<li>
|
||||
<a href="citations/BriefLookFront.pdf">
|
||||
Cook (1999). A Brief Look at the New Look in error, safety, and
|
||||
failure of complex systems. (Chicago: CtL).
|
||||
</a>
|
||||
</li>
|
||||
<li>
|
||||
<a
|
||||
href="citations/Perspectives_on_Human_Error.pdf"
|
||||
>
|
||||
Woods & Cook (1999). Perspectives on Human Error: Hindsight
|
||||
Biases and Local Rationality. In Durso, Nickerson, et al., eds.,
|
||||
<em>Handbook of Applied Cognition.</em> (New York: Wiley) pp.
|
||||
141-171.
|
||||
</a>
|
||||
</li>
|
||||
<li>
|
||||
<a href="citations/CharacteristicsPatientSafety.pdf">
|
||||
Woods & Cook (1998). Characteristics of Patient Safety: Five
|
||||
Principles that Underlie Productive Work. (Chicago: CtL)
|
||||
</a>
|
||||
</li>
|
||||
<li>
|
||||
<a href="citations/operatingatthesharp.pdf">
|
||||
Cook & Woods (1994), “Operating at the Sharp End: The Complexity
|
||||
of Human Error,” in MS Bogner, ed., <em>Human Error in Medicine</em> ,
|
||||
Hillsdale, NJ; pp. 255-310
|
||||
</a>
|
||||
</li>
|
||||
<li>
|
||||
<a href="citations/SOAR-BHE-original.pdf">
|
||||
Woods, Johannesen, Cook, & Sarter (1994),
|
||||
<em>Behind Human Error: Cognition, Computers and Hindsight</em> ,
|
||||
Wright Patterson AFB: CSERIAC.
|
||||
</a>
|
||||
</li>
|
||||
<li>
|
||||
<a
|
||||
href="citations/TaleOfTwoStories.pdf"
|
||||
>
|
||||
Cook, Woods, & Miller (1998),
|
||||
<em
|
||||
>A Tale of Two Stories: Contrasting Views of Patient Safety,
|
||||
Chicago, IL: NPSF</em
|
||||
>
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
<footer>
|
||||
<a
|
||||
href="https://www.adaptivecapacitylabs.com/HowComplexSystemsFail.pdf"
|
||||
>Original</a
|
||||
>
|
||||
Copyright © 1998, 1999, 2000 by R.I.Cook, MD, for CtL
|
||||
</footer>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user