Files
nexus/sreweekly/articles/96/04-how-can-we-apply-the-principles-of-chaos-engineering-to-aws-lambda.html
2026-09-12 17:23:01 +08:00

176 lines
133 KiB
HTML
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8" data-next-head=""/><meta name="viewport" content="width=device-width" data-next-head=""/><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-ECJJ2Q2SJQ"></script><title data-next-head=""></title><link rel="preconnect" href="https://bridge.hackernoon.com" data-next-head=""/><link rel="preconnect" href="https://cdn.hackernoon.com" data-next-head=""/><link rel="preconnect" href="https://hackernoon.imgix.net" data-next-head=""/><link rel="dns-prefetch" href="https://cdn.hackernoon.com" data-next-head=""/><meta name="description" content="This is the first of a multipart series that explores ideas on how we could apply the principles of chaos engineering to serverless architectures built around Lambda functions." data-next-head=""/><meta property="og:title" content="How can we apply the principles of chaos engineering to AWS Lambda? | HackerNoon" data-next-head=""/><meta property="og:description" content="This is the first of a multipart series that explores ideas on how we could apply the principles of chaos engineering to serverless architectures built around Lambda functions." data-next-head=""/><meta name="image" property="og:image" content="https://hackernoon.imgix.net/fallback-feat.png" data-next-head=""/><meta property="twitter:title" content="How can we apply the principles of chaos engineering to AWS Lambda? | HackerNoon" data-next-head=""/><meta property="twitter:description" content="This is the first of a multipart series that explores ideas on how we could apply the principles of chaos engineering to serverless architectures built around Lambda functions." data-next-head=""/><meta property="twitter:image" content="https://hackernoon.imgix.net/fallback-feat.png" data-next-head=""/><meta name="twitter:card" content="summary_large_image" data-next-head=""/><meta name="twitter:site" content="@hackernoon" data-next-head=""/><link rel="canonical" href="https://hackernoon.com/how-can-we-apply-the-principles-of-chaos-engineering-to-aws-lambda-80f87e3237e2" data-next-head=""/><link rel="preload" as="font" href="/fonts/HackerNoonFont/hackernoonv1-regular-webfont.woff2" type="font/woff2" crossorigin="anonymous"/><link rel="preconnect" href="https://fonts.googleapis.com"/><link rel="preconnect" href="https://fonts.gstatic.com" crossorigin="anonymous"/><link data-next-font="" rel="preconnect" href="/" crossorigin="anonymous"/><link rel="preload" href="/_next/static/css/121be0391d0b0ef0.css" as="style"/><link rel="preload" href="/_next/static/css/6d530d6069fd563f.css" as="style"/><link rel="preload" as="image" imageSrcSet="https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=640 640w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=750 750w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=828 828w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=1080 1080w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=1200 1200w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=1920 1920w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=2048 2048w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=3840 3840w" imageSizes="(max-width: 768px) 100vw, 900px" data-next-head=""/><script type="application/ld+json" data-next-head="">{"@context":"http://schema.org","@type":"Article","name":"How can we apply the principles of chaos engineering to AWS Lambda?","headline":"How can we apply the principles of chaos engineering to AWS Lambda?","author":{"@type":"Person","name":"Yan Cui"},"datePublished":"2017-09-01","image":"https://hackernoon.imgix.net/fallback-feat.png","articleSection":"aws","articleBody":"All the way back in 2011, Simon Wardley had identified Chaos Engines as a practice that will be employed by the next generation of tech companies, along with continuous deployment, being data-driven, and organised around small, autonomous teams (think microservices &amp; inverse-conway’s law). This is the first of a multipart series that explores ideas on how we could apply the principles of chaos engineering to serverless architectures built around Lambda functions. part 1: how can we apply principles of chaos engineering to Lambda? part 2: applying latency injection for APIs part 3: dealing with latency spikes and timeouts (coming 21/11/2017) part 4: applying fault injection for Lambda functions (coming ?) There’s no question about it, Netflix has popularised the of chaos engineering. By open sourcing some of their tools — notably the — they have also helped others build confidence in their system’s capability to withstand turbulent conditions in production. principles Simian Army There seems to be a renewed interest in chaos engineering recently. As Russ Miles noted in a recent , perhaps many companies have finally come to understand that chaos engineering is not about “hurting production”, but to build better understanding of, and confidence in a system’s resilience through experiments. post controlled This trend has been helped by the valuable (and freely available) information that Netflix has published, such as the e-book, and . Chaos Engineering principlesofchaos.org Tools such as by (the folks behind the load test tool) look to replicate Netflix’s , but execute from inside a Lambda function instead of an EC2 instance — hence bringing you the cost saving and convenience Lambda offers. chaos-lambda Shoreditch Ops Artillery Chaos Monkey I want to ask a different question however: how can one apply the principles of chaos engineering and some of the current practices to a serverless architecture comprised of Lambda functions? When your system runs on EC2 instances, then naturally, you build resilience by designing for the most likely failure mode — server crashes (due to both hardware and software issues). Hence, a controlled experiment to validate the resilience of your system would artificially recreate the failure scenario by terminating EC2 instances, and then AZs, and then entire regions. AWS Lambda, however, is a higher-level abstraction and has different failure modes to its EC2 counterparts. Hypothesis that focus on no longer apply as the platform handles these failure modes for you out of the box. “what if we lose these EC2 instances” We need to ask different questions in order to understand the weaknesses within our serverless architectures. More inherent chaos, not less “ . Systemic weaknesses could take the form of: improper fallback settings when a service is unavailable; retry storms from improperly tuned timeouts; outages when a downstream dependency receives too much traffic; cascading failures when a single point of failure crashes; etc. We must address the most significant weaknesses proactively, before they affect our customers in production. , take advantage of increasing flexibility and velocity, and ” We need to identify weaknesses before they manifest in system-wide, aberrant behaviors We need a way to manage the chaos inherent in these systems have confidence in our production deployments despite the complexity that they represent. — Principles of Chaos Engineering Having , I have some understanding of the dangers awaiting you in this new world. built and operated a non-trivial serverless architecture If anything, there are a lot more inherent chaos and complexity in these systems built around Lambda functions. modularity (unit of deployment) shifts from “services” to “functions”, and there are a lot more of them it’s harder to harden around the boundaries, because you need to harden around each function as opposed to a service which encapsulates a set of related functionalities there are more intermediary services (eg. Kinesis, SNS, API Gateway just to name a few), each with their own failure modes there are more configurations overall (timeout, IAM permissions, etc.), and therefore more opportunities for misconfiguration Also, since we have traded off more control of our infrastructure* it means we now face more unknown failure modes** and often there’s little we can do when an outage does occur***. * For better scalability, availability, cost efficiency and more convenience, which I for one, think it’s a fair trade in most cases . ** Everything the platform does for you — scheduling containers, scaling, polling Kinesis, retry failed invocations, etc. — have their own failure modes. These are often not obvious to us since they’re implementation details that are typically undocumented and are prone to change without notice. *** For example, if an outage happens and prevents Lambda functions from processing Kinesis events, then we have no meaningful alternative than to wait for AWS to fix the problem. Since the current position on the shards is abstracted away and unavailable to us, we can’t even replace the Lambda functions with KCL processors that run on EC2. Applying chaos to AWS Lambda A good exercise regime would continuously push you to your limits but never actually put you over the limit and cause injury. If there’s an exercise that is clearly beyond your current abilities then surely you would not attempt it as the only possible outcome is getting yourself hurt! . In order to “know” what the experiments tell us about the resilience of our system we also need to decide what metrics to monitor — ideally using client-side metrics, since the most important metric is the quality of service our users experience. The same common sense should be applied when designing controlled experiments for your serverless architecture There are plenty of failure modes that we know about and can design for, and we can run simple experiments to validate our design. For example, since a serverless architecture is (almost always) also a microservice architecture, many of its inherent failure modes still apply: improperly tuned timeouts, especially for intermediate services, which can cause services at the edge to also timeout Intermediate services should have more strict timeout settings compared to services at the edge. missing error handling, which allows exceptions from downstream services to escape missing fallbacks for when a downstream service is unavailable or experiences an outage Over the next couple of posts, we will explore how we can apply the practices of latency and fault injection to Lambda functions in order to simulate these failure modes and validate our design. Further readings: ChAP: Chaos Automation Platform Deploying the Netflix API The Netflix Simian Army Testing in production: Yes you can (and should) Design for latency issues 1000 actors, one chaos monkey… and everything’s OK [Ebook] Chaos Engineering [Book] Release it! [Book] Drift into Failure principlesofchaos.org Hi, my name is . I’m an and the author of . I have run production workload at scale in AWS for nearly 10 years and I have been an architect or principal engineer with a variety of industries ranging from banking, e-commerce, sports streaming to mobile gaming. I currently work as an independent consultant focused on AWS and serverless. Yan Cui AWS Serverless Hero Production-Ready Serverless You can contact me via , and . Email Twitter LinkedIn Check out my new course, . Complete Guide to AWS Step Functions In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. Including basic concepts, HTTP and event triggers, activities, design patterns and best practices. Get your copy . here Come learn about operational for AWS Lambda: CI/CD, testing &amp; debugging functions locally, logging, monitoring, distributed tracing, canary deployments, config management, authentication &amp; authorization, VPC, security, error handling, and more. BEST PRACTICES You can also get off the face price with the code . 40% ytcui Get your copy . here"}</script><link href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;700&amp;family=IBM+Plex+Sans:wght@400;700&amp;family=Inter:wght@400;600;900&amp;family=Source+Code+Pro:wght@400;500;600;700&amp;display=swap" rel="stylesheet" media="print"/><noscript><link href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;700&amp;family=IBM+Plex+Sans:wght@400;700&amp;family=Inter:wght@400;600;900&amp;family=Source+Code+Pro:wght@400;500;600;700&amp;display=swap" rel="stylesheet"/></noscript> <!-- --><script id="ga4-init">
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
// Consent Mode: default to denied
gtag('consent', 'default', {
'ad_storage': 'denied',
'analytics_storage': 'denied',
'ad_user_data': 'denied',
'ad_personalization': 'denied'
});
gtag('js', new Date());
gtag('config', 'G-ECJJ2Q2SJQ');
</script><script id="iubenda-init">
function initIubenda() {
(async function () {
try {
const res = await fetch("https://geolocation-db.com/json/");
const data = await res.json();
const country = data && data.country_code;
const GDPR_COUNTRIES = [
"AT","BE","BG","HR","CY","CZ","DK","EE","FI","FR","DE","GR","HU",
"IE","IT","LV","LT","LU","MT","NL","PL","PT","RO","SK","SI","ES",
"SE","IS","LI","NO","UK","GB"
];
var isGdpr = GDPR_COUNTRIES.indexOf(country) > -1;
window._iub = window._iub || [];
window._iub.csConfiguration = {
siteId: 1848357,
cookiePolicyId: 18778700,
lang: "en",
enableTcf: false,
googleAdditionalConsentMode: true,
banner: {
position: "bottom",
rejectButtonDisplay: true,
explicitWithdrawal: true,
customizeButtonDisplay: true,
acceptButtonDisplay: true,
showTotalNumberOfProviders: false,
display: isGdpr
}
};
var iubScript = document.createElement("script");
iubScript.src = "https://cdn.iubenda.com/cs/iubenda_cs.js";
iubScript.async = true;
document.head.appendChild(iubScript);
if (!isGdpr) {
gtag('consent', 'update', {
'ad_storage': 'granted',
'analytics_storage': 'granted',
'ad_user_data': 'granted',
'ad_personalization': 'granted'
});
}
} catch (e) {
console.error("Iubenda geolocation failed", e);
}
})();
}
// Defer until browser is idle — never blocks initial render
if (typeof requestIdleCallback !== 'undefined') {
requestIdleCallback(initIubenda, { timeout: 3000 });
} else {
setTimeout(initIubenda, 1000);
}
</script><script id="iubenda-consent-bridge">
window.addEventListener("iubenda_consent_given", function () {
gtag('consent', 'update', {
'ad_storage': 'granted',
'analytics_storage': 'granted',
'ad_user_data': 'granted',
'ad_personalization': 'granted'
});
gtag('event', 'page_view', {
page_title: document.title,
page_location: location.href,
page_path: location.pathname + location.search
});
});
</script><link rel="stylesheet" href="/_next/static/css/121be0391d0b0ef0.css" data-n-g=""/><link rel="stylesheet" href="/_next/static/css/6d530d6069fd563f.css" data-n-p=""/><noscript data-n-css=""></noscript><script defer="" noModule="" src="/_next/static/chunks/polyfills-42372ed130431b0a.js"></script><script src="https://accounts.google.com/gsi/client" defer="" data-nscript="beforeInteractive"></script><script defer="" src="/_next/static/chunks/7618.8cb06698e4978306.js"></script><script defer="" src="/_next/static/chunks/3213.7a85381e883f5859.js"></script><script defer="" src="/_next/static/chunks/7127.9c425c4d3409a6ac.js"></script><script defer="" src="/_next/static/chunks/3304.7c3523eee5ba4042.js"></script><script defer="" src="/_next/static/chunks/1826.1ab3736f712279dc.js"></script><script defer="" src="/_next/static/chunks/b6790ad6-21a72b711b29c9a6.js"></script><script defer="" src="/_next/static/chunks/4829-32ed3fa27f9fba8a.js"></script><script defer="" src="/_next/static/chunks/8145-56bb137bc815feca.js"></script><script defer="" src="/_next/static/chunks/7878-038a9b85114cf28d.js"></script><script defer="" src="/_next/static/chunks/9407-02afcd7299ecf2e9.js"></script><script defer="" src="/_next/static/chunks/1866-36cbb79df614e742.js"></script><script defer="" src="/_next/static/chunks/997.043403d1cfb4d583.js"></script><script defer="" src="/_next/static/chunks/9997.0bcf115a883cf0c1.js"></script><script defer="" src="/_next/static/chunks/9752.5dc7ee3796d8a882.js"></script><script defer="" src="/_next/static/chunks/1486.6875746f7e7b3bd6.js"></script><script defer="" src="/_next/static/chunks/2348.6ff1c8ab2b7f714e.js"></script><script src="/_next/static/chunks/webpack-b0b0e650ef898a91.js" defer=""></script><script src="/_next/static/chunks/framework-594babcea68f40f6.js" defer=""></script><script src="/_next/static/chunks/main-f4c6b80eccf8d9c3.js" defer=""></script><script src="/_next/static/chunks/pages/_app-e9f748c6202c8d20.js" defer=""></script><script src="/_next/static/chunks/4004-46cbf9060446c734.js" defer=""></script><script src="/_next/static/chunks/8230-f743aded49f5ec53.js" defer=""></script><script src="/_next/static/chunks/3363-3997af8403818196.js" defer=""></script><script src="/_next/static/chunks/5857-ec7c040b0c106d7c.js" defer=""></script><script src="/_next/static/chunks/7871-49db796a808d2c70.js" defer=""></script><script src="/_next/static/chunks/3261-28f7d7d5ddf137c8.js" defer=""></script><script src="/_next/static/chunks/1902-922299f711a16f76.js" defer=""></script><script src="/_next/static/chunks/8581-1c63d69fe79360dc.js" defer=""></script><script src="/_next/static/chunks/4581-8f149836b5130c5d.js" defer=""></script><script src="/_next/static/chunks/4680-486b209e44768b17.js" defer=""></script><script src="/_next/static/chunks/2562-5aa3cd1aa6e464a5.js" defer=""></script><script src="/_next/static/chunks/8373-1c61bad6a8e1fdcb.js" defer=""></script><script src="/_next/static/chunks/7225-8cb630c00db3e752.js" defer=""></script><script src="/_next/static/chunks/pages/%5Bslug%5D-e18135e2a995d794.js" defer=""></script><script src="/_next/static/qqTckmfliewRaBLUp_8jP/_buildManifest.js" defer=""></script><script src="/_next/static/qqTckmfliewRaBLUp_8jP/_ssgManifest.js" defer=""></script></head><body><link rel="preload" as="image" imageSrcSet="https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=640 640w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=750 750w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=828 828w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=1080 1080w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=1200 1200w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=1920 1920w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=2048 2048w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=3840 3840w" imageSizes="(max-width: 768px) 100vw, 1200px" fetchPriority="high"/><link rel="preload" as="image" href="https://hackernoon.imgix.net/avatars/robot-b5.png"/><div id="__next"><div class="bg-light text-lightText font-[ibm-plex-mono]"><main><header class="font-[ibm-plex-sans] fixed top-0 left-0 w-full z-50 transition-all duration-500 ease-in-out translate-y-0"><div class="flex items-center justify-between bg-primary lg:navbar h-[50px] sm:min-h-[75px] transition-all duration-100 shadow-md w-full"><div class="hidden lg:flex navbar-start h-full items-center ml-1"><button class="flex items-center hover:scale-[1.01] justify-center rounded-lg text-base px-4 font-bold py-2 border-none bg-primary-content text-primaryContentText">Discover Anything<i class="hn hn-search text-lg ml-4 text-primaryContentText "></i></button></div><div class="nav-start lg:navbar-center ml-2 lg:ml-0 min-w-0 flex-shrink"><a class="relative z-10 flex items-center space-x-2 hover:scale-[1.02]" aria-label="HackerNoon Homepage" href="/"><svg class="w-[180px] xs:w-[200px] sm:w-[240px] lg:w-[260px] h-auto" viewBox="0 0 2150 260" fill="none" xmlns="http://www.w3.org/2000/svg" preserveAspectRatio="xMidYMid meet" style="transition:fill 150ms ease"><g style="transition:fill 150ms ease"><path d="M269.997 20.0005V0H130V20.0005V40.0011V60.0016H150H169.995V40.0011H189.995H229.996V60.0016H249.997H269.997V40.0011V20.0005Z" fill="transparent"></path><path d="M130.006 80.003V60.0024H110.006V80.003V100.004H130.006V80.003Z" fill="transparent"></path><path d="M110 119.998V100.003H90V119.998V139.998V159.999H110V139.998V119.998Z" fill="transparent"></path><path d="M270 100.004H290V80.003V60.0024H270V80.003V100.004Z" fill="transparent"></path><path d="M310 119.997V100.002H290V119.997V139.998V159.998H310V139.998H330.001V119.997H310Z" fill="transparent"></path><path d="M130 159.998H110V179.998V199.999H130V179.998V159.998Z" fill="transparent"></path><path d="M270 179.998V199.999H290V179.998V159.998H270V179.998Z" fill="transparent"></path><path d="M130 260V240V219.999V199.999H150H169.995V219.999H189.995H209.996H229.996V199.999H249.997H269.997V219.999V240V260H130Z" fill="transparent"></path><path d="M210.415 39.74V59.7405V79.7411V99.7416V119.736V139.737H190.415V119.736V99.7416V79.7411V59.7405V39.74H210.415Z" fill="transparent"></path><path d="M390 200V60H417.801V116.676H501.206V60H530V200H501.206V144.517H417.801V200H390Z" fill="transparent"></path><path d="M672.199 116.676V88.8352H588.794V116.676H672.199ZM560 200V60H700V200H672.199V144.517H588.794V200H560Z" fill="transparent"></path><path d="M730 200V60H870V88.8352H758.794V172.159H870V200H730Z" fill="transparent"></path><path d="M900 200V60H928.794V116.276H984.397V143.724H928.794V200H900ZM1012.2 171.368H984.397V143.724H1012.2V171.368ZM1012.2 171.368H1040V199.013H1012.2V171.368ZM1012.2 88.6319V116.276H984.397V88.6319H1012.2ZM1012.2 88.6319V60H1040V88.6319H1012.2Z" fill="transparent"></path><path d="M1070 200V60H1210V88.8352H1098.79V116.676H1154.4V144.517H1098.79V172.159H1210V200H1070Z" fill="transparent"></path><path d="M1351.24 116.519V88.7589H1267.76V116.519H1351.24ZM1240 200V60H1380V144.479H1351.24V172.24H1380V200H1351.24V172.24H1323.48V144.479H1267.76V200H1240Z" fill="transparent"></path><path d="M1410 200V60H1550V200H1522.24V88.7589H1438.76V200H1410Z" fill="transparent"></path><path d="M1692.24 172.24V88.7589H1608.76V172.24H1692.24ZM1580 200V60H1720V200H1580Z" fill="transparent"></path><path d="M1862.04 172.24V88.7589H1778.97V172.24H1862.04ZM1750 200V60H1890V200H1750Z" fill="transparent"></path><path d="M1920 200V60H2060V200H2032.24V88.7589H1948.76V200H1920Z" fill="transparent"></path></g></svg></a></div><div class="navbar-end h-full flex items-center min-w-[100px] lg:min-w-[200px] space-x-4 mr-2"><div class=" h-[40px] flex items-center justify-center"></div><div class="hidden sm:flex space-x-4 "><button class="px-4 font-bold text-base py-1 sm:py-2 bg-primary-content text-primaryContentText rounded-md transition-all duration-300">Signup</button><a class="px-4 hover:scale-105 font-bold text-base py-2 bg-primary-content text-primaryContentText rounded-md " href="/new">Write</a></div><button class="btn border-none p-0 m-0 lg:hidden bg-transparent hover:bg-transparent text-primary-content hover:scale-110"><i class="hn hn-search text-xl mr-2 "></i></button><button class="relative lg:flex hidden items-center hover:scale-110 text-primary-content" aria-label="Notifications"><i width="20" class="hn hn-bell text-2xl w-4 h-4 sm:w-6 sm:h-6"></i></button><button class="flex items-center hover:scale-110 text-primary-content" aria-label="Menu"><i class="hn hn-bars text-2xl text-primary-content"></i></button></div></div><div class="z-20 hidden lg:block h-[52px] transition-all duration-500 ease-in-out"><nav class="h-[52px] bg-secondary animate-pulse"></nav></div><div class="flex items-center "><div class="bg-accent text-accent-content flex items-center h-[62px] sm:min-h-[80px] font-[ibm-plex-sans] w-full relative z-10 opacity-100"><div class="h-[62px] sm:h-[80px] bg-[transparent] animate-pulse"></div></div></div></header><div class="transition-all duration-200 pt-[112px] sm:pt-[155px] lg:pt-[207px]"><div data-rht-toaster="" style="position:fixed;z-index:9999;top:16px;left:16px;right:16px;bottom:16px;pointer-events:none"></div><div class=""><div class="bg-light text-lightText h-auto xl:mx-2 "><div class="col-span-12"><div class="max-w-[1200px] mx-auto px-2 xs:px-4 xl: xl:px-0 mt-10"><div class=""><div><div class="text-xs"><div class="mb-4 flex gap-2"><span class="bg-lightAlt p-2 rounded-lg inline-flex gap-2 items-center justify-start"><i class="hn hn-star-solid"></i> <!-- -->4,250<!-- --> <!-- -->reads</span></div><h1 class="font-bold line-clamp-4 leading-snug text-lightTextStrong
text-xl sm:text-2xl xl:text-3xl tracking-normal
">How can we apply the principles of chaos engineering to AWS Lambda?</h1><div class="flex flex-wrap border-y border-lightBorder my-4 sm:my-2 sm:border-t-0 py-2 items-center justify-between text-lightTextLight text-sm sm:text-base xl:text-xl "><div class="flex flex-wrap justify-between w-full xs:w-auto items-center gap-2"><span class="flex items-center flex-wrap gap-2 mr-10 sm:mr-0 ">by<div class="dropdown dropdown-hover "><label tabindex="0"><a aria-label="View profile of Yan Cui" href="/u/theburningmonk"><strong class="...">Yan Cui</strong></a></label><div class="dropdown-content z-[1] pt-2 sm:pt-1 left-[-40px] w-[280px] xs:w-[320px] sm:w-[400px] bg-transparent menu rounded "><div class="w-full "><div class=" p-4 border border-lightBorder bg-light rounded-lg"><a href="/u/theburningmonk" target="_blank" rel="noopener noreferrer" class="flex items-start text-sm rounded-lg group gap-2"><div class=""><img alt="Yan Cui" loading="lazy" width="40" height="40" decoding="async" data-nimg="1" class="w-10 h-9 border-solid border border-lightBorder rounded-full object-contain" style="color:transparent" srcSet="https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=48 1x, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=96 2x" src="https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=96"/></div><span class="flex flex-col min-w-0 w-full justify-center"><span class="flex items-center gap-1 text-ellipsis overflow-hidden whitespace-nowrap"><span class="font-bold group-hover:underline text-sm truncate"><span class="text-xs font-light mr-1">by</span>Yan Cui</span><span class="text-xs font-light opacity-50">|</span><span class="text-sm false text-ellipsis overflow-hidden whitespace-nowrap" title="@theburningmonk">@<!-- -->theburningmonk</span></span></span></a><p class="text-sm overflow-x-auto mt-2 text-bodyTxtLight">AWS Serverless Hero. Independent Consultant. Developer Advocate at Lumigo.</p><div class="mt-4"><div class="w-full flex justify-start"><div class="w-full"><form class="w-full flex flex-col items-start gap-2 "><div class="flex w-full"><input class="p-2 flex-grow border rounded-l-md text-lightText bg-light focus:outline-none focus:ring-0 focus:ring-transparent border-lightBorder w-full text-base px-2}
}" placeholder="name@company.com" type="email" required="" name="email" value=""/><button type="submit" class="text-base}
bg-lightAlt border border-l-0 border-lightBorder hover:bg-green-700 text-lightText hover:bg-dark hover:text-darkText px-2 py-1 rounded-r-md font-bold">Subscribe</button></div></form></div></div></div></div></div></div></div></span><div class="flex gap-2 items-center cursor-pointer"><span class="hidden sm:block w-1 h-1 bg-lightTextLight mx-4 rounded-full"></span><a href="/archives/2017/09/01"><span class="text-xs xs:text-sm lg:text-base">September 1st, 2017</span></a></div></div><div class="hidden xl:block"><div class=" flex items-center gap-4 "><span class="tooltip tooltip-left tooltip-left w-7 h-7 flex items-center justify-center cursor-pointer" data-tip="Terminal Reader"><img alt="Read on Terminal Reader" loading="lazy" width="20" height="20" decoding="async" data-nimg="1" style="color:transparent" srcSet="https://hackernoon.imgix.net/computer.png?auto=format%2Ccompress&amp;w=32 1x, https://hackernoon.imgix.net/computer.png?auto=format%2Ccompress&amp;w=48 2x" src="https://hackernoon.imgix.net/computer.png?auto=format%2Ccompress&amp;w=48"/></span><span class="tooltip tooltip-left tooltip-left w-7 h-7 flex items-center justify-center cursor-pointer" data-tip="Print this story"><img alt="Print this story" data-tip="true" data-for="print-page" loading="lazy" width="20" height="20" decoding="async" data-nimg="1" style="color:transparent" srcSet="https://hackernoon.imgix.net/images/Print%20Icon%20%4025px.png?auto=format%2Ccompress&amp;w=32 1x, https://hackernoon.imgix.net/images/Print%20Icon%20%4025px.png?auto=format%2Ccompress&amp;w=48 2x" src="https://hackernoon.imgix.net/images/Print%20Icon%20%4025px.png?auto=format%2Ccompress&amp;w=48"/></span><span class="tooltip tooltip-left tooltip-left w-7 h-7 flex items-center justify-center cursor-pointer" data-tip="Read this story w/o Javascript"><img alt="Read this story w/o Javascript" data-tip="true" data-for="arweave-backup" loading="lazy" width="20" height="20" decoding="async" data-nimg="1" style="color:transparent" srcSet="https://hackernoon.imgix.net/images/Lite%20Icon%20%4025px.png?auto=format%2Ccompress&amp;w=32 1x, https://hackernoon.imgix.net/images/Lite%20Icon%20%4025px.png?auto=format%2Ccompress&amp;w=48 2x" src="https://hackernoon.imgix.net/images/Lite%20Icon%20%4025px.png?auto=format%2Ccompress&amp;w=48"/></span></div></div></div></div><div class="mb-2 "><div class="flex justify-between "><button class="flex m-1 px-2 xl:px-4 h-[40px] items-center font-[hackernoon2] bg-dark text-darkText text-xs sm:text-sm rounded-lg border border-lightBorder">TLDR <i class="hn hn-angle-right text-base ml-1 "></i></button><div class="flex items-center gap-2"><div class="hidden sm:block xl:hidden"><div class=" flex items-center gap-4 "><span class="tooltip tooltip-left undefined w-7 h-7 flex items-center justify-center cursor-pointer" data-tip="Terminal Reader"><img alt="Read on Terminal Reader" loading="lazy" width="20" height="20" decoding="async" data-nimg="1" style="color:transparent" srcSet="https://hackernoon.imgix.net/computer.png?auto=format%2Ccompress&amp;w=32 1x, https://hackernoon.imgix.net/computer.png?auto=format%2Ccompress&amp;w=48 2x" src="https://hackernoon.imgix.net/computer.png?auto=format%2Ccompress&amp;w=48"/></span><span class="tooltip tooltip-left undefined w-7 h-7 flex items-center justify-center cursor-pointer" data-tip="Print this story"><img alt="Print this story" data-tip="true" data-for="print-page" loading="lazy" width="20" height="20" decoding="async" data-nimg="1" style="color:transparent" srcSet="https://hackernoon.imgix.net/images/Print%20Icon%20%4025px.png?auto=format%2Ccompress&amp;w=32 1x, https://hackernoon.imgix.net/images/Print%20Icon%20%4025px.png?auto=format%2Ccompress&amp;w=48 2x" src="https://hackernoon.imgix.net/images/Print%20Icon%20%4025px.png?auto=format%2Ccompress&amp;w=48"/></span><span class="tooltip tooltip-left undefined w-7 h-7 flex items-center justify-center cursor-pointer" data-tip="Read this story w/o Javascript"><img alt="Read this story w/o Javascript" data-tip="true" data-for="arweave-backup" loading="lazy" width="20" height="20" decoding="async" data-nimg="1" style="color:transparent" srcSet="https://hackernoon.imgix.net/images/Lite%20Icon%20%4025px.png?auto=format%2Ccompress&amp;w=32 1x, https://hackernoon.imgix.net/images/Lite%20Icon%20%4025px.png?auto=format%2Ccompress&amp;w=48 2x" src="https://hackernoon.imgix.net/images/Lite%20Icon%20%4025px.png?auto=format%2Ccompress&amp;w=48"/></span></div></div><div class="xl:hidden"></div></div><div class="hidden xl:flex items-center flex-wrap gap-2"></div></div></div></div></div></div><div class="max-w-[1200px] mx-auto"><div class="flex items-center justify-center w-full h-full"><div class="relative group cursor-zoom-in transition-transform hover:scale-[1.01] max-w-full mx-auto px-[14px] md:px-0 w-full"><button class="absolute top-2 right-5 z-10 w-6 h-6 rounded flex items-center justify-center opacity-0 group-hover:opacity-100 transition-opacity"><i class="hn hn-download text-darkText bg-dark p-2 rounded-xl text-base"></i></button><img alt="featured image - How can we apply the principles of chaos engineering to AWS Lambda?" fetchPriority="high" loading="eager" width="1000" height="500" decoding="async" data-nimg="1" class="w-full h-auto object-contain rounded-lg shadow-lg my-0" style="color:transparent;background-size:cover;background-position:50% 50%;background-repeat:no-repeat;background-image:url(&quot;data:image/svg+xml;charset=utf-8,%3Csvg xmlns=&#x27;http://www.w3.org/2000/svg&#x27; viewBox=&#x27;0 0 1000 500&#x27;%3E%3Cfilter id=&#x27;b&#x27; color-interpolation-filters=&#x27;sRGB&#x27;%3E%3CfeGaussianBlur stdDeviation=&#x27;20&#x27;/%3E%3CfeColorMatrix values=&#x27;1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 100 -1&#x27; result=&#x27;s&#x27;/%3E%3CfeFlood x=&#x27;0&#x27; y=&#x27;0&#x27; width=&#x27;100%25&#x27; height=&#x27;100%25&#x27;/%3E%3CfeComposite operator=&#x27;out&#x27; in=&#x27;s&#x27;/%3E%3CfeComposite in2=&#x27;SourceGraphic&#x27;/%3E%3CfeGaussianBlur stdDeviation=&#x27;20&#x27;/%3E%3C/filter%3E%3Cimage width=&#x27;100%25&#x27; height=&#x27;100%25&#x27; x=&#x27;0&#x27; y=&#x27;0&#x27; preserveAspectRatio=&#x27;none&#x27; style=&#x27;filter: url(%23b);&#x27; href=&#x27;data:image/svg+xml;base64,CiAgICA8c3ZnIHdpZHRoPSIxMDAwIiBoZWlnaHQ9IjUwMCIgdmVyc2lvbj0iMS4xIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIj4KICAgICAgPGRlZnM+CiAgICAgICAgPGxpbmVhckdyYWRpZW50IGlkPSJnIj4KICAgICAgICAgIDxzdG9wIHN0b3AtY29sb3I9IiMyMjIiIG9mZnNldD0iMjAlIiAvPgogICAgICAgICAgPHN0b3Agc3RvcC1jb2xvcj0iIzBmMCIgb2Zmc2V0PSI1MCUiIC8+CiAgICAgICAgICA8c3RvcCBzdG9wLWNvbG9yPSIjMjIyIiBvZmZzZXQ9IjgwJSIgLz4KICAgICAgICA8L2xpbmVhckdyYWRpZW50PgogICAgICA8L2RlZnM+CiAgICAgIDxyZWN0IHdpZHRoPSIxMDAwIiBoZWlnaHQ9IjUwMCIgZmlsbD0iIzIyMiIgLz4KICAgICAgPHJlY3QgaWQ9InIiIHdpZHRoPSIxMDAwIiBoZWlnaHQ9IjUwMCIgZmlsbD0idXJsKCNnKSIgLz4KICAgICAgPGFuaW1hdGUgeGxpbms6aHJlZj0iI3IiIGF0dHJpYnV0ZU5hbWU9IngiIGZyb209Ii0xMDAwIiB0bz0iMTAwMCIgZHVyPSIxcyIgcmVwZWF0Q291bnQ9ImluZGVmaW5pdGUiICAvPgogICAgPC9zdmc+&#x27;/%3E%3C/svg%3E&quot;)" sizes="(max-width: 768px) 100vw, 1200px" srcSet="https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=640 640w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=750 750w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=828 828w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=1080 1080w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=1200 1200w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=1920 1920w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=2048 2048w, https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=3840 3840w" src="https://hackernoon.imgix.net/fallback-feat.png?auto=format%2Ccompress&amp;w=3840"/></div></div></div><div class="px-2 xs:px-4 3xl:px-0 my-4 sm:mt-4 sm:mb-6 max-w-[1200px] 6xl:max-w-[1200px] mx-auto "><div class="w-full flex flex-col justify-center rounded-lg"><audio src="https://storage.googleapis.com/hackernoon/audios/how-can-we-apply-the-principles-of-chaos-engineering-to-aws-lambda-80f87e3237e2-en-US-Wavenet-I-MALE--f3d60ec0be2cf.mp3" preload="metadata">Your browser does not support the <code>audio</code> element.</audio><div class="hidden sm:flex justify-between items-center "><span></span></div><div class="flex gap-2 items-center"><div class="flex items-center justify-center mx-auto gap-2 xs:gap-4 lg:gap-6 flex-1"><button aria-label="play/pause" class="text-darkAccent max-w-[40px] max-h-[40px] sm:min-w-[48px] sm:min-h-[48px] border order-1 border-darkBorder bg-dark p-2 rounded-full flex items-center justify-center" title="Play/Pause"><i class="hn hn-play-solid text-base xs:text-lg sm:text-2xl "></i></button><div class="dropdown order-2 dropdown-hover"><label tabindex="0" class="flex items-center hn hn-playlist-solid text-base xs:text-lg sm:text-xl lg:text-2xl rounded-lg " title="Speed &amp; Voice"></label><ul tabindex="0" class="dropdown-content border z-40 menu p-4 shadow bg-light rounded-box w-60"><div class="text-lightText flex bg-light p-2 rounded w-full mb-2 items-center justify-between"><span class="text-xs font-bold">Speed</span><button class="bg-lightAlt ml-2 px-4 py-2 rounded-full text-sm font-bold min-w-[100px]">1x</button></div><div class="text-lightText flex flex-col bg-light p-2 rounded w-full"><span class="text-xs font-bold mb-2">Voice</span><div class="flex flex-col gap-2 max-h-60 overflow-auto pr-1"><button class="bg-lightAlt px-3 py-2 rounded-lg text-sm font-bold text-left flex items-center justify-between ring-2 ring-green-600"><span class="truncate mr-2">Dr. One </span><img src="https://hackernoon.imgix.net/avatars/robot-b5.png" alt="Dr. One (en-US)" class="w-6 h-6 rounded-full"/></button></div></div></ul></div><div class="flex gap-2 order-3 sm:items-center sm:flex-row w-full"><div class="rounded-lg flex-1 bg-lightAccentTextAlt relative"><div class="hidden lg:block"><div class="relative max-w-[1000px] h-full flex items-center cursor-pointer rounded-lg "><canvas class="bg-transparent absolute top-0 left-0 w-full h-full rounded-lg "></canvas><div></div><div class=" top-0 left-0 h-full overflow-hidden bg-lightAccentAlt border rounded-l-lg" style="width:0px"><canvas class="bg-transparent w-full h-full text-green-500"></canvas></div></div></div><div class="hidden sm:block lg:hidden"><div class="relative max-w-[1000px] h-full flex items-center cursor-pointer rounded-lg "><canvas class="bg-transparent absolute top-0 left-0 w-full h-full rounded-lg "></canvas><div></div><div class=" top-0 left-0 h-full overflow-hidden bg-lightAccentAlt border rounded-l-lg" style="width:0px"><canvas class="bg-transparent w-full h-full text-green-500"></canvas></div></div></div><div class="w-full sm:hidden"><div class="relative max-w-[1000px] h-full flex items-center cursor-pointer rounded-lg "><canvas class="bg-transparent absolute top-0 left-0 w-full h-full rounded-lg "></canvas><div></div><div class=" top-0 left-0 h-full overflow-hidden bg-lightAccentAlt border rounded-l-lg" style="width:0px"><canvas class="bg-transparent w-full h-full text-green-500"></canvas></div></div></div></div><div class="hidden lg:ml-2 sm:block"></div><div class=" flex items-center sm:hidden"></div></div></div></div></div></div><div class=" flex xl:hidden bg-light z-10 mx-auto gap-4 items-center border-y sticky top-[62px] sm:top-[80px] py-2 sm:py-0 "><div class="w-full max-w-[1200px] mx-auto px-4 flex justify-between items-center"><div class="6xl:hidden dropdown dropdown-bottom dropdown-hover"><div tabindex="0" role="button" class="flex text-sm rounded-lg py-2"><div class="mr-2 flex -space-x-2 items-center "><div class=""><img alt="Yan Cui" loading="lazy" width="48" height="48" decoding="async" data-nimg="1" class="w-10 h-10 sm:w-12 sm:h-12 bg-light relative border border-lightBorder rounded-full object-contain" style="color:transparent;z-index:1" srcSet="https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=48 1x, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=96 2x" src="https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=96"/></div></div><div class="flex-col hidden sm:flex flex-wrap"><span class="font-bold mx-2 text-sm"><span class="text-xs font-light mr-1">by</span>Yan Cui</span><span class="text-sm ml-2">@<!-- -->theburningmonk</span></div></div><ul tabindex="0" class="dropdown-content menu w-[300px] py-1 left-[-20px] bg-light px-1 ml-4 rounded-b-lg 3xl:border-none rounded-boxabsolute z-50"><div class="flex w-full flex-col gap-4"><div><div class="w-full "><div class=" p-4 border border-lightBorder bg-light rounded-lg"><a href="/u/theburningmonk" target="_blank" rel="noopener noreferrer" class="flex items-start text-sm rounded-lg group gap-2"><div class=""><img alt="Yan Cui" loading="lazy" width="40" height="40" decoding="async" data-nimg="1" class="w-10 h-9 border-solid border border-lightBorder rounded-full object-contain" style="color:transparent" srcSet="https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=48 1x, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=96 2x" src="https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=96"/></div><span class="flex flex-col min-w-0 w-full justify-center"><span class="flex items-center gap-1 text-ellipsis overflow-hidden whitespace-nowrap"><span class="font-bold group-hover:underline text-sm truncate"><span class="text-xs font-light mr-1">by</span>Yan Cui</span><span class="text-xs font-light opacity-50">|</span><span class="text-sm false text-ellipsis overflow-hidden whitespace-nowrap" title="@theburningmonk">@<!-- -->theburningmonk</span></span></span></a><p class="text-sm overflow-x-auto mt-2 text-bodyTxtLight">AWS Serverless Hero. Independent Consultant. Developer Advocate at Lumigo.</p><div class="mt-4"><div class="w-full flex justify-start"><div class="w-full"><form class="w-full flex flex-col items-start gap-2 "><div class="flex w-full"><input class="p-2 flex-grow border rounded-l-md text-lightText bg-light focus:outline-none focus:ring-0 focus:ring-transparent border-lightBorder w-full text-base px-2}
}" placeholder="name@company.com" type="email" required="" name="email" value=""/><button type="submit" class="text-base}
bg-lightAlt border border-l-0 border-lightBorder hover:bg-green-700 text-lightText hover:bg-dark hover:text-darkText px-2 py-1 rounded-r-md font-bold">Subscribe</button></div></form></div></div></div></div></div></div></div></ul></div><div class="w-[200px] 6xl:hidden"><div class=" flex flex-row flex-row-reverse items-start gap-4 "><span class="tooltip tooltip-left cursor-pointer" data-tip="Bookmark"><button class="3xl:hover:bg-lightAlt hover:bg-light p-1 md:p-2 rounded h-[40px] w-[40px] flex items-center justify-center border border-lightBorder"><i class="hn hn-bookmark text-lightText text-2xl"></i></button></span><span class="tooltip tooltip-left cursor-pointer" data-tip="Comment"><button class="3xl:hover:bg-lightAlt hover:bg-light p-1 md:p-2 rounded h-[40px] w-[40px] flex items-center justify-center border border-lightBorder"><i class="hn hn-comment text-lightText text-2xl"></i></button></span><div class="dropdown dropdown-bottom dropdown-hover group "><label tabindex="0" class="flex items-center cursor-pointer justify-center border border-lightBorder 3xl:group-hover:bg-lightAlt group-hover:bg-light h-[40px] w-[40px] p-2 rounded "><i class="hn hn-share text-2xl"></i></label><ul tabindex="0" class="dropdown-content bg-light z-[1] py-4 px-4 3xl:px-0 3xl:py-2 border 3xl:border-none flex flex-col items-center justify-center gap-2 "><button class="border p-2 rounded hover:bg-lightAlt"><i class=" hn hn-copy text-lightText text-2xl "></i></button><button class="border p-2 rounded hover:bg-lightAlt"><i class="hn hn-facebook-round text-lightText text-2xl"></i></button><button class="border p-2 rounded hover:bg-lightAlt"><i class="hn hn-x text-lightText text-2xl"></i></button><button class="border p-2 rounded hover:bg-lightAlt"><i class="hn hn-linkedin text-lightText text-2xl"></i></button><a href="mailto:?subject=I&#x27;d like to share a link with you &amp;body=" class="border p-2 rounded inline-block hover:bg-lightAlt"><i class="hn hn-envelope text-lightText text-2xl"></i></a></ul></div></div></div></div></div><div class="flex w-full min-w-0 max-w-[1100px] 2xl:max-w-[1200px] mx-auto justify-center flex-row gap-4 items-start mt-10 px-4 xl:px-0"><div class="hidden xl:flex xl:flex-col self-stretch"><div class="sticky top-[99px] z-40"><div class="dropdown z-25 dropdown-right dropdown-hover"><div class="mr-2 flex gap-2 flex-col justify-center flex-wrap items-center "><div class="relative h-12 w-12 bg-black rounded-full overflow-hidden flex-shrink-0"><img alt="Yan Cui" loading="lazy" decoding="async" data-nimg="fill" class="rounded-full object-contain " style="position:absolute;height:100%;width:100%;left:0;top:0;right:0;bottom:0;color:transparent;z-index:1" sizes="100vw" srcSet="https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=640 640w, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=750 750w, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=828 828w, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=1080 1080w, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=1200 1200w, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=1920 1920w, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=2048 2048w, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=3840 3840w" src="https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=3840"/></div></div><ul tabindex="0" class="dropdown-content w-[280px] xs:w-[320px] sm:w-[400px] rounded-lg z-[100] bg-light flex flex-col items-center justify-center gap-2"><div class="flex w-full flex-col gap-4"><div><div class="w-full "><div class=" p-4 border border-lightBorder bg-light rounded-lg"><a href="/u/theburningmonk" target="_blank" rel="noopener noreferrer" class="flex items-start text-sm rounded-lg group gap-2"><div class=""><img alt="Yan Cui" loading="lazy" width="40" height="40" decoding="async" data-nimg="1" class="w-10 h-9 border-solid border border-lightBorder rounded-full object-contain" style="color:transparent" srcSet="https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=48 1x, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=96 2x" src="https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=96"/></div><span class="flex flex-col min-w-0 w-full justify-center"><span class="flex items-center gap-1 text-ellipsis overflow-hidden whitespace-nowrap"><span class="font-bold group-hover:underline text-sm truncate"><span class="text-xs font-light mr-1">by</span>Yan Cui</span><span class="text-xs font-light opacity-50">|</span><span class="text-sm false text-ellipsis overflow-hidden whitespace-nowrap" title="@theburningmonk">@<!-- -->theburningmonk</span></span></span></a><p class="text-sm overflow-x-auto mt-2 text-bodyTxtLight">AWS Serverless Hero. Independent Consultant. Developer Advocate at Lumigo.</p><div class="mt-4"><div class="w-full flex justify-start"><div class="w-full"><form class="w-full flex flex-col items-start gap-2 "><div class="flex w-full"><input class="p-2 flex-grow border rounded-l-md text-lightText bg-light focus:outline-none focus:ring-0 focus:ring-transparent border-lightBorder w-full text-base px-2}
}" placeholder="name@company.com" type="email" required="" name="email" value=""/><button type="submit" class="text-base}
bg-lightAlt border border-l-0 border-lightBorder hover:bg-green-700 text-lightText hover:bg-dark hover:text-darkText px-2 py-1 rounded-r-md font-bold">Subscribe</button></div></form></div></div></div></div></div></div></div></ul></div></div></div><div class="flex-1 flex flex-col justify-center min-w-0 w-full"><div class=""><div class="story-body font-sans w-full min-w-0"><div class="prose max-w-[1020px] w-full min-w-0 xs:p-0 lg:px-0 prose-a:break-words prose-table:table prose-div:bg-transparent prose-table:!table prose-table:max-w-full prose-table:w-full prose-td_a:whitespace-nowrap prose-td_a:break-keep prose-td_a:overflow-wrap-normal prose-td_a:word-break-normal [&amp;_th]:!hyphens-none [&amp;_td]:!hyphens-none [&amp;_th]:!break-normal [&amp;_td]:!break-normal [&amp;_th]:![overflow-wrap:normal] [&amp;_td]:![overflow-wrap:break-word] [&amp;_th]:whitespace-nowrap xs:prose-table:mx-auto prose-table:overflow-x-auto leading-relaxed prose-p:text-lightTextLight prose-strong:text-lightTextStrong prose-strong:font-bold prose-small:text-lightTextLight prose-small:font-light prose-a:text-lightTextLight prose-p:mx-0 prose-p:my-2 prose [&amp;_.line-space]:my-0 prose-p:my-2 prose-p:text-base sm:prose-p:text-lg prose-blockquote:my-0 prose-blockquote:border-l-[5px] prose-blockquote:border-lightTextAccent prose-blockquote:pl-4 prose-blockquote:leading-relaxed prose-blockquote:text-lightText prose-h2:text-lightTextStrong prose-li:marker:text-lightText prose-h2:text-xl sm:prose-h2:text-3xl prose-h2:font-bold prose-h2:my-6 prose-h3:text-lightTextStrong prose-hr:m-2 prose-h3:text-xl sm:prose-h3:text-2xl prose-h3:font-bold prose-h3:my-5 prose-h4:text-xl prose-h4:font-bold prose-h4:my-4 prose-td:text-lightTextLight prose-td:border prose-td:border-lightBorder prose-td:px-2 prose-td:[&amp;_p]:my-0 prose-th:[&amp;_p]:my-0 prose-th:text-lightTextLight prose-th:border prose-th:border-lightBorder prose-th:px-2 prose-li:text-lg prose-li:text-lightTextLight prose-li:px-0 prose-li:mb-3 prose-li:ml-3 prose-li:leading-relaxed prose-ul:pl-2 prose-ul:sm:pl-8 prose-ol:pl-2 prose-ol:sm:pl-8 hover:prose-a:text-lightTextAccent prose-a:rounded prose-code:text-lightTextLight prose-code:break-all prose-pre:rounded-lg prose-pre:text-sm prose-pre:my-4 prose-pre:p-3 prose-pre:overflow-x-scroll prose-pre:whitespace-pre-wrap prose-pre:break-words "><div class="w-full flex items-center justify-center "><div><div class="w-full my-4"><div class="flex items-center justify-center w-full h-full"><div class="relative group cursor-zoom-in transition-transform hover:scale-[1.01] max-w-full mx-auto flex flex-col items-center"><button class="absolute top-2 right-5 z-10 w-6 h-6 rounded flex items-center justify-center opacity-0 group-hover:opacity-100 transition-opacity"><i class="hn hn-download text-darkText bg-dark p-2 rounded-xl text-base"></i></button><img alt="Yan Cui&#x27;s image-ff4928" fetchPriority="low" loading="lazy" width="940" height="830" decoding="async" data-nimg="1" class="w-full h-auto object-contain rounded-lg shadow-lg my-0" style="color:transparent;background-size:cover;background-position:50% 50%;background-repeat:no-repeat;background-image:url(&quot;data:image/svg+xml;charset=utf-8,%3Csvg xmlns=&#x27;http://www.w3.org/2000/svg&#x27; viewBox=&#x27;0 0 940 830&#x27;%3E%3Cfilter id=&#x27;b&#x27; color-interpolation-filters=&#x27;sRGB&#x27;%3E%3CfeGaussianBlur stdDeviation=&#x27;20&#x27;/%3E%3CfeColorMatrix values=&#x27;1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 100 -1&#x27; result=&#x27;s&#x27;/%3E%3CfeFlood x=&#x27;0&#x27; y=&#x27;0&#x27; width=&#x27;100%25&#x27; height=&#x27;100%25&#x27;/%3E%3CfeComposite operator=&#x27;out&#x27; in=&#x27;s&#x27;/%3E%3CfeComposite in2=&#x27;SourceGraphic&#x27;/%3E%3CfeGaussianBlur stdDeviation=&#x27;20&#x27;/%3E%3C/filter%3E%3Cimage width=&#x27;100%25&#x27; height=&#x27;100%25&#x27; x=&#x27;0&#x27; y=&#x27;0&#x27; preserveAspectRatio=&#x27;none&#x27; style=&#x27;filter: url(%23b);&#x27; href=&#x27;data:image/svg+xml;base64,CiAgICA8c3ZnIHdpZHRoPSI5NDAiIGhlaWdodD0iODMwIiB2ZXJzaW9uPSIxLjEiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgeG1sbnM6eGxpbms9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkveGxpbmsiPgogICAgICA8ZGVmcz4KICAgICAgICA8bGluZWFyR3JhZGllbnQgaWQ9ImciPgogICAgICAgICAgPHN0b3Agc3RvcC1jb2xvcj0iIzIyMiIgb2Zmc2V0PSIyMCUiIC8+CiAgICAgICAgICA8c3RvcCBzdG9wLWNvbG9yPSIjMGYwIiBvZmZzZXQ9IjUwJSIgLz4KICAgICAgICAgIDxzdG9wIHN0b3AtY29sb3I9IiMyMjIiIG9mZnNldD0iODAlIiAvPgogICAgICAgIDwvbGluZWFyR3JhZGllbnQ+CiAgICAgIDwvZGVmcz4KICAgICAgPHJlY3Qgd2lkdGg9Ijk0MCIgaGVpZ2h0PSI4MzAiIGZpbGw9IiMyMjIiIC8+CiAgICAgIDxyZWN0IGlkPSJyIiB3aWR0aD0iOTQwIiBoZWlnaHQ9IjgzMCIgZmlsbD0idXJsKCNnKSIgLz4KICAgICAgPGFuaW1hdGUgeGxpbms6aHJlZj0iI3IiIGF0dHJpYnV0ZU5hbWU9IngiIGZyb209Ii05NDAiIHRvPSI5NDAiIGR1cj0iMXMiIHJlcGVhdENvdW50PSJpbmRlZmluaXRlIiAgLz4KICAgIDwvc3ZnPg==&#x27;/%3E%3C/svg%3E&quot;)" srcSet="https://hackernoon.imgix.net/hn-images/1*56lZO-ri61SY-ndXv5PePA.png?auto=format%2Ccompress&amp;w=1080 1x, https://hackernoon.imgix.net/hn-images/1*56lZO-ri61SY-ndXv5PePA.png?auto=format%2Ccompress&amp;w=1920 2x" src="https://hackernoon.imgix.net/hn-images/1*56lZO-ri61SY-ndXv5PePA.png?auto=format%2Ccompress&amp;w=1920"/></div></div></div></div>
<p>All the way back in 2011, Simon Wardley had identified Chaos Engines as a practice that will be employed by the next generation of tech companies, along with continuous deployment, being data-driven, and organised around small, autonomous teams (think microservices &amp; inverse-conway’s law).</p>
<p>This is the first of a multipart series that explores ideas on how we could apply the principles of chaos engineering to serverless architectures built around Lambda functions.</p>
<ul>
<li><strong>part 1: how can we apply principles of chaos engineering to Lambda?</strong></li>
<li>part 2: <a href="https://hackernoon.com/chaos-engineering-and-aws-lambda-latency-injection-ddeb4ff8d983?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">applying latency injection for APIs</a></li>
<li>part 3: dealing with latency spikes and timeouts (coming 21/11/2017)</li>
<li>part 4: applying fault injection for Lambda functions (coming ?)</li>
</ul>
<p>There’s no question about it, Netflix has popularised the <em>principles</em> of chaos engineering. By open sourcing some of their tools — notably the <a href="https://github.com/Netflix/SimianArmy?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">Simian Army</a> — they have also helped others build confidence in their system’s capability to withstand turbulent conditions in production.</p>
<p>There seems to be a renewed interest in chaos engineering recently. As Russ Miles noted in a recent <a href="https://medium.com/russmiles/chaos-engineering-why-the-label-matters-35ddbb974fa5?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">post</a>, perhaps many companies have finally come to understand that chaos engineering is not about “hurting production”, but to build better understanding of, and confidence in a system’s resilience through <strong><em>controlled</em></strong> experiments.</p>
<p>This trend has been helped by the valuable (and freely available) information that Netflix has published, such as the <a href="http://oreil.ly/2tZU1Sn?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">Chaos Engineering</a> e-book, and <a href="http://principlesofchaos.org/?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">principlesofchaos.org</a>.</p>
<div><div class="w-full my-4"><div class="flex items-center justify-center w-full h-full"><div class="relative group cursor-zoom-in transition-transform hover:scale-[1.01] max-w-full mx-auto flex flex-col items-center"><button class="absolute top-2 right-5 z-10 w-6 h-6 rounded flex items-center justify-center opacity-0 group-hover:opacity-100 transition-opacity"><i class="hn hn-download text-darkText bg-dark p-2 rounded-xl text-base"></i></button><img alt="Yan Cui&#x27;s image-11926" fetchPriority="low" loading="lazy" width="1600" height="1085" decoding="async" data-nimg="1" class="w-full h-auto object-contain rounded-lg shadow-lg my-0" style="color:transparent;background-size:cover;background-position:50% 50%;background-repeat:no-repeat;background-image:url(&quot;data:image/svg+xml;charset=utf-8,%3Csvg xmlns=&#x27;http://www.w3.org/2000/svg&#x27; viewBox=&#x27;0 0 1600 1085&#x27;%3E%3Cfilter id=&#x27;b&#x27; color-interpolation-filters=&#x27;sRGB&#x27;%3E%3CfeGaussianBlur stdDeviation=&#x27;20&#x27;/%3E%3CfeColorMatrix values=&#x27;1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 100 -1&#x27; result=&#x27;s&#x27;/%3E%3CfeFlood x=&#x27;0&#x27; y=&#x27;0&#x27; width=&#x27;100%25&#x27; height=&#x27;100%25&#x27;/%3E%3CfeComposite operator=&#x27;out&#x27; in=&#x27;s&#x27;/%3E%3CfeComposite in2=&#x27;SourceGraphic&#x27;/%3E%3CfeGaussianBlur stdDeviation=&#x27;20&#x27;/%3E%3C/filter%3E%3Cimage width=&#x27;100%25&#x27; height=&#x27;100%25&#x27; x=&#x27;0&#x27; y=&#x27;0&#x27; preserveAspectRatio=&#x27;none&#x27; style=&#x27;filter: url(%23b);&#x27; href=&#x27;data:image/svg+xml;base64,CiAgICA8c3ZnIHdpZHRoPSIxNjAwIiBoZWlnaHQ9IjEwODUiIHZlcnNpb249IjEuMSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayI+CiAgICAgIDxkZWZzPgogICAgICAgIDxsaW5lYXJHcmFkaWVudCBpZD0iZyI+CiAgICAgICAgICA8c3RvcCBzdG9wLWNvbG9yPSIjMjIyIiBvZmZzZXQ9IjIwJSIgLz4KICAgICAgICAgIDxzdG9wIHN0b3AtY29sb3I9IiMwZjAiIG9mZnNldD0iNTAlIiAvPgogICAgICAgICAgPHN0b3Agc3RvcC1jb2xvcj0iIzIyMiIgb2Zmc2V0PSI4MCUiIC8+CiAgICAgICAgPC9saW5lYXJHcmFkaWVudD4KICAgICAgPC9kZWZzPgogICAgICA8cmVjdCB3aWR0aD0iMTYwMCIgaGVpZ2h0PSIxMDg1IiBmaWxsPSIjMjIyIiAvPgogICAgICA8cmVjdCBpZD0iciIgd2lkdGg9IjE2MDAiIGhlaWdodD0iMTA4NSIgZmlsbD0idXJsKCNnKSIgLz4KICAgICAgPGFuaW1hdGUgeGxpbms6aHJlZj0iI3IiIGF0dHJpYnV0ZU5hbWU9IngiIGZyb209Ii0xNjAwIiB0bz0iMTYwMCIgZHVyPSIxcyIgcmVwZWF0Q291bnQ9ImluZGVmaW5pdGUiICAvPgogICAgPC9zdmc+&#x27;/%3E%3C/svg%3E&quot;)" srcSet="https://hackernoon.imgix.net/hn-images/1*JlnapsrFEDH771EYRel0VA.png?auto=format%2Ccompress&amp;w=1920 1x, https://hackernoon.imgix.net/hn-images/1*JlnapsrFEDH771EYRel0VA.png?auto=format%2Ccompress&amp;w=3840 2x" src="https://hackernoon.imgix.net/hn-images/1*JlnapsrFEDH771EYRel0VA.png?auto=format%2Ccompress&amp;w=3840"/></div></div></div></div>
<p>Tools such as <a href="https://github.com/shoreditch-ops/chaos-lambda?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">chaos-lambda</a> by <em>Shoreditch Ops</em> (the folks behind the <a href="https://artillery.io/?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">Artillery</a> load test tool) look to replicate Netflix’s <a href="https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">Chaos Monkey</a>, but execute from inside a Lambda function instead of an EC2 instance — hence bringing you the cost saving and convenience Lambda offers.</p>
<p>I want to ask a different question however: <strong>how can one apply the principles of chaos engineering and some of the current practices to a serverless architecture comprised of Lambda functions?</strong></p>
<p>When your system runs on EC2 instances, then naturally, you build resilience by designing for the most likely failure mode — server crashes (due to both hardware and software issues). Hence, a controlled experiment to validate the resilience of your system would artificially recreate the failure scenario by terminating EC2 instances, and then AZs, and then entire regions.</p>
<p>AWS Lambda, however, is a higher-level abstraction and has different failure modes to its EC2 counterparts. Hypothesis that focus on <em>“what if we lose these EC2 instances”</em> no longer apply as the platform handles these failure modes for you out of the box.</p>
<p>We need to ask different questions in order to understand the weaknesses within our serverless architectures.</p>
<h3>More inherent chaos, not less</h3>
<blockquote>
<p>“<strong>We need to identify weaknesses before they manifest in system-wide, aberrant behaviors</strong>. Systemic weaknesses could take the form of: improper fallback settings when a service is unavailable; retry storms from improperly tuned timeouts; outages when a downstream dependency receives too much traffic; cascading failures when a single point of failure crashes; etc. We must address the most significant weaknesses proactively, before they affect our customers in production. <strong>We need a way to manage the chaos inherent in these systems</strong>, take advantage of increasing flexibility and velocity, and <strong>have confidence in our production deployments despite the complexity that they represent.</strong>”</p>
</blockquote>
<blockquote>
<p>— <a href="http://principlesofchaos.org/?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">Principles of Chaos Engineering</a></p>
</blockquote>
<p>Having <a href="https://hackernoon.com/yubls-road-to-serverless-part-1-overview-ca348370acde?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">built and operated a non-trivial serverless architecture</a>, I have some understanding of the dangers awaiting you in this new world.</p>
<p>If anything, there are a lot more inherent chaos and complexity in these systems built around Lambda functions.</p>
<ul>
<li>modularity (unit of deployment) shifts from “services” to “functions”, and there are a lot more of them</li>
<li>it’s harder to harden around the boundaries, because you need to harden around each function as opposed to a service which encapsulates a set of related functionalities</li>
<li>there are more intermediary services (eg. Kinesis, SNS, API Gateway just to name a few), each with their own failure modes</li>
<li>there are more configurations overall (timeout, IAM permissions, etc.), and therefore more opportunities for misconfiguration</li>
</ul>
<p>Also, since we have traded off more control of our infrastructure* it means we now face more unknown failure modes** and often there’s little we can do when an outage does occur***.</p>
<p><em>* For better scalability, availability, cost efficiency and more convenience, which I for one, think it’s a fair trade in</em> <strong><em>most cases</em></strong><em>.</em></p>
<p><em>** Everything the platform does for you — scheduling containers, scaling, polling Kinesis, retry failed invocations, etc. — have their own failure modes. These are often not obvious to us since they’re implementation details that are typically undocumented and are prone to change without notice.</em></p>
<p><em>*** For example, if an outage happens and prevents Lambda functions from processing Kinesis events, then we have no meaningful alternative than to wait for AWS to fix the problem. Since the current position on the shards is abstracted away and unavailable to us, we can’t even replace the Lambda functions with KCL processors that run on EC2.</em></p>
<h3>Applying chaos to AWS Lambda</h3>
<p>A good exercise regime would continuously push you to your limits but never actually put you over the limit and cause injury. If there’s an exercise that is clearly beyond your current abilities then surely you would not attempt it as the only possible outcome is getting yourself hurt!</p>
<p><strong>The same common sense should be applied when designing controlled experiments for your serverless architecture</strong>. In order to “know” what the experiments tell us about the resilience of our system we also need to decide what metrics to monitor — ideally using client-side metrics, since the most important metric is the quality of service our users experience.</p>
<p>There are plenty of failure modes that we know about and can design for, and we can run simple experiments to validate our design. For example, since a serverless architecture is (almost always) also a microservice architecture, many of its inherent failure modes still apply:</p>
<ul>
<li>improperly tuned timeouts, especially for intermediate services, which can cause services at the edge to also timeout</li>
</ul>
<div><div class="w-full my-4"><div class="flex items-center justify-center w-full h-full"><div class="relative group cursor-zoom-in transition-transform hover:scale-[1.01] max-w-full mx-auto flex flex-col items-center"><button class="absolute top-2 right-5 z-10 w-6 h-6 rounded flex items-center justify-center opacity-0 group-hover:opacity-100 transition-opacity"><i class="hn hn-download text-darkText bg-dark p-2 rounded-xl text-base"></i></button><img alt="Yan Cui&#x27;s image-dae0b8" fetchPriority="low" loading="lazy" width="1544" height="978" decoding="async" data-nimg="1" class="w-full h-auto object-contain rounded-lg shadow-lg my-0" style="color:transparent;background-size:cover;background-position:50% 50%;background-repeat:no-repeat;background-image:url(&quot;data:image/svg+xml;charset=utf-8,%3Csvg xmlns=&#x27;http://www.w3.org/2000/svg&#x27; viewBox=&#x27;0 0 1544 978&#x27;%3E%3Cfilter id=&#x27;b&#x27; color-interpolation-filters=&#x27;sRGB&#x27;%3E%3CfeGaussianBlur stdDeviation=&#x27;20&#x27;/%3E%3CfeColorMatrix values=&#x27;1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 100 -1&#x27; result=&#x27;s&#x27;/%3E%3CfeFlood x=&#x27;0&#x27; y=&#x27;0&#x27; width=&#x27;100%25&#x27; height=&#x27;100%25&#x27;/%3E%3CfeComposite operator=&#x27;out&#x27; in=&#x27;s&#x27;/%3E%3CfeComposite in2=&#x27;SourceGraphic&#x27;/%3E%3CfeGaussianBlur stdDeviation=&#x27;20&#x27;/%3E%3C/filter%3E%3Cimage width=&#x27;100%25&#x27; height=&#x27;100%25&#x27; x=&#x27;0&#x27; y=&#x27;0&#x27; preserveAspectRatio=&#x27;none&#x27; style=&#x27;filter: url(%23b);&#x27; href=&#x27;data:image/svg+xml;base64,CiAgICA8c3ZnIHdpZHRoPSIxNTQ0IiBoZWlnaHQ9Ijk3OCIgdmVyc2lvbj0iMS4xIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIj4KICAgICAgPGRlZnM+CiAgICAgICAgPGxpbmVhckdyYWRpZW50IGlkPSJnIj4KICAgICAgICAgIDxzdG9wIHN0b3AtY29sb3I9IiMyMjIiIG9mZnNldD0iMjAlIiAvPgogICAgICAgICAgPHN0b3Agc3RvcC1jb2xvcj0iIzBmMCIgb2Zmc2V0PSI1MCUiIC8+CiAgICAgICAgICA8c3RvcCBzdG9wLWNvbG9yPSIjMjIyIiBvZmZzZXQ9IjgwJSIgLz4KICAgICAgICA8L2xpbmVhckdyYWRpZW50PgogICAgICA8L2RlZnM+CiAgICAgIDxyZWN0IHdpZHRoPSIxNTQ0IiBoZWlnaHQ9Ijk3OCIgZmlsbD0iIzIyMiIgLz4KICAgICAgPHJlY3QgaWQ9InIiIHdpZHRoPSIxNTQ0IiBoZWlnaHQ9Ijk3OCIgZmlsbD0idXJsKCNnKSIgLz4KICAgICAgPGFuaW1hdGUgeGxpbms6aHJlZj0iI3IiIGF0dHJpYnV0ZU5hbWU9IngiIGZyb209Ii0xNTQ0IiB0bz0iMTU0NCIgZHVyPSIxcyIgcmVwZWF0Q291bnQ9ImluZGVmaW5pdGUiICAvPgogICAgPC9zdmc+&#x27;/%3E%3C/svg%3E&quot;)" srcSet="https://hackernoon.imgix.net/hn-images/1*OFKVj5FywcNs-CpsXFPi0w.png?auto=format%2Ccompress&amp;w=1920 1x, https://hackernoon.imgix.net/hn-images/1*OFKVj5FywcNs-CpsXFPi0w.png?auto=format%2Ccompress&amp;w=3840 2x" src="https://hackernoon.imgix.net/hn-images/1*OFKVj5FywcNs-CpsXFPi0w.png?auto=format%2Ccompress&amp;w=3840"/></div></div></div></div>
<p>Intermediate services should have more strict timeout settings compared to services at the edge.</p>
<ul>
<li>missing error handling, which allows exceptions from downstream services to escape</li>
</ul>
<div><div class="w-full my-4"><div class="flex items-center justify-center w-full h-full"><div class="relative group cursor-zoom-in transition-transform hover:scale-[1.01] max-w-full mx-auto flex flex-col items-center"><button class="absolute top-2 right-5 z-10 w-6 h-6 rounded flex items-center justify-center opacity-0 group-hover:opacity-100 transition-opacity"><i class="hn hn-download text-darkText bg-dark p-2 rounded-xl text-base"></i></button><img alt="Yan Cui&#x27;s image-af44" fetchPriority="low" loading="lazy" width="1500" height="782" decoding="async" data-nimg="1" class="w-full h-auto object-contain rounded-lg shadow-lg my-0" style="color:transparent;background-size:cover;background-position:50% 50%;background-repeat:no-repeat;background-image:url(&quot;data:image/svg+xml;charset=utf-8,%3Csvg xmlns=&#x27;http://www.w3.org/2000/svg&#x27; viewBox=&#x27;0 0 1500 782&#x27;%3E%3Cfilter id=&#x27;b&#x27; color-interpolation-filters=&#x27;sRGB&#x27;%3E%3CfeGaussianBlur stdDeviation=&#x27;20&#x27;/%3E%3CfeColorMatrix values=&#x27;1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 100 -1&#x27; result=&#x27;s&#x27;/%3E%3CfeFlood x=&#x27;0&#x27; y=&#x27;0&#x27; width=&#x27;100%25&#x27; height=&#x27;100%25&#x27;/%3E%3CfeComposite operator=&#x27;out&#x27; in=&#x27;s&#x27;/%3E%3CfeComposite in2=&#x27;SourceGraphic&#x27;/%3E%3CfeGaussianBlur stdDeviation=&#x27;20&#x27;/%3E%3C/filter%3E%3Cimage width=&#x27;100%25&#x27; height=&#x27;100%25&#x27; x=&#x27;0&#x27; y=&#x27;0&#x27; preserveAspectRatio=&#x27;none&#x27; style=&#x27;filter: url(%23b);&#x27; href=&#x27;data:image/svg+xml;base64,CiAgICA8c3ZnIHdpZHRoPSIxNTAwIiBoZWlnaHQ9Ijc4MiIgdmVyc2lvbj0iMS4xIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIj4KICAgICAgPGRlZnM+CiAgICAgICAgPGxpbmVhckdyYWRpZW50IGlkPSJnIj4KICAgICAgICAgIDxzdG9wIHN0b3AtY29sb3I9IiMyMjIiIG9mZnNldD0iMjAlIiAvPgogICAgICAgICAgPHN0b3Agc3RvcC1jb2xvcj0iIzBmMCIgb2Zmc2V0PSI1MCUiIC8+CiAgICAgICAgICA8c3RvcCBzdG9wLWNvbG9yPSIjMjIyIiBvZmZzZXQ9IjgwJSIgLz4KICAgICAgICA8L2xpbmVhckdyYWRpZW50PgogICAgICA8L2RlZnM+CiAgICAgIDxyZWN0IHdpZHRoPSIxNTAwIiBoZWlnaHQ9Ijc4MiIgZmlsbD0iIzIyMiIgLz4KICAgICAgPHJlY3QgaWQ9InIiIHdpZHRoPSIxNTAwIiBoZWlnaHQ9Ijc4MiIgZmlsbD0idXJsKCNnKSIgLz4KICAgICAgPGFuaW1hdGUgeGxpbms6aHJlZj0iI3IiIGF0dHJpYnV0ZU5hbWU9IngiIGZyb209Ii0xNTAwIiB0bz0iMTUwMCIgZHVyPSIxcyIgcmVwZWF0Q291bnQ9ImluZGVmaW5pdGUiICAvPgogICAgPC9zdmc+&#x27;/%3E%3C/svg%3E&quot;)" srcSet="https://hackernoon.imgix.net/hn-images/1*DSKdK5TqOy8FZ4F9xUmK8g.png?auto=format%2Ccompress&amp;w=1920 1x, https://hackernoon.imgix.net/hn-images/1*DSKdK5TqOy8FZ4F9xUmK8g.png?auto=format%2Ccompress&amp;w=3840 2x" src="https://hackernoon.imgix.net/hn-images/1*DSKdK5TqOy8FZ4F9xUmK8g.png?auto=format%2Ccompress&amp;w=3840"/></div></div></div></div>
<ul>
<li>missing fallbacks for when a downstream service is unavailable or experiences an outage</li>
</ul>
<div><div class="w-full my-4"><div class="flex items-center justify-center w-full h-full"><div class="relative group cursor-zoom-in transition-transform hover:scale-[1.01] max-w-full mx-auto flex flex-col items-center"><button class="absolute top-2 right-5 z-10 w-6 h-6 rounded flex items-center justify-center opacity-0 group-hover:opacity-100 transition-opacity"><i class="hn hn-download text-darkText bg-dark p-2 rounded-xl text-base"></i></button><img alt="Yan Cui&#x27;s image-bf9088" fetchPriority="low" loading="lazy" width="1332" height="674" decoding="async" data-nimg="1" class="w-full h-auto object-contain rounded-lg shadow-lg my-0" style="color:transparent;background-size:cover;background-position:50% 50%;background-repeat:no-repeat;background-image:url(&quot;data:image/svg+xml;charset=utf-8,%3Csvg xmlns=&#x27;http://www.w3.org/2000/svg&#x27; viewBox=&#x27;0 0 1332 674&#x27;%3E%3Cfilter id=&#x27;b&#x27; color-interpolation-filters=&#x27;sRGB&#x27;%3E%3CfeGaussianBlur stdDeviation=&#x27;20&#x27;/%3E%3CfeColorMatrix values=&#x27;1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 100 -1&#x27; result=&#x27;s&#x27;/%3E%3CfeFlood x=&#x27;0&#x27; y=&#x27;0&#x27; width=&#x27;100%25&#x27; height=&#x27;100%25&#x27;/%3E%3CfeComposite operator=&#x27;out&#x27; in=&#x27;s&#x27;/%3E%3CfeComposite in2=&#x27;SourceGraphic&#x27;/%3E%3CfeGaussianBlur stdDeviation=&#x27;20&#x27;/%3E%3C/filter%3E%3Cimage width=&#x27;100%25&#x27; height=&#x27;100%25&#x27; x=&#x27;0&#x27; y=&#x27;0&#x27; preserveAspectRatio=&#x27;none&#x27; style=&#x27;filter: url(%23b);&#x27; href=&#x27;data:image/svg+xml;base64,CiAgICA8c3ZnIHdpZHRoPSIxMzMyIiBoZWlnaHQ9IjY3NCIgdmVyc2lvbj0iMS4xIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIj4KICAgICAgPGRlZnM+CiAgICAgICAgPGxpbmVhckdyYWRpZW50IGlkPSJnIj4KICAgICAgICAgIDxzdG9wIHN0b3AtY29sb3I9IiMyMjIiIG9mZnNldD0iMjAlIiAvPgogICAgICAgICAgPHN0b3Agc3RvcC1jb2xvcj0iIzBmMCIgb2Zmc2V0PSI1MCUiIC8+CiAgICAgICAgICA8c3RvcCBzdG9wLWNvbG9yPSIjMjIyIiBvZmZzZXQ9IjgwJSIgLz4KICAgICAgICA8L2xpbmVhckdyYWRpZW50PgogICAgICA8L2RlZnM+CiAgICAgIDxyZWN0IHdpZHRoPSIxMzMyIiBoZWlnaHQ9IjY3NCIgZmlsbD0iIzIyMiIgLz4KICAgICAgPHJlY3QgaWQ9InIiIHdpZHRoPSIxMzMyIiBoZWlnaHQ9IjY3NCIgZmlsbD0idXJsKCNnKSIgLz4KICAgICAgPGFuaW1hdGUgeGxpbms6aHJlZj0iI3IiIGF0dHJpYnV0ZU5hbWU9IngiIGZyb209Ii0xMzMyIiB0bz0iMTMzMiIgZHVyPSIxcyIgcmVwZWF0Q291bnQ9ImluZGVmaW5pdGUiICAvPgogICAgPC9zdmc+&#x27;/%3E%3C/svg%3E&quot;)" srcSet="https://hackernoon.imgix.net/hn-images/1*1ZK4_SQEk3X1g4PowOOCIg.png?auto=format%2Ccompress&amp;w=1920 1x, https://hackernoon.imgix.net/hn-images/1*1ZK4_SQEk3X1g4PowOOCIg.png?auto=format%2Ccompress&amp;w=3840 2x" src="https://hackernoon.imgix.net/hn-images/1*1ZK4_SQEk3X1g4PowOOCIg.png?auto=format%2Ccompress&amp;w=3840"/></div></div></div></div>
<p>Over the next couple of posts, we will explore how we can apply the practices of latency and fault injection to Lambda functions in order to simulate these failure modes and validate our design.</p>
<h3>Further readings:</h3>
<ul>
<li><a href="https://medium.com/netflix-techblog/chap-chaos-automation-platform-53e6d528371f?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">ChAP: Chaos Automation Platform</a></li>
<li><a href="https://medium.com/netflix-techblog/deploying-the-netflix-api-79b6176cc3f0?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">Deploying the Netflix API</a></li>
<li><a href="https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab116?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">The Netflix Simian Army</a></li>
<li><a href="https://opensource.com/article/17/8/testing-production?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">Testing in production: Yes you can (and should)</a></li>
<li><a href="http://theburningmonk.com/2015/04/design-for-latency-issues/?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">Design for latency issues</a></li>
<li><a href="https://erlangcentral.org/blog/presentations/1000-actors-one-chaos-monkey-and-everything-ok/?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">1000 actors, one chaos monkey… and everything’s OK</a></li>
<li>[Ebook] <a href="http://www.oreilly.com/webops-perf/free/chaos-engineering.csp?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">Chaos Engineering</a></li>
<li>[Book] <a href="http://amzn.to/1pedVvt?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">Release it!</a></li>
<li>[Book] <a href="http://amzn.to/1CB0I6D?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">Drift into Failure</a></li>
<li><a href="http://principlesofchaos.org/?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">principlesofchaos.org</a></li>
</ul>
<div><div class="w-full my-4"><div class="flex items-center justify-center w-full h-full"><div class="relative group cursor-zoom-in transition-transform hover:scale-[1.01] max-w-full mx-auto flex flex-col items-center"><button class="absolute top-2 right-5 z-10 w-6 h-6 rounded flex items-center justify-center opacity-0 group-hover:opacity-100 transition-opacity"><i class="hn hn-download text-darkText bg-dark p-2 rounded-xl text-base"></i></button><img alt="Yan Cui&#x27;s image-ea6f08" fetchPriority="low" loading="lazy" width="309" height="313" decoding="async" data-nimg="1" class="w-full h-auto object-contain rounded-lg shadow-lg my-0" style="color:transparent;background-size:cover;background-position:50% 50%;background-repeat:no-repeat;background-image:url(&quot;data:image/svg+xml;charset=utf-8,%3Csvg xmlns=&#x27;http://www.w3.org/2000/svg&#x27; viewBox=&#x27;0 0 309 313&#x27;%3E%3Cfilter id=&#x27;b&#x27; color-interpolation-filters=&#x27;sRGB&#x27;%3E%3CfeGaussianBlur stdDeviation=&#x27;20&#x27;/%3E%3CfeColorMatrix values=&#x27;1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 100 -1&#x27; result=&#x27;s&#x27;/%3E%3CfeFlood x=&#x27;0&#x27; y=&#x27;0&#x27; width=&#x27;100%25&#x27; height=&#x27;100%25&#x27;/%3E%3CfeComposite operator=&#x27;out&#x27; in=&#x27;s&#x27;/%3E%3CfeComposite in2=&#x27;SourceGraphic&#x27;/%3E%3CfeGaussianBlur stdDeviation=&#x27;20&#x27;/%3E%3C/filter%3E%3Cimage width=&#x27;100%25&#x27; height=&#x27;100%25&#x27; x=&#x27;0&#x27; y=&#x27;0&#x27; preserveAspectRatio=&#x27;none&#x27; style=&#x27;filter: url(%23b);&#x27; href=&#x27;data:image/svg+xml;base64,CiAgICA8c3ZnIHdpZHRoPSIzMDkiIGhlaWdodD0iMzEzIiB2ZXJzaW9uPSIxLjEiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgeG1sbnM6eGxpbms9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkveGxpbmsiPgogICAgICA8ZGVmcz4KICAgICAgICA8bGluZWFyR3JhZGllbnQgaWQ9ImciPgogICAgICAgICAgPHN0b3Agc3RvcC1jb2xvcj0iIzIyMiIgb2Zmc2V0PSIyMCUiIC8+CiAgICAgICAgICA8c3RvcCBzdG9wLWNvbG9yPSIjMGYwIiBvZmZzZXQ9IjUwJSIgLz4KICAgICAgICAgIDxzdG9wIHN0b3AtY29sb3I9IiMyMjIiIG9mZnNldD0iODAlIiAvPgogICAgICAgIDwvbGluZWFyR3JhZGllbnQ+CiAgICAgIDwvZGVmcz4KICAgICAgPHJlY3Qgd2lkdGg9IjMwOSIgaGVpZ2h0PSIzMTMiIGZpbGw9IiMyMjIiIC8+CiAgICAgIDxyZWN0IGlkPSJyIiB3aWR0aD0iMzA5IiBoZWlnaHQ9IjMxMyIgZmlsbD0idXJsKCNnKSIgLz4KICAgICAgPGFuaW1hdGUgeGxpbms6aHJlZj0iI3IiIGF0dHJpYnV0ZU5hbWU9IngiIGZyb209Ii0zMDkiIHRvPSIzMDkiIGR1cj0iMXMiIHJlcGVhdENvdW50PSJpbmRlZmluaXRlIiAgLz4KICAgIDwvc3ZnPg==&#x27;/%3E%3C/svg%3E&quot;)" srcSet="https://hackernoon.imgix.net/hn-images/0*b_1R345KzKSaI8sg.png?auto=format%2Ccompress&amp;w=384 1x, https://hackernoon.imgix.net/hn-images/0*b_1R345KzKSaI8sg.png?auto=format%2Ccompress&amp;w=640 2x" src="https://hackernoon.imgix.net/hn-images/0*b_1R345KzKSaI8sg.png?auto=format%2Ccompress&amp;w=640"/></div></div></div></div>
<p>Hi, my name is <strong>Yan Cui</strong>. I’m an <a href="https://aws.amazon.com/developer/community/heroes/yan-cui/?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc"><strong>AWS Serverless Hero</strong></a> and the author of <a href="https://bit.ly/production-ready-serverless?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc"><strong>Production-Ready Serverless</strong></a>. I have run production workload at scale in AWS for nearly 10 years and I have been an architect or principal engineer with a variety of industries ranging from banking, e-commerce, sports streaming to mobile gaming. I currently work as an independent consultant focused on AWS and serverless.</p>
<p>You can contact me via <a href="http://mailto:theburningmonk.com" target="_blank" rel="noopener noreferrer ugc">Email</a>, <a href="https://twitter.com/theburningmonk?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">Twitter</a> and <a href="https://www.linkedin.com/in/theburningmonk/?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">LinkedIn</a>.</p>
<p>Check out my new course, <a href="https://theburningmonk.thinkific.com/courses/complete-guide-to-aws-step-functions?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc"><strong>Complete Guide to AWS Step Functions</strong></a>.</p>
<p>In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. Including basic concepts, HTTP and event triggers, activities, design patterns and best practices.</p>
<p>Get your copy <a href="https://theburningmonk.thinkific.com/courses/complete-guide-to-aws-step-functions?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">here</a>.</p>
<div><div class="w-full my-4"><div class="flex items-center justify-center w-full h-full"><div class="relative group cursor-zoom-in transition-transform hover:scale-[1.01] max-w-full mx-auto flex flex-col items-center"><button class="absolute top-2 right-5 z-10 w-6 h-6 rounded flex items-center justify-center opacity-0 group-hover:opacity-100 transition-opacity"><i class="hn hn-download text-darkText bg-dark p-2 rounded-xl text-base"></i></button><img alt="Yan Cui&#x27;s image-70a08" fetchPriority="low" loading="lazy" width="1024" height="368" decoding="async" data-nimg="1" class="w-full h-auto object-contain rounded-lg shadow-lg my-0" style="color:transparent;background-size:cover;background-position:50% 50%;background-repeat:no-repeat;background-image:url(&quot;data:image/svg+xml;charset=utf-8,%3Csvg xmlns=&#x27;http://www.w3.org/2000/svg&#x27; viewBox=&#x27;0 0 1024 368&#x27;%3E%3Cfilter id=&#x27;b&#x27; color-interpolation-filters=&#x27;sRGB&#x27;%3E%3CfeGaussianBlur stdDeviation=&#x27;20&#x27;/%3E%3CfeColorMatrix values=&#x27;1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 100 -1&#x27; result=&#x27;s&#x27;/%3E%3CfeFlood x=&#x27;0&#x27; y=&#x27;0&#x27; width=&#x27;100%25&#x27; height=&#x27;100%25&#x27;/%3E%3CfeComposite operator=&#x27;out&#x27; in=&#x27;s&#x27;/%3E%3CfeComposite in2=&#x27;SourceGraphic&#x27;/%3E%3CfeGaussianBlur stdDeviation=&#x27;20&#x27;/%3E%3C/filter%3E%3Cimage width=&#x27;100%25&#x27; height=&#x27;100%25&#x27; x=&#x27;0&#x27; y=&#x27;0&#x27; preserveAspectRatio=&#x27;none&#x27; style=&#x27;filter: url(%23b);&#x27; href=&#x27;data:image/svg+xml;base64,CiAgICA8c3ZnIHdpZHRoPSIxMDI0IiBoZWlnaHQ9IjM2OCIgdmVyc2lvbj0iMS4xIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIj4KICAgICAgPGRlZnM+CiAgICAgICAgPGxpbmVhckdyYWRpZW50IGlkPSJnIj4KICAgICAgICAgIDxzdG9wIHN0b3AtY29sb3I9IiMyMjIiIG9mZnNldD0iMjAlIiAvPgogICAgICAgICAgPHN0b3Agc3RvcC1jb2xvcj0iIzBmMCIgb2Zmc2V0PSI1MCUiIC8+CiAgICAgICAgICA8c3RvcCBzdG9wLWNvbG9yPSIjMjIyIiBvZmZzZXQ9IjgwJSIgLz4KICAgICAgICA8L2xpbmVhckdyYWRpZW50PgogICAgICA8L2RlZnM+CiAgICAgIDxyZWN0IHdpZHRoPSIxMDI0IiBoZWlnaHQ9IjM2OCIgZmlsbD0iIzIyMiIgLz4KICAgICAgPHJlY3QgaWQ9InIiIHdpZHRoPSIxMDI0IiBoZWlnaHQ9IjM2OCIgZmlsbD0idXJsKCNnKSIgLz4KICAgICAgPGFuaW1hdGUgeGxpbms6aHJlZj0iI3IiIGF0dHJpYnV0ZU5hbWU9IngiIGZyb209Ii0xMDI0IiB0bz0iMTAyNCIgZHVyPSIxcyIgcmVwZWF0Q291bnQ9ImluZGVmaW5pdGUiICAvPgogICAgPC9zdmc+&#x27;/%3E%3C/svg%3E&quot;)" srcSet="https://hackernoon.imgix.net/hn-images/0*ZYcHhOOzUf5VB-Ri.png?auto=format%2Ccompress&amp;w=1080 1x, https://hackernoon.imgix.net/hn-images/0*ZYcHhOOzUf5VB-Ri.png?auto=format%2Ccompress&amp;w=2048 2x" src="https://hackernoon.imgix.net/hn-images/0*ZYcHhOOzUf5VB-Ri.png?auto=format%2Ccompress&amp;w=2048"/></div></div></div></div>
<p>Come learn about operational <strong>BEST PRACTICES</strong> for AWS Lambda: CI/CD, testing &amp; debugging functions locally, logging, monitoring, distributed tracing, canary deployments, config management, authentication &amp; authorization, VPC, security, error handling, and more.</p>
<p>You can also get <strong>40%</strong> off the face price with the code <strong>ytcui</strong>.</p>
<p>Get your copy <a href="https://bit.ly/production-ready-serverless?ref=hackernoon.com" target="_blank" rel="noopener noreferrer ugc">here</a>.</p></div></div></div></div></div><div class="hidden xl:flex xl:flex-col self-stretch"><div class="sticky top-[99px] px-3"><div class=" flex flex-col flex-row-reverse items-start gap-4 "><span class="tooltip tooltip-left cursor-pointer" data-tip="Bookmark"><button class="3xl:hover:bg-lightAlt hover:bg-light p-1 md:p-2 rounded h-[40px] w-[40px] flex items-center justify-center border border-lightBorder"><i class="hn hn-bookmark text-lightText text-2xl"></i></button></span><span class="tooltip tooltip-left cursor-pointer" data-tip="Comment"><button class="3xl:hover:bg-lightAlt hover:bg-light p-1 md:p-2 rounded h-[40px] w-[40px] flex items-center justify-center border border-lightBorder"><i class="hn hn-comment text-lightText text-2xl"></i></button></span><div class="dropdown dropdown-bottom dropdown-hover group "><label tabindex="0" class="flex items-center cursor-pointer justify-center border border-lightBorder 3xl:group-hover:bg-lightAlt group-hover:bg-light h-[40px] w-[40px] p-2 rounded "><i class="hn hn-share text-2xl"></i></label><ul tabindex="0" class="dropdown-content bg-light z-[1] py-4 px-4 3xl:px-0 3xl:py-2 border 3xl:border-none flex flex-col items-center justify-center gap-2 "><button class="border p-2 rounded hover:bg-lightAlt"><i class=" hn hn-copy text-lightText text-2xl "></i></button><button class="border p-2 rounded hover:bg-lightAlt"><i class="hn hn-facebook-round text-lightText text-2xl"></i></button><button class="border p-2 rounded hover:bg-lightAlt"><i class="hn hn-x text-lightText text-2xl"></i></button><button class="border p-2 rounded hover:bg-lightAlt"><i class="hn hn-linkedin text-lightText text-2xl"></i></button><a href="mailto:?subject=I&#x27;d like to share a link with you &amp;body=" class="border p-2 rounded inline-block hover:bg-lightAlt"><i class="hn hn-envelope text-lightText text-2xl"></i></a></ul></div></div></div></div></div><div class="px-4 lg:px-0 mx-auto w-full lg:max-w-[1000px] flex-col flex items-center justify-center "><div id="commentSection" class=" font-sans max-w-[1000px] mt-4 mb-10 px-4 sm:px-0 items-center rounded-xl w-full flex flex-col"><div class="flex w-full flex-col xs:flex-row items-stretch justify-between gap-5 "><a href="/capture-and-forward-correlation-ids-through-different-lambda-event-sources-220c227c65f5" rel="external" class="flex xs:w-1/2 flex-col group justify-between no-underline border border-lightBorder rounded-[5px] transition-all duration-300 hover:scale-[1.03]"><div class="flex-grow p-3 text-lightText"><span class="font-bold hover:text-lightTextStrong">← Previous</span><p class="mt-2 font-light hover:underline">Capture and forward correlation IDs through different Lambda event sources</p></div></a><a href="/mind-the-75gb-limit-on-aws-lambda-deployment-packages-163b93c8eb72" rel="external" class="flex w-full xs:w-1/2 flex-col group justify-between no-underline border border-lightBorder rounded-[5px] transition-all duration-300 hover:scale-[1.03]"><div class="flex-grow p-3 "><span class="font-bold hover:text-lightTextStrong">Up Next →</span><p class="mt-2 font-light hover:underline ">Mind the 75GB limit on AWS Lambda deployment packages</p></div></a></div></div></div><div id="aboutCard" class=" max-w-[1000px] mx-auto flex flex-col items-center gap-6 "><div class="w-full lg:border border-lightBorder rounded-2xl"><div class=" w-full px-4 py-3 sm:px-8 sm:py-6 "><h3 class="text-xl xs:text-2xl sm:text-3xl font-bold mb-6">About Author</h3><div class="flex flex-col items-start"><div class="flex gap-4 flex-row items-start w-full"><div class="relative shadow-md rounded-full flex-shrink-0 min-w-[50px] w-[50px] h-[50px] sm:min-w-[75px] sm:h-[75px] ring-4 ring-gray-300"><a href="/u/theburningmonk"><img alt="Yan Cui HackerNoon profile picture" loading="lazy" decoding="async" data-nimg="fill" class="rounded-full" style="position:absolute;height:100%;width:100%;left:0;top:0;right:0;bottom:0;object-fit:cover;color:transparent" sizes="100vw" srcSet="https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=640 640w, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=750 750w, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=828 828w, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=1080 1080w, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=1200 1200w, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=1920 1920w, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=2048 2048w, https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=3840 3840w" src="https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&amp;w=3840"/></a></div><div class="flex-1 min-w-0 flex flex-col justify-center"><div class="flex flex-col"><div class="flex flex-wrap items-center gap-1 sm:gap-2 text-base text-bodyTxtLight"><span class="text-xs font-light mr-1">by</span><a class="hover:underline" href="/u/theburningmonk"><strong class="font-bold text-lightTextStrong">Yan Cui</strong></a><span class="text-xs font-light opacity-50">|</span><a class="hover:underline text-sm sm:text-base text-bodyTxtLight" href="/u/theburningmonk">@<!-- -->theburningmonk</a></div></div></div></div><p class="text-sm text-bodyTxtLight break-words overflow-wrap mt-4 mb-4 w-full">AWS Serverless Hero. Independent Consultant. Developer Advocate at Lumigo.</p><div class="w-full mb-4"><div class="w-full flex justify-start"><div class="w-full"><form class="w-full flex flex-col items-start gap-2 "><div class="flex w-full"><input class="p-2 flex-grow border rounded-l-md text-lightText bg-light focus:outline-none focus:ring-0 focus:ring-transparent border-lightBorder w-full text-base px-2}
}" placeholder="name@company.com" type="email" required="" name="email" value=""/><button type="submit" class="text-base}
bg-lightAlt border border-l-0 border-lightBorder hover:bg-green-700 text-lightText hover:bg-dark hover:text-darkText px-2 py-1 rounded-r-md font-bold">Subscribe</button></div></form></div></div></div><div class="flex-1 w-full"><div class="flex flex-col flex-wrap gap-2 items-start justify-start mt-2 mb-2"></div><div class="flex flex-col sm:flex-row gap-4 w-full"><a class="text-base flex-1 px-4 py-2 font-bold rounded-lg border-2 border-lightBorder transition text-center w-full sm:w-auto bg-light hover:bg-lightAlt text-lightText hover:bg-bodyAccent hover:text-bodyAccentTxt " href="/u/theburningmonk">Read my stories</a><a class="text-base break-words flex-1 break-all px-4 py-2 font-bold rounded-lg border-2 border-lightBorder transition text-center w-full sm:w-auto bg-light hover:bg-lightAlt text-lightText hover:bg-bodyAccent hover:text-bodyAccentTxt " href="/about/theburningmonk">Learn More</a></div></div></div></div></div><span id="aboutCard" class="hidden"></span><section class="w-full py-3 px-4 sm:px-0 sm:py-6 "><h4 class="text-xl xs:text-2xl sm:text-3xl font-bold mb-4 sm:mb-6">TOPICS</h4><div class="flex flex-wrap gap-2"><div class=" flex flex-wrap items-center gap-2 border-lightBorder"><a href="/c/ai" target="_blank" rel="noopener noreferrer" class="text-lg border-lightBorder hover:bg-lightAccent hover:text-lightAccentText hover:border-lightAccentText bg-lightAlt text-lightText flex items-center px-2 py-1 border rounded"><span class="mr-2"><i class="hn hn-machine-learning !leading-[inherit]"></i></span><span>ai-and-ml</span></a></div><a href="/tagged/aws" target="_blank" rel="noopener noreferrer" class="text-sm xs:text-base sm:text-lg flex items-center px-2 py-1 border hover:bg-lightAlt border-lightBorder rounded">#<!-- -->aws</a><a href="/tagged/aws-lambda" target="_blank" rel="noopener noreferrer" class="text-sm xs:text-base sm:text-lg flex items-center px-2 py-1 border hover:bg-lightAlt border-lightBorder rounded">#<!-- -->aws-lambda</a><a href="/tagged/serverless" target="_blank" rel="noopener noreferrer" class="text-sm xs:text-base sm:text-lg flex items-center px-2 py-1 border hover:bg-lightAlt border-lightBorder rounded">#<!-- -->serverless</a><a href="/tagged/chaos-engineering" target="_blank" rel="noopener noreferrer" class="text-sm xs:text-base sm:text-lg flex items-center px-2 py-1 border hover:bg-lightAlt border-lightBorder rounded">#<!-- -->chaos-engineering</a><a href="/tagged/cloud-computing" target="_blank" rel="noopener noreferrer" class="text-sm xs:text-base sm:text-lg flex items-center px-2 py-1 border hover:bg-lightAlt border-lightBorder rounded">#<!-- -->cloud-computing</a></div></section></div></div></div></div></div><div class="min-h-[200px]"></div></main><div class="flex flex-col gap-4 hidden"><button class="mr-auto"><i class="hn-sun hn text-2xl"></i></button><h2 class="text-sm font-semibold text-darkText">Light-Mode</h2><div class="cursor-pointer p-3 rounded-lg hover:scale-105 transition-transform "><h3 class="text-sm uppercase mb-2 ">Classic</h3><div class="flex space-x-1"><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#0F0"></span><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#F5EC43"></span><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#212428"></span></div></div><div class="cursor-pointer p-3 rounded-lg hover:scale-105 transition-transform "><h3 class="text-sm uppercase mb-2 ">Newspaper</h3><div class="flex space-x-1"><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#FFFFFF"></span><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#F5F5F5"></span><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#454545"></span></div></div><div class="cursor-pointer p-3 rounded-lg hover:scale-105 transition-transform "><h3 class="text-sm uppercase mb-2 ">Proof of Usefulness</h3><div class="flex space-x-1"><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#FFFFFF"></span><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#26AB5C"></span><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#D2FBE2"></span></div></div><h2 class="text-sm font-semibold text-darkText">Dark-Mode</h2><div class="cursor-pointer p-3 rounded-lg hover:scale-105 transition-transform "><h3 class="text-sm uppercase mb-2 ">Neon Noir</h3><div class="flex space-x-1"><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#1E1E1E"></span><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#0F0"></span><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#F5EC43"></span></div></div><div class="cursor-pointer p-3 rounded-lg hover:scale-105 transition-transform "><h3 class="text-sm uppercase mb-2 ">Minty</h3><div class="flex space-x-1"><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#061F19"></span><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#2AAA74"></span><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#63FF86"></span></div></div><div class="cursor-pointer p-3 rounded-lg hover:scale-105 transition-transform "><h3 class="text-sm uppercase mb-2 ">Startups of the Year</h3><div class="flex space-x-1"><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#08085E"></span><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#A2EF44"></span><span class="w-6 h-6 rounded border border-darkBorder" style="background-color:#1B1B95"></span></div></div></div></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"data":{"pageLang":"en","datePublished":"2017-09-01","slug":"how-can-we-apply-the-principles-of-chaos-engineering-to-aws-lambda-80f87e3237e2","articleBody":"All the way back in 2011, Simon Wardley had identified Chaos Engines as a practice that will be employed by the next generation of tech companies, along with continuous deployment, being data-driven, and organised around small, autonomous teams (think microservices \u0026 inverse-conway’s law). This is the first of a multipart series that explores ideas on how we could apply the principles of chaos engineering to serverless architectures built around Lambda functions. part 1: how can we apply principles of chaos engineering to Lambda? part 2: applying latency injection for APIs part 3: dealing with latency spikes and timeouts (coming 21/11/2017) part 4: applying fault injection for Lambda functions (coming ?) There’s no question about it, Netflix has popularised the of chaos engineering. By open sourcing some of their tools — notably the — they have also helped others build confidence in their system’s capability to withstand turbulent conditions in production. principles Simian Army There seems to be a renewed interest in chaos engineering recently. As Russ Miles noted in a recent , perhaps many companies have finally come to understand that chaos engineering is not about “hurting production”, but to build better understanding of, and confidence in a system’s resilience through experiments. post controlled This trend has been helped by the valuable (and freely available) information that Netflix has published, such as the e-book, and . Chaos Engineering principlesofchaos.org Tools such as by (the folks behind the load test tool) look to replicate Netflix’s , but execute from inside a Lambda function instead of an EC2 instance — hence bringing you the cost saving and convenience Lambda offers. chaos-lambda Shoreditch Ops Artillery Chaos Monkey I want to ask a different question however: how can one apply the principles of chaos engineering and some of the current practices to a serverless architecture comprised of Lambda functions? When your system runs on EC2 instances, then naturally, you build resilience by designing for the most likely failure mode — server crashes (due to both hardware and software issues). Hence, a controlled experiment to validate the resilience of your system would artificially recreate the failure scenario by terminating EC2 instances, and then AZs, and then entire regions. AWS Lambda, however, is a higher-level abstraction and has different failure modes to its EC2 counterparts. Hypothesis that focus on no longer apply as the platform handles these failure modes for you out of the box. “what if we lose these EC2 instances” We need to ask different questions in order to understand the weaknesses within our serverless architectures. More inherent chaos, not less “ . Systemic weaknesses could take the form of: improper fallback settings when a service is unavailable; retry storms from improperly tuned timeouts; outages when a downstream dependency receives too much traffic; cascading failures when a single point of failure crashes; etc. We must address the most significant weaknesses proactively, before they affect our customers in production. , take advantage of increasing flexibility and velocity, and ” We need to identify weaknesses before they manifest in system-wide, aberrant behaviors We need a way to manage the chaos inherent in these systems have confidence in our production deployments despite the complexity that they represent. — Principles of Chaos Engineering Having , I have some understanding of the dangers awaiting you in this new world. built and operated a non-trivial serverless architecture If anything, there are a lot more inherent chaos and complexity in these systems built around Lambda functions. modularity (unit of deployment) shifts from “services” to “functions”, and there are a lot more of them it’s harder to harden around the boundaries, because you need to harden around each function as opposed to a service which encapsulates a set of related functionalities there are more intermediary services (eg. Kinesis, SNS, API Gateway just to name a few), each with their own failure modes there are more configurations overall (timeout, IAM permissions, etc.), and therefore more opportunities for misconfiguration Also, since we have traded off more control of our infrastructure* it means we now face more unknown failure modes** and often there’s little we can do when an outage does occur***. * For better scalability, availability, cost efficiency and more convenience, which I for one, think it’s a fair trade in most cases . ** Everything the platform does for you — scheduling containers, scaling, polling Kinesis, retry failed invocations, etc. — have their own failure modes. These are often not obvious to us since they’re implementation details that are typically undocumented and are prone to change without notice. *** For example, if an outage happens and prevents Lambda functions from processing Kinesis events, then we have no meaningful alternative than to wait for AWS to fix the problem. Since the current position on the shards is abstracted away and unavailable to us, we can’t even replace the Lambda functions with KCL processors that run on EC2. Applying chaos to AWS Lambda A good exercise regime would continuously push you to your limits but never actually put you over the limit and cause injury. If there’s an exercise that is clearly beyond your current abilities then surely you would not attempt it as the only possible outcome is getting yourself hurt! . In order to “know” what the experiments tell us about the resilience of our system we also need to decide what metrics to monitor — ideally using client-side metrics, since the most important metric is the quality of service our users experience. The same common sense should be applied when designing controlled experiments for your serverless architecture There are plenty of failure modes that we know about and can design for, and we can run simple experiments to validate our design. For example, since a serverless architecture is (almost always) also a microservice architecture, many of its inherent failure modes still apply: improperly tuned timeouts, especially for intermediate services, which can cause services at the edge to also timeout Intermediate services should have more strict timeout settings compared to services at the edge. missing error handling, which allows exceptions from downstream services to escape missing fallbacks for when a downstream service is unavailable or experiences an outage Over the next couple of posts, we will explore how we can apply the practices of latency and fault injection to Lambda functions in order to simulate these failure modes and validate our design. Further readings: ChAP: Chaos Automation Platform Deploying the Netflix API The Netflix Simian Army Testing in production: Yes you can (and should) Design for latency issues 1000 actors, one chaos monkey… and everything’s OK [Ebook] Chaos Engineering [Book] Release it! [Book] Drift into Failure principlesofchaos.org Hi, my name is . I’m an and the author of . I have run production workload at scale in AWS for nearly 10 years and I have been an architect or principal engineer with a variety of industries ranging from banking, e-commerce, sports streaming to mobile gaming. I currently work as an independent consultant focused on AWS and serverless. Yan Cui AWS Serverless Hero Production-Ready Serverless You can contact me via , and . Email Twitter LinkedIn Check out my new course, . Complete Guide to AWS Step Functions In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. Including basic concepts, HTTP and event triggers, activities, design patterns and best practices. Get your copy . here Come learn about operational for AWS Lambda: CI/CD, testing \u0026 debugging functions locally, logging, monitoring, distributed tracing, canary deployments, config management, authentication \u0026 authorization, VPC, security, error handling, and more. BEST PRACTICES You can also get off the face price with the code . 40% ytcui Get your copy . here","arweave":"A8KFGykZJgP7dRNflFRDus6-ZEES-Ro8C1j_3YbRSac","createdAt":"2017-09-01T01:57:05.161Z","excerpt":"This is the first of a multipart series that explores ideas on how we could apply the principles of chaos engineering to serverless architectures built around Lambda functions.","featureImgColor":"rgb(4 251 4 / 70%)","firstSeenAt":false,"id":"how-can-we-apply-the-principles-of-chaos-engineering-to-aws-lambda-80f87e3237e2","imageSizes":{"https://hackernoon.com/hn-images/1*DSKdK5TqOy8FZ4F9xUmK8g.png":{"width":1500,"height":782},"https://hackernoon.com/hn-images/0*ZYcHhOOzUf5VB-Ri.png":{"width":1024,"height":368},"https://hackernoon.com/hn-images/1*JlnapsrFEDH771EYRel0VA.png":{"width":1600,"height":1085},"https://hackernoon.com/hn-images/1*OFKVj5FywcNs-CpsXFPi0w.png":{"width":1544,"height":978},"https://hackernoon.com/hn-images/0*b_1R345KzKSaI8sg.png":{"width":309,"height":313},"https://hackernoon.com/hn-images/1*56lZO-ri61SY-ndXv5PePA.png":{"width":940,"height":830},"https://hackernoon.com/hn-images/1*1ZK4_SQEk3X1g4PowOOCIg.png":{"width":1332,"height":674}},"mainImage":"https://hackernoon.imgix.net/fallback-feat.png","mainImageHeight":500,"mainImageWidth":1000,"mentions":[{"image":"http://logo.bigpicture.io/logo/amazon.com","name":"Amazon","collection":"companies","id":"amazon","manual":false},{"image":"http://logo.bigpicture.io/logo/netflix.com","name":"Netflix","id":"netflix","collection":"companies","manual":false},{"image":"http://logo.bigpicture.io/logo/twitter.com","name":"Twitter","collection":"companies","id":"twitter","manual":false},{"image":null,"name":"Velocity","id":"velocity","collection":"companies","manual":false},{"image":"https://hackernoon.com/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg","displayName":"Yan Cui","handle":"theburningmonk","collection":"profiles","manual":false,"url":"https://hackernoon.com/u/theburningmonk"}],"owner":"YjziSuGeDecNvCZR0j4bivrb6V13","parentCategory":"cloud","parsed":"\u003cp\u003e\u003cimg src=\"https://hackernoon.com/hn-images/1*56lZO-ri61SY-ndXv5PePA.png\" alt=\"\"\u003e\u003c/p\u003e\n\u003cp\u003eAll the way back in 2011, Simon Wardley had identified Chaos Engines as a practice that will be employed by the next generation of tech companies, along with continuous deployment, being data-driven, and organised around small, autonomous teams (think microservices \u0026amp; inverse-conway’s law).\u003c/p\u003e\n\u003cp\u003eThis is the first of a multipart series that explores ideas on how we could apply the principles of chaos engineering to serverless architectures built around Lambda functions.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003epart 1: how can we apply principles of chaos engineering to Lambda?\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003epart 2: \u003ca href=\"https://hackernoon.com/chaos-engineering-and-aws-lambda-latency-injection-ddeb4ff8d983\"\u003eapplying latency injection for APIs\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003epart 3: dealing with latency spikes and timeouts (coming 21/11/2017)\u003c/li\u003e\n\u003cli\u003epart 4: applying fault injection for Lambda functions (coming ?)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThere’s no question about it, Netflix has popularised the \u003cem\u003eprinciples\u003c/em\u003e of chaos engineering. By open sourcing some of their tools — notably the \u003ca href=\"https://github.com/Netflix/SimianArmy\"\u003eSimian Army\u003c/a\u003e — they have also helped others build confidence in their system’s capability to withstand turbulent conditions in production.\u003c/p\u003e\n\u003cp\u003eThere seems to be a renewed interest in chaos engineering recently. As Russ Miles noted in a recent \u003ca href=\"https://medium.com/russmiles/chaos-engineering-why-the-label-matters-35ddbb974fa5\"\u003epost\u003c/a\u003e, perhaps many companies have finally come to understand that chaos engineering is not about “hurting production”, but to build better understanding of, and confidence in a system’s resilience through \u003cstrong\u003e\u003cem\u003econtrolled\u003c/em\u003e\u003c/strong\u003e experiments.\u003c/p\u003e\n\u003cp\u003eThis trend has been helped by the valuable (and freely available) information that Netflix has published, such as the \u003ca href=\"http://oreil.ly/2tZU1Sn\"\u003eChaos Engineering\u003c/a\u003e e-book, and \u003ca href=\"http://principlesofchaos.org/\"\u003eprinciplesofchaos.org\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"https://hackernoon.com/hn-images/1*JlnapsrFEDH771EYRel0VA.png\" alt=\"\"\u003e\u003c/p\u003e\n\u003cp\u003eTools such as \u003ca href=\"https://github.com/shoreditch-ops/chaos-lambda\"\u003echaos-lambda\u003c/a\u003e by \u003cem\u003eShoreditch Ops\u003c/em\u003e (the folks behind the \u003ca href=\"https://artillery.io/\"\u003eArtillery\u003c/a\u003e load test tool) look to replicate Netflix’s \u003ca href=\"https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey\"\u003eChaos Monkey\u003c/a\u003e, but execute from inside a Lambda function instead of an EC2 instance — hence bringing you the cost saving and convenience Lambda offers.\u003c/p\u003e\n\u003cp\u003eI want to ask a different question however: \u003cstrong\u003ehow can one apply the principles of chaos engineering and some of the current practices to a serverless architecture comprised of Lambda functions?\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWhen your system runs on EC2 instances, then naturally, you build resilience by designing for the most likely failure mode — server crashes (due to both hardware and software issues). Hence, a controlled experiment to validate the resilience of your system would artificially recreate the failure scenario by terminating EC2 instances, and then AZs, and then entire regions.\u003c/p\u003e\n\u003cp\u003eAWS Lambda, however, is a higher-level abstraction and has different failure modes to its EC2 counterparts. Hypothesis that focus on \u003cem\u003e“what if we lose these EC2 instances”\u003c/em\u003e no longer apply as the platform handles these failure modes for you out of the box.\u003c/p\u003e\n\u003cp\u003eWe need to ask different questions in order to understand the weaknesses within our serverless architectures.\u003c/p\u003e\n\u003ch3\u003eMore inherent chaos, not less\u003c/h3\u003e\n\u003cblockquote\u003e\n\u003cp\u003e“\u003cstrong\u003eWe need to identify weaknesses before they manifest in system-wide, aberrant behaviors\u003c/strong\u003e. Systemic weaknesses could take the form of: improper fallback settings when a service is unavailable; retry storms from improperly tuned timeouts; outages when a downstream dependency receives too much traffic; cascading failures when a single point of failure crashes; etc. We must address the most significant weaknesses proactively, before they affect our customers in production. \u003cstrong\u003eWe need a way to manage the chaos inherent in these systems\u003c/strong\u003e, take advantage of increasing flexibility and velocity, and \u003cstrong\u003ehave confidence in our production deployments despite the complexity that they represent.\u003c/strong\u003e”\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cblockquote\u003e\n\u003cp\u003e— \u003ca href=\"http://principlesofchaos.org/\"\u003ePrinciples of Chaos Engineering\u003c/a\u003e\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eHaving \u003ca href=\"https://hackernoon.com/yubls-road-to-serverless-part-1-overview-ca348370acde\"\u003ebuilt and operated a non-trivial serverless architecture\u003c/a\u003e, I have some understanding of the dangers awaiting you in this new world.\u003c/p\u003e\n\u003cp\u003eIf anything, there are a lot more inherent chaos and complexity in these systems built around Lambda functions.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003emodularity (unit of deployment) shifts from “services” to “functions”, and there are a lot more of them\u003c/li\u003e\n\u003cli\u003eit’s harder to harden around the boundaries, because you need to harden around each function as opposed to a service which encapsulates a set of related functionalities\u003c/li\u003e\n\u003cli\u003ethere are more intermediary services (eg. Kinesis, SNS, API Gateway just to name a few), each with their own failure modes\u003c/li\u003e\n\u003cli\u003ethere are more configurations overall (timeout, IAM permissions, etc.), and therefore more opportunities for misconfiguration\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAlso, since we have traded off more control of our infrastructure* it means we now face more unknown failure modes** and often there’s little we can do when an outage does occur***.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e* For better scalability, availability, cost efficiency and more convenience, which I for one, think it’s a fair trade in\u003c/em\u003e \u003cstrong\u003e\u003cem\u003emost cases\u003c/em\u003e\u003c/strong\u003e\u003cem\u003e.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e** Everything the platform does for you — scheduling containers, scaling, polling Kinesis, retry failed invocations, etc. — have their own failure modes. These are often not obvious to us since they’re implementation details that are typically undocumented and are prone to change without notice.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e*** For example, if an outage happens and prevents Lambda functions from processing Kinesis events, then we have no meaningful alternative than to wait for AWS to fix the problem. Since the current position on the shards is abstracted away and unavailable to us, we can’t even replace the Lambda functions with KCL processors that run on EC2.\u003c/em\u003e\u003c/p\u003e\n\u003ch3\u003eApplying chaos to AWS Lambda\u003c/h3\u003e\n\u003cp\u003eA good exercise regime would continuously push you to your limits but never actually put you over the limit and cause injury. If there’s an exercise that is clearly beyond your current abilities then surely you would not attempt it as the only possible outcome is getting yourself hurt!\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eThe same common sense should be applied when designing controlled experiments for your serverless architecture\u003c/strong\u003e. In order to “know” what the experiments tell us about the resilience of our system we also need to decide what metrics to monitor — ideally using client-side metrics, since the most important metric is the quality of service our users experience.\u003c/p\u003e\n\u003cp\u003eThere are plenty of failure modes that we know about and can design for, and we can run simple experiments to validate our design. For example, since a serverless architecture is (almost always) also a microservice architecture, many of its inherent failure modes still apply:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eimproperly tuned timeouts, especially for intermediate services, which can cause services at the edge to also timeout\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cimg src=\"https://hackernoon.com/hn-images/1*OFKVj5FywcNs-CpsXFPi0w.png\" alt=\"\"\u003e\u003c/p\u003e\n\u003cp\u003eIntermediate services should have more strict timeout settings compared to services at the edge.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003emissing error handling, which allows exceptions from downstream services to escape\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cimg src=\"https://hackernoon.com/hn-images/1*DSKdK5TqOy8FZ4F9xUmK8g.png\" alt=\"\"\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003emissing fallbacks for when a downstream service is unavailable or experiences an outage\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cimg src=\"https://hackernoon.com/hn-images/1*1ZK4_SQEk3X1g4PowOOCIg.png\" alt=\"\"\u003e\u003c/p\u003e\n\u003cp\u003eOver the next couple of posts, we will explore how we can apply the practices of latency and fault injection to Lambda functions in order to simulate these failure modes and validate our design.\u003c/p\u003e\n\u003ch3\u003eFurther readings:\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://medium.com/netflix-techblog/chap-chaos-automation-platform-53e6d528371f\"\u003eChAP: Chaos Automation Platform\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://medium.com/netflix-techblog/deploying-the-netflix-api-79b6176cc3f0\"\u003eDeploying the Netflix API\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab116\"\u003eThe Netflix Simian Army\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://opensource.com/article/17/8/testing-production\"\u003eTesting in production: Yes you can (and should)\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://theburningmonk.com/2015/04/design-for-latency-issues/\"\u003eDesign for latency issues\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://erlangcentral.org/blog/presentations/1000-actors-one-chaos-monkey-and-everything-ok/\"\u003e1000 actors, one chaos monkey… and everything’s OK\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e[Ebook] \u003ca href=\"http://www.oreilly.com/webops-perf/free/chaos-engineering.csp\"\u003eChaos Engineering\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e[Book] \u003ca href=\"http://amzn.to/1pedVvt\"\u003eRelease it!\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e[Book] \u003ca href=\"http://amzn.to/1CB0I6D\"\u003eDrift into Failure\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://principlesofchaos.org/\"\u003eprinciplesofchaos.org\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cimg src=\"https://hackernoon.com/hn-images/0*b_1R345KzKSaI8sg.png\" alt=\"\"\u003e\u003c/p\u003e\n\u003cp\u003eHi, my name is \u003cstrong\u003eYan Cui\u003c/strong\u003e. I’m an \u003ca href=\"https://aws.amazon.com/developer/community/heroes/yan-cui/\"\u003e\u003cstrong\u003eAWS Serverless Hero\u003c/strong\u003e\u003c/a\u003e and the author of \u003ca href=\"https://bit.ly/production-ready-serverless\"\u003e\u003cstrong\u003eProduction-Ready Serverless\u003c/strong\u003e\u003c/a\u003e. I have run production workload at scale in AWS for nearly 10 years and I have been an architect or principal engineer with a variety of industries ranging from banking, e-commerce, sports streaming to mobile gaming. I currently work as an independent consultant focused on AWS and serverless.\u003c/p\u003e\n\u003cp\u003eYou can contact me via \u003ca href=\"mailto:theburningmonk.com\"\u003eEmail\u003c/a\u003e, \u003ca href=\"https://twitter.com/theburningmonk\"\u003eTwitter\u003c/a\u003e and \u003ca href=\"https://www.linkedin.com/in/theburningmonk/\"\u003eLinkedIn\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eCheck out my new course, \u003ca href=\"https://theburningmonk.thinkific.com/courses/complete-guide-to-aws-step-functions\"\u003e\u003cstrong\u003eComplete Guide to AWS Step Functions\u003c/strong\u003e\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eIn this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. Including basic concepts, HTTP and event triggers, activities, design patterns and best practices.\u003c/p\u003e\n\u003cp\u003eGet your copy \u003ca href=\"https://theburningmonk.thinkific.com/courses/complete-guide-to-aws-step-functions\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"https://hackernoon.com/hn-images/0*ZYcHhOOzUf5VB-Ri.png\" alt=\"\"\u003e\u003c/p\u003e\n\u003cp\u003eCome learn about operational \u003cstrong\u003eBEST PRACTICES\u003c/strong\u003e for AWS Lambda: CI/CD, testing \u0026amp; debugging functions locally, logging, monitoring, distributed tracing, canary deployments, config management, authentication \u0026amp; authorization, VPC, security, error handling, and more.\u003c/p\u003e\n\u003cp\u003eYou can also get \u003cstrong\u003e40%\u003c/strong\u003e off the face price with the code \u003cstrong\u003eytcui\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eGet your copy \u003ca href=\"https://bit.ly/production-ready-serverless\"\u003ehere\u003c/a\u003e.\u003c/p\u003e","profile":{"about_page_settings":{"blocked":false,"createdAt":"2021-12-16T04:44:48.785Z","owner":"YjziSuGeDecNvCZR0j4bivrb6V13","published":true,"style":{"headline_pos":"center","layout":0,"skin":0},"updatedAt":"2021-12-16T04:44:48.785Z"},"adLink":"https://theburningmonk.com/hire-me/","adText":"Ask me for help about serverless","avatar":"https://hackernoon.com/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg","bio":"AWS Serverless Hero. Independent Consultant. Developer Advocate at Lumigo.","callToActions":[{"active":true,"icon":"far fa-smile","id":"98d3912fa2ee6-brand","name":"Ask me for help about serverless","url":"https://theburningmonk.com/hire-me/"},{"active":true,"icon":"fa fa-book","id":"e11673721d0c1","name":"Read My Stories","url":"https://hackernoon.com/u/theburningmonk"},{"active":true,"icon":"far fa-smile","id":"2c5ef23d3285d-brand","name":"Ask me for help about serverless","url":"https://theburningmonk.com/hire-me/"}],"displayName":"Yan Cui","handle":"theburningmonk","currentJob":{},"isBrand":false,"jobHistory":[],"isTrusted":false,"allowSubscribers":true},"publishedAt":1504231025.161,"tags":["aws","aws-lambda","serverless","chaos-engineering","cloud-computing"],"title":"How can we apply the principles of chaos engineering to AWS Lambda?","backlinks":{"fetched":"2025-05-05T01:05:16.726Z","urls":["https://habr.com/ru/companies/flant/articles/477994/","https://toppodcast.com/podcast_feeds/serverless-chats/","https://www.pvsm.ru/testirovanie-it-sistem/340545","https://savepearlharbor.com/?p=296066","https://hnwaybackmachine.aryan.app/2017/10/27/","https://educosta.dev/blog/weekly-cloud-native-digest-9/","http://forpes.ru/post/89267","https://share.transistor.fm/s/6670b4dc","https://weekly-geekly-es.imtqy.com/articles/zh-CN480802/index.html"]},"draftId":"how-can-we-apply-the-principles-of-chaos-engineering-to-aws-lambda-80f87e3237e2","tldr":"This is the first of a multipart series that explores ideas on how we could apply the principles of chaos engineering to serverless architectures built around Lambda functions.","super_category":"ai-and-ml","annotations":[],"coAuthorProfiles":[],"commentsCount":1,"fromMongo":true,"relatedStories":[{"title":"AWS AppSync Queries and Mutations with Lambda","mainImage":"https://cdn-images-1.medium.com/max/1024/1*5qIzAsIrEPvJlz69oABiqg.png","slug":"aws-appsync-queries-and-mutations-with-lambda-2aee303c66b0","tags":["aws-lambda","aws","aws-appsync","aws-amplify","graphql"],"excerpt":"I’m using AWS AppSync to build \u003ca href=\"https://floom.app\"\u003eFloom\u003c/a\u003e. 90% of the time it’s great, but there are still some pretty glaring feature gaps. One of the biggest is not being able to use a private key to access GraphQL operations through AWSAppSyncClient (for example, when doing backend operations with Lambda). It seems like there should be some kind of out-of-the-box admin access that allows you to mutate and query the database without needing to authenticate using AWS\u0026nbsp;Cognito.","publishedAt":1557086052000,"profile":{"handle":"rob_moore","avatar":"https://hackernoon.com/fallback-profile.png","displayName":"Rob Moore"},"recommended":true},{"title":"AWS ECS vs AWS Lambda Compared","mainImage":"https://cdn.hackernoon.com/images/JT4BgXlfxveziEP9szyBKsEXoFf2-or037ks.jpeg","slug":"aws-ecs-vs-aws-lambda-compared","tags":["aws","aws-lambda","ecs","elasticsearch","microservices","software-development","software-engineering","cloud-computing"],"excerpt":"Comparing cloud services? Read our Lambda vs ECS guide. Consider programming language, pricing, and the benefits.","publishedAt":1639520245919,"profile":{"handle":"taavi-rehemagi","avatar":"https://hackernoon.com/images/avatars/JT4BgXlfxveziEP9szyBKsEXoFf2.jpg","displayName":"Taavi Rehemägi"},"recommended":true},{"title":"AWS Lambda — build yourself a URL shortener in 2 hours","mainImage":"https://hackernoon.com/hn-images/0*eBQwpVbKVcRdYdZF.","slug":"aws-lambda-build-yourself-a-url-shortener-in-2-hours-94f9627ed4a6","tags":["aws","aws-lambda","serverless","serverless-architecture","cloud"],"excerpt":"An interesting requirement came up at work this week where we discussed potentially having to run our own URL Shortener because the \u003cem\u003eUniversal Links\u003c/em\u003e mechanism (in iOS 9 and above) \u003ca href=\"http://blog.hokolinks.com/how-to-implement-apple-universal-links-on-ios-9/\" target=\"_blank\"\u003erequires a JSON manifest\u003c/a\u003e at","publishedAt":1491171492113,"profile":{"handle":"theburningmonk","avatar":"https://hackernoon.com/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg","displayName":"Yan Cui","isBrand":false},"recommended":true},{"title":"AWS Lambda — how best to manage shared code and shared infrastructure","mainImage":"https://hackernoon.com/fallback-feat.png","slug":"aws-lambda-how-best-to-manage-shared-code-and-shared-infrastructure-827bed395eb7","tags":["aws","aws-lambda","serverless","cloud","cloud-computing"],"excerpt":"In the \u003ca href=\"https://hackernoon.com/aws-lambda-should-you-have-few-monolithic-functions-or-many-single-purposed-functions-8c3872d4338f\" target=\"_blank\"\u003elast post\u003c/a\u003e I discussed the pros \u0026amp; cons of following the \u003cem\u003eSingle Responsibility Principle\u003c/em\u003e (SRP) when moving to the serverless paradigm.","publishedAt":1517125714546,"profile":{"handle":"theburningmonk","avatar":"https://hackernoon.com/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg","displayName":"Yan Cui","isBrand":false},"recommended":true},{"title":"AWS Lambda — should you have few monolithic functions or many single-purposed functions?","mainImage":"https://hackernoon.com/fallback-feat.png","slug":"aws-lambda-should-you-have-few-monolithic-functions-or-many-single-purposed-functions-8c3872d4338f","tags":["serverless","aws","aws-lambda","cloud","cloud-computing","hackernoon-es"],"excerpt":"A funny moment (at 38:50) happened during Tim Bray’s session (\u003ca href=\"https://www.youtube.com/watch?v=sMaqd5J69Ns\" target=\"_blank\"\u003eSRV306\u003c/a\u003e) at re:invent 2017, when he asked the audience if we should have many simple, single-purposed functions, or fewer monolithic functions, and the room was pretty much split in half.","publishedAt":1515428909479,"profile":{"handle":"theburningmonk","avatar":"https://hackernoon.com/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg","displayName":"Yan Cui","isBrand":false},"recommended":true},{"title":"AWS Lambda — use the invocation context to better handle slow HTTP responses","mainImage":"https://hackernoon.com/fallback-feat.png","slug":"aws-lambda-use-the-invocation-context-to-better-handle-slow-http-responses-16fbfecfa4e9","tags":["aws-lambda","serverless","cloud-computing","aws","resilience"],"excerpt":"With API Gateway and Lambda, you’re forced to use relatively short timeouts on the server-side:","publishedAt":1515034923459,"profile":{"handle":"theburningmonk","avatar":"https://hackernoon.com/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg","displayName":"Yan Cui","isBrand":false},"recommended":true},{"title":"AWS Lambda Alias: An Essential Guide ","mainImage":"https://cdn.hackernoon.com/images/TlU0qyilehYV4XrjqgkglSwSK413-h9g3ni8.png","slug":"aws-lambda-alias-an-essential-guide","tags":["aws","aws-lambda","serverless","serverless-architecture","aws-lambda-best-practices","tutorial","tutorial-for-beginners","guide"],"excerpt":"Why you should be using AWS Lambda alias for your versioning strategy\n\n","publishedAt":1652345875697,"profile":{"handle":"jerrychang","avatar":"https://cdn.hackernoon.com/images/TlU0qyilehYV4XrjqgkglSwSK413-8m93rxs.jpeg","displayName":"Jerry"},"recommended":true},{"title":"AWS Lambda Compared with Other Alternatives to Deploy your Function","mainImage":"https://cdn.hackernoon.com/images/3nkg3vxl.jpg","slug":"aws-lambda-compared-with-other-alternatives-to-deploy-your-function-0e6k3aex","tags":["faas","serverless","java","grpc","amazon","aws","aws-lamdba","hackernoon-top-story"],"excerpt":"Lambda vs GRPC vs Choreography vs HTTP / REST","publishedAt":1580148322953,"profile":{"handle":"cleuton-sampaio","avatar":"https://cdn.hackernoon.com/images/avatars/BhJEeRh64dRZDVWSwvuqi0gN6xP2.jpg","displayName":"Cleuton Sampaio"},"recommended":true},{"title":"AWS Lambda For Dummies [Part 2]","mainImage":"https://firebasestorage.googleapis.com/v0/b/hackernoon-app.appspot.com/o/images%2FJT4BgXlfxveziEP9szyBKsEXoFf2-78f3t14.jpeg?alt=media\u0026token=9cedee72-e0f8-4c02-b690-65ec8a01b286","slug":"aws-lambda-for-dummies-part-2-xe1l341j","tags":["serverless","aws-lambda","aws","ultimate-guides","amazon","use-cases","api-gateway","backend"],"excerpt":"In Part 1 of our Complete AWS Lambda Handbook for Beginners, we gave a refresher on the fundamentals of AWS Lambda and what is AWS Lambda. In this post, we’ll look at AWS Lambda pricing, some interesting Lambda facts and examples of great AWS Lambda use cases in your serverless application. ","publishedAt":1622282407725,"profile":{"handle":"taavi-rehemagi","avatar":"https://hackernoon.com/images/avatars/JT4BgXlfxveziEP9szyBKsEXoFf2.jpg","displayName":"Taavi Rehemägi"},"recommended":true},{"title":"AWS Lambda Functions in Java","mainImage":"https://hackernoon.com/fallback-feat.png","slug":"aws-lambda-functions-in-java-24a2e41fa736","tags":["aws","lambda","java","aws-lambda-functions","aws-lambda-in-java"],"excerpt":"In the past supporting a web application required purchasing one or more computer systems to act as web servers. Additional hardware might include a load balancer to distribute web application load between the web servers and a database server. These computer systems would need to be housed in an air conditioned facility. An alarm system and other security measures would be needed to provide physical security. Enough Internet bandwidth would need to be purchased to support anticipated peak usage.","publishedAt":1519419822471,"profile":{"handle":"nderground_net","avatar":"https://cdn.hackernoon.com/avatars/robot-a2.png","displayName":"nderground"},"recommended":true},{"title":"AWS Lambda Go vs. Node.js performance benchmark: updated 🔥","mainImage":"https://hackernoon.com/fallback-feat.png","slug":"aws-lambda-go-vs-node-js-performance-benchmark-1c8898341982","tags":["aws-lambda","aws","nodejs","golang","tech"],"excerpt":"Just this week AWS announced the release of Go for their Lambda service. This is pretty exciting as Go straddles a great niche between Java and Node.JS with regard to type safety, programming model and performance.","publishedAt":1516087688102,"profile":{"handle":"tim_nolet","avatar":"https://cdn.hackernoon.com/avatars/robot-a3.png","displayName":"Tim Nolet"},"recommended":true},{"title":"AWS Lambda with AWS Chalice","mainImage":"https://hackernoon.com/fallback-feat.png","slug":"aws-lambda-651cb3e17b1c","tags":["aws-lambda","aws","serverless","aws-chalice"],"excerpt":"Some notes while working with AWS Lambda using Python3","publishedAt":1533403970791,"profile":{"handle":"trannotes","avatar":"https://cdn.hackernoon.com/avatars/robot-b6.png","displayName":"trannotes"},"recommended":true},{"title":"AWS Lambda with Kinesis Trigger: 6 Pitfalls and How to Fix Them","mainImage":"https://cdn.hackernoon.com/images/JT4BgXlfxveziEP9szyBKsEXoFf2-3q35340t.jpeg","slug":"aws-lambda-with-kinesis-trigger-6-pitfalls-and-how-to-fix-them-hk3o34kp","tags":["amazon-kinesis","kinesis","aws-lambda","aws","aws-services","serverless","sre","devops"],"excerpt":"In this article, we discuss the 6 most common pitfalls of AWS Lambda with Kinesis trigger that are usually spotted only in production. ","publishedAt":1622192712404,"profile":{"handle":"taavi-rehemagi","avatar":"https://hackernoon.com/images/avatars/JT4BgXlfxveziEP9szyBKsEXoFf2.jpg","displayName":"Taavi Rehemägi"},"recommended":true}],"previousRead":{"slug":"capture-and-forward-correlation-ids-through-different-lambda-event-sources-220c227c65f5","mainImage":"https://hackernoon.com/hn-images/1*hlat_2akk4kxA_US-1bPSg.png","owner":"YjziSuGeDecNvCZR0j4bivrb6V13","title":"Capture and forward correlation IDs through different Lambda event sources"},"nextRead":{"slug":"mind-the-75gb-limit-on-aws-lambda-deployment-packages-163b93c8eb72","mainImage":"https://hackernoon.com/hn-images/1*h6dJ3WFL16CgEUCcg_22SQ.png","owner":"YjziSuGeDecNvCZR0j4bivrb6V13","title":"Mind the 75GB limit on AWS Lambda deployment packages"},"staticData":{"frLangTooltip":"Lisez cette histoire en Français!","about":"About","enLangTooltip":"Read this story in the original language, English!","loggedOutBookmark":"Create an account to store your bookmarks","learnMore":"Learn More","stats":"Stats","editStory":"Edit Story","audioPresented":"Audio Presented by","by":"by","audioTranslationText":null,"newStory":"New Story","loggedInBookmark":"Bookmark story","esLangTooltip":"Lee esta historia en Español!","relatedStories":"RELATED STORIES","addComment":"Add Comment","ptLangTooltip":"Leia esta história em português!","hiLangTooltip":"इस कहानी को हिंदी में पढ़ें!","comments":"Comments","removeBookmark":"Remove bookmark","commentReply":"Reply","minutes":"min","reads":"reads","trLangTooltip":"Bu hikayeyi Türkçe okuyun!","tags":"TOPICS","jaLangTooltip":"この物語を日本語で読んでください!","bnLangTooltip":"এই গল্পটি বাংলায় পড়ুন!","storyMentions":"MENTIONED IN THIS STORY","ruLangTooltip":"Прочтите эту историю на русском языке!","deLangTooltip":"Lesen Sie diese Geschichte auf Deutsch!","featuredIn":"THIS ARTICLE WAS FEATURED IN","tldrTitle":"Too Long; Didn't Read","koLangTooltip":"이 이야기를 한국어로 읽어보세요!","zhLangTooltip":"用繁體中文閱讀這個故事!","viLangTooltip":"Đọc bài viết này bằng tiếng Việt!"},"searchTopics":["aws lambda"],"stats":{"pageviews":4250},"socialPreviewImage":"https://hackernoon.imgix.net/fallback-feat.png","gptZeroMsg":"This story is mostly human.","audioData":[{"avatar":"https://cdn.hackernoon.com/avatars/robot-b5.png","url":"https://storage.googleapis.com/hackernoon/audios/how-can-we-apply-the-principles-of-chaos-engineering-to-aws-lambda-80f87e3237e2-en-US-Wavenet-I-MALE--f3d60ec0be2cf.mp3","audioPath":"audios/how-can-we-apply-the-principles-of-chaos-engineering-to-aws-lambda-80f87e3237e2-en-US-Wavenet-I-MALE--f3d60ec0be2cf.mp3","nickname":"Dr. One (en-US)"}]},"slug":"how-can-we-apply-the-principles-of-chaos-engineering-to-aws-lambda-80f87e3237e2"},"__N_SSG":true},"page":"/[slug]","query":{"slug":"how-can-we-apply-the-principles-of-chaos-engineering-to-aws-lambda-80f87e3237e2"},"buildId":"qqTckmfliewRaBLUp_8jP","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[77618,63213,87127,71206,89752,41116,31486,42348],"gsp":true,"scriptLoader":[]}</script><script>(function(){function c(){var b=a.contentDocument||(a.contentWindow&&a.contentWindow.document);if(b){var d=b.createElement('script');d.innerHTML="window.__CF$cv$params={r:'a39380223e3e1172',t:'MTc4OTA5NjQ0OA=='};var a=document.createElement('script');a.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js';document.getElementsByTagName('head')[0].appendChild(a);";b.getElementsByTagName('head')[0].appendChild(d)}}if(document.body){var a=document.createElement('iframe');a.height=1;a.width=1;a.style.position='absolute';a.style.top=0;a.style.left=0;a.style.border='none';a.style.visibility='hidden';document.body.appendChild(a);if('loading'!==document.readyState)c();else if(window.addEventListener)document.addEventListener('DOMContentLoaded',c);else{var e=document.onreadystatechange||function(){};document.onreadystatechange=function(b){e(b);'loading'!==document.readyState&&(document.onreadystatechange=e,c())}}}})();</script></body></html>