520 lines
33 KiB
HTML
520 lines
33 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en-us" dir="ltr">
|
||
<head>
|
||
<meta charset="UTF-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
<meta name="description" content="
|
||
|
||
Posted: Oct 2025
|
||
|
||
|
||
|
||
Github: https://github.com/brk0v/trixter
|
||
|
||
|
||
Contents
|
||
|
||
Chaos Engineering and Network Fault Injection
|
||
Introducing Trixter – A Chaos Monkey for TCP
|
||
Why Trixter vs GNU/Linux tc netem (Kernel Network Emulator)
|
||
Using Trixter: Examples of Injecting Chaos
|
||
|
||
Example 1: Adding latency and packet loss
|
||
Example 2: Throttling bandwidth
|
||
Example 3: Running in CI/CD with chaos
|
||
Example 4: Control failures on per connection in runtime
|
||
|
||
|
||
Documentation, recipes, reference
|
||
|
||
|
||
|
||
Break your network before production does">
|
||
<meta name="theme-color" media="(prefers-color-scheme: light)" content="#ffffff">
|
||
<meta name="theme-color" media="(prefers-color-scheme: dark)" content="#343a40">
|
||
<meta name="color-scheme" content="light dark"><meta property="og:url" content="https://biriukov.dev/posts/trixter-chaos-proxy/">
|
||
<meta property="og:site_name" content="Viacheslav Biriukov">
|
||
<meta property="og:title" content="Trixter: A Chaos Proxy for Simulating Network Faults">
|
||
<meta property="og:description" content="Posted: Oct 2025 Github: https://github.com/brk0v/trixter Contents
|
||
Chaos Engineering and Network Fault Injection Introducing Trixter – A Chaos Monkey for TCP Why Trixter vs GNU/Linux tc netem (Kernel Network Emulator) Using Trixter: Examples of Injecting Chaos Example 1: Adding latency and packet loss Example 2: Throttling bandwidth Example 3: Running in CI/CD with chaos Example 4: Control failures on per connection in runtime Documentation, recipes, reference Break your network before production does">
|
||
<meta property="og:locale" content="en_us">
|
||
<meta property="og:type" content="article">
|
||
<meta property="article:section" content="posts">
|
||
<title>Trixter: A Chaos Proxy for Simulating Network Faults | Viacheslav Biriukov</title>
|
||
<link rel="manifest" href="/manifest.json">
|
||
<link rel="icon" href="/favicon.png" >
|
||
<link rel="stylesheet" href="/book.min.1e13c2d8521416f2409b2e0e47b515c4ee90f2718cddd31ea96d2f739bc9d7e1.css" integrity="sha256-HhPC2FIUFvJAmy4OR7UVxO6Q8nGM3dMeqW0vc5vJ1+E=" crossorigin="anonymous">
|
||
<script defer src="/flexsearch.min.js"></script>
|
||
<script defer src="/en.search.min.7ef6667ebf3ba3c5fee7df6027d59f0842e90281b8952c037b4487bc40271d61.js" integrity="sha256-fvZmfr87o8X+599gJ9WfCELpAoG4lSwDe0SHvEAnHWE=" crossorigin="anonymous"></script>
|
||
<!--
|
||
Made with Book Theme
|
||
https://github.com/alex-shpak/hugo-book
|
||
-->
|
||
|
||
<script
|
||
async
|
||
src="https://www.googletagmanager.com/gtag/js?id=G-599VSLESJL"
|
||
></script>
|
||
<script>
|
||
window.dataLayer = window.dataLayer || [];
|
||
function gtag() {
|
||
dataLayer.push(arguments);
|
||
}
|
||
gtag("js", new Date());
|
||
|
||
gtag("config", "G-599VSLESJL");
|
||
</script>
|
||
|
||
<link rel="stylesheet" href="/my_css/cookie.css" />
|
||
<link rel="stylesheet" href="/my_css/copy-code.css" />
|
||
<link
|
||
rel="stylesheet"
|
||
href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.5.0/font/bootstrap-icons.css"
|
||
/>
|
||
|
||
</head>
|
||
<body dir="ltr">
|
||
<input type="checkbox" class="hidden toggle" id="menu-control" />
|
||
<input type="checkbox" class="hidden toggle" id="toc-control" />
|
||
<main class="container flex">
|
||
<aside class="book-menu">
|
||
<div class="book-menu-content">
|
||
|
||
<nav>
|
||
<h2 class="book-brand">
|
||
<a class="flex align-center" href="/"><span>Viacheslav Biriukov</span>
|
||
</a>
|
||
</h2>
|
||
|
||
|
||
<div class="book-search">
|
||
<input type="text" id="book-search-input" placeholder="Search" aria-label="Search" maxlength="64" data-hotkeys="s/" />
|
||
<div class="book-search-spinner hidden"></div>
|
||
<ul id="book-search-results"></ul>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<ul>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
</ul>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<div style="margin-top: 30px; margin-bottom: 30px">
|
||
<b>Blog:</b>
|
||
<ul>
|
||
<li><a href="/posts/async-rust-gocha-tokio-cancelation-select-future-then/">Async Rust gotcha: evolving tokio::select! code <span style="padding:0 2px;border-radius:2px ;background-color:#e84118;color: aliceblue;">new</span></li>
|
||
</ul>
|
||
</div>
|
||
|
||
<div style="margin-top: 30px; margin-bottom: 30px">
|
||
<b>More recent series:</b>
|
||
<ul>
|
||
<li><a href="/rust-tokio-io/">1. Async Rust & Tokio I/O Streams: Backpressure, Concurrency, and Ergonomics</li>
|
||
<li><a href="/docs/resolver-dual-stack-application/0-sre-should-know-about-gnu-linux-resolvers-and-dual-stack-applications/">2. Resolvers and Dual-Stack applications</li>
|
||
<li><a href="/docs/page-cache/0-linux-page-cache-for-sre/">3. Linux Page Cache mini book</li>
|
||
<li><a href="/docs/fd-pipe-session-terminal/0-sre-should-know-about-gnu-linux-shell-related-internals-file-descriptors-pipes-terminals-user-sessions-process-groups-and-daemons">4. File descriptors, pipes, terminals, user sessions, process groups and daemons</li>
|
||
</ul>
|
||
</div>
|
||
|
||
<div style="margin-top: 30px; margin-bottom: 30px">
|
||
<b>Open Source Projects</b>
|
||
<ul>
|
||
<li>
|
||
<a href="/posts/trixter-chaos-proxy/"> • trixter chaos proxy</a>
|
||
</li>
|
||
<li><a href="https://crates.io/crates/tokio-netem"> • tokio-netem</a></li>
|
||
</ul>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<ul>
|
||
|
||
<li>
|
||
<a href="https://twitter.com/brk0v/" target="_blank" rel="noopener"><i class="bi bi-twitter"></i>
|
||
Twitter
|
||
</a>
|
||
</li>
|
||
|
||
<li>
|
||
<a href="https://www.linkedin.com/in/biriukov/" target="_blank" rel="noopener"><i class="bi bi-linkedin"></i>
|
||
Linkedin
|
||
</a>
|
||
</li>
|
||
|
||
<li>
|
||
<a href="https://github.com/brk0v/" target="_blank" rel="noopener"><i class="bi bi-github"></i>
|
||
Github
|
||
</a>
|
||
</li>
|
||
|
||
</ul>
|
||
|
||
|
||
|
||
|
||
|
||
<div style="margin-top: 30px;">
|
||
<p xmlns:cc="http://creativecommons.org/ns#">
|
||
This content is licensed under
|
||
<a
|
||
href="http://creativecommons.org/licenses/by-nc/4.0/?ref=chooser-v1" target="_blank"
|
||
rel="license noopener noreferrer" style="display:inline-block; ">CC BY-NC 4.0<img
|
||
style="height:22px!important;margin-left:3px;vertical-align:text-bottom;"
|
||
src="https://mirrors.creativecommons.org/presskit/icons/cc.svg?ref=chooser-v1"><img
|
||
style="height:22px!important;margin-left:3px;vertical-align:text-bottom;"
|
||
src="https://mirrors.creativecommons.org/presskit/icons/by.svg?ref=chooser-v1"><img
|
||
style="height:22px!important;margin-left:3px;vertical-align:text-bottom;"
|
||
src="https://mirrors.creativecommons.org/presskit/icons/nc.svg?ref=chooser-v1">
|
||
</a>
|
||
</p>
|
||
</div>
|
||
</nav>
|
||
|
||
|
||
|
||
|
||
<script>(function(){var e=document.querySelector("aside .book-menu-content");addEventListener("beforeunload",function(){localStorage.setItem("menu.scrollTop",e.scrollTop)}),e.scrollTop=localStorage.getItem("menu.scrollTop")})()</script>
|
||
|
||
|
||
|
||
</div>
|
||
</aside>
|
||
|
||
<div class="book-page">
|
||
<header class="book-header">
|
||
|
||
<div class="flex align-center justify-between">
|
||
<label for="menu-control">
|
||
<img src="/svg/menu.svg" class="book-icon" alt="Menu" />
|
||
</label>
|
||
|
||
<strong>Trixter: A Chaos Proxy for Simulating Network Faults</strong>
|
||
|
||
<label for="toc-control">
|
||
|
||
</label>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
</header>
|
||
|
||
|
||
|
||
<article class="markdown book-post">
|
||
<h1>
|
||
<a href="/posts/trixter-chaos-proxy/">Trixter: A Chaos Proxy for Simulating Network Faults</a>
|
||
</h1>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<p class="updated-right">
|
||
<i>
|
||
<time datetime="2025-10">Posted: Oct 2025</time>
|
||
</i>
|
||
</p>
|
||
<p style="float1:right" class="text">
|
||
<i class="bi bi-github"></i> Github: <a href="https://github.com/brk0v/trixter">https://github.com/brk0v/trixter</a>
|
||
</p>
|
||
<hr/>
|
||
<p><strong>Contents</strong></p>
|
||
<ul>
|
||
<li><a href="posts/trixter-chaos-proxy/#chaos-engineering-and-network-fault-injection">Chaos Engineering and Network Fault Injection</a></li>
|
||
<li><a href="/posts/trixter-chaos-proxy/#introducing-trixter--a-chaos-monkey-for-tcp">Introducing Trixter – A Chaos Monkey for TCP</a></li>
|
||
<li><a href="/posts/trixter-chaos-proxy/#why-trixter-vs-gnulinux-tc-netem-kernel-network-emulator">Why Trixter vs GNU/Linux <code>tc</code> <code>netem</code> (Kernel Network Emulator)</a></li>
|
||
<li><a href="/posts/trixter-chaos-proxy/#using-trixter-examples-of-injecting-chaos">Using Trixter: Examples of Injecting Chaos</a>
|
||
<ul>
|
||
<li><a href="/posts/trixter-chaos-proxy/#example-1-adding-latency-and-packet-loss">Example 1: Adding latency and packet loss</a></li>
|
||
<li><a href="/posts/trixter-chaos-proxy/#example-2-throttling-bandwidth">Example 2: Throttling bandwidth</a></li>
|
||
<li><a href="/posts/trixter-chaos-proxy/#example-3-running-in-cicd-with-chaos">Example 3: Running in CI/CD with chaos</a></li>
|
||
<li><a href="/posts/trixter-chaos-proxy/#example-4-control-failures-on-per-connection-in-runtime">Example 4: Control failures on per connection in runtime</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="/posts/trixter-chaos-proxy/#documentation-recipes-reference">Documentation, recipes, reference</a></li>
|
||
</ul>
|
||
<hr/>
|
||
<blockquote>
|
||
<p><em>Break your network before production does</em></p>
|
||
</blockquote>
|
||
<h2 id="chaos-engineering-and-network-fault-injection">
|
||
Chaos Engineering and Network Fault Injection
|
||
<a class="anchor" href="#chaos-engineering-and-network-fault-injection">#</a>
|
||
</h2>
|
||
<p>Chaos engineering is the practice of <strong>deliberately injecting controlled failures</strong> into systems to observe their behavior and <strong>improve their resilience</strong>. One common chaos experiment is network-level fault injection, which involves artificial inserting network problems like latency, packet loss, or corruption to test how services cope. This is valuable because many modern systems are distributed (at least have clients and servers); introducing network slowness or errors can <strong>reveal hidden bugs, timeout issues, or insufficient retry logic in microservices</strong>. By simulating unreliable networks in a controlled way, SREs can ensure their applications are resilient against real-world network chaos (like spikes in latency or occasional outages).</p>
|
||
<h2 id="introducing-trixter--a-chaos-monkey-for-tcp">
|
||
Introducing Trixter – A Chaos Monkey for TCP
|
||
<a class="anchor" href="#introducing-trixter--a-chaos-monkey-for-tcp">#</a>
|
||
</h2>
|
||
<img class="img-center" style="float:right; padding-top:5px;" alt="Trixter: A Chaos Proxy for Simulating Network Faults" src="../images/trixter.png" width="30%" >
|
||
<p><code><a href="https://github.com/brk0v/trixter" target="_blank" rel="noopener">Trixter</a></code> is a <strong>high-performance chaos proxy designed for injecting network faults at the TCP</strong> layer. In essence, it’s a TCP proxy that sits between a client and server, forwarding traffic but intentionally sabotaging it according to your specifications.</p>
|
||
<p>Trixter is <strong>runtime-tunable too</strong>, so you can adjust fault parameters on the fly per connection without restarting the proxy.</p>
|
||
<p><em>How it works</em>: You run Trixter as a proxy on a specified listening port, pointing it to an upstream service. Under normal conditions, it simply relays traffic. But you can configure Trixter to inject various network failures in transit. For example, <strong>add artificial delay</strong> to each packet, <strong>throttle the bandwidth</strong>, <strong>drop a connection</strong> with some probability, <strong>corrupt a percentage of packets</strong> by injecting a trash, or even <strong>terminate connections</strong> to simulate outages by some timeout or by calling a REST API call.</p>
|
||
<blockquote class="book-hint info">
|
||
<p><strong>Note</strong></p>
|
||
<p>Think of Trixter as a minimalist, blazing-fast layer written in Rust using the async <a href="https://tokio.rs/" target="_blank" rel="noopener">Tokio</a> framework for efficiency.</p>
|
||
<p>Internally, it leverages the <code><a href="https://github.com/brk0v/trixter/tree/main/tokio-netem" target="_blank" rel="noopener">tokio-netem</a></code> library (<code>netem</code> = network emulator), which provides pluggable async I/O adaptors to simulate latency, bandwidth limits, packet slicing, connection termination, and data corruption at the stream level.</p>
|
||
|
||
</blockquote>
|
||
|
||
<p>This user-space approach means <strong>no kernel modules or root access</strong> is needed – the chaos is applied in the application layer (within the proxy) rather than the OS network stack.</p>
|
||
<p>What makes Trixter interesting is its combination of <strong>performance, portability, and simplicity</strong>. Being written in Rust, it’s <strong>memory-safe and built for speed</strong>. Configuration is straightforward: Trixter uses simple cli arguments and REST JSON API for per connection runtime changes, making it easy to emulate chaos scenarios.</p>
|
||
<p>The binary size is also quite small (3.3M), making it easy to bootstrap in every test suite run:</p>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>$ ls -lh ./target/release/trixter
|
||
</span></span><span style="display:flex;"><span>-rwxrwxr-x <span style="color:#ae81ff">2</span> user user 3.3M Oct <span style="color:#ae81ff">10</span> 21:39 ./target/release/trixter
|
||
</span></span></code></pre></div><p>If you want to jump straight to the <a href="/posts/trixter-chaos-proxy/#using-trixter-examples-of-injecting-chaos">examples</a> and skip the comparison to Linux kernel capabilities.</p>
|
||
<h2 id="why-trixter-vs-gnulinux-tc-netem-kernel-network-emulator">
|
||
Why Trixter vs GNU/Linux <code>tc</code> <code>netem</code> (Kernel Network Emulator)
|
||
<a class="anchor" href="#why-trixter-vs-gnulinux-tc-netem-kernel-network-emulator">#</a>
|
||
</h2>
|
||
<p>If you’ve ever simulated network faults on GNU/Linux, you might know <code>tc</code> <code>netem</code>, the traffic control tool’s network emulator. <code>netem</code> is powerful – it can impose delay, packet loss, duplication, reordering, and bandwidth limits at the kernel level. However, using <code>tc</code> <code>netem</code> in practice has some drawbacks for developers and SREs:</p>
|
||
<ul>
|
||
<li>
|
||
<p><strong>Usability</strong>: <code>netem</code> must be configured via the <code>tc</code> command syntax, which can be arcane. For example, to add 100ms latency and 5% packet loss on interface <code>eth0</code>, you’d run something like: <code>tc qdisc add dev eth0 root netem delay 100ms loss 5%</code>. It’s a manual process, and applying it to specific traffic (say, only one service or container) often <strong>requires setting up traffic control filters or isolation networks</strong>. Trixter, on the other hand, is as easy as running a proxy and setting cli arguments with the faults. You point your service or client to the proxy address, and only that traffic gets the chaos. No special networking setup needed.</p>
|
||
</li>
|
||
<li>
|
||
<p><strong>Root Privileges</strong>: Configuring <code>tc netem</code> requires root (<code>CAP_NET_ADMIN</code>) privileges on the host. This is fine in a local VM but problematic in environments where you can’t easily get root (Kubernetes pods, developer laptops on macOS/Windows, CI pipelines, etc.). Trixter requires no root – it runs in userland. Anyone can use it in a dev environment or CI job, and it can even be packaged as a container sidecar to inject faults for a specific application.</p>
|
||
</li>
|
||
<li>
|
||
<p><strong>Portability</strong>: <code>netem</code> is Linux-only (built into the kernel). If you’re developing on macOS or running tests on Windows, <code>tc</code> <code>netem</code> isn’t natively available. Trixter’s user-space proxy approach works across platforms, since it’s just a Rust binary. This makes it accessible to a wider range of use cases and teams.</p>
|
||
</li>
|
||
<li>
|
||
<p><strong>Dynamic Control</strong>: While you can adjust <code>netem</code> parameters by running more <code>tc</code> commands, it’s not designed for frequent on-the-fly changes (and each change affects the whole interface traffic globally). There are ways, but they are not ergonomic and usually requires some <code>iptables</code> configuration. Trixter is designed to be dynamically tunable. It provides an optional API endpoint you can run (e.g. REST JSON API) to tweak fault settings at runtime. In chaos experiments, being able to script fault injection (start with 0ms latency, then gradually increase to 500ms, etc.) is very useful – Trixter was built with this in mind.</p>
|
||
</li>
|
||
<li>
|
||
<p><strong>Scope and Limitations</strong>: <code>netem</code> operates at the packet level and can affect any protocol (TCP, UDP, ICMP, etc.) on the chosen interface. Trixter focuses on TCP streams (it’s a TCP proxy), which covers common cases like HTTP calls, gRPC, database connections, etc. The benefit is that Trixter targets exactly the connection you care about, rather than messing with the entire network stack of a host.</p>
|
||
</li>
|
||
</ul>
|
||
<p>In short, Trixter trades some of the breadth of kernel-level simulation for ease-of-use and flexibility. It’s an ideal choice for testing how your service handles network blips without the complexity of system-wide tools.</p>
|
||
<h2 id="using-trixter-examples-of-injecting-chaos">
|
||
Using Trixter: Examples of Injecting Chaos
|
||
<a class="anchor" href="#using-trixter-examples-of-injecting-chaos">#</a>
|
||
</h2>
|
||
<p>Using Trixter is straightforward. First, you’ll run the proxy pointing to a real service. Then you define what network conditions to impose. Below are a couple of examples.</p>
|
||
<p>For your test and play you have two main options:</p>
|
||
<ol>
|
||
<li>Setup global failures with cli arguments;</li>
|
||
<li><em>and/or</em> control failures per connection with REST JSON API.</li>
|
||
</ol>
|
||
<h3 id="example-1-adding-latency-and-packet-loss">
|
||
Example 1: Adding latency and packet loss
|
||
<a class="anchor" href="#example-1-adding-latency-and-packet-loss">#</a>
|
||
</h3>
|
||
<p>Suppose you have a service running on <code>localhost:3000</code> that you want to test with high latency and some connection loss. You could run Trixter with a config like this:</p>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>$ docker run --network host -it --rm ghcr.io/brk0v/trixter <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --listen 0.0.0.0:8080 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --upstream 127.0.0.1:3000 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --api 127.0.0.1:8888 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --delay-ms <span style="color:#ae81ff">1000</span> <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --terminate-probability-rate 0.001 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --connection-duration-ms <span style="color:#ae81ff">5000</span>
|
||
</span></span></code></pre></div><p>Here the new port to connect is 8080, every read and write will be delayed by 1 second, any I/0 on 0.1% probability fails, and every connection is terminated with TCP RST packet after 5 seconds.</p>
|
||
<p>This could be used, for example, to see how your app’s retry logic handles a few lost responses or how a web UI behaves on a slow connection (does a loading spinner show up, do requests time out gracefully, etc.).</p>
|
||
<h3 id="example-2-throttling-bandwidth">
|
||
Example 2: Throttling bandwidth
|
||
<a class="anchor" href="#example-2-throttling-bandwidth">#</a>
|
||
</h3>
|
||
<p>Trixter can also emulate bandwidth constraints. Let’s say you want to simulate a slow network (congested mobile network). You can configure a throttle in bytes per second. For instance:</p>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>$ docker run --network host -it --rm ghcr.io/brk0v/trixter <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --listen 0.0.0.0:8080 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --upstream 127.0.0.1:3000 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --api 127.0.0.1:8888 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --throttle-rate-bytes <span style="color:#ae81ff">1048576</span>
|
||
</span></span></code></pre></div><p>This will make Trixter buffer and drip-feed data to achieve roughly 1 MB/s, regardless of how fast the real upstream is. It’s a great way to test video streaming or large file downloads under limited network conditions. Your service might start queuing or compressing data differently once this bottleneck is introduced. This is a excellent way to test your application under network backpressure conditions.</p>
|
||
<h3 id="example-3-running-in-cicd-with-chaos">
|
||
Example 3: Running in CI/CD with chaos
|
||
<a class="anchor" href="#example-3-running-in-cicd-with-chaos">#</a>
|
||
</h3>
|
||
<p>A powerful way to integrate Trixter into your pipeline is to <strong>run it automatically during integration or E2E tests</strong> with periodic connection drops and random read/write bytes injections.</p>
|
||
<p>This lets you uncover non-deterministic failure patterns – but still reproduce them later.</p>
|
||
<p>For example, create an integration test with something like:</p>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>$ docker run --network host -it --rm ghcr.io/brk0v/trixter <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --listen 0.0.0.0:8080 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --upstream 127.0.0.1:3000 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --api 127.0.0.1:8888 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --terminate-probability-rate 0.001 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --corrupt-probability-rate 0.001
|
||
</span></span></code></pre></div><p>Each run it would pick a random seed for chaos parameters. If a test fails, you can open stdout.log and look for a line like:</p>
|
||
<pre tabindex="0"><code>2025-10-10T20:38:43.925064Z INFO trixter: random seed: 10382352052268666911
|
||
</code></pre><p>Then, <strong>reproduce the exact same chaos locally</strong>:</p>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>$ docker run --network host -it --rm ghcr.io/brk0v/trixter <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --listen 0.0.0.0:8080 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --upstream 127.0.0.1:3000 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --api 127.0.0.1:8888 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --terminate-probability-rate 0.001 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --corrupt-probability-rate 0.01 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --random-seed <span style="color:#ae81ff">10382352052268666911</span>
|
||
</span></span></code></pre></div><p>This pattern makes chaos deterministic, reproducible, and CI-friendly – like <strong>property-based testing for your network</strong>.</p>
|
||
<h3 id="example-4-control-failures-on-per-connection-in-runtime">
|
||
Example 4: Control failures on per connection in runtime
|
||
<a class="anchor" href="#example-4-control-failures-on-per-connection-in-runtime">#</a>
|
||
</h3>
|
||
<p>Spin up the Trixter proxy without default failures:</p>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>$ docker run --network host -it --rm ghcr.io/brk0v/trixter <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --listen 0.0.0.0:8080 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --upstream 127.0.0.1:3000 <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --api 127.0.0.1:8888
|
||
</span></span></code></pre></div><p>Make a connection with your client/application/service, and discover it with REST JSON API:</p>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>$ curl -s http://127.0.0.1:8888/connections | jq
|
||
</span></span><span style="display:flex;"><span><span style="color:#f92672">[</span>
|
||
</span></span><span style="display:flex;"><span> <span style="color:#f92672">{</span>
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"conn_info"</span>: <span style="color:#f92672">{</span>
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"id"</span>: <span style="color:#e6db74">"1J8UO7eCuqUMqdoP5KmvN"</span>,
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"downstream"</span>: <span style="color:#e6db74">"127.0.0.1:45528"</span>,
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"upstream"</span>: <span style="color:#e6db74">"127.0.0.1:3000"</span>
|
||
</span></span><span style="display:flex;"><span> <span style="color:#f92672">}</span>,
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"delay"</span>: <span style="color:#f92672">{</span>
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"secs"</span>: 0,
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"nanos"</span>: <span style="color:#ae81ff">0</span>
|
||
</span></span><span style="display:flex;"><span> <span style="color:#f92672">}</span>,
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"throttle_rate"</span>: 0,
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"slice_size"</span>: 0,
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"terminate_probability_rate"</span>: 0.0,
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"corrupt_probability_rate"</span>: 0.0
|
||
</span></span><span style="display:flex;"><span> <span style="color:#f92672">}</span>
|
||
</span></span><span style="display:flex;"><span><span style="color:#f92672">]</span>
|
||
</span></span></code></pre></div><p>Store the connection <code>ID</code> in variable to future usage:</p>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>$ ID<span style="color:#f92672">=</span><span style="color:#66d9ef">$(</span>curl -s http://127.0.0.1:8888/connections | jq -r <span style="color:#e6db74">'.[0].conn_info.id'</span><span style="color:#66d9ef">)</span>
|
||
</span></span></code></pre></div><p>Now we can easily inject failures – for instance, add a 1 second latency:</p>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>$ curl -i -X PATCH <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> http://127.0.0.1:8888/connections/$ID/delay <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> <span style="color:#ae81ff">\
|
||
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> -d <span style="color:#e6db74">'{"delay_ms":1000}'</span>
|
||
</span></span></code></pre></div><p>And check that it applied:</p>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>$ curl -s http://127.0.0.1:8888/connections | jq
|
||
</span></span><span style="display:flex;"><span><span style="color:#f92672">[</span>
|
||
</span></span><span style="display:flex;"><span> <span style="color:#f92672">{</span>
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"conn_info"</span>: <span style="color:#f92672">{</span>
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"id"</span>: <span style="color:#e6db74">"1J8UO7eCuqUMqdoP5KmvN"</span>,
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"downstream"</span>: <span style="color:#e6db74">"127.0.0.1:45528"</span>,
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"upstream"</span>: <span style="color:#e6db74">"127.0.0.1:3000"</span>
|
||
</span></span><span style="display:flex;"><span> <span style="color:#f92672">}</span>,
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"delay"</span>: <span style="color:#f92672">{</span>
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"secs"</span>: 1, <span style="color:#75715e"># <-------------------------- Changed</span>
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"nanos"</span>: <span style="color:#ae81ff">0</span>
|
||
</span></span><span style="display:flex;"><span> <span style="color:#f92672">}</span>,
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"throttle_rate"</span>: 0,
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"slice_size"</span>: 0,
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"terminate_probability_rate"</span>: 0.0,
|
||
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">"corrupt_probability_rate"</span>: 0.0
|
||
</span></span><span style="display:flex;"><span> <span style="color:#f92672">}</span>
|
||
</span></span><span style="display:flex;"><span><span style="color:#f92672">]</span>
|
||
</span></span></code></pre></div><p>This way, you can build a flexible test setup or gradually introduce latency, throttling, and other network conditions.</p>
|
||
<h2 id="documentation-recipes-reference">
|
||
Documentation, recipes, reference
|
||
<a class="anchor" href="#documentation-recipes-reference">#</a>
|
||
</h2>
|
||
<p>For more examples, API reference for controlling per connection settings in run time and recipes please go to <a href="https://github.com/brk0v/trixter" target="_blank" rel="noopener">https://github.com/brk0v/trixter</a>.</p>
|
||
<h2 id="summary">
|
||
Summary
|
||
<a class="anchor" href="#summary">#</a>
|
||
</h2>
|
||
<p>Trixter is a lightweight, blazing-fast chaos proxy for SREs and developers.</p>
|
||
<p>It bridges the gap between kernel tools like <code>tc</code> <code>netem</code> and high-level testing frameworks, letting you <strong>inject network chaos safely and precisely</strong>.</p>
|
||
<p>Run it locally, use it in CI with random seeds, and reproduce failures with one command – all without root privileges.</p>
|
||
<p>If you care about resilience and performance under adverse conditions – or just want to break things the smart way – give Trixter a try.</p>
|
||
</article>
|
||
|
||
|
||
|
||
<footer class="book-footer">
|
||
|
||
<div class="flex flex-wrap justify-between">
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
|
||
|
||
|
||
|
||
|
||
<script defer src="/my_js/copy-code.js"></script>
|
||
|
||
</footer>
|
||
|
||
|
||
|
||
<div class="book-comments">
|
||
|
||
</div>
|
||
|
||
|
||
|
||
<label for="menu-control" class="hidden book-menu-overlay"></label>
|
||
</div>
|
||
|
||
|
||
</main>
|
||
|
||
|
||
<div class="cookie-container">
|
||
<p>
|
||
This website uses "<b>cookies</b>".
|
||
Using this website means you're OK with this.
|
||
If you are <b>NOT</b>, please close the site page.
|
||
</p>
|
||
<button class="cookie-btn">
|
||
ACCEPT AND CLOSE
|
||
</button>
|
||
</div>
|
||
<script src="/my_js/cookie.js"></script>
|
||
</body>
|
||
</html>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|