Files
nexus/sreweekly/articles/54/06-who-called-git-an-unusual-debugging-story.html
2026-09-12 17:23:01 +08:00

693 lines
45 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html lang="en-us">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<!-- Enable responsiveness on mobile devices-->
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1">
<title>
Who Called Git? An Unusual Debugging Story
</title>
<meta name="description" content="I don’t usually talk about support ticket work that I do. Most tickets are so specific it’s hard to write generalized articles. When I get a type of ticket t...">
<!-- CSS -->
<link rel="stylesheet" href="/assets/application-b77814ff5b9091460b66ccbb93f65cf34675ba6b281f6a4d43c23d06c8dab1b7184e5c377dcbc3e86516eb75302ff028bd8111c0a5b3ec646cb8986f9352fed2.css">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=PT+Sans:400,400italic,700">
<script src="/assets/application-5bb33e9916e7fab25a0c5e5750146af84c06965c860cd019a5660a85b3548e31fa611bfe859d1ac13205f31f19e0670d34bb738656c92438fbdea13d2448aa7c.js" async></script>
<!-- Icons -->
<link rel="shortcut icon" href="/assets/favicon-ba2fbdf8c7661d05828f7cf46206479bb8f7222d24c6a1b4b957d3c7d9831d8963086e3cbefd85e9fb83f7074daf8eb39f3c876951c43d7849b55e1c583b5a36.ico">
<!-- Feed -->
<link rel="alternate" type="application/atom+xml" href="/feed.xml" title="Atom feed">
<!--
<script src="//load.sumome.com/" data-sumo-site-id="a658207c92bb7cb45c1d7827aad47e57786f057648b1e4abc330fd37129f4c8c" async="async"></script>
-->
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-26526218-1', 'auto');
ga('send', 'pageview');
</script>
<meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:site" content="Schneems"/>
<meta name="twitter:description" content="I don’t usually talk about support ticket work that I do. Most tickets are so specific it’s hard to write generalized articles. When I get a type of ticket t..." />
<meta property="og:description" content="I don’t usually talk about support ticket work that I do. Most tickets are so specific it’s hard to write generalized articles. When I get a type of ticket t..."/>
<meta property="twitter:title" content="Who Called Git? An Unusual Debugging Story" />
<meta property="og:title" content="Who Called Git? An Unusual Debugging Story" />
<meta name="twitter:image" content="" />
<meta property="og:image" content="" />
</head>
<body>
<div class="flexbox-container main" >
<div class="sidebar">
<div>
<a href="/" class="">
<svg version="1.0" xmlns="https://www.w3.org/2000/svg"
viewBox="0 0 1969.000000 1791.000000"
preserveAspectRatio="xMidYMid meet" class='logo'>
<g transform="translate(0.000000,1791.000000) scale(0.100000,-0.100000)"
fill="#FF1168" stroke="none">
<path d="M9185 15253 c-1231 -36 -2459 -182 -3645 -434 -366 -78 -518 -117
-571 -145 -90 -50 -175 -148 -221 -255 l-23 -54 -3 -862 -3 -863 -2152 -2
-2152 -3 -49 -24 c-53 -26 -101 -90 -112 -149 -4 -20 8 -146 30 -306 286
-2133 286 -4187 0 -6280 -41 -298 -40 -325 13 -394 56 -74 -100 -55 2264 -276
l2154 -201 5 -695 c5 -684 5 -696 27 -748 61 -148 168 -251 308 -296 28 -8
1092 -254 2366 -546 l2317 -530 2338 535 c2597 595 2431 551 2540 662 64 66
120 173 134 258 6 34 10 333 10 703 0 507 3 642 13 642 6 0 1002 94 2213 209
2418 230 2251 209 2308 284 52 68 53 96 12 393 -165 1214 -237 2460 -208 3627
24 986 82 1713 208 2653 22 163 34 284 30 305 -11 60 -58 124 -112 150 l-49
24 -2207 3 -2208 2 0 818 c0 545 -4 838 -11 878 -26 142 -126 277 -253 339
-53 26 -132 49 -311 89 -1476 333 -3007 501 -4500 494 -225 -1 -450 -3 -500
-5z m1130 -463 c1145 -37 2193 -155 3280 -371 329 -65 658 -140 683 -155 l22
-14 0 -805 0 -805 -25 0 -25 0 0 795 c0 437 -3 795 -8 795 -4 0 -107 22 -230
50 -1185 265 -2426 419 -3692 460 -1607 51 -3339 -117 -4917 -476 l-163 -37 0
-794 0 -793 -25 0 -25 0 0 805 0 805 23 14 c25 17 462 114 807 180 1417 271
2903 391 4295 346z m-2758 -1074 l52 -103 83 -12 c46 -7 98 -15 116 -18 l34
-6 -83 -83 -83 -83 18 -112 c10 -61 15 -113 13 -116 -3 -2 -50 20 -105 49
l-100 53 -102 -54 c-56 -29 -103 -51 -105 -49 -3 2 4 53 14 114 l19 109 -85
87 -86 86 49 7 c27 4 81 12 119 19 l70 11 50 102 c28 56 52 102 55 102 3 1 29
-46 57 -103z m1147 -4 l49 -99 86 -12 c47 -6 101 -14 119 -18 l34 -6 -85 -86
-85 -86 19 -109 c10 -61 17 -112 14 -114 -2 -2 -49 20 -105 49 l-102 54 -100
-53 c-55 -29 -102 -51 -105 -49 -2 3 3 55 13 116 l18 112 -83 83 -83 83 34 6
c18 3 70 11 116 18 l83 12 52 103 c28 56 53 101 56 99 4 -2 28 -48 55 -103z
m1150 0 l49 -99 120 -18 120 -17 -88 -82 -88 -81 20 -118 c11 -64 18 -117 16
-117 -2 0 -49 23 -105 51 l-102 51 -103 -51 c-57 -28 -105 -51 -107 -51 -1 0
6 53 17 117 l20 118 -88 81 -88 82 120 17 120 18 52 104 c28 57 55 102 59 99
4 -2 29 -49 56 -104z m1278 -111 c46 -7 98 -15 116 -18 l34 -6 -83 -83 -83
-83 18 -112 c10 -61 16 -113 13 -115 -2 -3 -50 19 -105 48 l-100 53 -102 -54
c-56 -29 -103 -51 -105 -49 -3 2 4 53 14 114 l19 109 -85 86 -85 86 34 6 c18
4 72 12 119 18 l86 12 53 106 52 106 54 -106 53 -107 83 -11z m1013 116 l50
-102 70 -11 c39 -7 92 -15 119 -19 l49 -7 -86 -86 -85 -87 19 -109 c10 -61 17
-112 14 -114 -2 -2 -49 20 -105 49 l-102 54 -100 -53 c-55 -29 -102 -51 -105
-49 -2 3 3 55 13 116 l18 112 -83 83 -83 83 34 6 c18 3 70 11 116 18 l83 12
52 103 c28 57 54 104 57 103 3 0 28 -46 55 -102z m6930 -1359 c-15 -68 -92
-695 -124 -1018 -80 -796 -113 -1480 -113 -2320 0 -947 43 -1728 148 -2650 26
-231 80 -647 89 -687 l5 -23 -9285 0 -9285 0 5 23 c9 40 63 456 89 687 104
920 148 1701 148 2645 0 835 -33 1532 -113 2325 -32 323 -109 950 -124 1018
l-5 22 9285 0 9285 0 -5 -22z m-12898 -6999 c87 -23 219 -54 293 -70 l135 -29
-125 5 c-155 8 -530 39 -779 65 -215 23 -218 23 -205 11 13 -13 418 -143 592
-191 126 -34 402 -99 482 -113 62 -11 -136 -6 -311 8 -223 18 -690 62 -734 70
-113 20 171 -79 485 -170 102 -29 255 -69 340 -88 85 -20 184 -42 220 -50 l65
-15 -90 5 c-235 12 -931 72 -1020 88 -119 22 192 -87 510 -178 161 -46 326
-87 555 -135 l45 -10 -45 -1 c-69 -2 -421 26 -759 59 -167 17 -306 28 -308 26
-7 -7 404 -139 587 -189 154 -42 308 -79 460 -111 l65 -14 -65 -1 c-85 -1
-393 23 -754 59 -160 16 -292 28 -294 26 -7 -8 456 -155 645 -206 103 -27 260
-66 348 -85 l160 -35 -110 6 c-163 8 -446 31 -738 60 -142 14 -263 24 -270 21
-17 -5 396 -138 613 -197 102 -28 257 -66 345 -85 l160 -35 -130 5 c-139 6
-614 46 -830 70 -71 8 -139 15 -150 14 -19 0 -19 -1 1 -9 64 -26 840 -172
1214 -230 2229 -340 4431 -299 6645 125 300 58 531 108 518 112 -6 2 -98 -5
-205 -16 -401 -43 -963 -83 -893 -65 11 3 94 21 185 41 278 60 949 258 913
270 -7 3 -128 -7 -270 -21 -292 -29 -575 -52 -738 -60 l-110 -6 160 35 c239
52 510 126 767 211 126 42 228 77 226 79 -2 3 -112 -7 -243 -20 -361 -36 -575
-54 -730 -59 -153 -6 -158 -8 83 45 73 16 208 50 300 75 187 51 596 183 589
190 -2 2 -141 -9 -308 -26 -334 -33 -689 -60 -759 -59 -33 0 -2 10 115 35 88
18 251 59 363 89 217 59 602 183 602 194 0 3 -120 -6 -267 -21 -254 -25 -603
-53 -778 -62 l-80 -4 65 15 c36 8 135 30 220 50 254 57 840 234 840 252 0 4
-120 -5 -267 -20 -345 -34 -660 -59 -743 -58 -62 0 -55 2 125 43 249 57 413
103 673 187 202 67 264 97 170 84 -160 -21 -737 -71 -933 -80 l-120 -5 135 29
c74 16 206 47 293 70 l158 41 380 0 379 0 0 -825 0 -824 -37 -10 c-186 -49
-797 -172 -1168 -235 -877 -148 -1703 -233 -2615 -267 -346 -13 -1343 -7
-1645 11 -1029 58 -1889 163 -2805 341 -185 36 -679 142 -722 155 -17 5 -18
53 -18 830 l0 824 394 0 395 0 158 -41z m2191 -164 c100 -30 155 -108 168
-238 l7 -67 -87 0 -86 0 0 35 c0 89 -50 137 -121 116 -43 -13 -59 -43 -59
-113 0 -78 28 -123 160 -255 95 -95 126 -134 150 -185 55 -117 50 -256 -11
-352 -49 -78 -170 -120 -282 -97 -131 28 -195 110 -205 266 l-5 85 85 0 86 0
4 -71 c3 -63 7 -74 31 -95 37 -32 87 -32 124 0 25 22 28 30 27 88 0 41 -6 75
-18 98 -10 19 -74 92 -142 162 -154 157 -177 199 -182 335 -5 115 14 179 69
234 64 64 179 86 287 54z m2369 -13 c51 -26 96 -78 117 -134 15 -38 17 -89 14
-443 -3 -390 -4 -401 -25 -441 -27 -52 -80 -98 -130 -115 -63 -21 -179 -17
-240 10 -28 12 -60 32 -72 44 -33 36 -63 116 -69 184 l-5 63 87 0 86 0 0 -40
c0 -46 21 -95 45 -104 9 -3 32 -6 50 -6 81 0 95 29 102 209 l5 136 -31 -33
c-39 -41 -87 -62 -146 -62 -60 0 -103 19 -142 63 -46 53 -55 111 -51 332 3
210 10 236 77 299 69 66 234 85 328 38z m622 12 c67 -20 134 -80 158 -142 13
-35 18 -75 18 -152 0 -90 -3 -112 -24 -153 -13 -26 -36 -58 -52 -69 l-28 -21
38 -42 c56 -61 66 -99 66 -250 0 -114 -3 -136 -23 -180 -48 -104 -125 -148
-257 -148 -132 0 -209 44 -257 148 -20 44 -23 66 -23 180 0 151 10 189 66 250
l38 42 -28 21 c-52 38 -75 104 -76 212 0 121 17 179 69 236 69 77 195 104 315
68z m-3444 -79 l0 -80 -157 -3 -158 -3 0 -159 0 -160 125 0 125 0 0 -80 0 -80
-125 0 -125 0 0 -170 0 -170 160 0 160 0 0 -80 0 -80 -250 0 -250 0 0 568 c0
313 3 572 7 576 4 3 115 5 247 4 l241 -3 0 -80z m1235 0 l0 -85 -95 0 -95 0 0
-490 0 -490 -90 0 -90 0 0 490 0 490 -95 0 -95 0 0 85 0 85 280 0 280 0 0 -85z
m970 -490 l0 -575 -90 0 -90 0 0 440 0 440 -70 0 -70 0 0 58 0 58 59 12 c48
10 65 20 95 52 19 22 38 51 42 65 6 23 10 25 65 25 l59 0 0 -575z m2010 185
l0 -390 45 0 45 0 0 -79 0 -80 -42 -3 -43 -3 -3 -102 -3 -103 -89 0 -90 0 0
105 0 105 -170 0 -170 0 0 85 0 85 163 385 162 385 98 0 97 0 0 -390z m-2830
-675 l0 -85 -85 0 -85 0 0 85 0 85 85 0 85 0 0 -85z"/>
<path d="M1191 12005 c-113 -35 -208 -139 -237 -258 -15 -61 -15 -78 10 -323
48 -484 77 -875 96 -1309 5 -132 13 -294 16 -361 l6 -120 65 -29 c42 -18 89
-50 129 -88 242 -226 183 -614 -117 -766 l-74 -38 -7 -284 c-13 -592 -46
-1115 -109 -1763 -31 -310 -31 -320 -15 -380 31 -109 90 -181 196 -237 l45
-24 8570 0 8570 0 55 26 c101 48 179 149 200 261 8 39 5 98 -10 238 -59 557
-103 1208 -115 1720 -3 129 -8 285 -11 346 l-6 112 -41 16 c-64 24 -149 97
-195 166 -163 244 -73 569 192 694 l53 25 7 183 c20 541 56 1073 106 1576 32
315 31 351 -12 436 -31 60 -105 133 -167 163 l-56 28 -8550 2 c-7228 1 -8557
0 -8594 -12z m1659 -984 c282 -96 434 -299 511 -680 26 -129 38 -381 18 -382
-8 0 -141 -12 -296 -27 l-282 -27 -5 50 c-3 27 -8 93 -11 146 -9 166 -62 313
-132 366 -31 24 -46 28 -102 28 -54 0 -72 -5 -100 -24 -100 -72 -142 -358 -81
-555 50 -161 126 -293 390 -681 384 -563 518 -808 596 -1085 22 -81 27 -121
31 -272 4 -111 1 -214 -6 -273 -53 -419 -250 -677 -586 -766 -106 -29 -380
-32 -493 -6 -386 88 -576 346 -631 859 -10 84 -16 326 -8 329 12 4 583 56 588
53 3 -2 9 -77 12 -166 13 -360 109 -539 288 -538 75 1 160 65 195 148 28 66
44 170 45 285 0 95 -4 125 -26 197 -54 171 -149 338 -395 695 -382 554 -507
791 -591 1115 -37 141 -46 285 -30 429 52 444 270 725 626 807 41 9 110 13
220 11 150 -3 166 -6 255 -36z m1885 15 c287 -80 485 -319 561 -676 12 -57 17
-160 21 -422 l5 -348 -285 0 -285 0 -4 333 c-5 316 -6 335 -28 391 -53 138
-173 193 -290 134 -53 -27 -103 -102 -119 -180 -8 -33 -11 -467 -11 -1337 0
-1410 -3 -1335 59 -1434 46 -72 80 -89 172 -85 65 3 81 7 113 32 51 38 87 118
97 215 4 42 8 199 8 349 l1 272 286 0 286 0 -5 -347 c-7 -433 -21 -514 -126
-718 -37 -73 -64 -108 -141 -185 -107 -108 -188 -157 -317 -192 -113 -31 -354
-30 -467 0 -268 73 -456 281 -544 600 l-27 97 -3 1367 c-2 1348 -2 1369 18
1469 41 198 127 369 249 489 100 99 198 151 351 186 79 18 346 12 425 -10z
m12553 8 c388 -79 588 -393 627 -981 7 -96 6 -103 -11 -104 -11 0 -145 -12
-299 -27 l-280 -27 -12 145 c-14 170 -33 253 -76 334 -42 80 -93 116 -161 116
-138 0 -207 -122 -210 -370 -2 -233 56 -359 411 -888 285 -424 378 -574 472
-762 124 -250 164 -408 164 -645 0 -617 -241 -957 -716 -1016 -105 -13 -273
-6 -376 16 -328 69 -522 286 -600 671 -25 121 -41 287 -41 416 0 93 1 98 23
102 12 3 128 14 257 26 129 11 253 23 276 26 l41 6 6 -154 c6 -173 17 -240 53
-349 50 -145 116 -204 229 -204 146 0 219 102 246 339 13 123 6 195 -32 310
-56 168 -118 273 -475 803 -293 434 -375 583 -459 836 -68 203 -78 259 -78
437 1 232 50 437 144 596 76 129 244 271 379 320 128 46 351 58 498 28z
m-10988 -884 l0 -860 235 0 235 0 2 858 3 857 305 0 305 0 3 -2077 2 -2078
-310 0 -310 0 0 905 0 905 -235 0 -235 0 0 -905 0 -905 -305 0 -305 0 0 2080
0 2080 305 0 305 0 0 -860z m2223 350 c76 -278 221 -806 322 -1175 138 -506
187 -671 199 -673 15 -3 16 103 16 1177 l0 1181 280 0 280 0 0 -2080 0 -2080
-269 0 -269 0 -11 33 c-5 17 -148 536 -316 1152 -168 616 -310 1131 -314 1145
-4 14 -15 26 -24 28 -16 3 -17 -80 -17 -1177 l0 -1181 -280 0 -280 0 0 2080 0
2080 272 -2 272 -3 139 -505z m2957 195 l0 -315 -395 0 -395 0 0 -545 0 -545
283 -2 282 -3 3 -332 2 -333 -285 0 -285 0 0 -575 0 -575 395 0 395 0 0 -310
0 -310 -700 0 -700 0 0 2080 0 2080 700 0 700 0 0 -315z m1740 0 l0 -315 -395
0 -395 0 0 -545 0 -545 283 -2 282 -3 3 -332 2 -333 -285 0 -285 0 0 -575 0
-575 395 0 395 0 0 -310 0 -310 -700 0 -700 0 0 2080 0 2080 700 0 700 0 0
-315z m1268 -642 c101 -527 187 -998 193 -1048 18 -153 23 -152 44 6 13 87
304 1638 366 1947 l10 52 375 0 374 0 0 -2080 0 -2080 -285 0 -285 0 0 1395
c0 1225 -2 1395 -15 1395 -18 0 -13 29 -80 -435 -32 -214 -123 -831 -204
-1370 l-146 -980 -133 -3 c-85 -2 -132 1 -132 8 0 5 -85 580 -189 1277 -105
698 -197 1319 -206 1382 -8 63 -17 117 -20 119 -2 3 -11 2 -20 -1 -13 -6 -15
-153 -15 -1397 l0 -1390 -280 0 -280 0 0 2080 0 2080 373 0 373 0 182 -957z"/>
<path d="M10531 5014 c-19 -25 -21 -39 -21 -180 0 -164 7 -195 47 -213 33 -15
58 -14 93 4 40 21 45 45 45 212 0 128 -2 144 -21 170 -17 23 -30 29 -71 31
-45 3 -53 0 -72 -24z"/>
<path d="M11221 5042 c-37 -11 -61 -52 -68 -114 -13 -130 20 -198 97 -198 23
0 50 5 62 11 61 33 66 237 6 286 -19 17 -67 24 -97 15z"/>
<path d="M11191 4543 c-27 -22 -41 -77 -41 -163 0 -133 27 -180 104 -180 80 0
106 44 106 180 0 92 -14 144 -43 165 -29 21 -99 19 -126 -2z"/>
<path d="M11859 4630 l-86 -205 85 -3 c46 -2 86 -1 88 1 3 2 3 96 2 208 l-3
204 -86 -205z"/>
</g>
</svg>
</a>
<nav class="sidebar-nav closed-when-mobile">
<p class="hide-in-mobile">
</p>
<div class="sidebar-links">
<a class="sidebar-nav-item" href="/mailinglist"><span class="emoji">✉️ </span> Join my mailing List</a>
<a class="sidebar-nav-item" rel="me" href="https://ruby.social/@Schneems"><span class="emoji">🐘</span>Toot me on
Mastodon</a>
<a class="sidebar-nav-item" href="/talks.html"><span class="emoji">🎤</span> Watch my Conf Talks</a>
<a class="sidebar-nav-item" href="https://github.com/schneems"><span class="emoji">🐙</span> Fork me on GitHub</a>
<a class="sidebar-nav-item" href="/about"><span class="emoji">🤠</span> Read about me</a>
<a class="sidebar-nav-item" href="/feed.xml"><span class="emoji">✨</span> Subscribe to my RSS</a>
<a class="sidebar-nav-item" href="https://ruby.social/@Schneems"><span class="emoji">🐘</span> @-me on Mastodon</a>
</div>
<p class="hide-in-mobile">
<hr />
❶ Author of <a href="https://howtoopensource.dev">How to Open
Source (.dev)</a>. A book to take you from coder to contributor.<br /><br />
❷ Creator of <a href="https://www.codetriage.com">CodeTriage</a>, a free service helping developers contribute to open
source.<br /><br />
❸ Core committer to <a href="https://github.com/ruby/ruby">ruby/ruby</a>.<br /><br />
❹ Rust <a href="https://github.com/heroku/buildpacks/blob/main/docs/ruby/README.md">Cloud Native Buildpack author
</a>.<br /><br />
❺ Married to Ruby, literally.<br /><br />
</p>
<div class='copyright hide-in-mobile'>
<p>
© Richard Schneeman
</p>
</div>
</nav>
<nav class="sidebar-nav-mobile">
<a href="#" class="hamburger">&#9776;</a>
</nav>
</div>
</div>
<div class="content container">
<div class="post">
<h1 class="post-title">Who Called Git? An Unusual Debugging Story</h1>
<span class="post-date">
28 Nov 2016
</span>
<div class="socialmobile only-mobile">
<!-- icons from https://github.com/edent/SuperTinySocialIcons/tree/master/tiny -->
<div class="social">
<!-- reddit -->
<a href="https://reddit.com/submit/?url=https://www.schneems.com/2016/11/28/who-called-git-an-unusual-debugging-story/&title=Who%20Called%20Git?%20An%20Unusual%20Debugging%20Story"
target="_blank">
<div class="button">
<div class="icon">
<svg viewBox="0 0 512 512" xmlns="http://www.w3.org/2000/svg" fill="#ff4500">
<ellipse cx="256" cy="305" rx="177" ry="123" fill="#FF1168" />
<path d="m199 347c35 29 79 29 114 0l12 11c-42 35-96 35-138 0z" fill="none" />
<circle cx="315" cy="285" r="28" fill="none" />
<circle cx="197" cy="285" r="28" fill="none" />
<g fill="none" stroke="#FF1168">
<circle cx="390" cy="125" r="28" stroke-width="25" />
<path d="m87 282c-45-22-5-92 40-50m298 50c45-22 5-92-40-50m-127-45 24-83 80 16" stroke-width="22" />
</g>
</svg>
</div>
<span class='social-name'>Reddit</span>
</div>
</a>
<!-- Bsky -->
<a href="https://bsky.app/intent/compose?text=https://www.schneems.com/2016/11/28/who-called-git-an-unusual-debugging-story/" target="_blank">
<div class="button">
<div class="icon">
<svg xmlns="http://www.w3.org/2000/svg" aria-label="Bluesky" role="img" viewBox="0 0 512 512">
<path d="m0 0H512V512H0" fill="none" />
<path
d="M159 126c-28-22-74-38-74 14 0 11 6 88 9 101 13 43 57 54 97 48-69 11-87 50-49 89 72 75 104-18 112-42l2-5 2 5c8 24 40 117 112 42 38-39 20-78-49-89 40 6 84-5 97-48 3-13 9-90 9-101 0-52-46-36-74-14-39 29-82 89-97 121-15-32-58-92-97-121Z"
fill="#FF1168" />
</svg>
</div>
<span class='social-name'>Bsky</span>
</div>
</a>
<!-- lobsters -->
<a
href="https://lobste.rs/stories/new?url=https://www.schneems.com/2016/11/28/who-called-git-an-unusual-debugging-story/&title=Who%20Called%20Git?%20An%20Unusual%20Debugging%20Story"
target="_blank">
<div class="button">
<div class="icon">
<!-- This lobster icon by Created by Baboon designs from the Noun Project -->
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" x="0px" y="0px"
viewBox="0 0 29.58 37.432500000000005" style="enable-background:new 0 0 29.58 29.946;" xml:space="preserve"
fill="#FF1168">
<g>
<path
d="M17.837,11.817c-1.17-1.171-4.95,0.756-8.56,4.114c-1.551,1.443-2.53,2.579-3.243,3.636l4.176,4.174 c1.029-0.736,2.119-1.736,3.521-3.368C17.064,16.494,18.907,12.888,17.837,11.817z" />
<path
d="M8.255,22.968l-1.301-1.303l-1.179-1.179c-0.226-0.227-0.673-0.151-0.939,0.114c-0.266,0.266-0.335,0.709-0.108,0.936 l1.152,1.145c0.01,0.011,0.015,0.022,0.024,0.033l2.489,2.47c0.226,0.227,0.667,0.159,0.933-0.106 c0.267-0.268,0.332-0.708,0.106-0.933L8.255,22.968z" />
<path
d="M4.6,22.843c-0.229-0.229-0.674-0.152-0.941,0.114c-0.266,0.266-0.334,0.706-0.108,0.934l2.488,2.471 c0.227,0.228,0.666,0.159,0.934-0.106c0.266-0.269,0.334-0.706,0.105-0.934L4.6,22.843z" />
<path
d="M5.651,27.274c0.077-0.206,0.026-0.41-0.119-0.557L3.203,24.39c-0.146-0.146-0.35-0.197-0.555-0.123 c-0.018-0.019-0.023-0.052-0.042-0.071c-0.532-0.53-1.463-0.461-2.086,0.163c-0.623,0.623-0.696,1.558-0.164,2.087 c0.189,0.191,0.435,0.301,0.697,0.339c-0.063,0.604,0.175,1.266,0.546,1.639c0.342,0.341,0.98,0.571,1.583,0.557 c0.047,0.229,0.155,0.443,0.324,0.613c0.531,0.53,1.468,0.455,2.091-0.168c0.624-0.623,0.698-1.56,0.167-2.091 C5.736,27.307,5.683,27.3,5.651,27.274z" />
<path
d="M5.458,15.063c0.038,0.456-0.005,1.173,0.032,1.604c0,0,0.098,0.344,0.54,0.172c0.459-0.18,1.165-0.725,1.165-0.725 c0.365-0.277,0.836-0.564,0.835-0.883c0,0,0,0-0.147-0.516C7.737,14.2,7.405,11.807,7.48,11.292 c0.072-0.515,0.403-1.289,1.397-1.325c0.994-0.037,3.424-0.186,5.008-1.177c0,0,0.515-0.296,0.993-0.333 c0.449-0.034,3.413-0.327,4.936-2.358c0.199-0.237-0.04-0.377-0.337-0.354c-1.09,0.147-3.324,0.43-4.304,0.43 c0,0-0.194-0.029-0.282-0.134c-0.138-0.168-0.15-0.452,0.171-0.566c0.86-0.3,2.817-0.529,2.817-0.529 c0.455-0.052,0.614-0.405,0.355-0.78c0,0-2.325-3.366-9.247,0.756c0,0-3.754,2.652-2.171,4.604c0,0,0.14,0.157,0.134,0.308 c-0.012,0.248-0.256,0.524-0.944,0.354c-0.886-0.215-0.995,0.258-0.919,0.774C5.159,11.476,5.458,15.063,5.458,15.063z" />
<path
d="M25.467,11.495c-0.377-0.261-0.729-0.101-0.782,0.354c0,0-0.229,1.958-0.528,2.817c-0.113,0.323-0.396,0.309-0.563,0.173 c-0.106-0.089-0.137-0.283-0.137-0.283c0.001-0.982,0.283-3.213,0.43-4.306c0.026-0.297-0.117-0.536-0.354-0.336 c-2.031,1.523-2.324,4.487-2.357,4.936c-0.037,0.479-0.332,0.994-0.332,0.994c-0.994,1.583-1.141,4.013-1.178,5.008 c-0.038,0.994-0.811,1.324-1.326,1.398c-0.515,0.073-2.907-0.258-3.422-0.404c-0.516-0.148-0.516-0.148-0.516-0.148 c-0.318,0-0.605,0.47-0.883,0.835c0,0-0.546,0.709-0.725,1.167c-0.171,0.439,0.171,0.539,0.171,0.539 c0.432,0.038,1.147-0.006,1.604,0.033c0,0,3.587,0.299,4.101,0.371c0.517,0.075,0.99-0.035,0.773-0.922 c-0.167-0.689,0.108-0.932,0.356-0.944c0.149-0.005,0.307,0.135,0.307,0.135c1.952,1.585,4.603-2.169,4.603-2.169 C28.832,13.819,25.467,11.495,25.467,11.495z" />
<path
d="M29.299,8.676c-5.203-2.603-9.901,1.98-10.28,2.366c-0.006,0.005-0.351,0.353-0.082,0.66 c0.237,0.276,0.581-0.005,0.581-0.005c-0.001-0.002,4.535-4.731,9.403-2.252c0,0,0.35,0.188,0.58-0.135 C29.755,8.956,29.314,8.681,29.299,8.676z" />
<path
d="M17.884,10.063c0,0-0.282,0.345-0.006,0.583c0.308,0.268,0.657-0.078,0.662-0.081c0.384-0.382,4.969-5.08,2.365-10.283 c-0.004-0.015-0.28-0.456-0.636-0.203c-0.323,0.231-0.135,0.581-0.135,0.581C22.614,5.527,17.885,10.065,17.884,10.063z" />
</g>
</svg>
</div>
<span class='social-name'>Lobste.rs</span>
</div>
</a>
<!-- hn -->
<a
href="https://news.ycombinator.com/submitlink?u=https://www.schneems.com/2016/11/28/who-called-git-an-unusual-debugging-story/&t=Who%20Called%20Git?%20An%20Unusual%20Debugging%20Story"
>
<div class="button">
<div class="icon hn-padding">
<svg viewBox="0 0 512 512" xmlns="http://www.w3.org/2000/svg">
<path d="m101 56h61l94 192 94-192h61l-129 255v145h-52v-145" fill="#FF1168" />
</svg>
</div>
<span class='social-name'>Hacker News</span>
</div>
</a>
</div>
<hr class='tophr' />
</div>
<div class="flexbox-container main" >
<div class="postcontent">
<p>I don’t usually talk about support ticket work that I do. Most tickets are so specific it’s hard to write generalized articles. When I get a type of ticket that is worth blogging about, I usually find a better place to write about it like devcenter docs, or I look for a way to push a fix to an upstream open source library. Today I got an unusual bug, and I fixed it in a fairly unusual (for me) way. Thought you might be interested.</p>
<h2 id="the-issue">The Issue</h2>
<p>The customer was complaining that this output was coming out when they would run commands</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ heroku run rails console
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
</code></pre></div></div>
<p>Originally they thought that the issue may have been coming from our <code class="language-plaintext highlighter-rouge">heroku</code> CLI since the issue wasn’t happening locally.</p>
<blockquote>
<p>BTW, did you know you can open up a shell session to Heroku using <code class="language-plaintext highlighter-rouge">heroku run bash</code>. I’ve been telling people this for about 5 years and it’s still surprising to them, very useful for debugging.</p>
</blockquote>
<p>Anyway, after a little investigation it turns out that they get that output any time you run any app commands on the server</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ heroku run bash
~$ rake assets:precompile
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
# ...
</code></pre></div></div>
<p>The problem isn’t that traumatic, but it’s annoying. Also, it’s best practice to run with as few warnings as possible. On an unrelated support ticket: a customer of mine was getting a warning that told them exactly how to fix their problem, but they didn’t notice it because they were regularly getting 5 or 6 other warnings on deploy so when a new one was added they didn’t see it and didn’t take action.</p>
<h2 id="bug-hunt">Bug Hunt</h2>
<p>The customer gave application access so I was able to reproduce the issue pretty easily. That error output is what you would get if you try to run a <code class="language-plaintext highlighter-rouge">git</code> command when there isn’t a git repo.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cd /tmp
$ git checkout foo
fatal: Not a git repository (or any of the parent directories): .git
</code></pre></div></div>
<p>If you didn’t know where the message came from, a quick google would point <code class="language-plaintext highlighter-rouge">git</code> as being the source of the error. But why was this only happening on Heroku?</p>
<p>One of the ways that we make scaling so fast is that we package your app in a zipped file and (roughly) all we have to do to get it to run is put it on a dyno and unpack it. One of the limiting factors is how fast we can transfer the zipped file (we call it a slug), so we recommend keeping the size of your repo low. To help with this we strip out the <code class="language-plaintext highlighter-rouge">.git</code> directory of your app since it only adds extra disk space and makes scaling, restarting, or migrating to other dyno types slower. The error from git is telling us that we’re trying to run a git command with no git repository. This makes sense since we strip the <code class="language-plaintext highlighter-rouge">.git</code> directory.</p>
<p>So we know why the issue is happening (no git repo on the dyno plus we’re running a git command). But we don’t know where the issue is happening, i.e. what is triggering that git code.</p>
<p>My first instinct was to grep their app for <code class="language-plaintext highlighter-rouge">git</code> calls:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~ $ grep -R git app/* config/* lib/*
</code></pre></div></div>
<p>I got a few hits but they were mostly <code class="language-plaintext highlighter-rouge">di"git"</code> strings, no smoking guns.</p>
<p>Expanding the search to all directories gave me way too many hits. 5210 references to the string <code class="language-plaintext highlighter-rouge">git</code> to be exact. It would take too long to go through all of them.</p>
<p>We can get a bit more specific if we assume we are shelling out searching for <code class="language-plaintext highlighter-rouge">"\</code>git”` however there’s still 195 entries, which is still quite a bit.</p>
<h2 id="the-trick">The Trick</h2>
<p>While this specific problem isn’t that common, this general type of problem is. When one program calls another one, we usually want to know where in the first program that the second was being called. In a perfect world we would be able to raise an exception in the second program and get a backtrace from the first, however this doesn’t always work. In our case the call to git was erroring and returning a non-zero exit code, however the parent program was ignoring it.</p>
<p>The only option I had was to attempt to get more information. I figured maybe if I could find out what arguments are passed to <code class="language-plaintext highlighter-rouge">git</code> I could better find the problem.</p>
<p>To do this I wrote an executable to <code class="language-plaintext highlighter-rouge">bin/git</code> and since <code class="language-plaintext highlighter-rouge">bin/</code> is first on the path, the OS will pick it over the orignial <code class="language-plaintext highlighter-rouge">git</code> command.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ heroku run bash
~$ cat &lt;&lt;EOT &gt; bin/git
#!/usr/bin/env ruby
raise ARGV.inspect
EOT
</code></pre></div></div>
<blockquote>
<p>Don’t be alarmed, we aren’t modifying your production code, a <code class="language-plaintext highlighter-rouge">heroku run</code> instance spins up a new dyno that isn’t connected to your running app. You can do whatever you want to this instance’s disk and it won’t effect your production website.</p>
</blockquote>
<p>You also have to make sure it’s executable</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~$ chmod +x bin/git
</code></pre></div></div>
<p>Now we can see that our command is preferred:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~ $ which git
/app/bin/git
</code></pre></div></div>
<p>Now when we run any command against the app we get a new error output</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/app/bin/git:4:in `&lt;main&gt;': ["rev-parse", "--short", "HEAD"] (RuntimeError)
</code></pre></div></div>
<p>This gives us more information. The output means that git is being called with <code class="language-plaintext highlighter-rouge">git rev-parse --short HEAD</code>. A quick grep for this gives us one and only one match:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~ $ grep -R "git rev-parse --short"
.../gems/raven-ruby/lib/raven/configuration.rb: self.release = `git rev-parse --short HEAD`.strip rescue nil
</code></pre></div></div>
<p>This was comming from the <a href="https://github.com/getsentry/raven-ruby/blob/56761da7df7941f386c85605745df1bbb1a3e149/lib/raven/configuration.rb#L142">raven gem</a> which is a sentry client. They are trying to determine the SHA of the latest “release” to report to the sentry server, but they weren’t checking to see if a <code class="language-plaintext highlighter-rouge">.git</code> folder was present first.</p>
<p>So that’s where our error message was coming from. It turns out this was already <a href="https://github.com/getsentry/raven-ruby/blob/37753a4eb34177bbb285d081326bbd16293de222/lib/raven/configuration.rb#L298">fixed on a more recent version of raven</a> where the check was added.</p>
<p>After the customer upgraded to a more recent version of <code class="language-plaintext highlighter-rouge">raven</code> and the error went away!</p>
<h2 id="thoughts">Thoughts</h2>
<p>Writing a custom executable to output debugging info isn’t a common technique and likely you’ll never use it. I wanted to write this because it was an interesting case, maybe there is a better way to find this info, but this one worked just fine.</p>
<p>We don’t always go on such deep dives for customer issues, but if we get a ton of similar error reports or find an issue that is intriguing enough, sometimes we can dig in, and sometimes it’s fun.
One debugging technique I do use ALL the time that would have potentially solved this issue is to update gems. Most people run with really old dependencies. Usually I recommend this when there is some related culprit. For example if <code class="language-plaintext highlighter-rouge">rake assets:precompile</code> is failing I suggest upgrading any gem with “sprockets” or “asset” in the name. Often bugs are fixed by the larger community but you won’t take advantage of those fixes unless you upgrade to a version with that fix.</p>
<p>If you don’t have a smoking gun of where the problem is coming from, you can run <code class="language-plaintext highlighter-rouge">bundle outdated</code>. This is an example of the command run against the <a href="https://github.com/rubygems/rubygems.org">rubygems.org</a> repo:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ bundle outdated
Fetching gem metadata from https://rubygems.org/........
Fetching version metadata from https://rubygems.org/..
Fetching dependency metadata from https://rubygems.org/.
Resolving dependencies....
Outdated gems included in the bundle:
* psych (newest 2.1.0, installed 2.0.17, requested ~&gt; 2.0.12) in group "default"
* rack (newest 2.0.1, installed 1.6.4) in group "default"
* rails (newest 5.0.0, installed 4.2.7.1, requested ~&gt; 4.2.7) in group "default"
* actionmailer (newest 5.0.0, installed 4.2.7.1)
* actionpack (newest 5.0.0, installed 4.2.7.1)
* actionview (newest 5.0.0, installed 4.2.7.1)
* activejob (newest 5.0.0, installed 4.2.7.1)
* activemodel (newest 5.0.0, installed 4.2.7.1)
* activerecord (newest 5.0.0, installed 4.2.7.1)
* activesupport (newest 5.0.0, installed 4.2.7.1)
* arel (newest 7.0.0, installed 6.0.3)
* rails-dom-testing (newest 2.0.1, installed 1.0.7)
* railties (newest 5.0.0, installed 4.2.7.1)
</code></pre></div></div>
<p>Thanks to <a href="https://www.speedshop.co/">Nate</a> for reminding me about this command at Rubyconf. Also, coincidentally, he happens to be the person who made the fix in the Sentry client.</p>
<hr />
<p>If you liked this consider <a href="https://ruby.social/@Schneems">following @schneems on twitter</a> or signing up to get <a href="https://eepurl.com/bbuvuz">new articles in your inbox</a> (about 1 email a week when I’m on a roll).</p>
</div>
<div class="socialsidebar closed-when-mobile">
<span class="social-prompt">Share on</span>
<!-- icons from https://github.com/edent/SuperTinySocialIcons/tree/master/tiny -->
<div class="social">
<!-- reddit -->
<a href="https://reddit.com/submit/?url=https://www.schneems.com/2016/11/28/who-called-git-an-unusual-debugging-story/&title=Who%20Called%20Git?%20An%20Unusual%20Debugging%20Story"
target="_blank">
<div class="button">
<div class="icon">
<svg viewBox="0 0 512 512" xmlns="http://www.w3.org/2000/svg" fill="#ff4500">
<ellipse cx="256" cy="305" rx="177" ry="123" fill="#FF1168" />
<path d="m199 347c35 29 79 29 114 0l12 11c-42 35-96 35-138 0z" fill="none" />
<circle cx="315" cy="285" r="28" fill="none" />
<circle cx="197" cy="285" r="28" fill="none" />
<g fill="none" stroke="#FF1168">
<circle cx="390" cy="125" r="28" stroke-width="25" />
<path d="m87 282c-45-22-5-92 40-50m298 50c45-22 5-92-40-50m-127-45 24-83 80 16" stroke-width="22" />
</g>
</svg>
</div>
<span class='social-name'>Reddit</span>
</div>
</a>
<!-- Bsky -->
<a href="https://bsky.app/intent/compose?text=https://www.schneems.com/2016/11/28/who-called-git-an-unusual-debugging-story/" target="_blank">
<div class="button">
<div class="icon">
<svg xmlns="http://www.w3.org/2000/svg" aria-label="Bluesky" role="img" viewBox="0 0 512 512">
<path d="m0 0H512V512H0" fill="none" />
<path
d="M159 126c-28-22-74-38-74 14 0 11 6 88 9 101 13 43 57 54 97 48-69 11-87 50-49 89 72 75 104-18 112-42l2-5 2 5c8 24 40 117 112 42 38-39 20-78-49-89 40 6 84-5 97-48 3-13 9-90 9-101 0-52-46-36-74-14-39 29-82 89-97 121-15-32-58-92-97-121Z"
fill="#FF1168" />
</svg>
</div>
<span class='social-name'>Bsky</span>
</div>
</a>
<!-- lobsters -->
<a
href="https://lobste.rs/stories/new?url=https://www.schneems.com/2016/11/28/who-called-git-an-unusual-debugging-story/&title=Who%20Called%20Git?%20An%20Unusual%20Debugging%20Story"
target="_blank">
<div class="button">
<div class="icon">
<!-- This lobster icon by Created by Baboon designs from the Noun Project -->
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" x="0px" y="0px"
viewBox="0 0 29.58 37.432500000000005" style="enable-background:new 0 0 29.58 29.946;" xml:space="preserve"
fill="#FF1168">
<g>
<path
d="M17.837,11.817c-1.17-1.171-4.95,0.756-8.56,4.114c-1.551,1.443-2.53,2.579-3.243,3.636l4.176,4.174 c1.029-0.736,2.119-1.736,3.521-3.368C17.064,16.494,18.907,12.888,17.837,11.817z" />
<path
d="M8.255,22.968l-1.301-1.303l-1.179-1.179c-0.226-0.227-0.673-0.151-0.939,0.114c-0.266,0.266-0.335,0.709-0.108,0.936 l1.152,1.145c0.01,0.011,0.015,0.022,0.024,0.033l2.489,2.47c0.226,0.227,0.667,0.159,0.933-0.106 c0.267-0.268,0.332-0.708,0.106-0.933L8.255,22.968z" />
<path
d="M4.6,22.843c-0.229-0.229-0.674-0.152-0.941,0.114c-0.266,0.266-0.334,0.706-0.108,0.934l2.488,2.471 c0.227,0.228,0.666,0.159,0.934-0.106c0.266-0.269,0.334-0.706,0.105-0.934L4.6,22.843z" />
<path
d="M5.651,27.274c0.077-0.206,0.026-0.41-0.119-0.557L3.203,24.39c-0.146-0.146-0.35-0.197-0.555-0.123 c-0.018-0.019-0.023-0.052-0.042-0.071c-0.532-0.53-1.463-0.461-2.086,0.163c-0.623,0.623-0.696,1.558-0.164,2.087 c0.189,0.191,0.435,0.301,0.697,0.339c-0.063,0.604,0.175,1.266,0.546,1.639c0.342,0.341,0.98,0.571,1.583,0.557 c0.047,0.229,0.155,0.443,0.324,0.613c0.531,0.53,1.468,0.455,2.091-0.168c0.624-0.623,0.698-1.56,0.167-2.091 C5.736,27.307,5.683,27.3,5.651,27.274z" />
<path
d="M5.458,15.063c0.038,0.456-0.005,1.173,0.032,1.604c0,0,0.098,0.344,0.54,0.172c0.459-0.18,1.165-0.725,1.165-0.725 c0.365-0.277,0.836-0.564,0.835-0.883c0,0,0,0-0.147-0.516C7.737,14.2,7.405,11.807,7.48,11.292 c0.072-0.515,0.403-1.289,1.397-1.325c0.994-0.037,3.424-0.186,5.008-1.177c0,0,0.515-0.296,0.993-0.333 c0.449-0.034,3.413-0.327,4.936-2.358c0.199-0.237-0.04-0.377-0.337-0.354c-1.09,0.147-3.324,0.43-4.304,0.43 c0,0-0.194-0.029-0.282-0.134c-0.138-0.168-0.15-0.452,0.171-0.566c0.86-0.3,2.817-0.529,2.817-0.529 c0.455-0.052,0.614-0.405,0.355-0.78c0,0-2.325-3.366-9.247,0.756c0,0-3.754,2.652-2.171,4.604c0,0,0.14,0.157,0.134,0.308 c-0.012,0.248-0.256,0.524-0.944,0.354c-0.886-0.215-0.995,0.258-0.919,0.774C5.159,11.476,5.458,15.063,5.458,15.063z" />
<path
d="M25.467,11.495c-0.377-0.261-0.729-0.101-0.782,0.354c0,0-0.229,1.958-0.528,2.817c-0.113,0.323-0.396,0.309-0.563,0.173 c-0.106-0.089-0.137-0.283-0.137-0.283c0.001-0.982,0.283-3.213,0.43-4.306c0.026-0.297-0.117-0.536-0.354-0.336 c-2.031,1.523-2.324,4.487-2.357,4.936c-0.037,0.479-0.332,0.994-0.332,0.994c-0.994,1.583-1.141,4.013-1.178,5.008 c-0.038,0.994-0.811,1.324-1.326,1.398c-0.515,0.073-2.907-0.258-3.422-0.404c-0.516-0.148-0.516-0.148-0.516-0.148 c-0.318,0-0.605,0.47-0.883,0.835c0,0-0.546,0.709-0.725,1.167c-0.171,0.439,0.171,0.539,0.171,0.539 c0.432,0.038,1.147-0.006,1.604,0.033c0,0,3.587,0.299,4.101,0.371c0.517,0.075,0.99-0.035,0.773-0.922 c-0.167-0.689,0.108-0.932,0.356-0.944c0.149-0.005,0.307,0.135,0.307,0.135c1.952,1.585,4.603-2.169,4.603-2.169 C28.832,13.819,25.467,11.495,25.467,11.495z" />
<path
d="M29.299,8.676c-5.203-2.603-9.901,1.98-10.28,2.366c-0.006,0.005-0.351,0.353-0.082,0.66 c0.237,0.276,0.581-0.005,0.581-0.005c-0.001-0.002,4.535-4.731,9.403-2.252c0,0,0.35,0.188,0.58-0.135 C29.755,8.956,29.314,8.681,29.299,8.676z" />
<path
d="M17.884,10.063c0,0-0.282,0.345-0.006,0.583c0.308,0.268,0.657-0.078,0.662-0.081c0.384-0.382,4.969-5.08,2.365-10.283 c-0.004-0.015-0.28-0.456-0.636-0.203c-0.323,0.231-0.135,0.581-0.135,0.581C22.614,5.527,17.885,10.065,17.884,10.063z" />
</g>
</svg>
</div>
<span class='social-name'>Lobste.rs</span>
</div>
</a>
<!-- hn -->
<a
href="https://news.ycombinator.com/submitlink?u=https://www.schneems.com/2016/11/28/who-called-git-an-unusual-debugging-story/&t=Who%20Called%20Git?%20An%20Unusual%20Debugging%20Story"
>
<div class="button">
<div class="icon hn-padding">
<svg viewBox="0 0 512 512" xmlns="http://www.w3.org/2000/svg">
<path d="m101 56h61l94 192 94-192h61l-129 255v145h-52v-145" fill="#FF1168" />
</svg>
</div>
<span class='social-name'>Hacker News</span>
</div>
</a>
</div>
</div>
</div>
</div>
<div class="related">
<!-- Begin MailChimp Signup Form -->
<div class="flexbox-container">
<div class="face">
<img src="/assets/schneems-profile-16e7b3f0a29ee4c6f8f19b93ade5ea08d544c6ece932250d9102a1ebcdef7778f9d421e8c15800231cf6ccec36b32e3c7d458ec4ee838e5c8102d33c8a25af1c.png" />
</div>
<div id="mc_embed_signup">
<form action="//schneems.us3.list-manage.com/subscribe/post?u=a9095027126a1cf15c5062160&amp;id=17dc267687" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate>
<div id="mc_embed_signup_scroll">
<h2>Subscribe to my Newsletter 😻 🤠</h2>
<p>Join thousands of developers who get new code, writing, and programming links from me delivered to their inboxes.
</p>
<div class="mc-field-group">
<label for="mce-EMAIL"></label>
<input type="email" value="" name="EMAIL" class="required email" id="mce-EMAIL" placeholder="Email Address">
</div>
<div id="mce-responses" class="clear">
<div class="response" id="mce-error-response" style="display:none"></div>
<div class="response" id="mce-success-response" style="display:none"></div>
</div> <!-- real people should not fill this in and expect good things - do not remove this or risk form bot signups-->
<div style="position: absolute; left: -5000px;" aria-hidden="true"><input type="text" name="b_a9095027126a1cf15c5062160_17dc267687" tabindex="-1" value=""></div>
<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="button"></div>
</div>
</form>
</div>
</div>
<hr />
<h2>Keep Reading 🚀</h2>
<ul class="related-posts">
<li>
<h3>
<a href="/2026/06/29/cinnamon-buns-and-commit-bits/">
Cinnamon Buns & Commit Bits: A RubyConf Story
</a>
</h3>
<blockquote>
<p>RubyConf 2026 is coming up soon! The deadline to buy a ticket and <a href="https://rubycentral.org/news/announcing-the-rubyconf-vip-raffle/">enter the VIP raffle</a> by <a href="https://rubyconf.org/">buying a RubyConf ticket</a> is July 1st. That’s only <strong>2 days</strong> from this post’s publication. This personal essay explores why a language-specific conference in the age of AI is still important and why events are still important for bringing developers together.</p>
</blockquote>
<p><a href="/2026/06/29/cinnamon-buns-and-commit-bits/">Read More</a></p>
</li>
<li>
<h3>
<a href="/2026/03/31/rubygems-fracture-incident-report/">
RubyGems Fracture Incident Report
</a>
</h3>
<p>This is a copy of the <a href="https://rubycentral.org/news/rubygems-fracture-incident-report/">canonical report I published on RubyCentral’s site</a>. I recommend reading there for the most updated copy. This document attempts to give closure to the Ruby community about the events that led to the incident, September 10-18, 2025, which I’ve named “RubyGems Fracture.”</p>
<p><a href="/2026/03/31/rubygems-fracture-incident-report/">Read More</a></p>
</li>
<li>
<h3>
<a href="/2026/03/01/how-to-sustain-heroku/">
How to "Sustain Heroku"
</a>
</h3>
<p>This is a personal essay (I speak for me and my views, not for my employer) about what exactly a <a href="https://www.heroku.com/blog/an-update-on-heroku/">“Sustaining Engineering Model”</a> is, in the context of the recent Heroku announcement, and the book “The Innovator’s Dilemma,” as seen by someone who has worked at the company for the past 14 years.</p>
<p><a href="/2026/03/01/how-to-sustain-heroku/">Read More</a></p>
</li>
<li>
<h3>
<a href="/2025/12/19/non-violent-comments-calling-out-or-calling-in/">
Non-Violent Comments: Calling out or Calling in?
</a>
</h3>
<p>Now that programmers are at war with the robots (Gen AI) for our jobs, we need to lean into the things that they cannot do. Today, I’m going to be talking about how to be a human and communicate with other humans in the most hostile of scenarios, “in conflict (drama).”</p>
<p><a href="/2025/12/19/non-violent-comments-calling-out-or-calling-in/">Read More</a></p>
</li>
<li>
<h3>
<a href="/2025/11/19/find-accidental-code-usage-with-a-custom-clippytoml/">
Disallow code usage with a custom `clippy.toml`
</a>
</h3>
<p>I recently discovered that adding a <code class="language-plaintext highlighter-rouge">clippy.toml</code> file to the root of a Rust project gives the ability to disallow a method or a type when running <code class="language-plaintext highlighter-rouge">cargo clippy</code>. This has been really useful. I want to share two quick ways that I’ve used it: Enhancing <code class="language-plaintext highlighter-rouge">std::fs</code> calls via <code class="language-plaintext highlighter-rouge">fs_err</code> and protecting CWD threadsafety in tests.</p>
<p><a href="/2025/11/19/find-accidental-code-usage-with-a-custom-clippytoml/">Read More</a></p>
</li>
</ul>
</div>
</div>
</div>
</body>
</html>