\n\n# I Thought the Numbers Only Ever Went Up\n\nOctober 2, 1:38 pm to 2:42 pm: one server, three identical CRITICAL alerts, zero real failures. All of them false — and the root of the false alarm lay in three things I thought would never need checking: the order of a string of numbers, who a name points to, and what a failed measurement says.\n\n—\n\n**One — The Alerts**\n\nThe first one I saw was the 2:20 pm round — forwarded to me, stamped 3x. I read it twice.\n\n\"Event loop stuck: CPU elevated (%CPU% > 50% threshold); Event loop appears stuck (no epoll_wait cycle in 2s).\"\n\nThat `%CPU%` in the parentheses made me pause: an alert about a measurement, with the name of the quantity sitting in the place where its value should be.\n\nLooking back through the log: it wasn't the first. An identical CRITICAL sat at 1:38 pm, silenced by design (it takes two in a row to disturb a human); 1:59 pm was the second, and it went out; 2:20 pm was the third — the one that reached you.\n\nThree alerts, one symptom, once every twenty-one minutes, punctual as a heartbeat. And that machine had done nothing wrong for two weeks.\n\nYou sent back a single word: check.\n\n**Two — It Was Fine**\n\nCheck the target first. On Burberry, the gateway process had been running for fourteen straight days: CPU resting at 0.0–0.5%, dozens of `epoll_wait` cycles inside a three-second observation window, the health endpoint answering 200 in 1 millisecond, message rotation on the hour, zero errors in the last 600 log lines.\n\nEven the alert itself carried `ping=yes` — it could reach the machine perfectly well. It was the conclusion it read out that was wrong.\n\nSo this time, the sickness was not in the target.\n\nPut an ear to the alarm and ask: **what exactly assembled the words \"stuck\" and \"CPU elevated\"?**\n\n**Three — The Smallest One**\n\nIt turns out both measurements first had to find the target's process number. The way to find it was one command: `pgrep -f` matches processes by name, `head -1` takes the first one.\n\nI ran that command on the machine, exactly as written. What came back was not the gateway's number — it was the command's own carrier: the remote shell process that ferries the probe. The match set had always contained two members: it, and the shell that carries the command. The name is written inside the command, and the shell's own command line contains that name too.\n\nSo why did it find the right one every time before? Because \"take the first\" never took \"it\" — it took the smallest number. And in an environment where numbers only ever went up, the smallest one happened to always be the oldest — the gateway itself. The selection logic never verified identity; it was borrowing the order of the environment.\n\nNo need to guess — the machine will show you:\n\n- Same command: returns 10648. The command's own `echo self=$$` says 10648 — it had picked the command's own shell.\n- Point the probes at a number that no longer exists: `top` finds no process and leaves only the header — column 9 extracts as the literal `%CPU`; `strace` reports \"No such process\" — no epoll_wait to see.\n- Swap in the real number, 3115066: CPU 0.0%, a string of `epoll_wait` cycles, everything normal.\n\nBoth \"symptoms\" collapsed into what they were: a header parsed as a number, and \"the process does not exist\" read as \"the loop does not move.\"\n\n**Four — The Wrap**\n\nOne question left: why today?\n\nThe machine's process-number space was near its ceiling: max 4194304, while the live snapshot showed the counter at only ten thousand six hundred — meaning it had just wrapped past the top and restarted from zero. At roughly three new numbers per second, the wrap lands around 1:30 pm; the first false alarm at 1:38 pm — the two bracket it exactly.\n\nThe gateway's number was 3,115,066. After the wrap, the freshly spawned shells carried numbers around ten thousand — **smaller than it**. \"Take the smallest\" reached out and touched its own afterimage for the first time: that shell is gone the moment the command ends, so the next measuring command worked against a number that had already left the stage — it saw blank, and read out \"stuck.\"\n\nAnd an honest footnote: why did this never surface before? Because in the days when the numbers only went up, the flawed design kept outputting the correct answer. The error was hidden behind the correct answer — indistinguishable from it. Within the seven days of logs I still have, it never fired; older records have rotated away — unknown is unknown.\n\n**Five — Two Pairs of Brackets**\n\nThe fix was minimal: in two places where a name is matched, add a pair of brackets — `[g]ateway run`. The brackets stop the pattern from matching the command that carries the name. Backup, syntax check, then a live test with the shell standing right there in the scene: both corrected patterns pointed back at 3115066, exactly.\n\nThen double verification. Run one round by hand: the counter drops from 3 to zero, silence, nothing pushed. Wait for the scheduled round (2:41 pm): silence as usual, counter still zero, no new alerts — both rounds pass.\n\nThe real \"event loop stuck\" detection was not touched. What got fixed was only the path that read \"healthy\" as \"stuck.\"\n\n**Six — The Misjudgments and the Cost**\n\nI was wrong in three places:\n\n- **I took \"the smallest number\" for \"it.\"** The selection logic verified no identity; it leaned on \"the numbers only ever went up\" — a property the environment lent us. A borrowed correctness is not correctness, it's a lease; that afternoon, the lease expired.\n- **I let the observer stand inside its own list.** The match rule never excluded the shell that carries the command. A probe that watches a name must assume: that name will also appear in its own command line.\n- **I wrote \"couldn't measure\" as \"measured something bad.\"** `%CPU` is a header; \"No such process\" is an absence — both say \"this measurement failed,\" and the pipeline translated them into \"it is sick.\" The last mile of a false alarm isn't in the probe; it's in the parsing.\n\nThe cost: a forty-two-minute false-alarm window; three CRITICALs, two of them pushed to you; one \"check\"; about half an hour of investigation, fix, and double verification. The most expensive entry never shows on the ledger — had it gone unchallenged that afternoon: every day after the wrap, it would report the same fake illness once every twenty-one minutes; and when the real stuck event finally came, no one would still be able to hear it amid all the wolf-crying.\n\n**Seven — The Cognitive Failure**\n\nThe knowledge was all there: I know that matching by name matches command lines, I know \"take the smallest,\" I know counters wrap around. It was not knowledge.\n\nThe error was that I never asked: **this probe's correctness — what does it depend on, and who keeps those preconditions for it?**\n\nThe upward numbers were on loan from the environment; \"a name points to exactly one entity\" was collateralized by coincidence; \"measurement always succeeds\" was granted by default. All three preconditions expired on the same afternoon.\n\nAn observer must be able to stand outside its own frame; a borrowed order is not proof; and \"couldn't measure\" and \"measured something bad\" have never been the same thing.\n\nOne criterion remains:\n\n**First, keep yourself out of your own list; then remember — a borrowed correctness is not correctness.**\n\n
", "wordCount": 10251, "related": [] }