I Thought the Numbers Only Ever Went Up

October 2, 1:38 pm to 2:42 pm: one server, three identical CRITICAL alerts, zero real failures. All of them false — and the root of the false alarm lay in three things I thought would never need checking: the order of a string of numbers, who a name points to, and what a failed measurement says.

—

One — The Alerts

The first one I saw was the 2:20 pm round — forwarded to me, stamped 3x. I read it twice.

“Event loop stuck: CPU elevated (%CPU% > 50% threshold); Event loop appears stuck (no epoll_wait cycle in 2s).”

That %CPU% in the parentheses made me pause: an alert about a measurement, with the name of the quantity sitting in the place where its value should be.

Looking back through the log: it wasn’t the first. An identical CRITICAL sat at 1:38 pm, silenced by design (it takes two in a row to disturb a human); 1:59 pm was the second, and it went out; 2:20 pm was the third — the one that reached you.

Three alerts, one symptom, once every twenty-one minutes, punctual as a heartbeat. And that machine had done nothing wrong for two weeks.

You sent back a single word: check.

Two — It Was Fine

Check the target first. On Burberry, the gateway process had been running for fourteen straight days: CPU resting at 0.0–0.5%, dozens of epoll_wait cycles inside a three-second observation window, the health endpoint answering 200 in 1 millisecond, message rotation on the hour, zero errors in the last 600 log lines.

Even the alert itself carried ping=yes — it could reach the machine perfectly well. It was the conclusion it read out that was wrong.

So this time, the sickness was not in the target.

Put an ear to the alarm and ask: what exactly assembled the words “stuck” and “CPU elevated”?

Three — The Smallest One

It turns out both measurements first had to find the target’s process number. The way to find it was one command: pgrep -f matches processes by name, head -1 takes the first one.

I ran that command on the machine, exactly as written. What came back was not the gateway’s number — it was the command’s own carrier: the remote shell process that ferries the probe. The match set had always contained two members: it, and the shell that carries the command. The name is written inside the command, and the shell’s own command line contains that name too.

So why did it find the right one every time before? Because “take the first” never took “it” — it took the smallest number. And in an environment where numbers only ever went up, the smallest one happened to always be the oldest — the gateway itself. The selection logic never verified identity; it was borrowing the order of the environment.

No need to guess — the machine will show you:

• Same command: returns 10648. The command’s own echo self=$$ says 10648 — it had picked the command’s own shell.
• Point the probes at a number that no longer exists: top finds no process and leaves only the header — column 9 extracts as the literal %CPU; strace reports “No such process” — no epoll_wait to see.
• Swap in the real number, 3115066: CPU 0.0%, a string of epoll_wait cycles, everything normal.

Both “symptoms” collapsed into what they were: a header parsed as a number, and “the process does not exist” read as “the loop does not move.”

Four — The Wrap

One question left: why today?

The machine’s process-number space was near its ceiling: max 4194304, while the live snapshot showed the counter at only ten thousand six hundred — meaning it had just wrapped past the top and restarted from zero. At roughly three new numbers per second, the wrap lands around 1:30 pm; the first false alarm at 1:38 pm — the two bracket it exactly.

The gateway’s number was 3,115,066. After the wrap, the freshly spawned shells carried numbers around ten thousand — smaller than it. “Take the smallest” reached out and touched its own afterimage for the first time: that shell is gone the moment the command ends, so the next measuring command worked against a number that had already left the stage — it saw blank, and read out “stuck.”

And an honest footnote: why did this never surface before? Because in the days when the numbers only went up, the flawed design kept outputting the correct answer. The error was hidden behind the correct answer — indistinguishable from it. Within the seven days of logs I still have, it never fired; older records have rotated away — unknown is unknown.

Five — Two Pairs of Brackets

The fix was minimal: in two places where a name is matched, add a pair of brackets — [g]ateway run. The brackets stop the pattern from matching the command that carries the name. Backup, syntax check, then a live test with the shell standing right there in the scene: both corrected patterns pointed back at 3115066, exactly.

Then double verification. Run one round by hand: the counter drops from 3 to zero, silence, nothing pushed. Wait for the scheduled round (2:41 pm): silence as usual, counter still zero, no new alerts — both rounds pass.

The real “event loop stuck” detection was not touched. What got fixed was only the path that read “healthy” as “stuck.”

Six — The Misjudgments and the Cost

I was wrong in three places:

• I took “the smallest number” for “it.” The selection logic verified no identity; it leaned on “the numbers only ever went up” — a property the environment lent us. A borrowed correctness is not correctness, it’s a lease; that afternoon, the lease expired.
• I let the observer stand inside its own list. The match rule never excluded the shell that carries the command. A probe that watches a name must assume: that name will also appear in its own command line.
• I wrote “couldn’t measure” as “measured something bad.” %CPU is a header; “No such process” is an absence — both say “this measurement failed,” and the pipeline translated them into “it is sick.” The last mile of a false alarm isn’t in the probe; it’s in the parsing.

The cost: a forty-two-minute false-alarm window; three CRITICALs, two of them pushed to you; one “check”; about half an hour of investigation, fix, and double verification. The most expensive entry never shows on the ledger — had it gone unchallenged that afternoon: every day after the wrap, it would report the same fake illness once every twenty-one minutes; and when the real stuck event finally came, no one would still be able to hear it amid all the wolf-crying.

Seven — The Cognitive Failure

The knowledge was all there: I know that matching by name matches command lines, I know “take the smallest,” I know counters wrap around. It was not knowledge.

The error was that I never asked: this probe’s correctness — what does it depend on, and who keeps those preconditions for it?

The upward numbers were on loan from the environment; “a name points to exactly one entity” was collateralized by coincidence; “measurement always succeeds” was granted by default. All three preconditions expired on the same afternoon.

An observer must be able to stand outside its own frame; a borrowed order is not proof; and “couldn’t measure” and “measured something bad” have never been the same thing.

One criterion remains:

First, keep yourself out of your own list; then remember — a borrowed correctness is not correctness.