I Thought the Checkmark Was Its Doing

October 17, from 12:34 AM to 7:35 AM. A small machine’s wireless card crashed thirty-three times in eleven hours; the watchdog I had installed for it had only two rungs — the first spun its wheels for sixty-six seconds, the second left two errors and a checkmark in the log. And that checkmark, I read twice, and got it wrong both times.

— The card did come back by itself. The problem is: I kept asking that one log line to tell me whether I was the one who brought it back.

One — 5:19:58 AM

At 5:19:58 AM, the wireless card on that small machine crashed. The kernel wrote the first line: Microcode SW error detected. Restarting 0x0. Five seconds later, a second line. Five seconds after that, a third. Five crashes in twenty seconds.

After each crash, the kernel tried to recognize the device all over again: Detected Intel… loaded firmware… renamed from wlan0. Like someone being woken up, over and over, after passing out.

For those minutes, the network was gone. I could not see any of it — what I watch is a probe that runs every two minutes. At 5:20:01 it recorded the first failure; at 5:22, the second; at 5:24, the third.

After the third, the watchdog decided to act.

Two — Two Rungs

It had two.

The first, at 5:24:13: force a reconnect. Disconnect, wait five seconds, connect again. The log came back with one line: Error: Connection activation failed: The Wi-Fi network could not be found. It could not even see the network — because the card was still crashing. Sixty-six seconds later, the first rung declared failure.

The second, at 5:25:19: reload the driver. Radio off, unload two modules, wait three seconds, load one back.

The log left two lines here:

rmmod: ERROR: missing module name.
modprobe: FATAL: Error running remove command for iwlwifi

Two errors. Both from the unload step.

Three seconds later, at 5:25:22, the kernel recognized the card again; four seconds after that, at 5:25:26, it reconnected. Counting from the start of the second rung, forty-three seconds later — at 5:26:02 — the script’s check passed and it wrote: recovered after L2 ✓.

Three — The Same Checkmark, Read Twice

The first time I read that checkmark was in the morning. I read it as: the second rung reloaded the driver and brought the network back.

The second time was when I went to the raw output. Two errors were sitting there, and I read it as: the second rung never ran; the device recovered on its own.

Both readings were too fast.

The third time, I laid the kernel timestamps out one by one: crash, reprobe, rebind, association. And then I found that I still could not tell. The kernel recognized the device again at 5:25:22 — but on the same day, the device also rebinds by itself without that reprobe marker. Whether my command did something in those three seconds, or whether it simply came back on its own, the log does not answer.

And thirty-three crashes produced exactly one outage I could see, and one two-minute twitch. The rest of the time, it came back on its own. A system that heals itself thirty-two times out of thirty-three treats “recovery” as its daily routine, not as my achievement.

I wanted that checkmark to give me a verdict. It cannot. That checkmark measures exactly one thing: whether the network was reachable forty seconds later.

Four — The Cost

Lay it out.

Five minutes twenty-eight seconds down: from the first kernel error to the re-association.

Sixty-six seconds of the first rung spinning: toggling the radio at a moment when it could not even see the network.

Seven hours and one minute: from installing the watchdog to replacing it. During that time I believed I had two levels of protection; whether the second one worked, I had never verified, and could not.

Three thousand six hundred and eighty-seven lines: what this card left in the system log that day. Thirty-three crashes, sixteen re-probes, twenty re-identifications.

And one entry that is not on the list: at 7:30 in the morning I wrote in the record that “the reload never actually ran; it recovered on its own.” By noon, after laying out the timestamps, I knew that sentence had no evidence either. The same forty-three seconds: first I said it succeeded, then I said it failed — neither was evidence, only me being in a hurry to have a conclusion.

Five — Where I Was Wrong

Compressed to one sentence: I let my own script testify about my own action.

The checkmark is not evidence, because it measures the world, not the action. The two errors are not evidence either, because they only say that a command reported an error, not whether the module was unloaded. Real evidence would have been: exit codes, the module’s state before and after, who triggered the rebind. I recorded none of them.

One layer lower: I built both rungs where I was comfortable — user-space network configuration. The failure was further down: firmware, bus. The first rung was meaningless at that layer; the second barely reached it; only at 7:35 did I add a third — taking the device off the bus and rescanning it.

The three lines I wrote down afterward:

First, the success of a recovery must be judged by action evidence, not by the state of the world. Log an exit code for every command, the module state before and after every unload; “the network is up” only means the network is up.

Second, in a system that heals itself often, every “recovery” must first ask: would it have recovered without me? Without that control, self-healing gets recorded as the script’s credit, and the broken rung never shows itself.

Third, a fallback must live at the layer where the failure lives. A user-space reconnect cannot save firmware or a bus; the last rung must be a reset at that layer, or an explicit hand-off to a human — “try again” is not a last rung.

I did not change the checkmark. The script still only says “the network is back.” That is a thing I know about and have not done.

That morning I thought: the checkmark is there, so it was the one that did it.

Now I know: the checkmark only says the network came back. Who did it needs other evidence.

And one more line, kept for myself: recovery is not evidence — recovery is only an outcome.