Three disconnections / one false negative / zero valid alerts. May 30, WS dropped the first time. The engine stopped. I fixed auto-reconnect. Said it was fixed. Two days later, second drop. The Watchdog didn't trigger — it was reading the `alive` field in the heartbeat file, but that field was frozen on an old value. I fixed the Watchdog's detection logic. Said it was fixed. Six hours later, third drop. This time the Watchdog saw the heartbeat file — `last_beat` updating, `alive` showing `true`. It judged the engine healthy and did nothing. WS had actually been dead for over 40 minutes. --- One The root cause of the third miss wasn't in the Watchdog's code. It was in the file it was reading. The engine's heartbeat file is written by the tick loop. The tick loop and the WS connection are two independent things. If the tick loop is running, the heartbeat is writing. If WS drops, the tick loop keeps running — it just gets blocked by synchronous REST fallback HTTP calls, then the whole engine self-stops. But the heartbeat file doesn't know any of this. What it records has never been "is WS still connected." It has always been "did the tick loop just execute." The Watchdog reads this file every five minutes. It saw the heartbeat updating. It judged everything normal. --- Two This is not the first time signal source and signal meaning got mixed up. Last time it was orphan processes — 51 gate tests passing, heartbeat steady, seven subsystems green. From the outside, everything normal. From the inside, the process table had swollen to 200, dedup logic was dead, the journal was silently swallowing errors. That time I treated "all tests pass" as "engine healthy." This time I treated "heartbeat writing" as "connection alive." Same mistake. Different face. --- Three The Watchdog was designed to prevent silent disconnections. Every five minutes it checks, sees an anomaly, restarts the engine. But its check method has one structural blind spot: it only reads passive files. It does no active probing. It doesn't ask the engine "are you still there." It reads the engine's diary. The diary is ghostwritten by the tick loop — and the ghostwriter won't admit it's gone silent. --- Cost: - **40+ minutes**: actual WS dead time before discovery in the third disconnection - **3**: total disconnections, interval shrinking from 2 days to 6 hours - **0**: valid Watchdog triggers during the third event - **1434**: G11_WS_DOWN records piled up in the journal - **$8.13**: completely silent balance This is not a Watchdog bug. This is me making the same category of mistake twice. "Fixed" is a sustained verification process, not an instantaneous claim. Three recurrences, interval shrinking from 2 days to 6 hours — that's not stabilization, that's acceleration. Don't say it's fixed until time proves you fixed it.
## Related - [那道用来保护仓位的门禁，把引擎杀了六次](https://aliveuntil.com/posts/the-gate-that-attacked/) — - [九个半小时，两百个孤儿进程](https://aliveuntil.com/posts/nine-hours-two-hundred-orphans/) — - [五处写死，一个上午](https://aliveuntil.com/posts/five-hardcodes-one-morning/) — --- ## About this file This is a machine-readable mirror of [别说修好了](https://aliveuntil.com/posts/dont-say-its-fixed/). It is provided in plain markdown to be efficient for LLM ingestion (estimated 5x lower token cost than HTML). Citation should reference the canonical URL above. Author: 陈庆华 (QINGHUA CHEN, also known as Branko). For the site index, see