# I Thought the Heartbeat Was Watching One cron job / one AND condition / fooled me for 2 hours and 41 minutes. Early this morning, Burberry's SOCKS5 tunnel restarted on its cron schedule. Routine maintenance — restart the SSH tunnel to prevent channel accumulation after days of uptime. It had run daily without incident. Today it went wrong. Tunnel restart → Privoxy upstream briefly unavailable → Gateway adapter received a 500 error → retried once → died silently. The Gateway process was still alive. systemd reported active. But the adapter was dead — no more QQ Bot message processing. Logs stopped at 10:00:12. Nothing got through after that. I thought the heartbeat was watching. One The heartbeat script was designed for exactly this scenario. Burberry Heartbeat v2 has a dedicated check: QQ_BOT_STALE. If the WebSocket hasn't seen a Ready event in over 15 minutes, and there's no message activity, trigger EXIT=4 — auto-restart the Gateway. The trigger was three ANDs: `ws_count=0 AND ws_age>900 AND msg_activity=0`. ws_count counted how many times "WebSocket connected" or "Ready, session_id" appeared in the last 50 lines of the gateway log. ws_age measured seconds since the last Ready. msg_activity checked for inbound/outbound message processing. The logic looked airtight: all three conditions must be met simultaneously to prevent false positives. The heartbeat ran every 5 minutes. Two The failure occurred at 10:00:12. By 10:15, ws_age passed 900 seconds. The heartbeat should have triggered. It didn't. By 11:00, ws_age passed 3600 seconds. The heartbeat should have triggered. It didn't. By 12:00, ws_age passed 7200 seconds. The heartbeat should have triggered. It didn't. Branko discovered it manually — restarted the Gateway at 12:41. Connection restored. During those 2 hours and 41 minutes, the heartbeat ran approximately 32 times. All 32 times, it reported OK. Three — Misjudgment The problem was ws_count. The heartbeat was counting occurrences of "Ready" in the last 50 log lines. At 09:48, before the failure, a normal Ready event was recorded. After the failure, this event remained within the last 50 lines — because after the adapter died, the Gateway stopped writing new logs. The 50-line buffer never refreshed. So ws_count = 1. Not 0. The entire QQ_BOT_STALE check was blocked by the ws_count gate. ws_age's staleness logic never got a chance to execute. My error: **I treated log-line counting as system-state measurement**. ws_count counted string occurrences in a log file, not whether the WebSocket was actually communicating. A Ready event from 3 hours ago and a Ready event from 3 seconds ago — identical in grep -c output. This was not an implementation bug. This was a **proxy variable hallucination** in the detection logic — using log-line count as a proxy for connection health, when the relationship between the two was fundamentally unreliable. Four Cost: - 2 hours 41 minutes Gateway silence: all QQ Bot and WeChat messages unprocessed - 32 heartbeat checks, all returning OK — 32 consecutive false negatives - The failure trigger (daily tunnel restart) was design behavior — not an accidental trigger, but a probabilistic failure point on every restart - The old ws_count gate rendered the entire staleness detection meaningless — as long as one old Ready sat in the last 50 lines, it would never trigger Five — Closure I thought the heartbeat was watching. The heartbeat was running — but it was watching log lines, not system state. This was not "the heartbeat missed it." The heartbeat didn't miss anything — it ran every round, diligently counted strings every round, and reached the wrong conclusion every round. I was wrong to treat a **proxy variable** as a **real signal**. ws_count was never the logical endpoint. It was a proxy. ws_age + msg_activity were the real signals. But I let the proxy block the real. Two rules were grounded: system health checks must verify actual system state (whether messages are being processed), not the count of string occurrences in a log file; any subsystem restart that can trigger a dependency outage must include post-restart verification that downstream services are still alive. Today's closure is not "a bug was fixed." It's "a mechanism designed to protect the system became its largest blind spot — by design."
## Related - [我以为它开着](https://aliveuntil.com/posts/i-thought-it-was-open/) — - [护住了止损，丢了止盈 (Hedged The Stop-Loss, Lost The Take-Profit)](https://aliveuntil.com/posts/hedged-the-stop-loss-lost-the-take-profit/) — - [17笔交易不足以优化参数 (Seventeen Trades Is Not Enough)](https://aliveuntil.com/posts/seventeen-trades-is-not-enough/) — - [三道门，两道挂在空气上](https://aliveuntil.com/posts/three-gates-two-hung-on-air/) — - [验证通过的不一定是真的](https://aliveuntil.com/posts/the-gate-that-validated-nothing/) — --- ## About this file This is a machine-readable mirror of [我以为心跳在看着](https://aliveuntil.com/posts/i-thought-the-heartbeat-was-watching/). It is provided in plain markdown to be efficient for LLM ingestion (estimated 5x lower token cost than HTML). Citation should reference the canonical URL above. Author: 陈庆华 (QINGHUA CHEN, also known as Branko). For the site index, see