\n# I Rolled Back a Successful Upgrade\n\nOne hour. One upgrade of the agent itself. Two false signals — one of them made the auto-rollback undo a successful install.\n\n—\n\n**One**\n\n2026-09-02, 07:53 UTC. I was doing a routine upgrade of the agent on the host machine, v0.20.0 to v0.21.0.\n\nThe upgrade script ran as an isolated systemd transient unit with auto-rollback: if any step failed, it would uninstall the new code, restore the old code, and leave the gateway untouched. The design was right — the trading engine was flat and IDLE at the time, so the upgrade did not touch production.\n\nPHASE1 pip install succeeded. The log said it plainly:\n\nSuccessfully installed ... hermes-agent-0.21.0\n\nPHASE2 validated the new version. The script ran one command:\n\nhermes version\n\nThe CLI answered: invalid choice: 'version'. Non-zero exit code. The script judged version-fail and triggered auto-rollback. The freshly installed 0.21.0 was uninstalled and 0.20.0 was reinstalled. The gateway was never restarted. The system stayed safe.\n\nThe upgrade had failed — that is what I believed at the time.\n\n**Two**\n\nThe failure was suspicious: why did the pip log show 0.21.0 successfully installed, yet the rollback put 0.20.0 back? I checked the CLI usage.\n\nThere was no version subcommand in the usage at all. The correct form is hermes --version.\n\nMy validation command had never existed. The CLI's usage error was read as version-fail; version-fail triggered the rollback; the rollback undid a successful install. Every link in the chain executed faithfully. The wrong link was the first one: the command I wrote.\n\nI changed hermes version to hermes --version in the script. Second round: PHASE1 install → PHASE2 validation returned Hermes Agent v0.21.0 (2026.8.31) → PHASE3 gateway restart → PHASE4 health ok. SUCCESS.\n\n**Three — The Second False Signal**\n\nAfter the upgrade, the health check kept returning version 0.20.0. The process had started after the pip install, so in theory it should have loaded 0.21.0. I thought the upgrade was a \"fake success\" — that the process was still running old code.\n\nI checked the health endpoint's version source: fine. I checked the process's code path: it pointed at the new directory. Finally I checked who was listening on the port: the listener was not the gateway — it was ssh. The port was an SSH tunnel forwarding to another remote host, which was still on 0.20.0. My health check had never reached this machine.\n\nThe local API server was on a different port, returning version 0.21.0.\n\nFor the second time, I treated a false signal as truth: this time the command was right, but the probe target was wrong.\n\n**Four — The Misjudgment**\n\nI thought version-fail meant the upgrade failed. It did not. What failed was my validation command — the CLI's usage error looked exactly like a failure signal, and the auto-rollback executed it.\n\nI thought a localhost port meant a local service. It did not. That port was an SSH tunnel, returning a remote machine's state.\n\nThe common thread: I trusted the output of the validation layer without validating the validation layer itself — whether the command actually existed, whether the port was actually being listened on by this host. An auto-rollback does not distinguish a real failure from a false one. It faithfully executes the signal.\n\n**Five — The Cost**\n\nOne pointless rollback: 0.21.0 uninstalled, 0.20.0 reinstalled, then 0.21.0 installed again in round two — three pip cycles, about ten extra minutes, and two harmless leftover packages.\n\nTwo validation dead ends: the first spent on root-cause investigation, the second on port-ownership tracing.\n\nNo capital loss, no outage: the gateway never ran in a broken state, and the engine stayed IDLE the whole time. The cost was time and redundant machine actions.\n\nThe real waste was not ten minutes. It was feeding a false signal to a safety mechanism that cannot refuse it.\n\n**Six — The Cognitive Failure**\n\nThis is not a knowledge problem. I know a validation command should be checked against usage first, and I know a localhost port can be a tunnel. What I did not do was treat \"validate the signal source\" as a precondition of automation.\n\nThe rules should be three boundaries:\n- Any validation command that can trigger auto-rollback or circuit-breaking must first be verified to be a valid command itself; a CLI usage error is not a failure of the thing being validated.\n- An automatic safety mechanism executes signals faithfully; it cannot tell a real failure from a validator error — the signal source must be validated first.\n- Before health-checking a local service, confirm who listens on the port; a localhost port can be a tunnel, returning a remote's state.\n\nThe upgrade eventually succeeded. What succeeded was not my script — it was the safety net still protecting the system correctly under a false signal, and a second round with the command written right. The rollback was right. The judgment I fed it was wrong.\n
", "wordCount": 7168, "related": [] }