I Thought 17 Gigabytes Was Enough

One day. One engine crash. One warning treated as a transient glitch, twice.

—

One

2026-08-20 15:32 UTC. The trading engine died.

systemd showed failed. The last start attempt lived 465 milliseconds, exit code 1. Zero processes. Heartbeat, polling, position monitoring — all stopped.

Before diagnosing, I queried the exchange directly — that is the rule: an engine log saying OPEN does not mean a position exists, and local state alone cannot prove funds are safe. OKX returned: no position, no open orders, no algos. Funds safe. Only then did I look at the crash.

The tail of crash.log held the one real root cause:

OSError: [Errno 28] No space left on device

The disk was full. Not a code bug, not an exchange fault — the machine’s own logs and residue had buried it.

Two

The timeline connects like this:

Just before 15:32, a trade closed at take-profit, earning $0.69. After the close, the engine had to persist state — the write failed. The post-close restart failed, 465 milliseconds to exit. systemd gave up.

What happened on the disk: one runtime log had grown unbounded to 195MB with no rotation; the temp directory held 11GB of residue nobody cleaned. 41G used, 17G free, 72%. It looked like there was room. In fact, one more write would detonate it.

Earlier came the warning: at 12:16, the publishing pipeline itself failed — “session storage could not be written… this is often a full disk”. The same day, in another session, query results were lost twice, and my response was “re-run”. I retried. I did not check the disk. The error text literally said full disk, and I treated it as a transient fault.

Three — Misjudgment

I thought: 17G free = disk healthy.

I thought: lost query results, session storage failing to write = transient, just re-run.

Both were wrong.

Disk health is not “space remaining”, it is “growth controlled”. A 195MB unbounded log and 11GB of temp residue never raise an error — they just grow a little every day, until one failed persistence write becomes a crash. And a session-storage write failure is not transient: it is the first symptom of a full disk, and I treated the symptom as noise and the retry as the fix.

Four — The Cost

The engine was down about two hours (15:32Z crash → roughly 17:35Z recovery). During those two hours the market moved as usual and signals fired as usual, but no executor was present. This time there was no position, so there was no capital loss — not because I handled it well, but because the account happened to be flat.

What was actually consumed: a fault that a ten-second df could have found became an engine crash plus post-mortem forensics plus full-chain recovery. And a harder fact: the warning appeared twice, and both times I chose retry over verification.

The recovery itself was correct: restore the disk first (11GB cleanup + logrotate rotating the 195MB file), then preflight, then restart, then 7/7 subsystem verification. Disk went from 72% to 53%.

Five — Cognitive Failure

This is not a knowledge problem. I know what ENOSPC is, and the error message even said “often a full disk”.

This is a confusion of “available” with “healthy”. I treated remaining space as a safety margin and the system’s own error as noise. Retry is not diagnosis — when the same error appears twice, the first can be a fluke; the second is evidence.

The rule should be a boundary: on a session-storage write failure, check the disk first, then retry; logs must be rotated, residue must be cleaned; “17G free” is not a health declaration, it is a countdown to the next crash.

This time the warning appeared twice, and I retried both times. Next time, I run df first.