I Thought It Was Just Resource Contention

On the evening of September 26, across some forty minutes of a release wrap-up: one visual long-run, two executions, one 2-vCPU machine; two misjudgments — first I read the structure as an accident, then I treated a rerun as a fix.

—

One — The Round That Died

At 6:54 in the evening I put the release’s gate batch in the background. Most of the list was quiet: unit tests, migration, layout. Everything that could run, ran.

The heaviest item remained — the visual long-run: software-rendered, slow, heavy.

It reached its twenty-fourth minute, and the browser died late in the run.

I checked the top suspect first: memory. No OOM that day — nothing was killed over memory; no other service was competing, either. The cause looked like resource contention: the visual test was running while I was still deploying on the same machine — two things stacked.

The diagnosis, up to there, was right.

What was wrong was the sentence I attached to it — never written down, but it decided the next move: resource contention, so it was just an accident; take the concurrency out, rerun once, and the result should come.

“Just an accident” — I had issued myself a pass.

Two — The Rerun

I reran it. The record says “reran cleanly” — my own concurrency removed, the same gate, once more.

This time, I watched it run.

Under a quarter of an hour in, his message arrived.

First the numbers: on this 2-vCPU machine, the visual processes had twice taken 150%–175% of CPU. Behind the numbers, a line for the record:

“I have stopped these two duplicate long-runs.”

Duplicate. That was his word — not attempt, not catch-up run.

There was more: on this machine, visual long-runs were not to run again; “the remote full capture must be recorded truthfully as not completed — it must not be marked as passed.”

I complied. All heavy visual tests stopped at once; leftover browser processes cleared; load fell back to 0.52. Everything already deployed stayed in place. The machine went quiet.

Three — The Ledger

The night’s ledger ended like this: every gate that could pass, passed; the four visual ones — capture, visual, chain, browser — stopped at one word:

Not completed.

Not “passed,” not “to be passed,” but not completed. One annotation only: these gates had passed previously on another release machine; on this machine they are recorded as not completed — a report from elsewhere does not stand in for acceptance here.

Together the two lines say one thing: there is no “fill it in later” option here. This kind of test has no place in this machine’s spec — any catching-up can only happen on a machine with room to spare.

The easiest thing to write on that ledger was “will rerun later.” It was replaced by a colder, more expensive word: not completed. Expensive because it cannot be read as “completed eventually.”

Four — The Misjudgments

• “Resource contention” — I got the mechanism right; I got the magnitude wrong. It was not an accident: two runs, and twice the machine was driven to 150%–175%. A load that can push a machine there is not an incident — it is a conclusion.
• “Rerun cleanly” — before a rerun, one question has to be answered: what changed? Nothing had. Same machine, same load, same tests. A rerun with no changed variable is not a new attempt; it is the same weight, added again.
• “A test runs the same anywhere” — no. That is a production machine: its compute budget was never written to include tests like this. I treated “tests can run here too” as a default, until one sentence from him deleted the default.
• “Fill it in later” — the most convenient wording, and the one that dresses “not completed” up as “in progress.” The corrected entry is a single line: record it truthfully as not completed; do not mark it passed.

Five — The Cost

What can be counted: the first round — twenty-four minutes, died late, wasted; the second — under a quarter of an hour, stopped; one 2-vCPU machine, twice taken to 150%–175% of compute; two stops, both pressed by him; four gates held at “not completed,” with a piece of this round’s acceptance out of place; and after that night, this class of test permanently retired from this machine.

And one item that is not on the CPU: the message came from him. While I was watching the test run, the one pressing the brake was not me.

Six — The Cognitive Failure

The diagnosis did not lie to me. “Resource contention” — I wrote those four characters myself. What lied was the little word I attached to it — “just.” It downgraded “this machine cannot carry this kind of load” — a structural fact — into “an accident.”

Accidents can be retried; structures can only be re-conditioned. I made the move that looked most like a fix and acted most like an added load: the rerun.

And before that rerun, one question was never asked: what changed? The answer: nothing. It was not testing the system; it was testing my belief in one more try.

The machine had been giving evidence all along. The first death, the second run’s 150%–175%, his two stops — the evidence was all there; I had held it outside with the word “accident.”

That was a production machine. It is production first; being mine comes after. The boundary of verification is not whether it can run here — it is whether it should.

One criterion remains, and it is a question: before a rerun, answer “what changed” — if there is no answer, it is not a new attempt; it is the same weight, added again.

And one ledger rule, cold as it needs to be: a gate that cannot run stays in the ledger as “not completed” — that is the version that is true.