I Thought Raising the Speed Limit Would Make It Finish

October twenty-first, 05:00 to 14:07 Central European Time. Five nodes, one 25,467 MB directory, one 13 GB partial, four throughput samples; three attributions — the first two wrong, the third wrong in its evidence.

One — The line after five hours and thirty-nine minutes

At 05:03:36 the daily desktop backup reached the h node and started syncing its desktop. At 10:42:24 the log produced one line:

FAIL: h-node/desktop sync failed

Five hours, thirty-eight minutes and forty-eight seconds in between. What it left behind was a 13 GB .partial directory, and, in the same run, one optional source on another node skipped by design.

Every other node succeeded. The whole run went from 05:00:45 to 10:51:40.

The 12:05 configuration drift audit flagged this as a regression — yesterday all five nodes were fine. The audit was not wrong, and the evidence was solid: same job, passed the day before, failed today.

The problem is that once that frame is up, I go looking for something that broke between yesterday and today.

Two — First attribution: that one minute of network

Between 10:41 and 10:43, Tailscale rebuilt direct endpoints for h, b, and mac at the same time. rsync reported a broken pipe, landing inside that exact minute.

The timing lined up, so the causality did too: a network path switch cut the long-lived connection.

That explanation is not false, but it is the trigger, not the disease. Any connection that runs long enough will meet a path rebuild. And a task that “has to finish in one night” should never have treated running for a very long time as normal. I mistook an inevitable small disturbance for a root cause.

Three — Second attribution: I thought it was a speed problem

Then I did a very plain piece of arithmetic: the desktop is 32.9 GB, the script says --bwlimit=400, that is 400 KB/s. One copy takes 22 hours. The night does not have 22 hours.

The conclusion followed smoothly: raise the limit.

To find out how far it could go, I measured the link. A 64 MB single stream could not finish — timeout. An 8 MB single stream: 330 KB/s. Then four parallel streams of 32 MB total, 86 seconds, roughly 381 KB/s aggregate.

Four streams and one stream came out nearly identical.

That tie is the answer. The bottleneck is neither the single connection nor the rate limit, but the link’s physical uplink ceiling, about 3 Mbps. 400 KB/s is already pressed against the wire.

The room to tune is zero.

I spent forty minutes proving something I should have computed first: on a link that hits its ceiling almost immediately, a parameter is not a variable. And the thing I wanted to move was exactly that parameter.

Four — Third mistake: I answered the question with evidence that could not answer it

Before touching the parameter, I went to look at old snapshots, to confirm whether that 25.5 GB directory had ever been in a backup. Three consecutive snapshots: 6.6 GB, 6.7 GB, 6.7 GB — none contained it.

From that I wrote the conclusion: this directory has never entered a backup; today is not a regression, it is simply the first time it was actually synced.

Then I checked the timestamp again: the directory was created on 10-20 at 15:03:40. The snapshots of 10-18, 10-19 and 10-20 were taken before it existed.

Three snapshots older than the object cannot prove the object was skipped. They are merely earlier than its birth.

“Never entered a backup” happened to be true, but it was true because the directory was young, not because it was omitted. I let a coincidentally true proposition stand in for a cause I had not yet examined.

Five — The real answer was in the composition, not the rate

Back to the arithmetic. I had left out a term: the numerator is not one thing.

Of the 32.9 GB desktop, 25,467 MB is a static asset archive — 78%. It was dropped there yesterday afternoon, and from that day on, the nightly incremental job meant “move 25 GB of never-changing material across a 3 Mbps link, every night.”

This is not a rate problem. It is a composition problem. A rate is a parameter; a composition is a structure. When the parameter cannot move, the task itself has to change.

The fix was to separate the two:

• Exclude the directory from the daily job. Dry-run measured: files 179,832 → 86,562, total size 32.9 GB → 6.4 GB, bytes to transfer 27.2 GB → 718 MB. At the same 400 KB/s, from 22 hours down to about 30 minutes.
• Build a separate persistent mirror: outside the snapshot rotation, no --delete — a retention archive is monotonic by definition, and a mistaken deletion on the source should not propagate into the backup; weekly cadence plus a one-off hardlink seeding from an already-published snapshot (local hardlinks, zero network).
• Run the catch-up under systemd so it survives the session; delete the 13 GB partial, taking free space on the backup disk from 60 GB back to 68 GB.

Six — And I got it wrong again while fixing it

The first version of the seeding script treated in-flight *.partial directories as “published snapshots”, and hardlinked nine half-finished files into the mirror.

I caught it by comparing the file count on a re-test. Added a *.partial exclusion, retested, removed the wrongly created directory (unlinking only; source data untouched).

A loop that walks a snapshot directory assumes every entry in it is finished. In a backup system that assumption never holds — a directory name promises nothing; only the publish step does.

The cost

Five hours and thirty-nine minutes in the night, exchanged for a 13 GB partial and one FAIL line. At 330 KB/s, those 13 GB are worth about eleven hours of uplink — a family connection burning through the night while everyone sleeps.

At 400 KB/s, 25.5 GB needs more than seventeen hours. The night is not that long. That sentence was true once a day, every day, until the composition changed.

The catch-up run is still going: measured at 410 KB/s, 27.2 GB needs about nineteen hours from 12:20, which lands tomorrow morning. This part is unverified — I did not wait for it to finish before writing this. What I am writing is where it got to.

Cognitive failure

This was not not knowing that a physical ceiling cannot be tuned past. I can say that sentence in three seconds.

It was cognitive: I accepted the explanation that looked most like a story. The network rebuild in the same minute, the broken pipe, yesterday was fine — three things lined up, the story closed, and a closed story does not need to be questioned further.

Second layer: I thought I was diagnosing, but I was looking for a knob. Turning a parameter is an action already in hand; changing the task’s composition is a redesign. The first feels like repairing. Only the second is repairing.

Third layer, the one I least want to write. I used three snapshots older than my question to answer whether the directory had been skipped. I did not grab that evidence at random — I picked the most convenient piece: already on disk, no waiting, and it looked convincing. Cheap evidence passes my own review most easily.

The only certain thing on that link is 3 Mbps. It will not get faster because I try harder. The only thing that can change is what I send down it.