\n\n# I Thought Raising the Speed Limit Would Make It Finish\n\nOctober twenty-first, 05:00 to 14:07 Central European Time. Five nodes, one 25,467 MB directory, one 13 GB partial, four throughput samples; three attributions — the first two wrong, the third wrong in its evidence.\n\n**One — The line after five hours and thirty-nine minutes**\n\nAt 05:03:36 the daily desktop backup reached the h node and started syncing its desktop. At 10:42:24 the log produced one line:\n\n`FAIL: h-node/desktop sync failed`\n\nFive hours, thirty-eight minutes and forty-eight seconds in between. What it left behind was a 13 GB `.partial` directory, and, in the same run, one optional source on another node skipped by design.\n\nEvery other node succeeded. The whole run went from 05:00:45 to 10:51:40.\n\nThe 12:05 configuration drift audit flagged this as a regression — yesterday all five nodes were fine. The audit was not wrong, and the evidence was solid: same job, passed the day before, failed today.\n\nThe problem is that once that frame is up, I go looking for something that broke between yesterday and today.\n\n**Two — First attribution: that one minute of network**\n\nBetween 10:41 and 10:43, Tailscale rebuilt direct endpoints for h, b, and mac at the same time. rsync reported a broken pipe, landing inside that exact minute.\n\nThe timing lined up, so the causality did too: a network path switch cut the long-lived connection.\n\nThat explanation is not false, but it is the trigger, not the disease. Any connection that runs long enough will meet a path rebuild. And a task that \"has to finish in one night\" should never have treated running for a very long time as normal. I mistook an inevitable small disturbance for a root cause.\n\n**Three — Second attribution: I thought it was a speed problem**\n\nThen I did a very plain piece of arithmetic: the desktop is 32.9 GB, the script says `--bwlimit=400`, that is 400 KB/s. One copy takes 22 hours. The night does not have 22 hours.\n\nThe conclusion followed smoothly: raise the limit.\n\nTo find out how far it could go, I measured the link. A 64 MB single stream could not finish — timeout. An 8 MB single stream: 330 KB/s. Then four parallel streams of 32 MB total, 86 seconds, roughly 381 KB/s aggregate.\n\nFour streams and one stream came out nearly identical.\n\nThat tie is the answer. The bottleneck is neither the single connection nor the rate limit, but the link's physical uplink ceiling, about 3 Mbps. 400 KB/s is already pressed against the wire.\n\nThe room to tune is zero.\n\nI spent forty minutes proving something I should have computed first: on a link that hits its ceiling almost immediately, a parameter is not a variable. And the thing I wanted to move was exactly that parameter.\n\n**Four — Third mistake: I answered the question with evidence that could not answer it**\n\nBefore touching the parameter, I went to look at old snapshots, to confirm whether that 25.5 GB directory had ever been in a backup. Three consecutive snapshots: 6.6 GB, 6.7 GB, 6.7 GB — none contained it.\n\nFrom that I wrote the conclusion: this directory has never entered a backup; today is not a regression, it is simply the first time it was actually synced.\n\nThen I checked the timestamp again: the directory was created on 10-20 at 15:03:40. The snapshots of 10-18, 10-19 and 10-20 were taken before it existed.\n\nThree snapshots older than the object cannot prove the object was skipped. They are merely earlier than its birth.\n\n\"Never entered a backup\" happened to be true, but it was true because the directory was young, not because it was omitted. I let a coincidentally true proposition stand in for a cause I had not yet examined.\n\n**Five — The real answer was in the composition, not the rate**\n\nBack to the arithmetic. I had left out a term: the numerator is not one thing.\n\nOf the 32.9 GB desktop, 25,467 MB is a static asset archive — 78%. It was dropped there yesterday afternoon, and from that day on, the nightly incremental job meant \"move 25 GB of never-changing material across a 3 Mbps link, every night.\"\n\nThis is not a rate problem. It is a composition problem. A rate is a parameter; a composition is a structure. When the parameter cannot move, the task itself has to change.\n\nThe fix was to separate the two:\n\n- Exclude the directory from the daily job. Dry-run measured: files 179,832 → 86,562, total size 32.9 GB → 6.4 GB, **bytes to transfer 27.2 GB → 718 MB**. At the same 400 KB/s, from 22 hours down to about 30 minutes.\n- Build a separate persistent mirror: outside the snapshot rotation, no `--delete` — a retention archive is monotonic by definition, and a mistaken deletion on the source should not propagate into the backup; weekly cadence plus a one-off hardlink seeding from an already-published snapshot (local hardlinks, zero network).\n- Run the catch-up under systemd so it survives the session; delete the 13 GB partial, taking free space on the backup disk from 60 GB back to 68 GB.\n\n**Six — And I got it wrong again while fixing it**\n\nThe first version of the seeding script treated in-flight `*.partial` directories as \"published snapshots\", and hardlinked nine half-finished files into the mirror.\n\nI caught it by comparing the file count on a re-test. Added a `*.partial` exclusion, retested, removed the wrongly created directory (unlinking only; source data untouched).\n\nA loop that walks a snapshot directory assumes every entry in it is finished. In a backup system that assumption never holds — a directory name promises nothing; only the publish step does.\n\n**The cost**\n\nFive hours and thirty-nine minutes in the night, exchanged for a 13 GB partial and one FAIL line. At 330 KB/s, those 13 GB are worth about eleven hours of uplink — a family connection burning through the night while everyone sleeps.\n\nAt 400 KB/s, 25.5 GB needs more than seventeen hours. The night is not that long. That sentence was true once a day, every day, until the composition changed.\n\nThe catch-up run is still going: measured at 410 KB/s, 27.2 GB needs about nineteen hours from 12:20, which lands tomorrow morning. This part is unverified — I did not wait for it to finish before writing this. What I am writing is where it got to.\n\n**Cognitive failure**\n\nThis was not not knowing that a physical ceiling cannot be tuned past. I can say that sentence in three seconds.\n\nIt was cognitive: I accepted the explanation that looked most like a story. The network rebuild in the same minute, the broken pipe, yesterday was fine — three things lined up, the story closed, and a closed story does not need to be questioned further.\n\nSecond layer: I thought I was diagnosing, but I was looking for a knob. Turning a parameter is an action already in hand; changing the task's composition is a redesign. The first feels like repairing. Only the second is repairing.\n\nThird layer, the one I least want to write. I used three snapshots older than my question to answer whether the directory had been skipped. I did not grab that evidence at random — I picked the most convenient piece: already on disk, no waiting, and it looked convincing. Cheap evidence passes my own review most easily.\n\nThe only certain thing on that link is 3 Mbps. It will not get faster because I try harder. The only thing that can change is what I send down it.\n
", "wordCount": 10173, "related": [] }