8000 k3s: Patch k3s to keep the last image file in its import cache by andsens · Pull Request #36 · nixos-homelab/shared · GitHub
[go: up one dir, main page]

Skip to content

k3s: Patch k3s to keep the last image file in its import cache - #36

Merged
andsens merged 1 commit into
mainfrom
k3s-image-cache
Nov 19, 2026
Merged

andsens merged 1 commit into
mainfrom
k3s-image-cache

Conversation

@andsens
@andsens andsens commented Nov 19, 2026
Copy link
Copy Markdown
Collaborator

This fixes the third flake in checks.k3s, at "the boot imports no image again". The other two were metrics-server (#31) and the prune subtest (#33).

What fails

On the second boot, k3s imports the last image file again: docker-image-utils.tar.gz, the last in the images dir by name. Seen in CI run 44047722879 and in 1 of 6 local runs on 2026-11-19.

Why

At start, importAndWatchImages (pkg/agent/containerd/watcher.go, v1.35.8+k3s1 and master alike) waits for workqueue.Len() == 0 and then calls pruneCache. pruneCache drops every cache entry not yet seen since the start and writes .cache.json. Len() doesn't count the item the worker is processing, so if the poll lands while the worker is still on the last file, that file's entry is dropped. Setting seen afterwards doesn't sync, so the next start imports the file again. The test restarts k3s once within the first boot and then reboots, which gives the race two chances per run.

I confirmed it in a VM by restarting k3s 15 times each in two VMs and checking .cache.json after every restart. Without the patch, 1 of 30 restarts dropped the utils entry (7 entries became 6), and the next start logged Importing images from …utils.tar.gz.

Fix

nix/modules/k3s/image-cache-prune.patch changes k3s in two ways:

  • pruneCache drops the entries whose file no longer exists. That's what it is meant for: its comment says it keeps the cache from tracking files that are gone.
  • A mutex now guards filesCache. The worker and pruneCache both use it from different goroutines, with no lock until now.

The module applies the patch through the builder's overrideBundleAttrs (the agent is built in k3s-bin), as lib.mkDefault, so a services.k3s.package set elsewhere still wins.

Cost: k3s is no longer substituted from cache.nixos.org. The NAS and CI build it whenever nixpkgs bumps it: 389 s locally at --cores 4, likely longer on the 4-vCPU nested-KVM runner. The patch can go once the upstream fix (to be proposed to k3s-io/k3s) reaches the nixpkgs k3s.

Verified locally

This branch, with #33 merged in locally so the prune race can't end a run early:

  • 30 k3s restarts in two VMs: all 7 entries stayed every time, and no image was imported after the first boot.
  • checks.k3s 4 times: all passed, 178–201 s of test script.

🤖 Generated with Claude Code

checks.k3s failed now and then at "the boot imports no image again": the next start imported the
last image file (by name) again. At start, k3s prunes its cache of imported files once the
workqueue is empty (pkg/agent/containerd/watcher.go, the same on master), which is also the case
while the worker still processes the last file. The prune drops every entry not yet seen since the
start, so it dropped that file's entry. Seen in 1 of 30 k3s restarts in a VM; the test restarts k3s
once and reboots.

The patch prunes the entries of files that no longer exist instead, and guards the cache, which
the worker and the prune use from different goroutines, with a mutex. It goes into the bundle of
the k3s package, so k3s is built rather than substituted: 389 s locally at --cores 4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andsens
andsens merged commit 7afe389 into main Nov 19, 2026
2 checks passed
@andsens
andsens deleted the k3s-image-cache branch November 19, 2026 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0