Repository navigation
k3s: Patch k3s to keep the last image file in its import cache - #36
Merged
Merged
Conversation
checks.k3s failed now and then at "the boot imports no image again": the next start imported the last image file (by name) again. At start, k3s prunes its cache of imported files once the workqueue is empty (pkg/agent/containerd/watcher.go, the same on master), which is also the case while the worker still processes the last file. The prune drops every entry not yet seen since the start, so it dropped that file's entry. Seen in 1 of 30 k3s restarts in a VM; the test restarts k3s once and reboots. The patch prunes the entries of files that no longer exist instead, and guards the cache, which the worker and the prune use from different goroutines, with a mutex. It goes into the bundle of the k3s package, so k3s is built rather than substituted: 389 s locally at --cores 4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes the third flake in
checks.k3s, at "the boot imports no image again". The other two were metrics-server (#31) and the prune subtest (#33).What fails
On the second boot, k3s imports the last image file again:
docker-image-utils.tar.gz, the last in the images dir by name. Seen in CI run 44047722879 and in 1 of 6 local runs on 2026-11-19.Why
At start,
importAndWatchImages(pkg/agent/containerd/watcher.go, v1.35.8+k3s1 and master alike) waits forworkqueue.Len() == 0and then callspruneCache.pruneCachedrops every cache entry not yetseensince the start and writes.cache.json.Len()doesn't count the item the worker is processing, so if the poll lands while the worker is still on the last file, that file's entry is dropped. Settingseenafterwards doesn't sync, so the next start imports the file again. The test restarts k3s once within the first boot and then reboots, which gives the race two chances per run.I confirmed it in a VM by restarting k3s 15 times each in two VMs and checking
.cache.jsonafter every restart. Without the patch, 1 of 30 restarts dropped theutilsentry (7 entries became 6), and the next start loggedImporting images from …utils.tar.gz.Fix
nix/modules/k3s/image-cache-prune.patchchanges k3s in two ways:pruneCachedrops the entries whose file no longer exists. That's what it is meant for: its comment says it keeps the cache from tracking files that are gone.filesCache. The worker andpruneCacheboth use it from different goroutines, with no lock until now.The module applies the patch through the builder's
overrideBundleAttrs(the agent is built ink3s-bin), aslib.mkDefault, so aservices.k3s.packageset elsewhere still wins.Cost: k3s is no longer substituted from cache.nixos.org. The NAS and CI build it whenever nixpkgs bumps it: 389 s locally at
--cores 4, likely longer on the 4-vCPU nested-KVM runner. The patch can go once the upstream fix (to be proposed to k3s-io/k3s) reaches the nixpkgs k3s.Verified locally
This branch, with #33 merged in locally so the prune race can't end a run early:
checks.k3s4 times: all passed, 178–201 s of test script.🤖 Generated with Claude Code