Repository navigation
k3s: Keep the takeover of the k3s rolebinding in its test past forced deploys - #33
Merged
Merged
Conversation
… deploys The prune subtest failed now and then waiting for kube-apiserver-kubelet-admin to have the owner of the test manifest (3 failed CI attempts since 2026-11-15, 2 of 5 local runs). Until one pass over the manifests goes through without an error, the deploy watcher of k3s applies every manifest again 15 s after each pass, by name and without comparing checksums. cert-manager.yaml fails those passes while the webhook of cert-manager is down, and in every failed run rolebindings.yaml was applied right after prune-test.yaml and took the binding back. The manifest is now takeover.yaml, which sorts after rolebindings.yaml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # nix/tests/k3s.nix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot
31DA
be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes the second flake in
checks.k3s, in the prune subtest. The first one, metrics-server, was fixed in #31.What fails
The subtest times out waiting for
kube-apiserver-kubelet-adminto carryowner-nametakeover(it was calledprune-testbefore this PR). That happened on 3 failed CI attempts (43762021909 attempt 1, 43769271952, 44046696278) and in 2 of 5 local runs on 2026-11-19. In every one of them, k3s logged an apply ofprune-test.yamland then, right after, an apply ofrolebindings.yaml, and that second apply took the binding back.Why
The deploy watcher in k3s (
pkg/deploy/controller.goat v1.35.8+k3s1) starts withforce := trueand only clears it after a pass with no errors. Whileforceis set, it applies every manifest again 15 s after each pass, insort.Stringsorder, without comparing checksums. In the test,cert-manager.yamlfails those passes until cert-manager's webhook answers. The k3s log showsfailed to create cert-manager/cert-manager-webhook-libdns … Certificateevery ~25 s right up to the link. So any forced pass after the link applies the test manifest first and k3s'srolebindings.yamlstraight after it.Fix
The test manifest is renamed to
takeover.yaml, which sorts afterrolebindings.yaml. It is then applied afterrolebindings.yamlin every forced pass, and on its own in normal passes. The prune checks are unchanged.Verified locally
I ran
checks.k3s6 times. Run 1 was this branch as-is. Runs 2–6 also had the #31 firewall rule, so the metrics race couldn't end a run before the prune subtest. The prune subtest passed in all 6. In 4 of them a forced pass came right after the link (rolebindings.yamlthentakeover.yaml), which is the case that used to fail.Run 3 failed later, at "the boot imports no image again". That is a third flake, also seen in CI 44047722879, and this PR doesn't touch it: on the second boot the last image of the first boot,
utils.tar.gz, gets imported again. My reading ofpkg/agent/containerd/watcher.go(not yet confirmed in a VM) is that k3s runspruneCacheonceworkqueue.Len()hits 0, butLen()leaves out the item the worker is still on. So on the k3s restart within the first boot, the last file's cache entry can be pruned before it is markedseen. k3s master still has the same code.🤖 Generated with Claude Code