8000 k3s: Keep the takeover of the k3s rolebinding in its test past forced deploys by andsens · Pull Request #33 · nixos-homelab/shared · GitHub
[go: up one dir, main page]

Skip to content

k3s: Keep the takeover of the k3s rolebinding in its test past forced deploys - #33

Merged
andsens merged 2 commits into
mainfrom
k3s-prune-flake
Nov 19, 2026
Merged

andsens merged 2 commits into
mainfrom
k3s-prune-flake

Conversation

@andsens
@andsens andsens commented Nov 19, 2026
Copy link
Copy Markdown
Collaborator

This fixes the second flake in checks.k3s, in the prune subtest. The first one, metrics-server, was fixed in #31.

What fails

The subtest times out waiting for kube-apiserver-kubelet-admin to carry owner-name takeover (it was called prune-test before this PR). That happened on 3 failed CI attempts (43762021909 attempt 1, 43769271952, 44046696278) and in 2 of 5 local runs on 2026-11-19. In every one of them, k3s logged an apply of prune-test.yaml and then, right after, an apply of rolebindings.yaml, and that second apply took the binding back.

Why

The deploy watcher in k3s (pkg/deploy/controller.go at v1.35.8+k3s1) starts with force := true and only clears it after a pass with no errors. While force is set, it applies every manifest again 15 s after each pass, in sort.Strings order, without comparing checksums. In the test, cert-manager.yaml fails those passes until cert-manager's webhook answers. The k3s log shows failed to create cert-manager/cert-manager-webhook-libdns … Certificate every ~25 s right up to the link. So any forced pass after the link applies the test manifest first and k3s's rolebindings.yaml straight after it.

Fix

The test manifest is renamed to takeover.yaml, which sorts after rolebindings.yaml. It is then applied after rolebindings.yaml in every forced pass, and on its own in normal passes. The prune checks are unchanged.

Verified locally

I ran checks.k3s 6 times. Run 1 was this branch as-is. Runs 2–6 also had the #31 firewall rule, so the metrics race couldn't end a run before the prune subtest. The prune subtest passed in all 6. In 4 of them a forced pass came right after the link (rolebindings.yaml then takeover.yaml), which is the case that used to fail.

Run 3 failed later, at "the boot imports no image again". That is a third flake, also seen in CI 44047722879, and this PR doesn't touch it: on the second boot the last image of the first boot, utils.tar.gz, gets imported again. My reading of pkg/agent/containerd/watcher.go (not yet confirmed in a VM) is that k3s runs pruneCache once workqueue.Len() hits 0, but Len() leaves out the item the worker is still on. So on the k3s restart within the first boot, the last file's cache entry can be pruned before it is marked seen. k3s master still has the same code.

🤖 Generated with Claude Code

… deploys

The prune subtest failed now and then waiting for kube-apiserver-kubelet-admin to have the owner
of the test manifest (3 failed CI attempts since 2026-11-15, 2 of 5 local runs). Until one pass
over the manifests goes through without an error, the deploy watcher of k3s applies every manifest
again 15 s after each pass, by name and without comparing checksums. cert-manager.yaml fails those
passes while the webhook of cert-manager is down, and in every failed run rolebindings.yaml was
applied right after prune-test.yaml and took the binding back.

The manifest is now takeover.yaml, which sorts after rolebindings.yaml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andsens
andsens merged commit 0a61357 into main Nov 19, 2026
2 checks passed
@andsens
andsens deleted the k3s-prune-flake branch November 19, 2026 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0