Repository navigation
Conversation
* ci: Add baremetal containerd functional test matrix and its installation steps. * build: Add `--container-runtime=containerd` to functional test start arguments. * feat: add functional test for baremetal-containerd-ubuntu-24.04-arm64
…_64` suffix and shortening `arm64` to `arm`. (kubernetes#22178)
…0956) * Addon portainer: Upgrade to version 2.27.7 * Add update portainer * CI: Add portainer workflow * change Makefile to run update-portainer to hack folder
kubernetes#22161) Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 7.0.11 to 8.0.0. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@22a9089...98357b1) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-version: 8.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…etes#22159) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 5.0.0 to 6.0.0. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@330a01c...b7c566a) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: 6.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…rnetes#22160) Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 6.0.0 to 7.0.0. - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@018cc2c...37930b1) --- updated-dependencies: - dependency-name: actions/download-artifact dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…21881) Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.37.0 to 0.38.0. - [Commits](golang/sys@v0.37.0...v0.38.0) --- updated-dependencies: - dependency-name: golang.org/x/sys dependency-version: 0.38.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…m v0.18.0 to v0.18.1 (kubernetes#22126)
* refactor: convert if/else if chains to switch statements and simplify function signatures. * revert helm changes
…#22180) * chore: Update Kubernetes dependencies to v0.34.1 and refresh other indirect modules. * feat: update Go dependencies, including Kubernetes modules to v0.34.1
* Build(deps): Bump github.com/spf13/viper from 1.20.1 to 1.21.0 Bumps [github.com/spf13/viper](https://github.com/spf13/viper) from 1.20.1 to 1.21.0. - [Release notes](https://github.com/spf13/viper/releases) - [Commits](spf13/viper@v1.20.1...v1.21.0) --- updated-dependencies: - dependency-name: github.com/spf13/viper dependency-version: 1.21.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * go mod tidy * feat: add dependabot-gomodtidy workflow to automatically update go.mod and go.sum --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
And change package from python to python3
…ubernetes#22191) * Build(deps): Bump github.com/hashicorp/go-getter from 1.8.0 to 1.8.3 Bumps [github.com/hashicorp/go-getter](https://github.com/hashicorp/go-getter) from 1.8.0 to 1.8.3. - [Release notes](https://github.com/hashicorp/go-getter/releases) - [Commits](hashicorp/go-getter@v1.8.0...v1.8.3) --- updated-dependencies: - dependency-name: github.com/hashicorp/go-getter dependency-version: 1.8.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * update go.mod and go.sum --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: minikube-bot <minikube-bot@google.com>
* drain stdout pipe concurrently to avoid Windows timeout in addons list test * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Medya Ghazizadeh <medyagh@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…ernetes#22194) * Build(deps): Bump k8s.io/cluster-bootstrap from 0.34.1 to 0.34.3 Bumps [k8s.io/cluster-bootstrap](https://github.com/kubernetes/cluster-bootstrap) from 0.34.1 to 0.34.3. - [Commits](kubernetes/cluster-bootstrap@v0.34.1...v0.34.3) --- updated-dependencies: - dependency-name: k8s.io/cluster-bootstrap dependency-version: 0.34.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * update go.mod and go.sum --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: minikube-bot <minikube-bot@google.com>
* Kicbase/ISO: Update buildkit from v0.18.1 to v0.26.3 * Updating kicbase image to v0.0.48-1765966054-22186 * Updating ISO to v1.37.0-1765965980-22186
* Build(deps): Bump k8s.io/api from 0.34.1 to 0.34.3 Bumps [k8s.io/api](https://github.com/kubernetes/api) from 0.34.1 to 0.34.3. - [Commits](kubernetes/api@v0.34.1...v0.34.3) --- updated-dependencies: - dependency-name: k8s.io/api dependency-version: 0.34.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * update go.mod and go.sum --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: minikube-bot <minikube-bot@google.com>
…o 28.5.2+incompatible (kubernetes#22192) * Build(deps): Bump github.com/docker/docker Bumps [github.com/docker/docker](https://github.com/docker/docker) from 28.3.3+incompatible to 28.5.2+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v28.3.3...v28.5.2) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-version: 28.5.2+incompatible dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * update go.mod and go.sum --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: minikube-bot <minikube-bot@google.com>
Skip the minimum CPU validation (requires 2 CPUs) when --no kubernetes is set, since Kubernetes is not being run. Also adds --cpus=1 to existing integration test to verify this behavior. Signed-off-by: Aryan Bagade <aryan@aryanbagade.com>
Disable the skip on high load since the load does not seem related to the failures. We get the same failures with high or minimal load. Run top for 60 seconds before the test to let the runner cool down after the build tasks. Top output will help to understand the load on the runner. Run top for 10 seconds after the tests to understand the load when the test completed or failed. Remove the stop and second start, we can add this when starting minikube works reliably. Tweak minikube arguments to match vment-helper integration tests, known to work reliably: - cpus: 1 - memory: 2600m
We see that spotlight processes (mds_stores, mds, mds_worker_shared)
consume lot of cpu time and memory just before we start the test.
Disabling indexing at the start of the build to avoid pointless indexing
during the build and in particular during the test.
Before:
Processes: 517 total, 8 running, 509 sleeping, 1516 threads
2025/12/17 01:49:28
Load Avg: 8.14, 9.31, 7.59
CPU usage: 63.62% user, 36.13% sys, 0.24% idle
SharedLibs: 884M resident, 120M data, 73M linkedit.
MemRegions: 0 total, 0B resident, 0B private, 676M shared.
PhysMem: 11G used (1507M wired, 0B compressor), 3569M unused.
VM: 16T vsize, 5224M framework vsize, 0(0) swapins, 0(0) swapouts.
Networks: packets: 557738/718M in, 86877/9409K out.
Disks: 122874/5709M read, 229653/18G written.
PID COMMAND %CPU TIME MEM
292 mds_stores 107.5 04:00.70 52M+
113 mds 32.4 01:39.27 13M+
32910 mdworker_shared 24.9 00:01.27 1348K-
32777 mdworker_shared 24.7 00:01.37 1492K+
32907 mdworker_shared 24.2 00:01.31 1488K-
32908 mdworker_shared 24.1 00:01.28 2332K+
32686 mdworker_shared 24.1 00:01.36 1420K+
32635 mdworker_shared 24.0 00:01.47 1456K-
32728 mdworker_shared 24.0 00:01.38 1352K-
32729 mdworker_shared 23.8 00:01.34 1404K
32632 mdworker_shared 23.3 00:01.46 1440K
32906 mdworker_shared 23.2 00:01.28 1460K-
0 kernel_task 5.9 01:15.77 1360K
163 contextstored 3.6 00:15.06 2636K
32360 top 2.3 00:00.91 2544K
19442 Runner.Worker 1.7 00:10.56 51M-
32861 mdworker_shared 1.2 00:00.29 1472K
782 provjobd39176879 0.6 01:41.50 11M
90 fseventsd 0.6 00:16.76 1832K
1 launchd 0.1 00:17.18 16M-
After:
Processes: 494 total, 2 running, 492 sleeping, 1416 threads
2025/12/17 02:22:47
Load Avg: 4.28, 4.54, 3.11
CPU usage: 1.4% user, 1.27% sys, 97.67% idle
SharedLibs: 882M resident, 121M data, 72M linkedit.
MemRegions: 0 total, 0B resident, 409M private, 593M shared.
PhysMem: 9002M used (1472M wired, 0B compressor), 5328M unused.
VM: 16T vsize, 5224M framework vsize, 0(0) swapins, 0(0) swapouts.
Networks: packets: 481856/612M in, 87771/9843K out.
Disks: 129207/5521M read, 206790/11G written.
PID COMMAND %CPU TIME MEM
135 dasd 2.4 00:09.17 3724K+
57171 top 1.7 00:00.55 2404K+
781 provjobd11303566 0.4 02:30.29 11M
90 fseventsd 0.4 00:14.70 1800K
0 kernel_task 0.3 01:16.46 312K-
1 launchd 0.3 00:17.97 16M+
47311 Runner.Worker 0.2 00:07.52 51M-
85 logd 0.2 00:06.20 7056K+
413 secd 0.2 00:00.75 3492K+
172 contextstored 0.1 00:08.01 2588K+
382 UserEventAgent 0.1 00:01.55 2740K+
435 cloudd 0.1 00:02.16 4312K+
124 opendirectoryd 0.1 00:09.24 3564K+
402 tccd 0.1 00:01.95 1716K+
329 audioclocksyncd 0.0 00:02.58 3800K
177 airportd 0.0 00:03.20 2224K
146 notifyd 0.0 00:03.65 1116K
152 tccd 0.0 00:02.27 2080K
514 com.apple.geod 0.0 00:02.94 7712K+
425 rapportd 0.0 00:00.67 3820K+
After we find the VM IP address, we logged:
I1217 20:01:45.672609 80789 main.go:141] libmachine: Waiting for VM to start (ssh -p 22 docker@192.168.64.9)...
And enter a loop trying to dial to the VM SSH port and read a byte from
the remote server. This loop had no logging, so it the SSH server is not
responding or the VM process terminated, we would loop for ever
silently until minikube timeout expires.
This horrible wait loop was also duplicated 3 times in qemu, vfkit, and
krunkit. The function was de-duplicated to common.WaitForSSHAccess() to
make the purpose more clear.
Improvements:
- Wait after dial error to avoid busy dial loop
- Move logging into common.WaitForSSHAccess() so we don't need to
duplicate logging in the drivers
- Remove the confusing "(ssh -p 22 docker@192.168.64.9)" part, since we
don't really use SSH or the user name.
- Log the total wait time, to make it easy to debug cases when dialing,
or reading is slow.
- Log before dialing or reading, to make it easy to debug when a call is
stuck for long time, or we enter a endless retry loop.
- Log dial or read errors. We expect errors during the wait, but we want
to know how we failed.
Testings shows that this wait can be very fast (vfkit: 5 milliseconds)
or extremely slow (qemu: 18 seconds).
Example logs when running locally with vfkit:
I1217 20:18:25.188500 86179 main.go:143] libmachine: Waiting until SSH server "192.168.64.10:22" is accessible
I1217 20:18:25.188505 86179 main.go:143] libmachine: Dialing to "192.168.64.10:22"
I1217 20:18:25.191354 86179 main.go:143] libmachine: Reading from SSH server "192.168.64.10:22"
I1217 20:18:25.196342 86179 main.go:143] libmachine: SSH server "192.168.64.10:22" is accessible in 0.008 seconds
Example logs when running locally with vfkit --network vmnet-shared:
I1217 20:34:40.142807 91672 main.go:143] libmachine: Waiting until SSH server "192.168.105.20:22" is accessible
I1217 20:34:40.142812 91672 main.go:143] libmachine: Dialing to "192.168.105.20:22"
I1217 20:34:40.144611 91672 main.go:143] libmachine: Reading from SSH server "192.168.105.20:22"
I1217 20:34:40.148252 91672 main.go:143] libmachine: SSH server "192.168.105.20:22" is accessible in 0.005 seconds
Example logs when running locally with krunkit:
I1217 20:20:03.210732 86755 main.go:143] libmachine: Waiting until SSH server "192.168.105.18:22" is accessible
I1217 20:20:03.210738 86755 main.go:143] libmachine: Dialing to "192.168.105.18:22"
I1217 20:20:04.218743 86755 main.go:143] libmachine: Reading from SSH server "192.168.105.18:22"
I1217 20:20:04.226358 86755 main.go:143] libmachine: SSH server "192.168.105.18:22" is accessible in 1.016 seconds
Example logs when running locally with qemu:
I1217 20:24:38.089783 88331 main.go:143] libmachine: Waiting until SSH server "192.168.105.19:22" is accessible
I1217 20:24:38.089787 88331 main.go:143] libmachine: Dialing to "192.168.105.19:22"
I1217 20:24:39.092261 88331 main.go:143] libmachine: Failed to dial: dial tcp 192.168.105.19:22: connect: connection refused
I1217 20:24:40.092861 88331 main.go:143] libmachine: Dialing to "192.168.105.19:22"
I1217 20:24:41.095662 88331 main.go:143] libmachine: Reading from SSH server "192.168.105.19:22"
I1217 20:24:41.106530 88331 main.go:143] libmachine: SSH server "192.168.105.19:22" is accessible in 3.017 seconds
Example logs when running locally with qemu --network builtin:
I1217 20:21:45.419219 87370 main.go:143] libmachine: Waiting until SSH server "127.0.0.1:53354" is accessible
I1217 20:21:45.419224 87370 main.go:143] libmachine: Dialing to "127.0.0.1:53354"
I1217 20:21:45.419914 87370 main.go:143] libmachine: Reading from SSH server "127.0.0.1:53354"
I1217 20:22:04.272831 87370 main.go:143] libmachine: SSH server "127.0.0.1:53354" is accessible in 18.854 seconds
We were running qemu with -daemonize and vfkit, and krunkit in a new
process group. This is similar to using the `&` operator in the shell to
run a process in the background.
Running in the background is useful when the user tries to terminate the
entire minikube process group using `killpg`, and want to keep the vm
process running. This is feature is used by ramen testing tool to
terminate child processes after an error.
In macOS 15, Apple introduced the *Local network privacy*[1], limiting
access to the local network. When an untrusted application (like
minikube, vfkit, krunkit, qemu, and vmnet-helper) is trying to access
the local network, the user is asked to approve the operation.
Approvals can be also managed via *System Settings -> Privacy & Security
-> Local Network*. There is no way to manage approvals programmatically,
only via the UI.
On GitHub actions, we run in a headless server, so there is no way to
allow access to local network. Apple mitigated this by automatically
allowing local network access to command-line tools run over SSH,
including any child processes they spawn.
However running a child process as a daemon (qemu) or in a new process
group (vfkit, krunkit) seems to break the relation between the sshd
process and child process, so network operations are blocked in the
child process. When this happens, the network operation will fail with
*no route to host*, as we see in GitHub CI:
I1217 19:10:10.289490 28945 main.go:143] libmachine: Waiting until SSH server "192.168.105.2:22" is accessible
I1217 19:10:10.289497 28945 main.go:143] libmachine: Dialing to "192.168.105.2:22"
I1217 19:10:10.290521 28945 main.go:143] libmachine: Failed to dial: dial tcp 192.168.105.2:22: connect: no route to host
I1217 19:10:11.291543 28945 main.go:143] libmachine: Dialing to "192.168.105.2:22"
I1217 19:10:11.291773 28945 main.go:143] libmachine: Failed to dial: dial tcp 192.168.105.2:22: connect: no route to host
I1217 19:10:12.292106 28945 main.go:143] libmachine: Dialing to "192.168.105.2:22"
I1217 19:10:12.292373 28945 main.go:143] libmachine: Failed to dial: dial tcp 192.168.105.2:22: connect: no route to host
...
I1217 19:14:56.609203 28945 main.go:143] libmachine: Failed to dial: dial tcp 192.168.105.2:22: connect: no route to host
I1217 19:14:56.705836 28945 start.go:128] duration metric: took 6m0.010647057s to createHost
W1217 19:14:56.705915 28945 start.go:715] error starting host: creating host: create host timed out in 360.000000 seconds
To fix the issue, we run qemu, vfkit, krunkit, and vment-helper in the
foreground, and they inherit the local network access privileges from
the sshd process.
This may cause issues to users assuming the previous behavior so we may
need to make this configurable. I'm starting with simple change to test
if this fixes the issues in GitHub actions.
[1] https://developer.apple.com/documentation/technotes/tn3179-understanding-local-network-privacy
qemu was started in an insane way. Simplify the code to start qemu in the same way on all platforms, and redirect qemu stderr to qemu.log, in the same way we start vfkit and krunkit.
Even after fixing possible local network privacy, we still fail with "no route to host". This may be caused by macOS firewall, so disable it when we start the job.
Since we solved the load issue, minimize top output to 2 samples with 10 seconds delay before and after the test. Log output of ifconfig before and after the test. This will show if the bridge is created by vmnet when starting the vm. Log routing table before and after the test. This will show if the bridge got the expected IP (192.168.105.1) and if there is a conflict with another interface using the same IP.
In previous test we had a configured bridge
bridge100: flags=8a63<UP,BROADCAST,SMART,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
options=3<RXCSUM,TXCSUM>
ether 46:df:b8:97:77:64
inet 192.168.105.1 netmask 0xffffff00 broadcast 192.168.105.255
inet6 fe80::44df:b8ff:fe97:7764%bridge100 prefixlen 64 scopeid 0xb
inet6 fd6c:a5f7:9262:9b09:1cef:42b1:8d7a:d8ec prefixlen 64 autoconf secured
Configuration:
id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
ipfilter disabled flags 0x0
member: vmenet0 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 10 priority 0 path cost 0
nd6 options=201<PERFORMNUD,DAD>
media: autoselect
status: active
But the routing table marked it as dead (!):
Internet:
Destination Gateway Flags Netif Expire
default 192.168.64.1 UGScg en0
default link#11 UCSIg bridge100 !
127 127.0.0.1 UCS lo0
127.0.0.1 127.0.0.1 UH lo0
169.254 link#5 UCS en0 !
192.168.64 link#5 UCS en0 !
192.168.64.1/32 link#5 UCS en0 !
192.168.64.1 16:9d:99:28:da:64 UHLWIir en0 1186
192.168.64.23/32 link#5 UCS en0 !
192.168.105 link#11 UC bridge100 !
224.0.0/4 link#5 UmCS en0 !
224.0.0.251 1:0:5e:0:0:fb UHmLWI en0
255.255.255.255/32 link#5 UCS en0 !
Gemini thinks that the kernel may reject the packets, and suggest to
delete the bridge, flush the routing table, and clear the arp cache
before test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.