8000 Debugging smoke tests by nirs · Pull Request #5 · nirs/minikube · GitHub
[go: up one dir, main page]

Skip to content

Debugging smoke tests - #5

Closed
nirs wants 8000 to merge 9 commits into
masterfrom
smoke-tests-macos
Closed

nirs wants to merge 9 commits into
masterfrom
smoke-tests-macos

Conversation

@nirs
@nirs nirs commented Dec 18, 2025
Copy link
Copy Markdown
Owner

No description provided.

AryanBagade and others added 9 commits December 18, 2025 00:53
Skip the minimum CPU validation (requires 2 CPUs) when --no kubernetes is set, since Kubernetes is not being run.
Also adds --cpus=1 to existing integration test to verify this behavior.

Signed-off-by: Aryan Bagade <aryan@aryanbagade.com>
Disable the skip on high load since the load does not seem related to
the failures. We get the same failures with high or minimal load.

Run top for 60 seconds before the test to let the runner cool down after
the build tasks. Top output will help to understand the load on the
runner.

Run top for 10 seconds after the tests to understand the load when the
test completed or failed.

Remove the stop and second start, we can add this when starting minikube
works reliably.

Tweak minikube arguments to match vment-helper integration tests, known
to work reliably:
- cpus: 1
- memory: 2600m
We see that spotlight processes (mds_stores, mds, mds_worker_shared)
consume lot of cpu time and memory just before we start the test.

Disabling indexing at the start of the build to avoid pointless indexing
during the build and in particular during the test.

Before:

    Processes: 517 total, 8 running, 509 sleeping, 1516 threads
    2025/12/17 01:49:28
    Load Avg: 8.14, 9.31, 7.59
    CPU usage: 63.62% user, 36.13% sys, 0.24% idle
    SharedLibs: 884M resident, 120M data, 73M linkedit.
    MemRegions: 0 total, 0B resident, 0B private, 676M shared.
    PhysMem: 11G used (1507M wired, 0B compressor), 3569M unused.
    VM: 16T vsize, 5224M framework vsize, 0(0) swapins, 0(0) swapouts.
    Networks: packets: 557738/718M in, 86877/9409K out.
    Disks: 122874/5709M read, 229653/18G written.

    PID    COMMAND          %CPU  TIME     MEM
    292    mds_stores       107.5 04:00.70 52M+
    113    mds              32.4  01:39.27 13M+
    32910  mdworker_shared  24.9  00:01.27 1348K-
    32777  mdworker_shared  24.7  00:01.37 1492K+
    32907  mdworker_shared  24.2  00:01.31 1488K-
    32908  mdworker_shared  24.1  00:01.28 2332K+
    32686  mdworker_shared  24.1  00:01.36 1420K+
    32635  mdworker_shared  24.0  00:01.47 1456K-
    32728  mdworker_shared  24.0  00:01.38 1352K-
    32729  mdworker_shared  23.8  00:01.34 1404K
    32632  mdworker_shared  23.3  00:01.46 1440K
    32906  mdworker_shared  23.2  00:01.28 1460K-
    0      kernel_task      5.9   01:15.77 1360K
    163    contextstored    3.6   00:15.06 2636K
    32360  top              2.3   00:00.91 2544K
    19442  Runner.Worker    1.7   00:10.56 51M-
    32861  mdworker_shared  1.2   00:00.29 1472K
    782    provjobd39176879 0.6   01:41.50 11M
    90     fseventsd        0.6   00:16.76 1832K
    1      launchd          0.1   00:17.18 16M-

After:

    Processes: 494 total, 2 running, 492 sleeping, 1416 threads
    2025/12/17 02:22:47
    Load Avg: 4.28, 4.54, 3.11
    CPU usage: 1.4% user, 1.27% sys, 97.67% idle
    SharedLibs: 882M resident, 121M data, 72M linkedit.
    MemRegions: 0 total, 0B resident, 409M private, 593M shared.
    PhysMem: 9002M used (1472M wired, 0B compressor), 5328M unused.
    VM: 16T vsize, 5224M framework vsize, 0(0) swapins, 0(0) swapouts.
    Networks: packets: 481856/612M in, 87771/9843K out.
    Disks: 129207/5521M read, 206790/11G written.
    PID    COMMAND          %CPU TIME     MEM
    135    dasd             2.4  00:09.17 3724K+
    57171  top              1.7  00:00.55 2404K+
    781    provjobd11303566 0.4  02:30.29 11M
    90     fseventsd        0.4  00:14.70 1800K
    0      kernel_task      0.3  01:16.46 312K-
    1      launchd          0.3  00:17.97 16M+
    47311  Runner.Worker    0.2  00:07.52 51M-
    85     logd             0.2  00:06.20 7056K+
    413    secd             0.2  00:00.75 3492K+
    172    contextstored    0.1  00:08.01 2588K+
    382    UserEventAgent   0.1  00:01.55 2740K+
    435    cloudd           0.1  00:02.16 4312K+
    124    opendirectoryd   0.1  00:09.24 3564K+
    402    tccd             0.1  00:01.95 1716K+
    329    audioclocksyncd  0.0  00:02.58 3800K
    177    airportd         0.0  00:03.20 2224K
    146    notifyd          0.0  00:03.65 1116K
    152    tccd             0.0  00:02.27 2080K
    514    com.apple.geod   0.0  00:02.94 7712K+
    425    rapportd         0.0  00:00.67 3820K+
After we find the VM IP address, we logged:

    I1217 20:01:45.672609   80789 main.go:141] libmachine: Waiting for VM to start (ssh -p 22 docker@192.168.64.9)...

And enter a loop trying to dial to the VM SSH port and read a byte from
the remote server. This loop had no logging, so it the SSH server is not
responding or the VM process terminated, we would loop for ever
silently until minikube timeout expires.

This horrible wait loop was also duplicated 3 times in qemu, vfkit, and
krunkit. The function was de-duplicated to common.WaitForSSHAccess() to
make the purpose more clear.

Improvements:
- Wait after dial error to avoid busy dial loop
- Move logging into common.WaitForSSHAccess() so we don't need to
  duplicate logging in the drivers
- Remove the confusing "(ssh -p 22 docker@192.168.64.9)" part, since we
  don't really use SSH or the user name.
- Log the total wait time, to make it easy to debug cases when dialing,
  or reading is slow.
- Log before dialing or reading, to make it easy to debug when a call is
  stuck for long time, or we enter a endless retry loop.
- Log dial or read errors. We expect errors during the wait, but we want
  to know how we failed.

Testings shows that this wait can be very fast (vfkit: 5 milliseconds)
or extremely slow (qemu: 18 seconds).

Example logs when running locally with vfkit:

    I1217 20:18:25.188500   86179 main.go:143] libmachine: Waiting until SSH server "192.168.64.10:22" is accessible
    I1217 20:18:25.188505   86179 main.go:143] libmachine: Dialing to "192.168.64.10:22"
    I1217 20:18:25.191354   86179 main.go:143] libmachine: Reading from SSH server "192.168.64.10:22"
    I1217 20:18:25.196342   86179 main.go:143] libmachine: SSH server "192.168.64.10:22" is accessible in 0.008 seconds

Example logs when running locally with vfkit --network vmnet-shared:

    I1217 20:34:40.142807   91672 main.go:143] libmachine: Waiting until SSH server "192.168.105.20:22" is accessible
    I1217 20:34:40.142812   91672 main.go:143] libmachine: Dialing to "192.168.105.20:22"
    I1217 20:34:40.144611   91672 main.go:143] libmachine: Reading from SSH server "192.168.105.20:22"
    I1217 20:34:40.148252   91672 main.go:143] libmachine: SSH server "192.168.105.20:22" is accessible in 0.005 seconds

Example logs when running locally with krunkit:

    I1217 20:20:03.210732   86755 main.go:143] libmachine: Waiting until SSH server "192.168.105.18:22" is accessible
    I1217 20:20:03.210738   86755 main.go:143] libmachine: Dialing to "192.168.105.18:22"
    I1217 20:20:04.218743   86755 main.go:143] libmachine: Reading from SSH server "192.168.105.18:22"
    I1217 20:20:04.226358   86755 main.go:143] libmachine: SSH server "192.168.105.18:22" is accessible in 1.016 seconds

Example logs when running locally with qemu:

    I1217 20:24:38.089783   88331 main.go:143] libmachine: Waiting until SSH server "192.168.105.19:22" is accessible
    I1217 20:24:38.089787   88331 main.go:143] libmachine: Dialing to "192.168.105.19:22"
    I1217 20:24:39.092261   88331 main.go:143] libmachine: Failed to dial: dial tcp 192.168.105.19:22: connect: connection refused
    I1217 20:24:40.092861   88331 main.go:143] libmachine: Dialing to "192.168.105.19:22"
    I1217 20:24:41.095662   88331 main.go:143] libmachine: Reading from SSH server "192.168.105.19:22"
    I1217 20:24:41.106530   88331 main.go:143] libmachine: SSH server "192.168.105.19:22" is accessible in 3.017 seconds

Example logs when running locally with qemu --network builtin:

    I1217 20:21:45.419219   87370 main.go:143] libmachine: Waiting until SSH server "127.0.0.1:53354" is accessible
    I1217 20:21:45.419224   87370 main.go:143] libmachine: Dialing to "127.0.0.1:53354"
    I1217 20:21:45.419914   87370 main.go:143] libmachine: Reading from SSH server "127.0.0.1:53354"
    I1217 20:22:04.272831   87370 main.go:143] libmachine: SSH server "127.0.0.1:53354" is accessible in 18.854 seconds
We were running qemu with -daemonize and vfkit, and krunkit in a new
process group. This is similar to using the `&` operator in the shell to
run a process in the background.

Running in the background is useful when the user tries to terminate the
entire minikube process group using `killpg`, and want to keep the vm
process running. This is feature is used by ramen testing tool to
terminate child processes after an error.

In macOS
8000
 15, Apple introduced the *Local network privacy*[1], limiting
access to the local network. When an untrusted application (like
minikube, vfkit, krunkit, qemu, and vmnet-helper) is trying to access
the local network, the user is asked to approve the operation.
Approvals can be also managed via *System Settings -> Privacy & Security
-> Local Network*. There is no way to manage approvals programmatically,
only via the UI.

On GitHub actions, we run in a headless server, so there is no way to
allow access to local network. Apple mitigated this by automatically
allowing local network access to command-line tools run over SSH,
including any child processes they spawn.

However running a child process as a daemon (qemu) or in a new process
group (vfkit, krunkit) seems to break the relation between the sshd
process and child process, so network operations are blocked in the
child process. When this happens, the network operation will fail with
*no route to host*, as we see in GitHub CI:

    I1217 19:10:10.289490   28945 main.go:143] libmachine: Waiting until SSH server "192.168.105.2:22" is accessible
    I1217 19:10:10.289497   28945 main.go:143] libmachine: Dialing to "192.168.105.2:22"
    I1217 19:10:10.290521   28945 main.go:143] libmachine: Failed to dial: dial tcp 192.168.105.2:22: connect: no route to host
    I1217 19:10:11.291543   28945 main.go:143] libmachine: Dialing to "192.168.105.2:22"
    I1217 19:10:11.291773   28945 main.go:143] libmachine: Failed to dial: dial tcp 192.168.105.2:22: connect: no route to host
    I1217 19:10:12.292106   28945 main.go:143] libmachine: Dialing to "192.168.105.2:22"
    I1217 19:10:12.292373   28945 main.go:143] libmachine: Failed to dial: dial tcp 192.168.105.2:22: connect: no route to host
    ...
    I1217 19:14:56.609203   28945 main.go:143] libmachine: Failed to dial: dial tcp 192.168.105.2:22: connect: no route to host
    I1217 19:14:56.705836   28945 start.go:128] duration metric: took 6m0.010647057s to createHost
    W1217 19:14:56.705915   28945 start.go:715] error starting host: creating host: create host timed out in 360.000000 seconds

To fix the issue, we run qemu, vfkit, krunkit, and vment-helper in the
foreground, and they inherit the local network access privileges from
the sshd process.

This may cause issues to users assuming the previous behavior so we may
need to make this configurable. I'm starting with simple change to test
if this fixes the issues in GitHub actions.

[1] https://developer.apple.com/documentation/technotes/tn3179-understanding-local-network-privacy
qemu was started in an insane way. Simplify the code to start qemu in
the same way on all platforms, and redirect qemu stderr to qemu.log, in
the same way we start vfkit and krunkit.
Even after fixing possible local network privacy, we still fail with "no
route to host". This may be caused by macOS firewall, so disable it when
we start the job.
Since we solved the load issue, minimize top output to 2 samples with 10
seconds delay before and after the test.

Log output of ifconfig before and after the test. This will show if the
bridge is created by vmnet when starting the vm.

Log routing table before and after the test. This will show if the
bridge got the expected IP (192.168.105.1) and if there is a conflict
with another interface using the same IP.
In previous test we had a configured bridge

    bridge100: flags=8a63<UP,BROADCAST,SMART,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
            options=3<RXCSUM,TXCSUM>
            ether 46:df:b8:97:77:64
            inet 192.168.105.1 netmask 0xffffff00 broadcast 192.168.105.255
            inet6 fe80::44df:b8ff:fe97:7764%bridge100 prefixlen 64 scopeid 0xb
            inet6 fd6c:a5f7:9262:9b09:1cef:42b1:8d7a:d8ec prefixlen 64 autoconf secured
            Configuration:
                    id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
                    maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
                    root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
                    ipfilter disabled flags 0x0
            member: vmenet0 flags=3<LEARNING,DISCOVER>
                    ifmaxaddr 0 port 10 priority 0 path cost 0
            nd6 options=201<PERFORMNUD,DAD>
            media: autoselect
            status: active

But the routing table marked it as dead (!):

    Internet:
    Destination        Gateway            Flags               Netif Expire
    default            192.168.64.1       UGScg                 en0
    default            link#11            UCSIg           bridge100      !
    127                127.0.0.1          UCS                   lo0
    127.0.0.1          127.0.0.1          UH                    lo0
    169.254            link#5             UCS                   en0      !
    192.168.64         link#5             UCS                   en0      !
    192.168.64.1/32    link#5             UCS                   en0      !
    192.168.64.1       16:9d:99:28:da:64  UHLWIir               en0   1186
    192.168.64.23/32   link#5             UCS                   en0      !
    192.168.105        link#11            UC              bridge100      !
    224.0.0/4          link#5             UmCS                  en0      !
    224.0.0.251        1:0:5e:0:0:fb      UHmLWI                en0
    255.255.255.255/32 link#5             UCS                   en0      !

Gemini thinks that the kernel may reject the packets, and suggest to
delete the bridge, flush the routing table, and clear the arp cache
before test.
@nirs nirs closed this Dec 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0