Repository navigation
ssh: Avoid unneeded sleep when waiting for SSH server - #23001
Conversation
|
/ok-to-test |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Pull request overview
This PR optimizes WaitForSSHAccess’s retry loop to avoid adding an extra fixed delay when the SSH dial attempt already blocks for a long time (common on GitHub macOS runners during early boot/network bring-up), while still preventing a busy loop when failures return immediately.
Changes:
- Measure per-attempt duration in
WaitForSSHAccess. - Skip the retry sleep when the attempt itself took at least
retryDelay. - Retain a delay for fast-failing cases (e.g. “host is down”, “connection refused”) to avoid tight retry loops.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This comment was marked as outdated.
This comment was marked as outdated.
The previous code slept unconditionally after each failed attempt. Sleep only if the check returned in under 500ms (retryThreshold); if the dial already blocked in the kernel, sleeping again just adds unnecessary delay. On local machines with vmnet, the dial typically blocks 1-5 seconds in the kernel before returning "connection refused". The sleep is skipped, saving 1 second per attempt. In GitHub runners, where the VM takes longer to boot, we observe three phases during the SSH wait: - "operation timed out": dial blocks 10-16 seconds in the kernel (host not up, no ARP response). The sleep is skipped, saving 1 second per attempt. - "host is down": kernel returns instantly after cached ARP failure. The sleep prevents a busy loop, retrying every 1 second. - "connection refused": guest network is up but sshd not yet listening. Returns in under 100ms. Same as above, we sleep 1 second.
|
kvm2 driver with docker runtime DetailsTimes for minikube start: 39.6s 35.7s 35.6s 36.7s 38.7s Times for minikube ingress: 15.3s 15.2s 15.7s 18.7s 15.2s docker driver with docker runtime DetailsTimes for minikube start: 21.0s 18.3s 21.2s 18.6s 18.4s Times for minikube ingress: 10.1s 12.6s 12.6s 12.6s 12.1s docker driver with containerd runtime DetailsTimes for minikube start: 16.4s 19.2s 16.3s 16.4s 16.0s Times for minikube ingress: 24.6s 25.1s 25.1s 24.6s 24.6s |
|
@nirs: The following tests failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: medyagh, nirs The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
The previous code slept unconditionally after each failed attempt.
Sleep only if the check returned in under 500ms (retryThreshold); if
the dial already blocked in the kernel, sleeping again just adds
unnecessary delay.
On local machines with vmnet, the dial typically blocks 1-5 seconds
in the kernel before returning "connection refused". The sleep is
skipped, saving 1 second per attempt.
In GitHub runners, where the VM takes longer to boot, we observe three
phases during the SSH wait:
"operation timed out": dial blocks 10-16 seconds in the kernel (host
not up, no ARP response). The sleep is skipped, saving 1 second per
attempt.
"host is down": kernel returns instantly after cached ARP failure.
The sleep prevents a busy loop, retrying every 1 second.
"connection refused": guest network is up but sshd not yet listening.
Returns in under 100ms. Same as above, we sleep 1 second.
Examples from GitHub Actions
Sleep eliminated — dial blocked in kernel
vfkit second boot — dial blocked 75s, retried immediately (no 1s sleep):
QEMU (socket_vmnet) second boot — dial blocked 16s, retried immediately:
Sleep kept — dial returned instantly
"host is down" returned in <1ms, slept 1s before retry:
"connection refused" returned in 50ms, slept 1s before retry:
Assisted by Cursor/Claude Opus 4.6