8000 Re-initalize failed Kubernetes clusters by tstromberg · Pull Request #7234 · kubernetes/minikube · GitHub
[go: up one dir, main page]

Skip to content

Re-initalize failed Kubernetes clusters< 8000 span class="sr-only"> - #7234

Merged
tstromberg merged 1 commit into
kubernetes:masterfrom
tstromberg:init-retry-port-conflict
Mar 26, 2020
Merged

tstromberg merged 1 commit into
kubernetes:masterfrom
tstromberg:init-retry-port-conflict

Conversation

@tstromberg
@tstromberg tstromberg commented Mar 25, 2020 •
Copy link
Copy Markdown
Contributor

If we fail to bring-up a Kubernetes cluster, this PR will reset it and bring it up again.

This helps primarily for two cases:

  • none driver initialization where untracked Kubernetes pods are running from a previous install.
  • clusters which have webhooks that prevent addons from being deployed.

Fixes #6425 and #6312 and numerous other errors which resemble:

	[ERROR Port-10251]: Port 10251 is in use
	[ERROR Port-10252]: Port 10252 is in use
	[ERROR Port-2380]: Port 2380 is in use

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 25, 2020
@k8s-ci-robot
Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tstromberg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 25, 2020
@medyagh
medyagh commented Mar 25, 2020
Copy link
Copy Markdown
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Mar 25, 2020
@minikube-pr-bot
Copy link
Copy Markdown

Error: running mkcmp: exit status 1

@tstromberg
Copy link
Copy Markdown
Contributor Author

/ok-to-test


if rr, err := k.c.RunCmd(exec.Command("/bin/bash", "-c", cmd)); err != nil {
return errors.Wrapf(err, "kubeadm reset: cmd: %q", rr.Command())
rr, derr := k.c.RunCmd(exec.Command("/bin/bash", "-c", cmd))
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code seems similar to Kic and None Driver Stop()... I wonder if we could get rid of those (not sure) since we have it here.

given if minikube stop calls delete cluster....we could get rid of those?

@medyagh
medyagh commented Mar 25, 2020
Copy link
Copy Markdown
Member

for this PR, we still see the bind problems:
https://storage.googleapis.com/minikube-builds/logs/7234/442a786/Docker_Linux.html#fail_TestStartStop%2fgroup%2fcontainerd

helpers.go:200: TestStartStop/group/containerd logs: * Problems detected in kube-apiserver [6f8628b796d086bbad75355810a66403e4d2dc6f26b1d59a02e709e0ee15a3b7]:
- Error: failed to create listener: failed to listen on 0.0.0.0:8444: listen tcp 0.0.0.0:8444: bind: address already in use
* Problems detected in kubelet:
- Mar 25 21:52:52 containerd-20200

@tstromberg
Copy link
Copy Markdown
Contributor Author

for this PR, we still see the bind problems:
https://storage.googleapis.com/minikube-builds/logs/7234/442a786/Docker_Linux.html#fail_TestStartStop%2fgroup%2fcontainerd

helpers.go:200: TestStartStop/group/containerd logs: * Problems detected in kube-apiserver [6f8628b796d086bbad75355810a66403e4d2dc6f26b1d59a02e709e0ee15a3b7]:
- Error: failed to create listener: failed to listen on 0.0.0.0:8444: listen tcp 0.0.0.0:8444: bind: address already in use
* Problems detected in kubelet:
- Mar 25 21:52:52 containerd-20200

Correct, this does not prevent the error from happening, it only provides recovery from that error.

There isn't enough detail available to show why the dashboard failed to deploy.

@tstromberg
tstromberg merged commit 095ccbe into kubernetes:master Mar 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

none driver: Port in use error during reuse

4 participants

0