8000 Odd behavior when tearing down VMs at end of scale_out test · Issue #349 · ibmcb/cbtool · GitHub
[go: up one dir, main page]

Skip to content

Odd behavior when tearing down VMs at end of scale_out test #349

Open

Description

@pdmazz

At the completion of the scale out test, all the AIs and VMs are destroyed. I have set the detach_parallelism to 3 for vm_defaults, aidrs_defaults, and ai_defaults in the osgcloud_rules.yaml file.

The tear downs will begin with 3 AIs and their associated VMs. When an AI-XXX is terminated/made undefined, another AI-YYY will begin the tear down process.

However, several times I have seen odd pauses in the tear down.

For example, there are three AIs in the process of being terminated: AI-2, AI-4, and AI-8.

The order of completion of termination is: AI-4, AI-8, then AI-2. Though AI-4 and AI-8 have terminated, no other AI terminates are initiated.

After AI-2 completes, it detects that it was successfully undefined and begins termination of AI-10. It did NOT spawn any other terminations [AI-1, AI-11, and AI-12 are waiting to be terminated].

AI-10 completes and it detected as successfully undefined and it now detects that AI-4 and AI-8 have completed/undefined [even though they completed several minutes prior]. AI-11, AI-12, and AI-1 terminations are spawned.

Activity

  1. mraygalaxy commented on Nov 12, 2019

    @mraygalaxy
    Collaborator

    Greetings @pdmazz Some

    questions:

    1. Did all of the terminations eventually fully complete?
    2. What cloud are you using?
    3. If you do $ grep ERROR /var/log/cloudbench/XXXX_operations.log, do you find any meaningful errors between cloudbench and the interations with your cloud provider?
    4. How many AIs in total are we talking about?
  2. pdmazz commented on Nov 15, 2019

    @pdmazz
    ContributorAuthor

    Michael,
    sorry for the tardy reply, the GitHub email notification went into my junk folder :-(

    1. Yes, everything eventually terminated successfully.
    2. I am using IBM Cloud (SoftLayer).
    3. There were 13 AIs in total.
    4. At the point of the tear down, there was this single message in the cbuser_operations.log file:

    Oct 23 03:29:06 pdmazz-spec2-mel01.ibmperf.com cloudbench [ERROR] base_operations.py/ActiveObjectOperations.initialize_object TEST_cbuser - AI object initialization failure: need to pass through the appropriate 'all' function

  3. mraygalaxy commented on Jun 1, 2020

    @mraygalaxy
    Collaborator

    I suspect something is going on with the softlayer adapter here. @maugustosilva comments?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0