You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Odd behavior when tearing down VMs at end of scale_out test #349
At the completion of the scale out test, all the AIs and VMs are destroyed. I have set the detach_parallelism to 3 for vm_defaults, aidrs_defaults, and ai_defaults in the osgcloud_rules.yaml file.
The tear downs will begin with 3 AIs and their associated VMs. When an AI-XXX is terminated/made undefined, another AI-YYY will begin the tear down process.
However, several times I have seen odd pauses in the tear down.
For example, there are three AIs in the process of being terminated: AI-2, AI-4, and AI-8.
The order of completion of termination is: AI-4, AI-8, then AI-2. Though AI-4 and AI-8 have terminated, no other AI terminates are initiated.
After AI-2 completes, it detects that it was successfully undefined and begins termination of AI-10. It did NOT spawn any other terminations [AI-1, AI-11, and AI-12 are waiting to be terminated].
AI-10 completes and it detected as successfully undefined and it now detects that AI-4 and AI-8 have completed/undefined [even though they completed several minutes prior]. AI-11, AI-12, and AI-1 terminations are spawned.
Did all of the terminations eventually fully complete?
What cloud are you using?
If you do $ grep ERROR /var/log/cloudbench/XXXX_operations.log, do you find any meaningful errors between cloudbench and the interations with your cloud provider?
Michael,
sorry for the tardy reply, the GitHub email notification went into my junk folder :-(
Yes, everything eventually terminated successfully.
I am using IBM Cloud (SoftLayer).
There were 13 AIs in total.
At the point of the tear down, there was this single message in the cbuser_operations.log file:
Oct 23 03:29:06 pdmazz-spec2-mel01.ibmperf.com cloudbench [ERROR] base_operations.py/ActiveObjectOperations.initialize_object TEST_cbuser - AI object initialization failure: need to pass through the appropriate 'all' function
Activity
mraygalaxy commentedon Nov 12, 2019
Greetings @pdmazz Some
questions:
pdmazz commentedon Nov 15, 2019
Michael,
sorry for the tardy reply, the GitHub email notification went into my junk folder :-(
Oct 23 03:29:06 pdmazz-spec2-mel01.ibmperf.com cloudbench [ERROR] base_operations.py/ActiveObjectOperations.initialize_object TEST_cbuser - AI object initialization failure: need to pass through the appropriate 'all' function
mraygalaxy commentedon Jun 1, 2020
I suspect something is going on with the softlayer adapter here. @maugustosilva comments?