Repository navigation
Separate heartbeat adoption decisions from their side effects - #16722
Closed
chrismeyersfsu wants to merge 1 commit into
Closed
chrismeyersfsu wants to merge 1 commit into
chrismeyersfsu wants to merge 1 commit into
Conversation
Move the decisions the heartbeat makes about lost instances and orphaned jobs into awx.main.tasks.adoption_decisions: which peers are lost, which lost instances the mesh gate lets through, what happens to a lost instance and its jobs, whether a running job is adopted, reaped or left alone, and whether an orphan is past the adoption deadline. The functions read plain attributes and touch no database, receptor or dispatcher, so they can be tested with simple objects. awx.main.tasks.system keeps the queries and side effects and calls the decisions. Behavior is unchanged. Co-Authored-By: Claude <noreply@anthropic.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueComment |
|
Member
Author
Codecov Report❌ Patch coverage is 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SUMMARY
The heartbeat's decisions about lost instances and orphaned jobs are mixed in with the queries, receptor calls and dispatcher calls that act on them. Testing a decision today means setting up all of that, which in practice means a running cluster.
This moves the decisions into
awx/main/tasks/adoption_decisions.py. They read plain attributes and touch no database, receptor or dispatcher:find_lost_instancesgate_lost_instanceslost_instance_job_actionlost_instance_dispositionstartup_job_actionrunning_job_actionadoption_deadline_passedHADR_JOB_ADOPTION_TIMEOUTawx/main/tasks/system.pykeeps its queries and side effects and calls these. Behavior is unchanged. The query in_process_running_jobsstill narrows the candidates;running_job_actionrepeats its conditions so the decision is complete and testable on its own.Why: cross-controller adoption (AAP-89602, #16669) adds several new decisions here: when a controller is really gone, whether there is capacity to adopt, and who owns a claimed job. Failure testing of #16669 on a docker-compose cluster turned up defects in exactly these decisions: a slow controller judged lost while still running, claims at zero capacity, and a silent playbook timed out as dead. Each of those took a cluster run of 5 to 25 minutes to reproduce. With the decisions separated, the same cases are table-driven unit tests that run in milliseconds. A few such cases are included as examples (slow controller, clock skew, quiet job). They document today's behavior; they do not change it.
ISSUE TYPE
COMPONENT NAME
STEPS TO REPRODUCE AND EXTRA INFO