[AAP-96820] Separate heartbeat adoption decisions from their side effects by chrismeyersfsu · Pull Request #16728 · ansible/awx

chrismeyersfsu · 2026-10-11T01:54:22Z

SUMMARY

The heartbeat's decisions about lost instances and orphaned jobs are mixed in with the queries, receptor calls and dispatcher calls that act on them. Testing a decision today means setting up all of that, which in practice means a running cluster.

This moves the decisions into awx/main/tasks/adoption_decisions.py. They read plain attributes and touch no database, receptor or dispatcher:

Function	Decides
`find_lost_instances`	Which peers are lost (this instance is judged separately)
`gate_lost_instances`	Which lost instances may be handled while the mesh re-establishes (execution and hop only)
`lost_instance_job_action`	What happens to a running job of a lost instance (reap today; the AAP-89602 adoption hook)
`lost_instance_disposition`	Deprovision, mark offline, or leave the lost instance
`startup_job_action`	Adopt or reap a running job at startup
`running_job_action`	Adopt, reap or leave a possibly orphaned job on a periodic heartbeat
`adoption_deadline_passed`	Whether an orphan has been silent past `HADR_JOB_ADOPTION_TIMEOUT`

awx/main/tasks/system.py keeps its queries and side effects and calls these. Behavior is unchanged. The query in _process_running_jobs still narrows the candidates; running_job_action repeats its conditions so the decision is complete and testable on its own.

Why: cross-controller adoption (AAP-89602, #16669) adds several new decisions here: when a controller is really gone, whether there is capacity to adopt, and who owns a claimed job. Failure testing of #16669 on a docker-compose cluster turned up defects in exactly these decisions: a slow controller judged lost while still running, claims at zero capacity, and a silent playbook timed out as dead. Each of those took a cluster run of 5 to 25 minutes to reproduce. With the decisions separated, the same cases are table-driven unit tests that run in milliseconds. A few such cases are included as examples (slow controller, clock skew, quiet job). They document today's behavior; they do not change it.

ISSUE TYPE

Bug, Docs Fix or other nominal change

COMPONENT NAME

Other (dispatcher heartbeat)

STEPS TO REPRODUCE AND EXTRA INFO

$ py.test awx/main/tests/unit/tasks/test_adoption_decisions.py awx/main/tests/unit/tasks/test_receptor_adoption.py \
    awx/main/tests/functional/tasks/test_tasks_system.py awx/main/tests/functional/test_dispatch.py
213 passed
$ py.test awx/main/tests/unit/tasks awx/main/tests/functional/tasks
32
8000
0 passed
$ ruff check / ruff format --check (changed files)
All checks passed!

coderabbitai · 2026-10-11T01:54:30Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

aap-pde-ci-bot · 2026-10-11T03:10:06Z

✅ Test Results - PASSED

Summary

Metric	Count
Total Tests	643
✅ Passed	571
❌ Failed	0
⚠️ Errors	0
⏭️ Skipped	72
⏱️ Duration	2921.80s

Pass Rate: 88.8%

chrismeyersfsu · 2026-10-11T10:07:54Z

/run-atf-tests

aap-pde-ci-bot · 2026-10-11T11:19:56Z

✅ Test Results - PASSED

Summary

Metric	Count
Total Tests	643
✅ Passed	571
❌ Failed	0
⚠️ Errors	0
⏭️ Skipped	72
⏱️ Duration	2825.07s

Pass Rate: 88.8%

aap-pde-ci-bot · 2026-10-12T14:26:26Z

❌ Test Results - FAILED

Summary

Metric	Count
Total Tests	643
✅ Passed	570
❌ Failed	1
⚠️ Errors	0
⏭️ Skipped	72
⏱️ Duration	2689.07s

Pass Rate: 88.6%

❌ Failed Tests

Test	Class
`test_project_sync_with_galaxy_disabled`	`opt.test-suite.tests.test_projects.TestProjectGalaxyInstall`

8000

aap-pde-ci-bot · 2026-10-13T12:17:58Z

❌ Test Results - FAILED

Summary

Metric	Count
Total Tests	643
✅ Passed	567
❌ Failed	2
⚠️ Errors	2
⏭️ Skipped	72
⏱️ Duration	3528.23s

Pass Rate: 88.2%

❌ Failed Tests

Test	Class
`test_user_can_view_inventories_with_role[Inventory Adhoc-200-Controller]`	`opt.test-suite.tests.rbac.test_inventories.TestControllerRBACInventories`
`test_team_can_change_inventories_with_role[Inventory Adhoc-403-Controller]`	`opt.test-suite.tests.rbac.test_inventories.TestControllerRBACInventories`
`test_team_can_view_credentials_with_role[None-403-Controller]`	`opt.test-suite.tests.rbac.test_credentials.TestControllerRBACCredentials`
`test_try_create_jt_using_ee_from_other_organization[Organization JobTemplate Admin-403-Controller]`	`opt.test-suite.tests.rbac.test_job_templates.TestControllerRBACJobTemplates`

aap-pde-ci-bot · 2026-10-14T16:19:54Z

❌ Test Results - FAILED

Summary

Metric	Count
Total Tests	643
✅ Passed	569
❌ Failed	2
⚠️ Errors	0
⏭️ Skipped	72
⏱️ Duration	2601.05s

Pass Rate: 88.5%

❌ Failed Tests

Test	Class
`test_project_gpg_signature_verification_valid`	`opt.test-suite.tests.test_projects.TestProjectGpgSignatureVerification`
`test_project_checksum_manifest_verification`	`opt.test-suite.tests.test_projects.TestProjectGpgSignatureVerification`

aap-pde-ci-bot · 2026-10-14T23:16:02Z

✅ Test Results - PASSED

Summary

Metric	Count
Total Tests	643
✅ Passed	571
❌ Failed	0
⚠️ Errors	0
⏭️ Skipped	72
⏱️ Duration	2901.64s

Pass Rate: 88.8%

aap-pde-ci-bot · 2026-10-21T14:23:18Z

✅ Test Results - PASSED

Summary

Metric	Count
Total Tests	643
✅ Passed	571
❌ Failed	0
⚠️ Errors	0
⏭️ Skipped	72
⏱️ Duration	2660.63s

Pass Rate: 88.8%

aap-pde-ci-bot · 2026-10-22T15:27:23Z

✅ Test Results - PASSED

Summary

Metric	Count
Total Tests	643
✅ Passed	571
❌ Failed	0
⚠️ Errors	0
⏭️ Skipped	72
⏱️ Duration	3358.35s

Pass Rate: 88.8%

aap-pde-ci-bot · 2026-10-23T15:35:02Z

✅ Test Results - PASSED

Summary

Metric	Count
Total Tests	643
✅ Passed	571
❌ Failed	0
⚠️ Errors	0
⏭️ Skipped	72
⏱️ Duration	3346.93s

Pass Rate: 88.8%

aap-pde-ci-bot · 2026-10-27T13:42:40Z

✅ Test Results - PASSED

Summary

Metric	Count
Total Tests	643
✅ Passed	571
❌ Failed	0
⚠️ Errors	0
⏭️ Skipped	72
⏱️ Duration	2726.64s

Pass Rate: 88.8%

aap-pde-ci-bot · 2026-10-27T15:36:06Z

✅ Test Results - PASSED

Summary

Metric	Count
Total Tests	643
✅ Passed	571
❌ Failed	0
⚠️ Errors	0
⏭️ Skipped	72
⏱️ Duration	2859.34s

Pass Rate: 88.8%

aap-pde-ci-bot · 2026-10-27T19:09:53Z

✅ Test Results - PASSED

Summary

Metric	Count
Total Tests	643
✅ Passed	571
❌ Failed	0
⚠️ Errors	0
⏭️ Skipped	72
⏱️ Duration	2590.05s

Pass Rate: 88.8%

aap-pde-ci-bot · 2026-10-28T23:45:57Z

❌ Test Results - FAILED

Summary

Metric	Count
Total Tests	643
✅ Passed	569
❌ Failed	2
⚠️ Errors	0
⏭️ Skipped	72
⏱️ Duration	2668.89s

Pass Rate: 88.5%

❌ Failed Tests

Test	Class
`test_user_can_delete_projects_with_role[Project Admin-204-Controller]`	`opt.test-suite.tests.rbac.test_projects.TestControllerProjectsRBAC`
`test_project_sync_with_galaxy_disabled`	`opt.test-suite.tests.test_projects.TestProjectGalaxyInstall`

aap-pde-ci-bot · 2026-11-03T13:58:24Z

✅ Test Results - PASSED

Summary

Metric	Count
Total Tests	643
✅ Passed	571
❌ Failed	0
⚠️ Errors	0
⏭️ Skipped	72
⏱️ Duration	2688.19s

Pass Rate: 88.8%

aap-pde-ci-bot · 2026-11-04T17:13:14Z

✅ Test Results - PASSED

Summary

Metric	Count
Total Tests	643
✅ Passed	571
❌ Failed	0
⚠️ Errors	0
⏭️ Skipped	72
⏱️ Duration	3410.17s

Pass Rate: 88.8%

Move the decisions the heartbeat makes about lost instances and orphaned jobs into awx.main.tasks.adoption_decisions: which peers are lost, which lost instances the mesh gate lets through, what happens to a lost instance and its jobs, whether a running job is adopted, reaped or left alone, and whether an orphan is past the adoption deadline. The functions read plain attributes and touch no database, receptor or dispatcher, so they can be tested with simple objects. awx.main.tasks.system keeps the queries and side effects and calls the decisions. Behavior is unchanged. Co-Authored-By: Claude <noreply@anthropic.com>

gate_lost_instances now takes the mesh check as a callable and skips it when nothing is lost, so the heartbeat has no branch of its own there. New tests cover a lost instance's job the decision doesn't reap, and a heartbeat job the decision skips. Co-Authored-By: Claude <noreply@anthropic.com>

Cases that differ only in their inputs are now rows of one test, each named by its id. Co-Authored-By: Claude <noreply@anthropic.com>

sonarqubecloud · 2026-11-05T18:58:31Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

aap-pde-ci-bot · 2026-11-05T20:08:54Z

✅ Test Results - PASSED

Summary

Metric	Count
Total Tests	643
✅ Passed	571
❌ Failed	0
⚠️ Errors	0
⏭️ Skipped	72
⏱️ Duration	3362.87s

Pass Rate: 88.8%

chrismeyersfsu mentioned this pull request Oct 11, 2026

[AAP-96820] Put receptor work-unit operations behind an interface with a fake #16729

Draft

github-actions Bot added the component:api label Oct 11, 2026

chrismeyersfsu added this pull request to stack #16732 October 11, 2026 01:54

chrismeyersfsu mentioned this pull request Oct 11, 2026

[AAP-96820] Separate heartbeat adoption decisions from their side effects #16723

Closed

chrismeyersfsu force-pushed the cmeyers-adoption-seam1-decisions branch from 7dba4c3 to ef5a174 Compare October 12, 2026 13:17

chrismeyersfsu force-pushed the cmeyers-adoption-seam1-decisions branch from 28b0bc8 to e2603f4 Compare October 13, 2026 10:40

chrismeyersfsu force-pushed the cmeyers-adoption-seam1-decisions branch from e2603f4 to e573392 Compare October 14, 2026 15:14

chrismeyersfsu force-pushed the cmeyers-adoption-seam1-decisions branch from e573392 to bc719b6 Compare October 14, 2026 22:00

chrismeyersfsu force-pushed the cmeyers-adoption-seam1-decisions branch 3 times, most recently from d073ad5 to b62f9f5 Compare October 21, 2026 13:17

chrismeyersfsu force-pushed the cmeyers-adoption-seam1-decisions branch 2 times, most recently from 90ad243 to 99a55bd Compare October 22, 2026 13:53

chrismeyersfsu force-pushed the cmeyers-adoption-seam1-decisions branch 3 times, most recently from 12733ad to 80530d6 Compare October 23, 2026 14:00

chrismeyersfsu force-pushed the cmeyers-adoption-seam1-decisions branch from 80530d6 to 53809c1 Compare October 27, 2026 12:32

chrismeyersfsu force-pushed the cmeyers-adoption-seam1-decisions branch from 53809c1 to d750810 Compare October 27, 2026 14:21

chrismeyersfsu force-pushed the cmeyers-adoption-seam1-decisions branch from d750810 to 97b9f22 Compare October 27, 2026 18:04

chrismeyersfsu force-pushed the cmeyers-adoption-seam1-decisions branch from 97b9f22 to c8ab93f Compare October 28, 2026 22:38

chrismeyersfsu force-pushed the cmeyers-adoption-seam1-decisions branch 2 times, most recently from 942011b to 6bd9565 Compare November 3, 2026 12:51

chrismeyersfsu force-pushed the cmeyers-adoption-seam1-decisions branch from 6bd9565 to e0dbc2a Compare November 4, 2026 15:34

ekachnycz-ciq mentioned this pull request Nov 4, 2026

Hand-off jobs to a new pod if ascender-task pod restarts. ctrliq/ascender#1180

Open

3 of 8 tasks

chrismeyersfsu and others added 3 commits November 5, 2026 18:34

Parametrize the heartbeat decision tests

110ad27

Cases that differ only in their inputs are now rows of one test, each named by its id. Co-Authored-By: Claude <noreply@anthropic.com>

chrismeyersfsu force-pushed the cmeyers-adoption-seam1-decisions branch from e0dbc2a to 110ad27 Compare November 5, 2026 18:35

Navigation Menu

Repository navigation

[AAP-96820] Separate heartbeat adoption decisions from their side effects - #16728

[AAP-96820] Separate heartbeat adoption decisions from their side effects#16728
chrismeyersfsu wants to merge 3 commits into
develfrom
cmeyers-adoption-seam1-decisions

Uh oh!

Draft PR not reviewed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

SUMMARY

ISSUE TYPE

COMPONENT NAME

STEPS TO REPRODUCE AND EXTRA INFO

Uh oh!

Uh oh!

Draft PR not reviewed

Uh oh!

✅ Test Results - PASSED

Summary

Uh oh!

Uh oh!

✅ Test Results - PASSED

Summary

Uh oh!

❌ Test Results - FAILED

Summary

❌ Failed Tests

Uh oh!

❌ Test Results - FAILED

Summary

❌ Failed Tests

Uh oh!

❌ Test Results - FAILED

Summary

❌ Failed Tests

Uh oh!

✅ Test Results - PASSED

Summary

Uh oh!

✅ Test Results - PASSED

Summary

Uh oh!

✅ Test Results - PASSED

Summary

Uh oh!

✅ Test Results - PASSED

Summary

Uh oh!

✅ Test Results - PASSED

Summary

Uh oh!

✅ Test Results - PASSED

Summary

Uh oh!

✅ Test Results - PASSED

Summary

Uh oh!

❌ Test Results - FAILED

Summary

❌ Failed Tests

Uh oh!

✅ Test Results - PASSED

Summary

Uh oh!

✅ Test Results - PASSED

Summary

Uh oh!

Quality Gate passed

Uh oh!

✅ Test Results - PASSED

Summary

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants