Repository navigation
[AAP-96820] Let the heartbeat and adoption take a clock - #16725
Closed
chrismeyersfsu wants to merge 1 commit into
Closed
chrismeyersfsu wants to merge 1 commit into
chrismeyersfsu wants to merge 1 commit into
Conversation
The heartbeat's judgment of lost peers and orphaned jobs, the adoption deadline, and an adopted job's finish time now read the time from a clock callable that defaults to django.utils.timezone.now. cluster_node_heartbeat delegates to run_heartbeat(binder, clock), and adopt_job_async to adopt_job(job_id, open_work, clock). Behavior is unchanged. awx.main.tests.fake_clock provides FakeClock. New functional tests check the lost-peer grace period to the second, clock skew in both directions (a fast judge finds a healthy peer lost; a dead peer with a fast clock is found late), and the adoption deadline end to end against the fake receptor. Co-Authored-By: Claude <noreply@anthropic.com>
This was referenced Oct 11, 2026
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueComment |
Member
Author
|
Replaced by #16730 (stack #16732). This PR closed when its branch was renamed to remove a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be app
30E3
lied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SUMMARY
Stacked on #16724.
The heartbeat and adoption read the current time in several places: judging lost peers, picking orphaned jobs, the adoption deadline (
HADR_JOB_ADOPTION_TIMEOUT), and an adopted job's finish time. They now read it from aclockcallable that defaults todjango.utils.timezone.now:cluster_node_heartbeat(binder)delegates torun_heartbeat(binder, clock=None).adopt_job_async(job_id)delegates toadopt_job(job_id, open_work, clock=None).open_workreturns theReceptorWorkfrom [AAP-96820] Put receptor work-unit operations behind an interface with a fake #16724, and is only called when the job is actually reattached.reattach_to_work_unitand_finalize_adopted_jobpass the clock through.Behavior is unchanged. The default is resolved at call time, so existing tests that patch
awx.main.tasks.system.nowstill work.Unlike patching the time module-wide, a clock argument lets one test give two simulated controllers different clocks, and set the time to the second instea 8000 d of sleeping.
awx/main/tests/fake_clock.pyaddsFakeClock. The new functional tests cover:last_seen;These record today's behavior. Failure testing of #16669 hit the fast-judge case on a live cluster: a controller with a fast clock marked its healthy peers lost.
ISSUE TYPE
COMPONENT NAME
STEPS TO REPRODUCE AND EXTRA INFO