Repository navigation
fix(logging): the hard-cap pacing period ends when the reader has stayed under the cap for 60 s, not at the next crossing -- the first acknowledgement after the quiet gap closes it, and the pacing-ended line names the state at the last release (Spec 01.05 section 3.4 item 6) - #1514
Merged
Merged
Conversation
…yed under the cap for 60 s, not at the next crossing -- the first acknowledgement after the quiet gap closes it, and the 'pacing ended' line names the state at the last release instead of the over-cap state of the wait that closed it (Spec 01.05 section 3.4 item 6) Observed on two connectors on the same build: (1) a connector whose reader met the cap once during a bulk load kept the period open for hours -- no 'pacing ended' line ever came, because the period was only closed by the NEXT wait, which never happened; (2) on a connector that re-crossed the cap after a 3-minute quiet gap, the line read 'pacing ended ...: 549999 row(s) in 11 unit(s) outstanding' and was followed in the same millisecond by a new WARN for the same 549,999 rows -- the 'ended' line was written at the next crossing and printed that crossing's over-cap counters. Fix: DebeziumOffsetManagement.endPause() records the outstanding rows/units/bytes at each release; closePacingPeriod() reports that snapshot; drainCompletedUnits() calls closePacingPeriodIfQuiet(), which closes the period once the last release is more than CAPACITY_PACING_REARM_MS ago and no wait is in progress (a pauseInProgress flag set by beginPause and cleared by endPause/reset: one wait longer than the window is the period continuing on a slow head unit, not ending). The lazy close from beginPause is kept as the fallback. Tests (HandoffHardCapLogPacingTest): theFirstAcknowledgementAfterAQuietGapEndsThePeriod (RED without the drainCompletedUnits hook), pacingEndedNamesTheStateAtTheLastRelease (RED without the snapshot), anAcknowledgementDuringALongWaitDoesNotEndThePeriod (the guard). Specs 01.05 s3.4 item 6 + s5 and 09.01 s5 updated; validate_specs PASSED (68 specs / 715 citations).
…off-cap-pacing-end
Collaborator
Author
|
The only red check here was the spec validator aborting with no merge base before reading a file. That is a workflow defect, not this change: the job fetches the base branch with --depth=1 after a full checkout, which grafts the repository at the base tip, so a PR branched from an older base commit (this one branched two commits behind) has no merge base. The fix is #1515. To let the gate run on this PR meanwhile, the branch now carries the current 2.11.0 tip as a merge commit (tree unchanged apart from the base); validate_specs.py --changed-base 2.11.0 passes (68 specifications, 775 citations) and the four handoff hard-cap test classes are green on the merged tree (23 tests). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Observed on two connectors running the same build with the hard-cap log pacing from #1504. (1) A connector whose reader met the cap once during a bulk load kept the pacing period open for hours: no 'pacing ended' line ever came, because the period was only closed by the NEXT wait, which never happened once the load finished. (2) On a connector that re-crossed the cap after a quiet gap of a few minutes, the line read 'pacing ended ...: 549999 row(s) in 11 unit(s) outstanding' and was followed in the same millisecond by a new WARN for the same 549,999 rows: the 'ended' line was written at the next crossing and printed that crossing's over-cap counters, not the state the period actually ended in. Fix (DebeziumOffsetManagement): endPause() records the outstanding rows/units/bytes at each release; closePacingPeriod() reports that snapshot ('... outstanding at the last release'); drainCompletedUnits() calls closePacingPeriodIfQuiet(), which closes the period once the last release is more than CAPACITY_PACING_REARM_MS ago and no wait is in progress. A pauseInProgress flag (set by beginPause, cleared by endPause and reset) keeps one wait longer than the window from being read as the period ending: that is the period continuing on a slow head unit. The lazy close from beginPause is kept as the fallback for a reader that never acknowledges again. Tests (HandoffHardCapLogPacingTest, 7 of 7 green): theFirstAcknowledgementAfterAQuietGapEndsThePeriod (fails without the drainCompletedUnits hook), pacingEndedNamesTheStateAtTheLastRelease (fails without the snapshot), anAcknowledgementDuringALongWaitDoesNotEndThePeriod (the guard). Each new test was checked red against its mutation before the fix was restored. Specs: 01.05 section 3.4 item 6 and section 5, 09.01 section 5 updated; validate_specs passed (68 specs, 715 citations). No behaviour change on the pause/release path itself: only when the pacing period is closed and what the closing line reports.