Repository navigation
fix(engine): a FATAL failure is terminal at once, never a retry (Spec 10.04 section 3.5 rule 4) - #1487
Merged
Conversation
… 10.04 section 3.5 rule 4) A dead worker whose cause chain is FATAL by the error classifier (an unrepresentable value under clamp.out.of.range=false, an unknown table or column, a type mismatch, a denied privilege) recreated the engine; the engine started, connectorStarted() refilled the retry budget, the same batch was redelivered to the same outcome and the worker died again. The log read 'Restarting the engine - retry 1 of 10' every SLEEP_TIME, forever -- measured every 12-25 s for hours on two connectors -- each turn re-reading the whole schema history from the target, re-issuing the startup catalog queries and re-logging every skipped row event with its full row image. The terminal exit was never reached and no supervisor was ever signalled. handleEngineCompletion now routes a failure that ClickHouseErrorClassifier.classify marks FATAL straight to onTerminalFailure: exit code 3, or the loud liveness failure with exit.on.terminal.failure=false. Only Exceptions are classified; an Error keeps the retry path, as do retriable and unclassifiable failures. Spec 10.04 section 3.5 gains rule 4 and the matching contract bullet. Tests: TerminalFailureExitTest fatalTerminalTypeIsNotRetried, fatalErrorCodeIsNotRetried, retriableFailureStillRetries.
Collaborator
Author
|
#1489 is stacked on this PR: it keeps rule 4 (a FATAL-classified failure is terminal at once) and adds rule 5 for every other failure shape -- the retry budget refills on an acknowledged offset (progress), never on a bare start -- so an unclassified failure that recurs on every start is bounded by MAX_RETRIES too. Once this merges, #1489 shrinks to a single commit. |
minguyen9988
added a commit
that referenced
this pull request
Sep 24, 2026
Resolves the conflicts with the squash-merged #1487 (a FATAL failure is terminal at once) and #1488 (per-batch progress lines at DEBUG): the acknowledgement counter increments next to the now-DEBUG 'BATCH marked as processed' line, and spec 10.04 section 3.5 keeps the progress-refill contract (rule 5) in place of the superseded start-resets-the-budget line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Defect
handleEngineCompletiondrew on the retry budget for every failure. A failure thatClickHouseErrorClassifier.classifymarks FATAL (spec 10.01 section 3.1: aValueOutOfRangeExceptionin the cause chain underclamp.out.of.range=false, an unknown table or column, a type mismatch, a denied privilege) is identical on every attempt, so the recreated engine redelivered the same batch to the same outcome. And because the engine did start,connectorStarted()refilled the budget, sonumRetriesnever passed 1.Observed on a 2.11.0 deployment whose source holds the
9999-12-31 23:59:59open-ended sentinel in a column replicated as DateTime64, withclamp.out.of.rangeunset:Restarting the engine - retry 1 of 10every 12-25 s for hours, on two connectors, 34 engine restarts in one 100 MB log. Each turn re-read the whole schema history from the target (7 s and 390 MiB per read), re-issued the startup catalog queries and re-logged every skipped row event with its full row image. The terminal exit of spec 10.04 section 3.5 was never reached and no supervisor was ever signalled.Fix
isDeterministicFailure(Throwable):throwable instanceof Exception && classify(...) == FATAL. AnErroris not a replication verdict and keeps the retry path; retriable and unclassifiable failures are unchanged.handleEngineCompletionroutes such a failure straight toonTerminalFailure: exit code 3, or withexit.on.terminal.failure=falsethe loud liveness failure (/statusreportsReplica_Running=false, a FATAL line names the failure).Tests
JDK 17,
mvn -o -pl sink-connector-lightweight -Dtest=TerminalFailureExitTest test:fatalTerminalTypeIsNotRetried: dead worker, thenFatal ClickHouse error, thenValueOutOfRangeExceptionat the root of the cause chain: 0 restarts, exit code 3, replication reported stopped.fatalErrorCodeIsNotRetried:Code: 60UNKNOWN_TABLE in the chain: 0 restarts, exit.retriableFailureStillRetries:Code: 252TOO_MANY_PARTS: 1 restart, no exit.Tests run: 7, Failures: 0, Errors: 0, Skipped: 0.Tests run: 7, Failures: 2. Exactly the two FATAL tests fail (expected 0 restarts, got 1), the retriable one still passes.python3 scripts/validate_specs.py --changed-base origin/2.11.0: Validation PASSED (61 specifications, 596 verification citations).Ordering invariants
No versioning, de-duplication or watermark path is touched. The failing batch is retained and no offset is committed past it, so a restart by the supervisor redelivers it from the last committed position exactly as before; a re-delivered event and a late-committing transaction behave as on the base branch.