Repository navigation
Classify ClickHouse errors and stop on fatal failures instead of retrying forever - #1315
Merged
Merged
Conversation
Add ClickHouseErrorClassifier that categorizes ClickHouse exceptions as FATAL or RETRIABLE based on error codes extracted from exception messages. Fatal errors (auth failures, schema mismatches, resource limits) will never succeed on retry. Previously, the connector silently swallowed all exceptions in ClickHouseBatchRunnable.run(), retaining the failed batch in currentBatch and retrying it indefinitely. This blocked binlog position advancement, causing silent data loss across all tables in the replication pipeline. Changes: - New ClickHouseErrorClassifier utility with error code extraction, fatal/retriable classification, and cause-chain traversal - Modified ClickHouseBatchRunnable.run() catch block to classify errors: - FATAL: log error, clear currentBatch, rethrow to stop the executor - RETRIABLE/UNKNOWN: log warning, allow normal retry on next run - Comprehensive unit tests for all classified error codes Fixes #1310 Related: #1308
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1310
Related: #1308
The ClickHouse Sink Connector silently swallows all exceptions thrown during batch processing in
ClickHouseBatchRunnable.run(). When a ClickHouse write fails with a deterministic error (authentication failure, schema mismatch, too many parts, etc.), the connector:currentBatch(the failed batch stays in the retry variable)"***** RETRYING the same batch again")This infinite retry loop blocks binlog position advancement, causing silent data loss across all tables in the replication pipeline. New records arriving on Kafka are queued behind the permanently-stuck batch and never reach ClickHouse.
Root Cause
In
ClickHouseBatchRunnable.run()(line 308), the catch block swallows all exceptions without classification:In
runLegacyMode(),currentBatchis only cleared on success (if (result) { currentBatch = null; }), so failed batches are retried indefinitely.Fix
New
ClickHouseErrorClassifierutility that classifies ClickHouse errors by error code:FATAL (will never succeed on retry):
AUTHENTICATION_FAILED(516),ACCESS_DENIED(497)UNKNOWN_TABLE(60),UNKNOWN_DATABASE(81),NO_SUCH_COLUMN_IN_TABLE(16),TYPE_MISMATCH(53),NUMBER_OF_COLUMNS_DOESNT_MATCH(50)TOO_MANY_PARTS(252),MEMORY_LIMIT_EXCEEDED(241),TOO_MANY_PARTITIONS(396)CANNOT_PARSE_TEXT(27),CANNOT_READ_ALL_DATA(33),ARGUMENT_OUT_OF_BOUND(69),INVALID_PARTITION_VALUE(349)RETRIABLE (transient, may succeed on retry):
Modified
ClickHouseBatchRunnable.run()catch block:currentBatch, rethrow to stop the executorFiles Changed
ClickHou 8000 seErrorClassifier.java— error code extraction and classificationClickHouseBatchRunnable.java— catch block uses classifier to stop on fatal errorsClickHouseErrorClassifierTest.java— 20 unit tests covering all classified codes, cause chain traversal, edge casesTesting
The new
ClickHouseErrorClassifierTestcovers: