Repository navigation
Antalya 26.8 - Backport of #118596 - Do not store ClickHouse unsigned integers in narrower Iceberg types - #2878
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 620a9ac98c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| case TypeIndex::UInt32: | ||
| case TypeIndex::Int64: | ||
| return {"long", true}; |
There was a problem hiding this comment.
Widen UInt32 in ORC data files too
For tables created with IcebergLocal(..., 'ORC') or another ORC-backed Iceberg engine, this now declares UInt32 as Iceberg long, but the unchanged ORC writer still declares it as TypeKind::INT and serializes it through Int64(Int32(value)). Inserting a value such as 4000000000 therefore writes a negative 32-bit value while the manifest schema and bounds describe a positive 64-bit value, producing incorrect reads or a schema mismatch for Iceberg consumers. The Iceberg ORC write path must emit UInt32 as ORC LONG and preserve the unsigned value, including for nested fields.
Useful? React with 👍 / 👎.
Fixes the tests that fail in CI after the backport of ClickHouse#118596: - `05303_optimize_empty_string_comparisons_distributed_low_cardinality` and `test_database_iceberg` create Iceberg tables with a `UInt64` column, which is now rejected. Use `Int64`. - `test_writes_modify_column` and `test_writes_modify_column_position` used `UInt32` as the type that Iceberg cannot tell apart from `Int32`. `UInt32` is now stored as `long`, so the change is recordable. Use `UInt16`, which is still `int`.
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
The port of ClickHouse#118596 matches upstream. Using the recursive forEachChild in checkNoUInt64 in place of anyInTypeTree is equivalent, and the fork-only Decimal/Time handling is kept. The UInt64 check runs only on CREATE, so existing tables still attach. A few things before merge:
-
Iceberg regression fails because of this change. In
iceberg_2,export partition/.../datatypes/export part/accepted/accepted: UInt64,Nullable(UInt64)andArray(UInt64)error out. The suite still expects aUInt64export into Iceberg to succeed, and that is now rejected on purpose. The real error is hidden by a harness bug:NameError: name 'error' is not definedatexport_operations.py:393. These scenarios need to be updated or xfailed in clickhouse-regression, ideally with theerror(...)helper fixed as well. -
Iceberg tables created before this change with a
UInt32column (stored as Icebergint). See the inline comments. Inserts take their types from the ClickHouse table definition, not from the Iceberg schema. For such tables, the column bounds and the Avro type of an identity partition field would now be written aslongwhile the schema saysint. Could you check this with an insert into such a table, read back by ClickHouse and by Spark or PyIceberg? Upstream has the same gap, so a known limitation is also an acceptable outcome. -
Commit
18c5cbac("fix tests") copies changes that are already onantalya-26.8from #2875, #2876 and #2877, plus thetest_database_gluediskassertion. It merges with no net effect but adds noise. Merging the base branch in would have been cleaner.
The other CI failures look unrelated: ThinLTO cache download, S3 503 in 04074_where_large_int_boolean_predicate, and test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts. DCO is red as well.
| case TypeIndex::Int16: | ||
| case TypeIndex::UInt32: | ||
| return dumpValue(static_cast<Int32>(applyVisitor(FieldVisitorConvertToNumber<Int64>(), field))); | ||
| case TypeIndex::UInt32: |
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
The bounds are now written as an 8-byte long whenever the ClickHouse column is UInt32. The type comes from sample_block, which is the ClickHouse table definition, not the Iceberg schema. A table created by an earlier build with u UInt32 has Iceberg type int for that field. Inserting into it after the upgrade would write 8-byte bounds into an int field. Readers that require exactly 4 bytes for int bounds, like PyIceberg, may reject the manifest. Could the size be chosen from the Iceberg field type, or could this be confirmed safe with a test?
There was a problem hiding this comment.
This is half true.
It is false for the regular insert path this comment is referring to because the methods updateExternalDynamicMetadataIfExists and buildStorageMetadataFromState are called and load the sample block based on the schema specified in the iceberg metadata files.
On the other hand, export part / partition is affected. The problem is that with export the schema is validated only at export time in a single node, and shall not change afterwards. I`ll see if I can come up with a fix for that.
| case TypeIndex::Date: | ||
| case TypeIndex::Date32: | ||
| return "int"; | ||
| case TypeIndex::UInt32: |
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
Same concern for the manifest's partition schema. On a table created before this change, an identity partition on a UInt32 column whose Iceberg type is int would now get Avro type long in new manifests. That doesn't match the table's partition type.
PR #2878 Audit Review
VerdictOne Medium defect, narrow trigger. The fix itself is sound. The PR does what it says. New Iceberg tables no longer get The defect: the replica now derives destination columns from that JSON using its own Confirmed defectsConfiguration & version skewMedium: export to an Iceberg table with a geometry/geography column passes scheduling, then fails on every replica
Lower-severity notes
What was checked and found clean
Coverage
Assumptions and limits. Static reading of Operational readinessScope.
|
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Do not store ClickHouse unsigned integers in narrower Iceberg types (ClickHouse#118596 by @scanhex12)
Documentation entry for user-facing changes
...
CI/CD Options
Exclude tests:
Regression jobs to run: