8000 Antalya 26.8 - Backport of #118596 - Do not store ClickHouse unsigned integers in narrower Iceberg types by arthurpassos · Pull Request #2878 · Altinity/ClickHouse · GitHub
[go: up one dir, main page]

Skip to content

Antalya 26.8 - Backport of #118596 - Do not store ClickHouse unsigned integers in narrower Iceberg types - #2878

Open
arthurpassos wants to merge 7 commits into
antalya-26.8from
backport/antalya-26.8/pr-118596-iceberg-unsigned-ints
Open

arthurpassos wants to merge 7 commits into
antalya-26.8from
backport/antalya-26.8/pr-118596-iceberg-unsigned-ints

Conversation

@arthurpassos
@arthurpassos arthurpassos commented Nov 16, 2026 •
Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Do not store ClickHouse unsigned integers in narrower Iceberg types (ClickHouse#118596 by @scanhex12)

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • 8000
  • Tiered Storage (2h)

@chatgpt-codex-connector
chatgpt-codex-connector Bot commented Nov 16, 2026 •
Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T20:17:25.515258Z 620a9ac PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions
github-actions Bot commented Nov 16, 2026 •
Copy link
Copy Markdown

Workflow [PR], commit [556c7c2]

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 620a9ac98c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +678 to 680
case TypeIndex::UInt32:
case TypeIndex::Int64:
return {"long", true};
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Widen UInt32 in ORC data files too

For tables created with IcebergLocal(..., 'ORC') or another ORC-backed Iceberg engine, this now declares UInt32 as Iceberg long, but the unchanged ORC writer still declares it as TypeKind::INT and serializes it through Int64(Int32(value)). Inserting a value such as 4000000000 therefore writes a negative 32-bit value while the manifest schema and bounds describe a positive 64-bit value, producing incorrect reads or a schema mismatch for Iceberg consumers. The Iceberg ORC write path must emit UInt32 as ORC LONG and preserve the unsigned value, including for nested fields.

Useful? React with 👍 / 👎.

Fixes the tests that fail in CI after the backport of
ClickHouse#118596:

- `05303_optimize_empty_string_comparisons_distributed_low_cardinality`
  and `test_database_iceberg` create Iceberg tables with a `UInt64`
  column, which is now rejected. Use `Int64`.
- `test_writes_modify_column` and `test_writes_modify_column_position`
  used `UInt32` as the type that Iceberg cannot tell apart from `Int32`.
  `UInt32` is now stored as `long`, so the change is recordable. Use
  `UInt16`, which is still `int`.
@mkmkme mkmkme left a comment
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [Posted by an LLM agent on behalf of @mkmkme]

The port of ClickHouse#118596 matches upstream. Using the recursive forEachChild in checkNoUInt64 in place of anyInTypeTree is equivalent, and the fork-only Decimal/Time handling is kept. The UInt64 check runs only on CREATE, so existing tables still attach. A few things before merge:

  1. Iceberg regression fails because of this change. In iceberg_2, export partition/.../datatypes/export part/accepted/accepted: UInt64, Nullable(UInt64) and Array(UInt64) error out. The suite still expects a UInt64 export into Iceberg to succeed, and that is now rejected on purpose. The real error is hidden by a harness bug: NameError: name 'error' is not defined at export_operations.py:393. These scenarios need to be updated or xfailed in clickhouse-regression, ideally with the error(...) helper fixed as well.

  2. Iceberg tables created before this change with a UInt32 column (stored as Iceberg int). See the inline comments. Inserts take their types from the ClickHouse table definition, not from the Iceberg schema. For such tables, the column bounds and the Avro type of an identity partition field would now be written as long while the schema says int. Could you check this with an insert into such a table, read back by ClickHouse and by Spark or PyIceberg? Upstream has the same gap, so a known limitation is also an acceptable outcome.

  3. Commit 18c5cbac ("fix tests") copies changes that are already on antalya-26.8 from #2875, #2876 and #2877, plus the test_database_glue disk assertion. It merges with no net effect but adds noise. Merging the base branch in would have been cleaner.

The other CI failures look unrelated: ThinLTO cache download, S3 503 in 04074_where_large_int_boolean_predicate, and test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts. DCO is red as well.

case TypeIndex::Int16:
case TypeIndex::UInt32:
return dumpValue(static_cast<Int32>(applyVisitor(FieldVisitorConvertToNumber<Int64>(), field)));
case TypeIndex::UInt32:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [Posted by an LLM agent on behalf of @mkmkme]

The bounds are now written as an 8-byte long whenever the ClickHouse column is UInt32. The type comes from sample_block, which is the ClickHouse table definition, not the Iceberg schema. A table created by an earlier build with u UInt32 has Iceberg type int for that field. Inserting into it after the upgrade would write 8-byte bounds into an int field. Readers that require exactly 4 bytes for int bounds, like PyIceberg, may reject the manifest. Could the size be chosen from the Iceberg field type, or could this be confirmed safe with a test?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is half true.

It is false for the regular insert path this comment is referring to because the methods updateExternalDynamicMetadataIfExists and buildStorageMetadataFromState are called and load the sample block based on the schema specified in the iceberg metadata files.

On the other hand, export part / partition is affected. The problem is that with export the schema is validated only at export time in a single node, and shall not change afterwards. I`ll see if I can come up with a fix for that.

case TypeIndex::Date:
case TypeIndex::Date32:
return "int";
case TypeIndex::UInt32:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [Posted by an LLM agent on behalf of @mkmkme]

Same concern for the manifest's partition schema. On a table created before this change, an identity partition on a UInt32 column whose Iceberg type is int would now get Avro type long in new manifests. That doesn't match the table's partition type.

@alsugiliazova
Copy link
Copy Markdown
Member

PR #2878 Audit Review

AI audit note: This review was generated by AI (Claude Opus 5.5).

Field Value
PR Altinity/ClickHouse#2878 — Antalya 26.8 - Backport of ClickHouse#118596 - Do not store ClickHouse unsigned integers in narrower Iceberg types
Upstream ClickHouse/ClickHouse#118596 (merged 2026-10-12)
Author arthurpassos
Base antalya-26.8
Head 533895f4b1295404e0e5586ee5f0342648264aef
State at review Open. CI in progress: 24 checks pass, 19 pending, and DCO fails (commit sign-off)
Size 6 commits, 33 files, +492 / −104. The upstream backport is the type mapping in Utils.cpp, IcebergWrites.cpp, and checkNoUInt64 in IcebergMetadata.cpp. The Altinity-only part (commit e94947a, "export check metadata json types") makes export use the destination types from the Iceberg metadata stored in the export manifest, not the destination table's declared columns
Scope reviewed Full diff at head. Read at head: MergeTreeData::exportPartToTable, StorageMergeTree / StorageReplicatedMergeTree::exportPartitionToTable, ExportPartTask::executeStep and addExportConvertingActions, ExportPartitionUtils::getContextCopyWithTaskSettings and verifyIcebergPartitionCompatibility, StorageObjectStorage::getExportDestinationMetadata / import / commitExportPartitionTransaction, IcebergMetadata::commitExportPartitionTransaction (schema/spec fencing), IcebergMetadata::createInitial, IcebergSchemaProcessor::getSimpleType, StorageObjectStorage::updateExternalDynamicMetadataIfExists, BinPackRewrite header construction, DatabaseReplicated::recoverLostReplica
Method Static review at head, walking the distributed-systems fault classes (initiator replica, executing replicas, Keeper export manifest, object storage, catalog). Nothing was built or run

Verdict

One Medium defect, narrow trigger. The fix itself is sound.

The PR does what it says. New Iceberg tables no longer get UInt32 as int or UInt64 as long. Export no longer depends on each replica's stale declared columns. The initiator validates against one Iceberg metadata JSON, stores it in the manifest, and every replica uses it for the cast check, the converting actions, the sink header and the commit. The commit is still fenced on schema id and partition spec id.

The defect: the replica now derives destination columns from that JSON using its own allow_experimental_geo_types_in_iceberg, which the manifest does not carry. If the destination has a geometry / geography field, scheduling succeeds under the initiator's query setting. Then every execution attempt, on every replica including the initiator's own background task, throws BAD_ARGUMENTS. The export cannot complete unless the server profile enables the setting. Before this PR, execution never parsed the schema, so it did not depend on that setting.

Confirmed defects

Configuration & version skew

Medium: export to an Iceberg table with a geometry/geography column passes scheduling, then fails on every replica

  • Impact. EXPORT PARTITION (and EXPORT PART scheduled from a partition task) to such a table is accepted, then every part task fails with BAD_ARGUMENTS: Using geometry/geography types is not allowed without enabled allow_experimental_geo_types_in_iceberg flag. The export never reaches COMPLETED. No data is lost, and nothing is committed.
  • Anchor.
    • IcebergMetadata::getExportDestinationColumns (IcebergMetadata.cpp:2814–2827) builds a scratch IcebergSchemaProcessor with context->getSettingsRef()[allow_experimental_geo_types_in_iceberg].
    • IcebergSchemaProcessor::getSimpleType (SchemaProcessor.cpp:639–645) throws for geometry/geography when that flag is off.
    • Callers on the execution path: ExportPartTask::addExportConvertingActions (ExportPartTask.cpp:130), StorageObjectStorage::import (StorageObjectStorage.cpp:893), StorageObjectStorage::commitExportPartitionTransaction (:989), and MergeTreeData::exportPartToTable (MergeTreeData.cpp:8674).
    • The execution context comes from storage.getContext() (ExportPartTask.cpp:235, ExportPartitionTaskScheduler.cpp:211/410). ExportPartitionUtils::getContextCopyWithTaskSettings (ExportPartitionUtils.cpp:259–318) re-applies the manifest settings, including allow_insert_into_iceberg, lossy-cast and iceberg_partition_timezone, but not allow_experimental_geo_types_in_iceberg.
  • Trigger. The table is created with allow_experimental_geo_types_in_iceberg = 1 and a Geometry column; getIcebergType maps it to geometry (Utils.cpp:802). Then ALTER TABLE src EXPORT PARTITION ID '...' TO TABLE ice SETTINGS allow_insert_into_iceberg = 1, allow_experimental_geo_types_in_iceberg = 1 runs on a cluster whose default profile leaves the geo flag at its default (false).
  • Why this is a defect. The PR's goal is that "the scheduling node and every replica that runs the export use the same destination types". Type derivation now depends on a setting that only the initiator's query context has. This is the same class of bug that getContextCopyWithTaskSettings already fixed for allow_insert_into_iceberg, where its comment warns about "an export silently stalls when the setting is only set at the query level".
  • Fix direction. Persist allow_experimental_geo_types_in_ic 8BD0 eberg in the manifest and re-apply it in getContextCopyWithTaskSettings. Alternatively, derive export columns without the geo gate, because the types were already validated at scheduling. iceberg_timezone_for_timestamptz has the same shape; see the clean-checks section for why it is harmless today.
  • Test direction. Add an EXPORT PARTITION to an Iceberg table with a Geometry column, with the geo setting enabled only at query level, and assert that the export reaches COMPLETED on a replica that did not schedule it.

Lower-severity notes

  • Low (version skew, rolling upgrade). An old-version replica that runs a manifest from a new initiator still uses its declared columns and the old mapping (UInt32 as int). If the destination was created by the new version (UInt32 as long) and that replica has not reloaded the schema, it writes 4-byte bounds and an int partition Avro type for a long field. The window is a mixed-version cluster during an upgrade. This PR cannot fix it on the old side. Release notes could say to avoid running Iceberg exports while the upgrade is in progress.
  • Low (usability, breaking DDL). UInt64 is now rejected in CREATE, even with IF NOT EXISTS over an existing table, and nested types like Array(UInt64) are rejected too because forEachChild recurses. This is documented, and it breaks idempotent deploy scripts that re-run CREATE TABLE IF NOT EXISTS ... (id UInt64) against tables created by earlier releases. It also breaks CREATE TABLE ... ENGINE = Iceberg... AS SELECT number / count() ..., because those produce UInt64 (many tests were changed for this reason). Existing tables keep working: the check runs only in CREATE mode (StorageObjectStorageCluster.cpp:146), so server restart and ATTACH are unaffected.
  • Low (export behavior change). Exports now cast to the Iceberg types, not the declared ones. Users may see two differences. First, DateTime destinations keep microseconds (the test_export_partition_all_iceberg_types_lossy expectation changed). Second, an export into a declared UInt32 column backed by an old Iceberg int field now fails without export_merge_tree_part_allow_lossy_cast when values exceed Int32. Previously such values silently wrapped. This is more correct, but some exports that used to succeed will now fail and need the opt-in.
  • Out of scope, not fixed by this PR. Tables created by earlier releases keep UInt32 columns as Iceberg int. After the schema reload the column is Int32, so INSERT of values above 2^31 still wraps through the default CAST. Similarly, UInt64 values above Int64 max written into a long field still wrap. The PR only protects newly created schemas.
  • Test gaps. No test covers a geo-typed destination for export (the defect above), and none covers mixed versions. The new replica test (test_export_partition_replica_uses_iceberg_schema_not_its_declared_columns) forces execution onto replica2 with SYSTEM STOP MOVES on replica1. That is a sound way to prove the replica uses the manifest types.

What was checked and found clean

  • One type source for every replica. The cast check (MergeTreeData.cpp:8674–8678), converting actions (ExportPartTask.cpp:130), sink header (StorageObjectStorage.cpp:893), and commit header (:989) all use getExportDestinationMetadata(manifest.iceberg_metadata_json). The scratch schema processor does not touch the table's live processor.
  • Schema drift between scheduling and commit. The commit compares the manifest's current-schema-id and default-spec-id with the latest metadata and fails with METADATA_MISMATCH if either changed (IcebergMetadata.cpp:2705–2716, and on retry at :2458–2467). Data cast to the pinned schema therefore cannot commit against a different schema.
  • Partition compatibility. verifyIcebergPartitionCompatibility rebuilds the key from the Iceberg spec and uses only the columns of the destination metadata (ExportPartitionUtils.cpp:996). Keeping the replica's declared partition key in the copied metadata has no effect.
  • Manifest format / Keeper. No new manifest field: iceberg_metadata_json already existed and was already written by older initiators. A new replica running an old manifest gets the same JSON.
  • Plain object storage destinations. getExportDestinationMetadata returns the in-memory metadata unchanged when the destination is not a data lake or the JSON is empty. StorageObjectStorageCluster forwards to pure_storage, like its other overrides.
  • Restart and replica recovery. checkNoUInt64 runs only from configuration->create, which only runs in LoadingStrictnessLevel::CREATE. DatabaseReplicated replay of an existing Iceberg path already fails with TABLE_ALREADY_EXISTS without IF NOT EXISTS, so UInt64 adds no new failure there.
  • Statistics width on INSERT. Iceberg always reloads columns from the schema (IcebergMetadata::shouldReloadSchemaForConsistency returns true), so the sink header is Int32 for an old int field and 4-byte bounds are written. test_writes_uint32_declared_over_int_field_writes_int_bounds covers this. BinPackRewrite builds its header from the Iceberg schema (BinPackRewrite.cpp:1488), and mutations refresh first (StorageObjectStorage.cpp:1280).
  • iceberg_timezone_for_timestamptz. It is also read from the replica context and not carried in the manifest. Casting between DateTime64 time zones keeps the epoch value, so the stored data is identical.
  • Backport fidelity. The upstream-only test changes (test_iceberg_disk_client_replacement, test_full_path_scheme, 05043, 05045, 05229, 05293, 05294) target files that do not exist on antalya-26.8. The Altinity-only Iceberg tests that used UInt64 / UInt32 were adapted.

Coverage

Fault class Status Note
Network / transport N/A No new RPC, retry or timeout. The initiator's one extra metadata read replaces an equivalent read later in the same function
Partition / availability N/A No quorum or leader logic changed. Each replica executes from Keeper as before
Timing / ordering Reviewed Schema drift between scheduling and commit is fenced by schema id and spec id
Partial / gray failure Reviewed The commit path is unchanged apart from the header source
Idempotency & duplicates Reviewed Retries rebuild the same columns from the same JSON, so the outcome is deterministic
Backpressure & resource limits Reviewed Each attempt parses the metadata JSON again (the destination's current Iceberg metadata, held in the export manifest). The cost is small next to writing a part
Recovery & rollback Reviewed Restart and ATTACH skip the new UInt64 check, and Replicated database replay behaves as before
Configuration & version skew Reviewed Defect 1 (geo setting not carried). Low note on the rolling-upgrade window
Security / trust N/A No new peer call or credential path
Observability Reviewed Error messages for UInt64 are clear and say which type to use instead. The defect's error names the flag but not that it must be set at server level for exports

Assumptions and limits. Static reading of 533895f. Tests were read but not run. CI had not finished at review time. I did not check whether Parquet writing of Geometry into Iceberg works end to end; the defect is about the export path, which now fails before reaching that code.

Operational readiness

Scope. EXPORT PART / EXPORT PARTITION from (Replicated)MergeTree to Iceberg, and CREATE TABLE ... ENGINE = Iceberg*. Persona: operator / DBA.

  1. Destination table declared with types that differ from the Iceberg schema. The export now casts to the Iceberg types on every replica. Values that do not fit fail validation at ALTER time unless export_merge_tree_part_allow_lossy_cast = 1.
  2. Destination with a geometry/geography column. Setting allow_experimental_geo_types_in_iceberg in the query is not enough: the export stalls (defect 1). Workaround: enable the setting in the server's default profile on every replica.
  3. Re-running deploy DDL with UInt64 after upgrade. CREATE TABLE IF NOT EXISTS fails with BAD_ARGUMENTS even though the table exists. Change the DDL to Int64 or Decimal(20, 0). Existing tables keep loading.
  4. New table with a UInt32 column. It reads back as Int64, and Spark sees bigint.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

0