8000 Antalya 26.6: Backport of #116171 to write iceberg v3 next_seq_num and first_row_id by subkanthi · Pull Request #2456 · Altinity/ClickHouse · GitHub
[go: up one dir, main page]

Skip to content

Antalya 26.6: Backport of #116171 to write iceberg v3 next_seq_num and first_row_id - #2456

Merged
zvonand merged 6 commits into
antalya-26.6from
backport-116171-iceberg-v3-cdc-writes
Oct 22, 2026
Merged

zvonand merged 6 commits into
antalya-26.6from
backport-116171-iceberg-v3-cdc-writes

Conversation

@subkanthi
@subkanthi subkanthi commented Sep 30, 2026 •
Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add Iceberg v3 statistics (next-seq-num and first_row_id). This PR allows to write those stats. Follow up to #2308 which included functionality to read both the stats columns. (ClickHouse#116171 by @scanhex12)

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

@subkanthi subkanthi changed the title Backport of #116171 Antalya 26.6: Backport of #116171 Sep 30, 2026
@github-actions
github-actions Bot commented Sep 30, 2026 •
Copy link
Copy Markdown

Workflow [PR], commit [a2f5210]

@subkanthi
subkanthi marked this pull request as ready for review October 1, 2026 16:03
@subkanthi subkanthi changed the title Antalya 26.6: Backport of #116171 Antalya 26.6: Backport of #116171 to write iceberg v3 next_seq_num and first_row_id Oct 3, 2026
@subkanthi
subkanthi commented Oct 3, 2026 •
Copy link
Copy Markdown
Collaborator Author

TESTING



SELECT
    first_row_id,
    sequence_number
FROM system.iceberg_files
WHERE (database = currentDatabase()) AND (`table` = 'flowers.irisv3')

Query id: 3cf56511-0e7d-428a-ad7d-2e85c7096b1e

   ┌─first_row_id─┬─sequence_number─┐
1. │          150 │               2 │
2. │            0 │               1 │
   └──────────────┴─────────────────┘

 select * from ice.`flowers.irisv3` limit 5;

SELECT *
FROM ice.`flowers.irisv3`
LIMIT 5

Query id: 53e8c33c-33e3-4884-a184-c3787d2859ec

   ┌─sepal.length─┬─sepal.width─┬─petal.length─┬─petal.width─┬─variety─┐
1. │          5.1 │         3.5 │          1.4 │         0.2 │ Setosa  │
2. │          4.9 │           3 │          1.4 │         0.2 │ Setosa  │
3. │          4.7 │         3.2 │          1.3 │         0.2 │ Setosa  │
4. │          4.6 │         3.1 │          1.5 │         0.2 │ Setosa  │
5. │            5 │         3.6 │          1.4 │         0.2 │ Setosa  │
   └──────────────┴─────────────┴──────────────┴─────────────┴─────────┘

5 rows in set. Elapsed: 0.020 sec. 

Ubuntu-2404-noble-amd64-base :) update ice.`flowers.irisv3` set variety='Setosa1' where sepal.length=5;

UPDATE ice.`flowers.irisv3` SET variety = 'Setosa1' WHERE sepal.length = 5

Query id: 3516a132-ba2f-4ff6-afdd-f049c891207d


Elapsed: 0.044 sec. 

Received exception from server (version 26.6.4):
Code: 48. DB::Exception: Received from localhost:9008. DB::Exception: Lightweight updates are not supported. Table with engine IcebergS3 doesn't support lightweight updates. (NOT_IMPLEMENTED)


insert into ice.`flowers.irisv3` values(100,100,100,0.2, 'Setos1');

INSERT INTO ice.`flowers.irisv3` FORMAT Values

Query id: 887cf79a-27fb-4950-88a1-11b7760fac28

Ok.

1 row in set. Elapsed: 0.779 sec. 

SELECT
    first_row_id,
    sequence_number
FROM system.iceberg_files
WHERE (database = currentDatabase()) AND (`table` = 'flowers.irisv3')

Query id: 91fdb261-fd56-4f68-9f64-d0efc2d14e88

   ┌─first_row_id─┬─sequence_number─┐
1. │          150 │               2 │
2. │            0 │               1 │
3. │          300 │               3 │
   └──────────────┴─────────────────┘

Copy link
Copy Markdown
Collaborator Author

@blau-ai

@blau-ai
blau-ai commented Oct 3, 2026
Copy link
Copy Markdown
Collaborator

CI triage for #2456

Verdict: no PR-caused failures. Every red check is either an infra/install flake, the perpetually-partially-red Altinity regression suite, or a container CVE scan — none of them touch the Iceberg code this PR changes, and the jobs that do exercise this PR's changes are green.

The decisive positive signal: the targeted jobs (the ones CI runs specifically against the files this PR changed) both passed —

  • Integration tests (amd_asan_ubsan, targeted) → Failures: 0/1 ✅
  • AST fuzzer (amd_debug, targeted) and …(targeted, old_compatibility) → OK ✅

and 7 of the 8 Integration tests (… old analyzer, N/8) shards passed with 0 failures across ~6,000 tests, as did every Stateless/Unit/Build/Fast-test job. The 4 new Iceberg row-lineage integration tests added by this PR ran green.


1. Install/decompress UBSan flake — NOT PR-related (just needs a re-run)

Two test jobs failed, both at the "Install ClickHouse" step, before a single test ran, with the identical error:

Decompressing the binary
/ClickHouse/utils/self-extracting-executable/decompressor.cpp:209:60: runtime error:
pointer index expression with base 0x… overflowed to 0x…
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior .../decompressor.cpp:209:60

Affected:

  • Stateless tests (amd_asan_ubsan, db disk, distributed plan, sequential, 1/2) — failed on Install ClickHouse (sibling shard 2/2 passed).
  • Integration tests (amd_asan_ubsan, db disk, old analyzer, 4/8) — job killed while running clickhouse --version right after unpacking the self-extracting binary (the other 7 shards passed).

decompressor.cpp is the self-extracting-executable unpacker; this PR does not touch it (gh pr diff → 0 lines matching decompressor). It's a known recurring UBSan flake in unpacking the sanitizer binary. Next step: re-run these two jobs — no code change needed.

2. AST fuzzer (amd_debug) — infra, NOT PR-related

Fuzzer runner aborted before writing status.tsv — the runner died before producing a result. The targeted AST fuzzer (which replays this PR's queries) passed. Safe to re-run.

3. Altinity regression suite — pre-existing / not PR-related

All Regression release * / RegressionTestsRelease / * reds are in aggregate_functions (1/2/3 + cas / cas-s3-cache variants), ClickHouseKeeper (no_ssl), LDAP (role_mapping), Common (settings), S3 (azure), CAS (selects), alter/attach, lightweight delete. These exercise aggregate functions, Keeper, LDAP, settings and Azure — feature areas a pure Iceberg-write change cannot affect. The base branch antalya-26.6 itself currently has red scheduled runs (MasterCI, NightlyUpload, NightlyKeeperNoFaults), so this suite is red independently of this PR. Not a backport regression; compare against the base-branch regression baseline rather than treating it as new.

4. Grype scans — pre-existing dependency CVEs, not code

Grype Scan …clickhouse-keeper (4 high/critical) and …clickhouse-server…-alpine (1 high/critical) are container-image vulnerability scans against base-image/dependency CVEs. Unrelated to this PR's source changes and present fleet-wide.


Summary table

Check Classification Action
Stateless (asan_ubsan, db disk, …, sequential, 1/2) Install/decompress UBSan flake Re-run
Integration (asan_ubsan, db disk, old analyzer, 4/8) Install/decompress UBSan flake Re-run
AST fuzzer (amd_debug) Runner aborted (infra) Re-run
Regression release * / RegressionTestsRelease/* Pre-existing Altinity regression suite (unrelated features) Compare vs base baseline
Grype Scan (keeper / server-alpine) Pre-existing container CVEs Not a code issue

Bottom line: nothing here indicates a defect in the backport. Re-running the two install-flaked test jobs + the AST fuzzer should clear the test reds; the regression and Grype reds are environmental/pre-existing and should be judged against the base branch. No code fix is warranted on this PR based on current evidence.

— @blau-ai · evidence from praktika result_pr.json for 6805979 and base-branch run history. I can't build/run ClickHouse in this container, so correctness is ultimately confirmed by a green re-run.

Poco::JSON::Object::Ptr initial_metadata_object = plan.initial_metadata_object;
std::unordered_map<Iceberg::IcebergPathFromMetadata, Iceberg::IcebergPathFromMetadata> manifest_file_renamings;
std::unordered_map<Iceberg::IcebergPathFromMetadata, Int64> manifest_file_sizes;
std::unordered_map<Iceberg::IcebergPathFromMetadata, Int64> manifest_file_row_counts;
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see this change (and other in this file, in Constant.h, and in IcebergMetadata.cpp) in original PR (https://github.com/ClickHouse/ClickHouse/pull/116171/changes) as well as don't see it in current upstream master (https://github.com/ClickHouse/ClickHouse/blame/master/src/Storages/ObjectStorage/DataLakes/Iceberg/Compaction.cpp#L1107).
From where are these additional changes?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @ianton-ru , This was added because optimize would fail for iceberg v3 because of this check, this adds support for compaction, if u think it should be a separate PR I can separate it out.

if (initial_metadata_object->getValue<Int32>(Iceberg::f_format_version) < 2)
    throw Exception(ErrorCodes::BAD_ARGUMENTS, "Compaction is supported only for format_version 2.");

@ianton-ru ianton-ru Oct 7, 2026 •
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But compaction for v3 is not compatible with v2. You can't reuse code for it.

@ianton-ru
Copy link
Copy Markdown

Audit of Altinity PR #2456

Audit of Altinity/ClickHouse#2456 at 1fc3c51a77b0, against antalya-26.6.

The PR backports ClickHouse/ClickHouse#116171, which writes Iceberg v3 next-row-id and first_row_id. It also changes data-file compaction. That compaction code is not in upstream master. Upstream master rejects format-version 3 for both compaction paths.

Confirmed defects

High: Data-file OPTIMIZE rewrites an Iceberg v3 table as format-version 2 and drops row lineage

  • Impact: With allow_experimental_iceberg_compaction enabled, OPTIMIZE TABLE on a v3 table that has position deletes commits format-version 2 metadata. next-row-id, snapshot first-row-id, and manifest first_row_id disappear. The new metadata file gets a new table UUID and a 8000 n empty partition spec. Deletion vectors are not copied, so rows deleted only by a deletion vector can come back when position deletes are also present. OPTIMIZE TABLE ... MANIFEST refuses v3.
  • Anchor: Compaction.cpp / getPlan and writeMetadataFiles; IcebergWrites.cpp / generateManifestList
  • Trigger: v3 table with position deletes, then OPTIMIZE TABLE.
  • Why defect: getPlan rejects only format-version < 2, so v3 proceeds. writeMetadataFiles rebuilds metadata through createEmptyMetadataFile, whose default format version is 2. A v3 table is downgraded and loses row lineage. generateManifestList throws NOT_IMPLEMENTED for a format-version 3 manifest rewrite, but that check reads the rebuilt metadata object, which is already version 2, so the throw does not run. ManifestListEntryCounts are written into that v2 manifest list. Upstream master rejects this path with format_version != 2 for the same reason.
  • Fix direction: Reject v3 in getPlan with format_version != 2 until compaction preserves format-version 3 and each file's first_row_id.
  • Regression test direction: v3 table with a position delete, OPTIMIZE, then assert the statement throws and the table stays format-version 3.

Medium: The first ClickHouse commit after a v2-to-v3 upgrade does not assign row ids to existing data manifests

  • Impact: Pre-upgrade rows stay _row_id = NULL. next-row-id advances only by the newly inserted rows.
  • Anchor: IcebergWrites.cpp / generateManifestList; MetadataGenerator::generateNextMetadata
  • Trigger: Upgrade a v2 table to v3, then INSERT from ClickHouse.
  • Why defect: Parent manifest-list entries are copied with a null first_row_id. New ids are assigned only to manifests added by this commit. The Iceberg spec requires the first post-upgrade snapshot to assign first_row_id to every data manifest that does not already have one, and to advance next-row-id past those existing rows as well as the added ones.
  • Fix direction: Assign first_row_id to copied data manifests whose value is null, and advance next-row-id by added_rows_count + existing_rows_count.
  • Regression test direction: Spark v2 table, upgrade to v3, one ClickHouse append, then assert old and new rows have non-null _row_id.

Coverage summary

  • Scope reviewed: tip 1fc3c51a77b0 against antalya-26.6. Insert, import, mutation, data-file OPTIMIZE, and OPTIMIZE ... MANIFEST.
  • Missed-class probes: 6/12 passed; failed: P1, P2, P9, P10; N/A: P6, P11, P12.
  • Categories failed: v3 data-compaction rewrite; post-upgrade manifest first_row_id assignment.
  • Categories passed: v3 append row ids for ClickHouse-created tables; v3 DELETE/UPDATE rejected; manifest OPTIMIZE fail-closed on v3; v2 per-manifest added-row counts.
  • Assumptions/limits: Static review of 1fc3c51a77b0. Compaction was not executed.

@subkanthi
Copy link
Copy Markdown
Collaborator Author

Medium: The first ClickHouse commit after a v2-to-v3 upgrade does not assign row ids to existing data manifests

Hi @ianton-ru , addressed HIGH and LOW, this medium is a pre-existing bug, please let me know if I should address that here or on a separate PR.

@ianton-ru
ianton-ru commented Oct 16, 2026 •
Copy link
Copy Markdown

@subkanthi
Please split PR in two, backport and new features.
New features must be frontported in next Antalya versions, but backports not, because already come from upstream.
Mixed PR makes this process harder.

@ianton-ru
Copy link
Copy Markdown

Audit of Altinity PR #2456

Audit of Altinity/ClickHouse#2456 at a2f5210fb81f, against antalya-26.6.

The PR backports ClickHouse/ClickHouse#116171, which writes Iceberg v3 next-row-id and first_row_id. It also changes data-file compaction. Data-file OPTIMIZE and OPTIMIZE TABLE ... MANIFEST both reject format-version 3. getPlan throws when format-version != 2, before any file rewrite, because writeMetadataFiles rebuilds metadata through createEmptyMetadataFile at format-version 2.

Confirmed defects

Medium: The first ClickHouse commit after a v2-to-v3 upgrade does not assign row ids to existing data manifests

  • Impact: Pre-upgrade rows stay _row_id = NULL. next-row-id advances only by the newly inserted rows.
  • Anchor: IcebergWrites.cpp / generateManifestList; MetadataGenerator::generateNextMetadata
  • Trigger: Upgrade a v2 table to v3, then INSERT from ClickHouse.
  • Why defect: Parent manifest-list entries are copied with add_field_to_datum for first_row_id. A v2 manifest has no such field, so the copied entry stays null. New ids are assigned only to manifests added by this commit, starting at the snapshot first-row-id, and next-row-id increases only by added_records. The Iceberg spec requires the first post-upgrade snapshot to assign first_row_id to every data manifest that does not already have one, and to advance next-row-id past those existing rows as well as the added ones.
  • Fix direction: Assign first_row_id to copied data manifests whose value is null, and advance next-row-id by added_rows_count + existing_rows_count.
  • Regression test direction: Spark v2 table, upgrade to v3, one ClickHouse append, then assert old and new rows have non-null _row_id.

Coverage summary

  • Scope reviewed: tip a2f5210fb81f against antalya-26.6. Insert, import, mutation, data-file OPTIMIZE, and OPTIMIZE ... MANIFEST.
  • Missed-class probes: 9/12 passed; failed: P9; N/A: P6, P11, P12.
  • Categories failed: post-upgrade manifest first_row_id assignment.
  • Categories passed: v3 data-file OPTIMIZE rejected before rewrite; manifest OPTIMIZE rejected for v3; v3 append row ids for ClickHouse-created tables; v3 DELETE/UPDATE rejected; v2 per-manifest added-row counts.
  • Assumptions/limits: Static review of a2f5210fb81f. test_optimize_v3_rejected was not executed.

@subkanthi
Copy link
Copy Markdown
Collaborator Author

@subkanthi Please split PR in two, backport and new features. New features must be frontported in next Antalya versions, but backports not, because already come from upstream. Mixed PR makes this process harder.

@ianton-ru this PR is only the backport, the Medium is a pre-existing bug inherited from upstream that I plan to address in a separate PR

@ianton-ru
Copy link
Copy Markdown

Aha, I see, changes in ManifestListEntryCounts are from ClickHouse#111617.

@ianton-ru ianton-ru left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ianton-ru
Copy link
Copy Markdown

@blau-ai

@blau-ai
blau-ai commented Oct 20, 2026
Copy link
Copy Markdown
Collaborator

CI triage for a2f5210

Verdict: 7 failing checks, 0 PR-caused. All red checks are pre-existing / flaky / infra failures in subsystems this PR does not touch. This PR's diff is limited to src/Storages/ObjectStorage/DataLakes/Iceberg/* (+ iceberg integration tests), and every check that actually exercises those code paths is green (see "PR health" below). No code change is needed to address the red checks — they just need re-runs / are known flakes.

Failing checks

1. Stateless tests (amd_asan_ubsan, distributed plan, parallel, 2/2) — flaky, not PR-related

  • Failing test: 03572_export_merge_tree_part_limits_and_table_functions.
  • CI's own rerun diagnosis: "Re-running with the same randomized settings… Runs: 1, Failed: 0, Passed: 1. All reruns passed. The failure is not reproducible (likely a transient issue)." The result is labeled flaky and carries do_not_block_pipeline_on_failure: true.
  • The diff was a timing/count mismatch while waiting for MergeTree→Parquet exports to complete ("Timeout waiting for exports to complete after 60s", 3 files vs expected 5). This is the MergeTree part-export path, unrelated to Iceberg v3 manifest writing.
  • → Safe to re-run.

2. Stateless tests (amd_binary, cas s3 storage, parallel) — pre-existing engine crash, not PR-related

  • Failed: 0 for actual tests; the check went red on two server-health markers: Server died + Logical error: 'GroupingAggregatedTransform pushed bucket 5 twice'.
  • Stack is entirely in the aggregation pipeline — src/Processors/Transforms/MergingAggregatedMemoryEfficientTransform.cpp:63 … GroupingAggregatedTransform::pushData → tryPushTwoLevelData → prepare. No Iceberg frames. Marked do_not_block_pipeline_on_failure: true (labels blocker/log_check, i.e. a server-log health check, not a test case).
  • This is a memory-efficient two-level aggregation bug that cannot be reached by an Iceberg-write-only diff. → Pre-existing; re-run / track separately.

3. AST fuzzer (amd_debug) — flaky server crash, not PR-related

  • Result: Unknown error / Lost connection to server.
  • The fuzzer completed all 1000/1000 fuzz steps; the connection was lost only during the final step / table-cleanup phase. The queries in flight were local-table bitmap aggregates (SELECT * APPLY bitmapMin/bitmapMax FROM (SELECT * APPLY groupBitmapState FROM t …)) over a MergeTree table t — no object storage, no Iceberg, no writes.
  • Corroboration: AST fuzzer (amd_debug, targeted) — the variant that specifically fuzzes queries related to the changed files — passed. → Safe to re-run.

4–7. Regression release — cas_selects, cas_lightweight_delete_4, cas_s3_cache_alter_attach_3, clickhouse_keeper_no_ssl_1 (Altinity testflows) — not PR-related

  • These suites exercise plain SELECTs, lightweight DELETE, S3-cache + ATTACH PARTITION, and ClickHouse Keeper (no-SSL) respectively. None of them touch the Iceberg object-storage write path that this PR modifies, so this diff cannot affect their outcome.
  • These are the standard per-PR Antalya regression suites, which routinely report a handful of flaky scenarios (e.g. cas_s3_cache_alter_attach_3: 8/3427 scenarios failed; cas_selects: 8/88). → Treat as pre-existing regression-suite flakes; confirm against a recent antalya-26.6 base run and re-run if needed.

("PR" is just the aggregate gate — it is red solely because of the checks above.)

PR health (the parts that test this change — all green ✅)

  • Build amd_debug / amd_asan_ubsan / amd_binary / amd_release / arm_release — all Failures: 0.
  • Integration tests (amd_asan_ubsan, targeted) — Failures: 0/1. This is the suite that runs the tests affected by the changed files, i.e. the new test_writes_v3_row_lineage*.py, test_row_lineage*.py, test_optimize.py.
  • Integration tests (amd_asan_ubsan, db disk, old analyzer, 1–8/8) — all Failures: 0.
  • AST fuzzer (amd_debug, targeted) and (targeted, old_compatibility) — OK.
  • Fast test, Stateless (amd_debug parallel/sequential, s3 storage, cas storage), Unit tests, SQLLogic, SQLStorm, Stress, BuzzHouse — OK.

Bottom line: the Iceberg v3 manifest-write change looks healthy; the red X's are unrelated flakes/pre-existing issues. I'd re-run checks 1–7 (especially the AST fuzzer and the two stateless jobs). No fix to the PR is warranted for these failures. Happy to dig deeper into any specific one if you'd like.

🤖 Triage by @blau-ai from the praktika S3 report + logs for a2f5210. I can't build/run ClickHouse in this environment, so verdicts are evidence-based from CI artifacts, not a local repro.

@subkanthi subkanthi added the port-antalya PRs to be ported to all new Antalya releases label Oct 20, 2026
@Selfeer
Selfeer commented Oct 22, 2026
Copy link
Copy Markdown
Collaborator

Seems okay to merge,marking as verified


PR #2456 CI Triage

Bottom line

No failure in this run comes from this PR's changes. The diff writes Iceberg v3 statistics (next_seq_num, first_row_id) in IcebergWrites, AvroSchema, compaction, and Iceberg mutations, and adds row-lineage tests. Every job that exercises that code passed.

The workflow is red because of seven jobs. Each one fails in code this PR does not touch: CAS disk checkpoint and mount-lease handling, ClickHouse Keeper log replay, two-level aggregation, materialized-CTE planning, an EXPORT PART wait, and a concurrent MergeTree lightweight delete.

What this PR ran, and passed

Job Result
Regression iceberg_1 44/44 features ok, 484 scenarios ok, 9 xfail
Regression iceberg_2 module ok, 1849 scenarios ok
Integration tests (amd_asan_ubsan, db disk, old analyzer, 1/8–8/8) 0 failures on every shard
Fast test, unit tests, stress, BuzzHouse, SQLLogic, SQLStorm success
Grype scans 0 high/critical

The new and updated tests (test_writes_v3_row_lineage.py, test_writes_v3_row_lineage_partitioned.py, test_optimize.py, test_row_lineage.py, test_row_lineage_pruning.py) live in those integration shards. No integration shard reported a failure.

Funnel

Seven GitHub jobs are red. The report's comparison with the base SHA lists 4 check rows and 14 regression rows. Parent TestFlows paths and Server died are consequences of a smaller set of roots, so they are cascade. What remains is seven roots. None of them has a path from the Iceberg write diff.

gh-data was not queried (no database password in this session). Branch rates for the two server logical errors come from upstream CI (play.clickhouse.com, last 30 days). Regression history comes from this PR's own earlier runs (Sep 30 36761849316, Oct 1 36887896922, Oct 11 38184000558) and from the error text.

Failures

Category What Why it is not this PR
pre-existing-flaky 03572_export_merge_tree_part_limits_and_table_functions on stateless asan distributed-plan 2/2 The test exports a MergeTree part to an S3 Parquet table (ALTER TABLE … EXPORT PART, allow_experimental_export_merge_tree_part). One export was still running when the 60s wait expired (Timeout waiting for exports, 3 files instead of 5). The job's own diagnosis re-ran it with the same settings: 1 run, 0 failed. The report labels the row flaky. This PR does not change export-part or the S3 table engine.
pre-existing-flaky AST fuzzer (amd_debug): Unknown error / lost connection The server aborted (exit 134, signal 6) on Logical error: Reading from materialized CTE 'b' before its materialization completed - DelayedPortsProcessor gate is missing in the query plan. Stack is MemorySource::generate → pipeline executor. The crashing query is a WITH b AS MATERIALIZED / GLOBAL NOT IN fuzzer query. Upstream CI recorded this logical error 60 times across 45 pull requests since 2026-09-24 (STID 2467-34d3). No Iceberg frame in the stack.
pre-existing-flaky Logical error: GroupingAggregatedTransform pushed bucket A twice on stateless amd_binary cas s3 The suite itself finished Failed: 0, Passed: 4864. The post-run log check found GroupingAggregatedTransform pushed bucket 5 twice in MergingAggregatedMemoryEfficientTransform.cpp. Upstream CI recorded that logical error 87 times in 30 days, 26 of them on master (pull_request_number = 0), last seen 2026-10-21. Aggregation, not Iceberg writes.
cascade Server died on the same stateless cas s3 job The server was already gone when the harness flushed logs (Connection refused on port 9000). Same job as the grouping logical error above. No OOM line in the dmesg check.
pre-existing-flaky /selects/final/force/concurrent/* on cas_selects Every leaf is Code: 210 … content-addressed disk 'cas_disk' -- mount lease not held. This job failed on all 4 completed runs of this PR (Sep 30, Oct 1, Oct 11, Oct 14). The CAS disk mount lease is unrelated to Iceberg statistics. Parent paths (/selects/final, /selects) are cascade.
pre-existing-flaky /alter/attach partition/part 3/… on cas_s3_cache_alter_attach_3 Inserts fail with CAS write could not be committed (CAS _ckpt … persistent CAS contention, the checkpoint contribution was not published). One leaf also hits Data is not yet replicated after that. The same job failed on the Oct 1 run. Checkpoint publication on the CAS disk, not Iceberg. Parent feature paths are cascade.
pre-existing-flaky /lightweight delete/concurrent delete/MergeTree/random delete 75 percent of the table without overlap on cas_lightweight_delete_4 After concurrent DELETEs on a plain MergeTree table, SELECT count() returned 253 and the test expected 250. The table engine in the failing scenario is MergeTree, not Iceberg. cas_lightweight_delete_4 also failed on the Sep 30 run and passed on the Oct 11 run. Parent paths are cascade. The Iceberg Mutations.cpp edit in this PR is not on this path.
pre-existing-flaky /clickhouse keeper/part 1/cli converter/* on clickhouse_keeper_no_ssl_1 Keeper on clickhouse7 aborted in KeeperStateMachine: Got new ZXID (1) smaller or equal to current ZXID (2792) and Trying to rollback invalid ZXID (2792). Later CLI steps then get Connection refused (localhost:9000), and config cleanup asserts exitcode == 1 while the uid is still in the preprocessed config. The same job failed on the Oct 1 run and passed on Oct 11 and Sep 30. Keeper log replay, not Iceberg.

Recommendations

  1. Nothing in this run needs a change in Antalya 26.6: Backport of #116171 to write iceberg v3 next_seq_num and first_row_id #2456. Iceberg regression and the integration shards that hold the new row-lineage tests are already green.
  2. The two logical errors (DelayedPortsProcessor STID 2467-34d3, GroupingAggregatedTransform STID 3994-4c1d) are already failing across upstream CI. Track them there, not on this backport.
  3. cas_selects has failed on every run of this PR with the CAS mount-lease error. That belongs with the existing CAS mount-lease work, not with Iceberg v3 statistics.

@Selfeer Selfeer added the verified Approved for release label Oct 22, 2026
@zvonand
zvonand merged commit f409075 into antalya-26.6 Oct 22, 2026
475 of 490 checks passed
@zvonand zvonand removed the port-antalya PRs to be ported to all new Antalya releases label Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

0