Repository navigation
Antalya 26.6: Backport of #116171 to write iceberg v3 next_seq_num and first_row_id - #2456
Conversation
|
TESTING |
CI triage for #2456Verdict: no PR-caused failures. Every red check is either an infra/install flake, the perpetually-partially-red Altinity regression suite, or a container CVE scan — none of them touch the Iceberg code this PR changes, and the jobs that do exercise this PR's changes are green. The decisive positive signal: the targeted jobs (the ones CI runs specifically against the files this PR changed) both passed —
and 7 of the 8 1. Install/decompress UBSan flake — NOT PR-related (just needs a re-run)Two test jobs failed, both at the "Install ClickHouse" step, before a single test ran, with the identical error: Affected:
2. AST fuzzer (amd_debug) — infra, NOT PR-related
3. Altinity regression suite — pre-existing / not PR-relatedAll 4. Grype scans — pre-existing dependency CVEs, not code
Summary table
Bottom line: nothing here indicates a defect in the backport. Re-running the two install-flaked test jobs + the AST fuzzer should clear the test reds; the regression and Grype reds are environmental/pre-existing and should be judged against the base branch. No code fix is warranted on this PR based on current evidence. — @blau-ai · evidence from praktika |
| Poco::JSON::Object::Ptr initial_metadata_object = plan.initial_metadata_object; | ||
| std::unordered_map<Iceberg::IcebergPathFromMetadata, Iceberg::IcebergPathFromMetadata> manifest_file_renamings; | ||
| std::unordered_map<Iceberg::IcebergPathFromMetadata, Int64> manifest_file_sizes; | ||
| std::unordered_map<Iceberg::IcebergPathFromMetadata, Int64> manifest_file_row_counts; |
There was a problem hiding this comment.
I don't see this change (and other in this file, in Constant.h, and in IcebergMetadata.cpp) in original PR (https://github.com/ClickHouse/ClickHouse/pull/116171/changes) as well as don't see it in current upstream master (https://github.com/ClickHouse/ClickHouse/blame/master/src/Storages/ObjectStorage/DataLakes/Iceberg/Compaction.cpp#L1107).
From where are these additional changes?
There was a problem hiding this comment.
Hi @ianton-ru , This was added because optimize would fail for iceberg v3 because of this check, this adds support for compaction, if u think it should be a separate PR I can separate it out.
if (initial_metadata_object->getValue<Int32>(Iceberg::f_format_version) < 2)
throw Exception(ErrorCodes::BAD_ARGUMENTS, "Compaction is supported only for format_version 2.");
There was a problem hiding this comment.
But compaction for v3 is not compatible with v2. You can't reuse code for it.
Audit of Altinity PR #2456Audit of Altinity/ClickHouse#2456 at The PR backports ClickHouse/ClickHouse#116171, which writes Iceberg v3 Confirmed defectsHigh: Data-file
|
Hi @ianton-ru , addressed HIGH and LOW, this medium is a pre-existing bug, please let me know if I should address that here or on a separate PR. |
|
@subkanthi |
Audit of Altinity PR #2456Audit of Altinity/ClickHouse#2456 at The PR backports ClickHouse/ClickHouse#116171, which writes Iceberg v3 Confirmed defectsMedium: The first ClickHouse commit after a v2-to-v3 upgrade does not assign row ids to existing data manifests
Coverage summary
|
@ianton-ru this PR is only the backport, the Medium is a pre-existing bug inherited from upstream that I plan to address in a separate PR |
|
Aha, I see, changes in |
CI triage for a2f5210Verdict: 7 failing checks, 0 PR-caused. All red checks are pre-existing / flaky / infra failures in subsystems this PR does not touch. This PR's diff is limited to Failing checks1. Stateless tests (amd_asan_ubsan, distributed plan, parallel, 2/2) — flaky, not PR-related
2. Stateless tests (amd_binary, cas s3 storage, parallel) — pre-existing engine crash, not PR-related
3. AST fuzzer (amd_debug) — flaky server crash, not PR-related
4–7. Regression release —
("PR" is just the aggregate gate — it is red solely because of the checks above.) PR health (the parts that test this change — all green ✅)
Bottom line: the Iceberg v3 manifest-write change looks healthy; the red X's are unrelated flakes/pre-existing issues. I'd re-run checks 1–7 (especially the AST fuzzer and the two stateless jobs). No fix to the PR is warranted for these failures. Happy to dig deeper into any specific one if you'd like. 🤖 Triage by @blau-ai from the praktika S3 report + logs for a2f5210. I can't build/run ClickHouse in this environment, so verdicts are evidence-based from CI artifacts, not a local repro. |
|
Seems okay to merge,marking as verified PR #2456 CI Triage
Bottom lineNo failure in this run comes from this PR's changes. The diff writes Iceberg v3 statistics ( The workflow is red because of seven jobs. Each one fails in code this PR does not touch: CAS disk checkpoint and mount-lease handling, ClickHouse Keeper log replay, two-level aggregation, materialized-CTE planning, an What this PR ran, and passed
The new and updated tests ( FunnelSeven GitHub jobs are red. The report's comparison with the base SHA lists 4 check rows and 14 regression rows. Parent TestFlows paths and
Failures
Recommendations
|
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add Iceberg v3 statistics (next-seq-num and first_row_id). This PR allows to write those stats. Follow up to #2308 which included functionality to read both the stats columns. (ClickHouse#116171 by @scanhex12)
CI/CD Options
Exclude tests:
Regression jobs to run: