8000 Antalya 26.8, backport of 126267: Use less s3 requests to get metadata file by ianton-ru · Pull Request #2834 · Altinity/ClickHouse · GitHub
[go: up one dir, main page]

Skip to content

Antalya 26.8, backport of 126267: Use less s3 requests to get metadata file - #2834

Open
ianton-ru wants to merge 1 commit into
antalya-26.8from
feature/antalya-26.8/iceberg-fewer-s3-metadata-requests
Open

ianton-ru wants to merge 1 commit into
antalya-26.8from
feature/antalya-26.8/iceberg-fewer-s3-metadata-requests

Conversation

@ianton-ru
Copy link
Copy Markdown

Backport of ClickHouse#126267 by @scanhex12.

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Use less s3 requests to get metadata file (ClickHouse#126267 by @scanhex12)

Documentation entry for user-facing changes

Reduce the number of S3 requests when reading Iceberg tables by reusing metadata lookups and caching parsed metadata.json files: one GET and one LIST instead of three GETs and two LISTs.

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

Backport ClickHouse#126267 onto `antalya-26.8`.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions
Copy link
Copy Markdown

Workflow [PR], commit [02113e6]

@ianton-ru
Copy link
Copy Markdown
Author

@blau-ai

@blau-ai
blau-ai commented Nov 16, 2026
Copy link
Copy Markdown
Collaborator

CI triage for 02113e654c

Summary: 5 failing checks. None are caused by this PR. 3 are known failures on the base branch, 1 is a job timeout that also hits unrelated PRs, and 1 is DCO (process only, not a code problem).

To check this I compared against #2846, an unrelated Decimal backport on antalya-26.8 that ran in the same CI window. It fails with the same test names and the same scenarios in every one of these checks.

Check Verdict
Integration (amd_asan_ubsan, db disk, old analyzer, 1/8): test_database_glue::test_create_table_engine_backend_mismatch_rejected Already failing on base (stale assertion); fix is in #2869
Integration (… 7/8): 2× test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts Already failing on base / infra
Stateless (amd_asan_ubsan, cas s3 storage, parallel, 1/2 and 2/2) Job timeout, not a test failure; same on other PRs
Regression release swarms Already failing on base
DCO Commit has no Signed-off-by line

1. test_database_glue/test.py::test_create_table_engine_backend_mismatch_rejected

E   AssertionError: Received exception from server (version 26.8.34):
E     Code: 36. DB::Exception: ... Table setting `disk` differs from its value in this DataLakeCatalog. ...
    5. src/Databases/DataLake/DatabaseDataLake.cpp:1213:23: DB::DatabaseDataLake::validateCreateTableEngine

The server rejects the query with the error it is supposed to raise; the test's assertion just doesn't match it any more. This PR doesn't touch DatabaseDataLake.cpp. The same test fails on #2846 and #2831. Next step: nothing to do here. Open PR #2869 fixes the assertion.

2. test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts.py (both tests)

File: test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts.py:51 - in started_cluster
E   Exception: Timed out while waiting for instance `node' with ip address 172.16.4.6 to start.

The cluster fixture fails before any test code runs. This is a backward-compatibility test with an old server, and the failure is in startup, not in Iceberg code. Both tests fail the same way on #2846. Next step: nothing PR-specific.

3. Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2 and 2/2): ERROR

WARNING: Timeout exceeded [12600] for [4621]
WARNING: Job timed out: [Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2)], timeout [12600], exit code [-15]

This is a job budget problem with this configuration (ASan plus the CAS S3 disk is slow). It doesn't come from this PR. Next step: nothing PR-specific. A CI-side fix would be more shards or a higher timeout for this job.

4. Regression release swarms

The failing scenarios are the same as on #2846:

✘ [ Fail ] '/swarms/feature/cluster discovery/check cluster discovery with wrong cluster name'
✘ [ Fail ] '/swarms/feature/node failure/cpu overload'
✘ [ Fail ] '/swarms/feature/node failure/cpu overload all swarm nodes'

Example error: Code: 279 ... Cannot connect to any replica for query execution. (ALL_CONNECTION_TRIES_FAILED). These cover cluster discovery and node overload, which this PR doesn't touch. Next step: none.

5. DCO

The only commit (02113e654c, author Konstantin Vedernikov) has no Signed-off-by: line. Other backports (#2846, #2845) fail DCO for the same reason.

Next step (optional): to turn DCO green, amend the commit with git commit --amend -s and push it. That needs a force-push, which I'm not allowed to do. The other option is to have a maintainer override DCO in the GitHub UI.


Bottom line: none of the failures need a code change in this PR. The new Iceberg tests pass in CI.

Builds and tests on the arm, tsan and msan variants were skipped by this PR's CI options, so this PR has no signal from them.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

antalya antalya-26.8 Session label (releasy session config) backport Backport

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

0