8000 Fix wrong condition for direct io on export part/ partition by arthurpassos · Pull Request #2824 · Altinity/ClickHouse · GitHub
[go: up one dir, main page]

Skip to content

Fix wrong condition for direct io on export part/ partition - #2824

Open
arthurpassos wants to merge 1 commit into
antalya-26.8from
fix/antalya-26.8/export-part-direct-io
Open

arthurpassos wants to merge 1 commit into
antalya-26.8from
fix/antalya-26.8/export-part-direct-io

Conversation

@arthurpassos
Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fix wrong condition for direct io on export part/ partition

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

@chatgpt-codex-connector
chatgpt-codex-connector Bot commented Nov 10, 2026 •
Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-10T13:54:59.289320Z 2a79f34 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@arthurpassos arthurpassos added antalya port-antalya PRs to be ported to all new Antalya releases antalya-26.8 Session label (releasy session config) labels Nov 10, 2026
@github-actions
Copy link
Copy Markdown

Workflow [PR], commit [2a79f34]

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2a79f343a0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

bool apply_deleted_mask = true;
bool read_with_direct_io = local_context->getSettingsRef()[Setting::min_bytes_to_use_direct_io] > manifest.data_part->getBytesOnDisk();
const UInt64 min_bytes_to_use_direct_io = local_context->getSettingsRef()[Setting::min_bytes_to_use_direct_io];
bool read_with_direct_io = min_bytes_to_use_direct_io != 0 && manifest.data_part->getBytesOnDisk() >= min_bytes_to_use_direct_io;
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Apply the threshold to the whole exported partition

For EXPORT PARTITION, both partition schedulers invoke exportPartToTable separately for each part, so this comparison evaluates each part in isolation rather than the total data volume of the export. For example, a partition containing two 60 MiB parts with min_bytes_to_use_direct_io = 100 MiB reads 120 MiB overall but disables direct I/O for both parts, contrary to the setting's total-volume semantics. Persist the aggregate partition size or a precomputed direct-I/O decision in the partition descriptor and use that decision for every part; retain this per-part calculation only for EXPORT PART.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm... not sure if this is a good idea, it might be

@mkmkme mkmkme self-assigned this Nov 10, 2026
@DimensionWieldr
Copy link
Copy Markdown
Collaborator

Possible issue found by AI, cleaned up for conciseness.

An export can still open column files with O_DIRECT after min_bytes_to_use_direct_io is set to zero or a value above the system profile. Page cache is bypassed, and the read can fail on filesystems that reject unaligned direct I/O.

  • Proposed fix: For export reads, set direct_io_threshold from the task setting (and set it to 0 when the part-level decision is false) instead of keeping the table context value.

Related lines of code:

src/Storages/MergeTree/ExportPartTask.cpp lines 344-345
src/Storages/MergeTree/MergeTreeSequentialSource.cpp lines 153-157
src/Disks/IO/createReadBufferFromFileBase.cpp lines 169-169

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

antalya antalya-26.8 Session label (releasy session config) port-antalya PRs to be ported to all new Antalya releases

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

0