Repository navigation
Conversation
…probe Add `object_storage_cluster_local_read_max_files` and `object_storage_cluster_local_read_probe_ms`. When enabled, the initiator lists the object storage files for a `*Cluster` query for up to the time budget. If the listing finishes within the budget and yields at most the file limit, the query is read locally. Otherwise the query keeps the cluster path. The default of `0` keeps the existing behavior. Co-authored-by: Cursor <cursoragent@cursor.com>
The `object_storage_cluster_local_read_max_files` probe listed files and then discarded them, so the local read or the cluster task distributor listed the same files again. Keep the probe iterator and the objects it already returned on the query info, and replay them in `ReadFromObjectStorageStep` and in `getTaskIteratorExtension`. Placeholder metadata from `skip_object_metadata` is dropped for local reads so they fetch the real size. Queries that read `_tags` still list again, because the probe does not request tags. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the local-or-cluster decision on `ObjectStorageClusterReadProbe` instead of a second flag on `SelectQueryInfo`. A finished listing is represented by a null iterator, so the separate exhausted flag is gone. `createObjectIteratorForRead` always sets `tasks_go_to_other_replicas` and emits profile events, so those parameters are removed. Placeholder metadata from `skip_object_metadata` is no longer cleared for local reads, because the read already treats an unfetched object as an unknown size. Co-authored-by: Cursor <cursoragent@cursor.com>
…e children A `Merge` passes one `SelectQueryInfo` to every child, so a single probe slot let the first child's local-read decision apply to later children. The probe is now keyed by the cluster storage, and `05429_object_storage_cluster_local_read_merge` checks a small child read locally next to a large child that stays on the cluster. Co-authored-by: Cursor <cursoragent@cursor.com>
The probe now counts the same tasks the cluster would run, so with `bucket` split granularity each bucket counts, not each file. The setting `object_storage_cluster_local_read_max_files` is renamed to `object_storage_cluster_local_read_max_tasks`. Co-authored-by: Cursor <cursoragent@cursor.com>
…probe The probe drops a finished iterator, so a local read lost the `_data_lake_snapshot_version` that the cluster read reports. The probe now stores the snapshot version before it releases the iterator, and the replay iterator returns it. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5021b15138
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| auto object = iterator->next(0); | ||
| if (!object) | ||
| { | ||
| probe->read_locally = true; |
There was a problem hiding this comment.
When iterator->next(0) starts just before the deadline but a slow object-store listing or bucket split returns nullptr after it, this branch marks the probe local without checking elapsed time again. Consequently object_storage_cluster_local_read_probe_ms neither bounds planning latency nor preserves the documented rule that a listing finishing outside the budget stays distributed; use a deadline-aware/cancellable iterator operation and verify the deadline before accepting exhaustion as a local-read decision.
Useful? React with 👍 / 👎.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Select plain or swarm mode for cluster query
Documentation entry for user-facing changes
We have a
object_storage_clusterfeature to make distributed query to table on object storage.But small query can be executed faster locally, directly on initiator.
This feature allow to select local or distributed way based on expected query size.
It does not give 100% accuracy, it's a some kind of heuristic.
Distributed query is split on tasks for remote nodes.
Now we start to do in on early stage. If we get only few tasks, not greater than
object_storage_cluster_local_read_max_tasksand spend not more thanobject_storage_cluster_local_read_probe_msmilliseconds, we execute query locally. In other case, execute on cluster.CI/CD Options
Exclude tests:
Regression jobs to run: