8000 Fix shard weight. Now we can set it to zero by maxistua · Pull Request #1192 · Altinity/clickhouse-operator · GitHub
[go: up one dir, main page]
Skip to content

Fix shard weight. Now we can set it to zero - #1192

Merged
sunsingerus merged 4 commits into
Altinity:0.23.0from
maxistua:shard-weight
Nov 22, 2023
Merged

Fix shard weight. Now we can set it to zero#1192
sunsingerus merged 4 commits into
Altinity:0.23.0from
maxistua:shard-weight

Conversation

@maxistua
@maxistua maxistua commented Jul 13, 2023
Copy link
Copy Markdown
Contributor

I can't set the shard weight to zero
We need it for some reasons...

Hopefully this MR will fix that

@UnamedRus
UnamedRus commented Jul 21, 2023
Copy link
Copy Markdown

It's problem with other settings as well AFAIK.
If value is 0, it will not be materialized in xml config by operator

alex-zaitsev
alex-zaitsev requested changes Oct 21, 2023
View reviewed changes
@alex-zaitsev alex-zaitsev left a comment
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please provide a usage example. Ideally a test, but even example would be helpful.

@maxistua
maxistua commented Nov 21, 2023
Copy link
Copy Markdown
Contributor Author

Yes, imagine a system where historical data is filled from shard to shard.

And now, when a new shard is added, we need to limit writing to the previous ones.

We can write data directly to the node (which is more efficient in terms of resources), but we can also write it to a distributed table.

So, in order to limit writing to already filled shards, we can set their weight to 0.

Currently, it is not possible to disable writing completely. The minimum weight that can be set is 1.

An example of the desired configuration will be below:

configuration:
  clusters:
    - name: "data-lake"
      secret:
        auto: "True"
      layout:
        shards:
          - name: "1"
            weight: 0 <--
            replicasCount: 1
            ...
          - name: "2"
            weight: 0
            replicasCount: 1
            ...
          - name: "3"
            weight: 0
            replicasCount: 1
            ...
          - name: "4"
            weight: 1
            replicasCount: 1
            ...
          - name: "5"
            weight: 1
            replicasCount: 1
            ...
...

This example will not work because a weight of 0 will be considered an empty value and will not be included in the final configuration.

In this case, the weight of all shards will be equal to 1.

@Slach
Slach changed the base branch from 0.21.3 to 0.23.0 November 21, 2023 10:26
@Slach
Slach commented Nov 21, 2023
Copy link
Copy Markdown
Collaborator

@maxistua thanks for explanation, could you resolve conflicts?

@antip00 could you write test for this use case? which change weight to 0 and insert data into distributed table and check data not inserted into zero-weighted shard?

@sunsingerus
sunsingerus commented Nov 21, 2023
Copy link
Copy Markdown
Collaborator

I'll take a look on this PR

@maxistua
maxistua commented Nov 21, 2023
Copy link
Copy Markdown
Contributor Author

@maxistua thanks for explanation, could you resolve conflicts?

@antip00 could you write test for this use case? which change weight to 0 and insert data into distributed table and check data not inserted into zero-weighted shard?

@Slach All conflicts have been resolved

@Slach
Slach requested a review from sunsingerus November 22, 2023 03:39
@sunsingerus
sunsingerus merged commit 5033f9e into Altinity:0.23.0 Nov 22, 2023
sunsingerus pushed a commit that referenced this pull request Nov 29, 2023
@maxistua @sunsingerus
Fix shard weight. Now we can set it to zero (#1192)
ad3d500 
* Shard weight can be set to zero

(cherry picked from commit 5033f9e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alex-zaitsev alex-zaitsev alex-zaitsev requested changes
@sunsingerus sunsingerus Awaiting requested review from sunsingerus

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@maxistua @UnamedRus @Slach @sunsingerus @alex-zaitsev
0