8000 add `Detached Parts` instead of `Inactive Parts` to Grafana dashboard by Slach · Pull Request #682 · Altinity/clickhouse-operator · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion Vagrantfile
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,8 @@ Vagrant.configure(2) do |config|
curl -sL https://github.com/liggitt/audit2rbac/releases/download/v${AUDIT2RBAC_VERSION}/audit2rbac-linux-amd64.tar.gz | tar -zxvf - -C /usr/local/bin

# minikube
MINIKUBE_VERSION=1.19.0
MINIKUBE_VERSION=1.18.1
# MINIKUBE_VERSION=1.19.0
wget -c --progress=bar:force:noscroll -O /usr/local/bin/minikube https://github.com/kubernetes/minikube/releases/download/v${MINIKUBE_VERSION}/minikube-linux-amd64
chmod +x /usr/local/bin/minikube
# required for k8s 1.18+
Expand Down
7 changes: 7 additions & 0 deletions deploy/prometheus/prometheus-alert-rules.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -489,6 +489,13 @@ spec:
Please check detached parts in log ```kubectl logs -n {{ $labels.exported_namespace }} pod/$(kubectl get pods -n {{ $labels.exported_namespace }} | grep $( echo {{ $labels.hostname }} | cut -d '.' -f 1) | cut -d " " -f 1) -c clickhouse-pod --since=1h | grep -i detach```
Read about how ATTACH / DROP detached parts
https://clickhouse.tech/docs/en/operations/system-tables/detached_parts/
Legend for reason:
`detached_by_user` - part was detached via ALTER TABLE ... DETACH ... query.
`clone` - old part was detached during clone replica (see `detach_old_local_parts_when_cloning_replica` system.settings)
`ignored` - part was unexpected with Zookeeper during restart clickhouse-server
`broken` - part was marks as broken during startup or merging (check disk, memory, network hardware faillures), look to clickhouse-server.log for details
`unexpected` - during replica startup, part was found in local file system, modification of part was less than 5 minutes, but not found in ZK
`noquorum` - part was detached, cause part was created during insert into Distributed table with quorum, but quorum was failed.

# based on https://blog.serverdensity.com/how-to-monitor-zookeeper/
# and https://github.com/apache/zookeeper/blob/master/zookeeper-contrib/zookeeper-contrib-monitoring/nagios/services.cfg
Expand Down
52 changes: 38 additions & 14 deletions grafana-dashboard/Altinity_ClickHouse_Operator_dashboard.json
8000
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@
"gnetId": 882,
"graphTooltip": 1,
"id": null,
"iteration": 1602226784521,
"iteration": 1619238572345,
"links": [],
"panels": [
{
Expand Down Expand Up @@ -2338,13 +2338,8 @@
"links": [
{
"targetBlank": true,
"title": "system.parts",
"url": "https://clickhouse.tech/docs/en/operations/system-tables/#system_tables-parts"
},
{
"targetBlank": true,
"title": "parts_to_delay_insert",
"url": "https://github.com/ClickHouse/ClickHouse/search?q=parts_to_delay_insert"
"title": "system.detached_parts",
"url": "https://clickhouse.tech/docs/en/operations/system-tables/detached_parts/"
}
],
"nullPointMode": "null",
Expand All @@ -2355,22 +2350,51 @@
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"seriesOverrides": [
{
"$$hashKey": "object:254",
"alias": "/.*detached_by_user.*/",
"color": "#CA95E5"
},
{
"$$hashKey": "object:285",
"alias": "/.*broken.*/",
"color": "#E02F44"
},
{
"$$hashKey": "object:355",
"alias": "/.*(clone|ignored).*/",
"color": "#FFEE52"
},
{
"$$hashKey": "object:447",
"alias": "/^Inactive/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum by(hostname) (chi_clickhouse_table_parts{exported_namespace=~\"$exported_namespace\",chi=~\"$chi\",hostname=~\"$hostname\",active=\"0\"})",
"legendFormat": "Parts {{hostname}}",
"expr": "sum by(hostname,reason) (chi_clickhouse_metric_DetachedParts{exported_namespace=~\"$exported_namespace\",chi=~\"$chi\",hostname=~\"$hostname\"})",
"interval": "",
"legendFormat": "{{reason}} {{hostname}} ",
"refId": "C"
},
{
"expr": "sum by(hostname) (chi_clickhouse_table_parts{exported_namespace=~\"$exported_namespace\",chi=~\"$chi\",hostname=~\"$hostname\",active=\"0\"})",
"hide": true,
"interval": "",
"legendFormat": "Inactive {{hostname}} ",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Inactive Parts",
"title": "Detached parts",
"tooltip": {
"shared": true,
"sort": 0,
Expand Down Expand Up @@ -2403,8 +2427,8 @@
}
],
"yaxis": {
"align": false,
"alignLevel": null
"align": true,
"alignLevel": 0
}
},
{
Expand Down
0