Repository navigation
Metrics exporter: read the key-value asynchronous metrics of ClickHouse 26.8 - #2136
lizardoluis wants to merge 3 commits into
Conversation
…se 26.8 ClickHouse 26.8 publishes the per CPU, block device, network interface and similar asynchronous metrics as one key-value metric each. Read them from the key_values column and publish them with a key label, keeping the exclusion regexps written for the old names working. Signed-off-by: Luís Lizardo <lizardo.luis@gmail.com>
There was a problem hiding this comment.
thanks for the PR, few comments inline, the main one is about keeping legacy metric names
| toString(kv.2) AS value, | ||
| '' AS description, | ||
| 'gauge' AS type, | ||
| kv.1 AS key |
There was a problem hiding this comment.
main issue: this breaks existing dashboards and alerts on 26.8+
DiskTotal_default, NetworkReceiveBytes_eth0, BlockReadBytes_sda, OSUserTimeCPU3 become DiskTotal{key="default"}, NetworkReceiveBytes{key="eth0"} etc.
please export legacy names for families which have them
ClickHouse keeps a fixed table of ~50 families in src/Common/AsynchronousMetrics.cpp (getLegacyAsynchronousMetricName), naming is not uniform:
- suffix:
OSUserTimeCPU+3 _suffix:CPUFrequencyMHz_0,Block*_sda,Network*_eth0,Disk*_<disk>- infix / renamed family:
EDACCorrectable->EDAC0_Correctable,AsyncLoggingQueueSize->AsyncLoggingTextLogQueueSize - prefix:
DeadBlobsQueueEstimate->s3_diskDeadBlobsQueueEstimate Temperature: numeric key ->Temperature3, otherwiseTemperature_<key>
port this table to Go and map (metric, key) -> legacy name, use key label only for families without legacy name (added after 26.8)
There was a problem hiding this comment.
Done now. I ported the getLegacyAsynchronousMetricName table to Go (async_metrics_legacy_names.go) and the fetcher now maps (metric, key) to the legacy name, so DiskTotal_default, NetworkReceiveBytes_eth0, OSUserTimeCPU3, EDAC0_Correctable and the rest keep their exported names. The key label is now only used for families without a legacy name. The test cases mirror ClickHouse's gtest_async_metrics_key_values_mode.cpp, including Temperature.
| if key == "" { | ||
| return false | ||
| } | ||
| return IsExcluded(filter, metric+key) || IsExcluded(filter, metric+"_"+key) |
There was a problem hiding this comment.
metric+key / metric+"_"+key covers only suffix families, for EDAC, AsyncLogging, Dead/MissingBlobsQueueEstimate it builds wrong legacy name, so user excludeRegexp for them silently stops working
with legacy name mapping from the comment above this helper becomes just IsExcluded(filter, legacyName)
There was a problem hiding this comment.
isKeyExcluded is removed. Exclusion is now checked against the legacy name returned by the mapping, so excludeRegexp entries for EDAC, AsyncLogging and the blob queue families work. There is a test with metric.EDAC0_Correctable excluded.
| 'gauge' AS type, | ||
| '' AS key | ||
| FROM system.asynchronous_metrics | ||
| WHERE empty(key_values) |
There was a problem hiding this comment.
with asynchronous_metrics_key_values_mode: both ClickHouse publishes legacy scalar rows (empty map) and family rows (with map) at the same time, so every value is exported twice: NetworkReceiveBytes_eth0 and NetworkReceiveBytes{key="eth0"}
please skip family rows when legacy rows for the same family exist (or dedup by legacy name after mapping)
| return f.clickHouseQueryScanRows( | ||
| ctx, | ||
| f.buildMetricsSQL(), | ||
| f.buildMetricsSQL(f.hasAsyncMetricsKeyValues(ctx)), |
There was a problem hiding this comment.
system.columns probe runs on every scrape for every host, could be cached per host in Exporter, column can't disappear without server restart
There was a problem hiding this comment.
Done, with one difference from the suggestion. The result is cached per host in Exporter. A positive answer is kept until a metrics query fails, which handles a downgrade. A negative answer expires after 5 minutes, because a 26.3 host upgraded in place to 26.8 would otherwise keep reading the NaN rows until the exporter restarts.
| w.writeSingleMetricToPrometheus(name, desc, metricType, value, nil) | ||
| var labels map[string]string | ||
| if len(metric) > 4 && metric[4] != "" { | ||
| labels = map[string]string{"key": metric[4]} |
There was a problem hiding this comment.
key is too generic and silently collides with a CHI label named key (MergeStringMapsOverwrite), if label still needed for families without legacy name, maybe use more specific name, e.g. async_metric_key
There was a problem hiding this comment.
Renamed the label to async_metric_key. Now it only appears on families that have no legacy name.
| '' AS description, | ||
| 'counter' AS type | ||
| 'counter' AS type, | ||
| '' AS key |
There was a problem hiding this comment.
nit: indentation of '' AS key differs from the rest of the query, same at L64, L72, L80, L88
…us metrics Port the legacy name table of ClickHouse to Go and export the elements of key-value asynchronous metrics under the names they had before 26.8 (DiskTotal_default, OSUserTimeCPU3, EDAC0_Correctable...), so existing dashboards, alerts and exclusion regexps keep working. Families without a legacy name keep the async_metric_key label. With asynchronous_metrics_key_values_mode = both a scalar row wins over the renamed key-value row of the same name, so the value is exported once. Cache the key_values column probe per host instead of running it on every scrape.
Fixes #2135
What
ClickHouse 26.8 publishes the per CPU, block device, network interface, disk and similar asynchronous metrics as one key-value metric each (
asynchronous_metrics_key_values_mode, defaultkey_values). Thevaluecolumn holdsnanand the values are in the newkey_valuesmap, which the exporter did not read, so it exposedNaN.system.columnsforasynchronous_metrics.key_values, cached per host (a positive answer until a metrics query fails, a negative one for 5 minutes). Without the column (ClickHouse before 26.8) the query is the same as before.ARRAY JOIN key_values AS kv.chi_clickhouse_metric_OSUserTimeCPU3,..._DiskTotal_default,..._EDAC0_Correctable), using a Go port of ClickHouse'sgetLegacyAsynchronousMetricNametable. A family without a legacy name (none exists today) is exported with anasync_metric_keylabel.asynchronous_metrics_key_values_mode: boththe server also publishes the legacy scalar rows, which win over the renamed key-value rows, so each value is exported once.Why
On 26.8 the per CPU, disk and network metrics were lost (exposed as
NaN). Keeping the pre 26.8 names means existing dashboards, alerts and exclusion regexps keep working.Manual tests
pkg/metrics/clickhouse.