8000 🧭 Add root-cause and correlation work with its dashboard support by abahmed · Pull Request #606 · abahmed/kwatch · GitHub
[go: up one dir, main page]

7FFF
Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
fa534a0
📝 Add the root-cause round copy and dashboard spec
abahmed Nov 7, 2026
ad57261
⏱️ Report a workload short of replicas sooner when its autoscaler is …
abahmed Nov 7, 2026
bb5d684
🔗 Read dependency hosts from ConfigMaps and name the source in messages
abahmed Nov 7, 2026
c3d5f6b
🔎 Flag error bursts in healthy pods and check the stated cause again …
abahmed Nov 8, 2026
cc4117d
⏲️ Join workloads that fail seconds apart under a shared node
abahmed Nov 8, 2026
c5e8e11
🧭 Name where users reach what is affected and show the chain as a tim…
abahmed Nov 8, 2026
7a29b7b
🛡️ Gate wrong roots when logs, metrics or RBAC are missing
abahmed Nov 8, 2026
80c1739
💥 Rank changes by how many failing pods are downstream of them
abahmed Nov 8, 2026
d6cfcee
🔬 Settle a close call between two causes with one read-only check
abahmed Nov 8, 2026
3f54a1a
🖥️ Show the round's findings on the issue page
abahmed Nov 8, 2026
2f64a10
🧹 Split the log sampler, reach and runtime helpers for the linters
abahmed Nov 8, 2026
0a1f890
🧪 Wait for the provider health itself in its test, not for a shared e…
abahmed Nov 8, 2026
0f2fab6
🩹 Fix the blast share, close-call wording, sample baseline and cache …
abahmed Nov 8, 2026
6abf6c8
🧪 Test the blast share and cluster-scoped overlap, and reach pods thr…
abahmed Nov 8, 2026
767d3b8
🧹 Drop the unused return of the close-call endpoint lookup
abahmed Nov 8, 2026
44417e2
📊 Give the round's findings their own places on the issue page
abahmed Nov 8, 2026
5b9022b
🔍 Hold back a workload cause when the config it reads is unreadable
abahmed Nov 8, 2026
3b03213
🧱 Install kind and the metrics-server manifest without GitHub release…
abahmed Nov 8, 2026
95926fe
Merge remote-tracking branch 'origin/main' into root-cause-correlatio…
abahmed Nov 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -243,11 +243,23 @@ jobs:
# lookup fails, and the chart tests need Helm 4.
version: v4.3.1

- name: Create disposable Kubernetes cluster
# kind comes from the Go module proxy (checksum-verified), not
# GitHub release downloads, which can fail with 504s.
- name: Set up Go for Kind
if: steps.changes.outputs.chart == 'true'
uses: helm/kind-action@ea822c7bf1874f08e5a491df991c228edffc3376 # v1.15.2
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
cluster_name: kwatch-chart
go-version-file: go.mod

- name: Create disposable Kubernetes cluster
if: steps.changes.outputs.chart == 'true'
env:
KIND_VERSION: v0.30.0
run: |
set -euo pipefail
go install "sigs.k8s.io/kind@${KIND_VERSION}"
echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH"
"$(go env GOPATH)/bin/kind" create cluster --name kwatch-chart --wait 120s

- name: CRD lifecycle integration test
if: steps.changes.outputs.chart == 'true'
Expand Down
17 changes: 13 additions & 4 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,7 @@ jobs:
rbac: [ full, least-privilege ]
secrets: [ true, false ]
env:
# One name for the cluster kind-action creates and the script loads
# One name for the cluster kind creates and the script loads
# the image into.
KIND_CLUSTER_NAME: kwatch-ops
steps:
Expand All @@ -301,10 +301,19 @@ jobs:
# lookup fails, and the chart tests need Helm 4.
version: v4.3.1

- name: Create disposable Kubernetes cluster
uses: helm/kind-action@ea822c7bf1874f08e5a491df991c228edffc3376 # v1.15.2
# kind comes from the Go module proxy (checksum-verified), not
# GitHub release downloads, which can fail with 504s.
- name: Set up Go for Kind
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
cluster_name: ${{ env.KIND_CLUSTER_NAME }}
go-version-file: go.mod

- name: Create disposable Kubernetes cluster
run: |
set -euo pipefail
go install "sigs.k8s.io/kind@${KIND_VERSION}"
echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH"
"$(go env GOPATH)/bin/kind" create cluster --name "$KIND_CLUSTER_NAME" --wait 120s

# The scenarios' Kwatch image, which the smoke test only needs under
# another name.
Expand Down
5 changes: 4 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,11 @@ ALERT_QUALITY_TEST := $(GOTEST) -count=1 -timeout 600s ./internal/scenarios \
-run '^TestScorecardGates$$' -v
# The mutation check: the gates must fail when part of the root-cause
# engine is switched off, so a gate that cannot see a regression is caught.
# The second test replays every labelled scenario with logs denied, without
# metrics and with a trimmed RBAC, and fails on a wrong root stated.
ALERT_QUALITY_ABLATION_TEST := $(GOTEST) -count=1 -timeout 600s \
./internal/scenarios -run '^TestScorecardGatesSeeRootCauseRegressions$$' -v
./internal/scenarios -v -run \
'^(TestScorecardGatesSeeRootCauseRegressions|TestMissingEvidenceNeverStatesAWrongRoot)$$'

# Race tests are slower than the default per-package timeout allows for
# the scenario replay.
Expand Down
1 change: 1 addition & 0 deletions deploy/chart/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,7 @@ Karpenter annotation with `--set-string`.
| `dashboard.tls.secretName` | `kubernetes.io/tls` Secret kwatch serves https from itself, for a LoadBalancer or NodePort with nothing in front. Renewals are picked up without a restart | `""` |
| `dashboard.existingSecret` | Existing Secret with the keys `password` (sign-in) and `token` (API bearer token). Without it the chart ships `<release>-dashboard` empty and kwatch writes a generated password (`XXXX-XXXX-XXXX-XXXX`) and token into it once, so every renderer (`helm template`, Argo CD, Flux) sees the same manifest and the values never change. kwatch gets and updates only that Secret. Edits take effect after a restart; changing the password signs everyone out | `""` |
| `config.investigation.logs` | Pro evidence reads container logs (redacted, bounded). `false` reads none and the dashboard shows the log section as turned off | `true` |
| `config.investigation.sampleLogs` | Samples a bounded slice of the logs of Running and Ready pods for a burst of errors (redacted). `false` reads none | `true` |
| `dashboard.networkPolicyFrom` | With `networkPolicy.enabled`: extra peers (your ingress controller, `ipBlock`s for load balancer clients) allowed to reach the dashboard port. The release namespace and `healthIngressFrom` always may | `[]` |

Sign-ins last 12 hours. A password change or a kwatch restart (upgrade,
Expand Down
5 changes: 5 additions & 0 deletions deploy/chart/values.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -787,6 +787,11 @@
"type": "boolean",
"default": true,
"description": "Read container logs for evidence; false reads none"
},
"sampleLogs": {
"type": "boolean",
"default": true,
"description": "Sample the logs of Running and Ready pods for a burst of errors; false reads none"
}
}
}
Expand Down
1 change: 1 addition & 0 deletions deploy/config-catalog.meta.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,4 @@ ignoreNodeMessages|Compatibility|Legacy node-message suppression field.|deprecat
kubelet.insecureSkipVerify|Security|Skip kubelet serving certificate verification for direct node stats reads; only for self-signed kubelet certificates.|active|
watch.secrets|Scope|Watch Secrets (values hashed, never stored); false removes Secret RBAC and Secret-based checks report that they cannot verify.|active|
investigation.logs|Scope|Read container logs for Pro evidence (redacted, bounded); false reads no log and shows the log section as turned off.|active|
investigation.sampleLogs|Scope|Sample a bounded slice of the logs of Running and Ready pods for a burst of errors (redacted); false reads none.|active|
1 change: 1 addition & 0 deletions deploy/config-catalog.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,4 @@ dashboard.tlsCertFile|string|empty|Security|Certificate file kwatch serves on th
dashboard.tlsKeyFile|string|empty|Security|Key file for dashboard.tlsCertFile.|active|
investigation.logs|boolean|true|Scope|Read container logs for Pro evidence (redacted, bounded); false reads no log and shows the log section as turned off.|active|
dashboard.trustedProxies|list|null|Security|CIDRs or addresses of the proxies that terminate TLS in front of the dashboard; X-Forwarded-For and X-Forwarded-Proto count only from them, and any entry makes the dashboard https only.|active|
investigation.sampleLogs|boolean|true|Scope|Sample a bounded slice of the logs of Running and Ready pods for a burst of errors (redacted); false reads none.|active|
1 change: 1 addition & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -559,6 +559,7 @@ listener, never on the health port. Everything about reaching it from outside
| `dashboard.trustedProxies` | `[]` | CIDRs or addresses of the proxies that end TLS in front of kwatch. `X-Forwarded-For` and `X-Forwarded-Proto` count only from them; any entry (or own TLS) makes the dashboard https only |
| `dashboard.tlsCertFile`, `dashboard.tlsKeyFile` | empty | Make kwatch serve TLS on the dashboard port itself. Set both or neither |
| `investigation.logs` | `true` | Reads container logs as evidence (redacted, bounded). `false` reads no log at all, and the dashboard shows the log section as turned off |
| `investigation.sampleLogs` | `true` | Samples a bounded slice of the log (the last 2 minutes, one pod per workload, at most 6 reads per 30 seconds) of Running and Ready pods and reports a burst of errors new against the workload's own earlier windows (`LogErrorBurst`: at least 20 error lines in the window, at least 5 times the earlier rate per window, with at least 10 minutes of earlier windows). Reads nothing when `investigation.logs` is `false`. A refused log read pauses it quietly for 5 minutes |

The bearer token and the sign-in password are not in the file. The chart
ships the Secret `<release>-dashboard` empty and kwatch fills both values
Expand Down
1 change: 1 addition & 0 deletions docs/kubernetes-coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,7 @@ Health is taken from the severity each detector assigns: warning is degraded, cr
| Draining | degraded | `NodeDraining` | `node` |
| EphemeralStorageHigh | varies | `ContainerEphemeralStorageUsageHigh` | `resources` |
| Error | varies | `Error` | `container`, `exit_codes` |
| ErrorBurst | degraded | `LogErrorBurst` | `log_errors` |
| Etcd.Large | degraded | `EtcdDatabaseLarge` | `control_plane_storage` |
| Evicted | varies | `Evicted` | `pod_failed` |
| Evicting | degraded | `NodeEvicting` | `kubelet_health` |
Expand Down
Loading
0