8000 Pro auto-evidence: kwatch runs the read-only checks and shows the result by abahmed · Pull Request #584 · abahmed/kwatch · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,7 @@ enforces, are in `docs/contributor-architecture.md`.
| `internal/pipeline` | Engine loop: observation queue, model update, detectors, incident manager, scope, investigation, downtime reconciliation, audit entries; the only producer of incident decisions |
| `internal/pipeline/announce` | Collecting steps: startup summary, digest, roll-up, namespace outage hold, listings; the pipeline wires it and it never imports the pipeline |
| `internal/pipeline/coverage` | Memory of the coverage backstop (failing workloads, hand-backs) and its timings |
| `internal/pipeline/investigate` | Evidence investigators (crash, node, scheduling, config, admission, registry) and the `Investigator` contract |
| `internal/pipeline/investigate` | Pro evidence: a registry of read-only checks run in parallel per incident (the root-kind investigators lead the bounded channel summary), the dashboard report sections, correlation (`correlate.go`) and the `Investigator` contract |
| `internal/delivery/*` | Delivery manager for notification messages: routing, retries, fallback, pacing, queue coalescing, transport, provider dispatch |
| `internal/delivery/api` | Neutral provider contract shared by delivery and the static catalog |
| `internal/alert/*` | Provider adapters; one subpackage per provider |
Expand Down
61 changes: 61 additions & 0 deletions RELEASES.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,67 @@
teaser when Pro found a likely cause. app.hideProHints hides the alert
line, the teaser and the startup notice; /status still counts.

### Evidence (Pro)

- kwatch now runs the read-only look a message used to suggest and shows
the result: pod state, logs (previous and current run, per container),
events, memory use, node and the pods on it, volume claims, Secrets and
ConfigMaps (key names only), Service and Ingress chains, webhooks,
rollouts, control plane readiness, finalizers, images and pull secrets,
certificates, quotas, autoscalers, Jobs and probes. Every applicable
check runs in parallel when an incident opens and on re-investigation,
under a 10-second budget with at most four API reads at once.
- A Pro message keeps only the steps that change the cluster; a read-only
step stays when its read was not allowed or timed out. When the gathered
evidence proves the cause, the message says so and the confidence is
"confirmed". When a provider's length limit cuts a message, it cuts the
proof and keeps the fix whole.
- The dashboard's issue page has an Evidence tab with one section per
check, a log viewer, the related objects and the correlation (the
failure signature and what it points to, the nearest change, the spread
and what a rollout changed). `GET /api/incidents/{id}` adds `evidence`
and `correlation`. The report is kept in memory only, not in the state
file.
- New settings: `investigation.logs` (default true; false reads no log)
and the Helm value `rbac.readyz` (default true). Memory and CPU use come
from the kubelet stats kwatch already reads (nodes/stats), so no add-on
such as metrics-server is needed. A read that is not allowed shows as
such 10BC8 and never fails the investigation. Audit entries add `evidence`
and `readSteps`.
- A Pro message for a workload whose container was killed at its memory
limit suggests a new limit: half again above the observed peak (or the
limit, when higher), rounded up to 64Mi, as a step that changes the
cluster.
- Teams cards and email lay a Pro message out as cause, what kwatch found,
the log quote and the fix; email adds an HTML part next to the plain
text. The webhook, n8n and Zapier payloads add an `evidence` object
(confirmed, checks, facts, quote, fix); every other field is unchanged.
- When a provider's limit cuts a Pro message, the log quote loses lines
first, then the proof, then the quote; the cause and the fix stay.
- The dashboard shows when each check read its section.
- Community messages are unchanged, except on Discord (below).

### Cluster DNS

- The cluster DNS servers are found through the DNS Service
(`kube-system/kube-dns`, `openshift-dns/dns-default`, or a kube-system
Service named for CoreDNS or selecting `k8s-app=kube-dns`), then its
pods and their owners of any kind. OpenShift's DaemonSet, RKE2's
`rke2-coredns-rke2-coredns` and Helm installs with a release prefix are
now recognised for DNS metrics and DNS outages; the coredns and kube-dns
names remain the fallback.

### Delivery

- Discord incidents are an embed again: the headline as the title, the
cause, proof, log quote and fix in the description, colour by status,
and kwatch, the cluster and the time in the footer. The embed stays
within Discord's limits (description 4096, embed 6000) by cutting the
proof, never the fix, and without one field per section, which broke
long messages before (#194, #416). Mentions are neutralized, thread_id
still works, and plain notices stay message content of at most 2000
characters. This changes Discord output in both editions.

### Detection

- A node that has not been Ready since it joined is now given the ten-minute
Expand Down
12 changes: 12 additions & 0 deletions deploy/chart/templates/rbac.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,13 @@ rules:
- apiGroups: [""]
resources: ["nodes/stats", "nodes/metrics", "pods/log"]
verbs: ["get"]
{{- if ne .Values.rbac.readyz false }}
# /readyz: the prober and Pro evidence read the API server's readiness
# checks (rbac.readyz).
- nonResourceURLs: ["/readyz", "/metrics"]
{{- else }}
- nonResourceURLs: ["/metrics"]
{{- end }}
verbs: ["get"]
- apiGroups: ["authorization.k8s.io"]
resources: ["selfsubjectaccessreviews"]
Expand Down Expand Up @@ -57,7 +63,13 @@ rules:
- apiGroups: [""]
resources: ["nodes/stats", "nodes/metrics", "pods/log"]
verbs: ["get"]
{{- if ne .Values.rbac.readyz false }}
# /readyz: the prober and Pro evidence read the API server's readiness
# checks (rbac.readyz).
- nonResourceURLs: ["/readyz", "/metrics"]
{{- else }}
- nonResourceURLs: ["/metrics"]
{{- end }}
verbs: ["get"]
- apiGroups: ["apps"]
resources: ["deployments", "replicasets", "statefulsets", "daemonsets", "controllerrevisions"]
Expand Down
5 changes: 5 additions & 0 deletions deploy/chart/values.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,11 @@
],
"default": "least-privilege",
"description": "least-privilege (default) grants an explicit list/watch set; full grants read-only list/watch on every resource"
},
"readyz": {
"type": "boolean",
"default": true,
"description": "Grant get on the API server's /readyz (prober and Pro evidence)"
}
},
"additionalProperties": false
Expand Down
4 changes: 4 additions & 0 deletions deploy/chart/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,10 @@ rbac:
# release namespace, in both modes and even with watch.secrets=false.
# Opt back into the wildcard with --set rbac.mode=full.
mode: least-privilege
# -- Grant get on the API server's /readyz. The prober and Pro evidence
# read its readiness checks; false removes the grant and the control
# plane evidence shows that kwatch isn't allowed to read it.
readyz: true

watch:
# -- Watch Secrets. They feed checks such as missing Secret references
Expand Down
1 change: 1 addition & 0 deletions deploy/config-catalog.meta.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,4 @@ ignoreNodeReasons|Compatibility|Legacy node-reason suppression field.|deprecated
ignoreNodeMessages|Compatibility|Legacy node-message suppression field.|deprecated|silences
kubelet.insecureSkipVerify|Security|Skip kubelet serving certificate verification for direct node stats reads; only for self-signed kubelet certificates.|active|
watch.secrets|Scope|Watch Secrets (values hashed, never stored); false removes Secret RBAC and Secret-based checks report that they cannot verify.|active|
investigation.logs|Scope|Read container logs for Pro evidence (redacted, bounded); false reads no log and shows the log section as turned off.|active|
1 change: 1 addition & 0 deletions deploy/config-catalog.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,4 @@ dashboard.mode|string|local|Security|How people reach the dashboard: local (port
dashboard.trustProxy|boolean|false|Security|Honour X-Forwarded-Proto and X-Forwarded-For from the proxy that terminates TLS in front of the dashboard.|active|
dashboard.tlsCertFile|string|empty|Security|Certificate file kwatch serves on the dashboard port itself (with tlsKeyFile); reloaded when it changes.|active|
dashboard.tlsKeyFile|string|empty|Security|Key file for dashboard.tlsCertFile.|active|
investigation.logs|boolean|true|Scope|Read container logs for Pro evidence (redacted, bounded); false reads no log and shows the log section as turned off.|active|
25 changes: 16 additions & 9 deletions internal/alert/discord/contract_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,14 @@ func TestContract(t *testing.T) {
}
providertest.RunContract(t, providertest.Contract{
Chat: true,
Send: c.SendIncident, Rec: rec, LimitBytes: maxContent,
Send: c.SendIncident, Rec: rec, LimitBytes: maxEmbedTotal,
Secrets: []string{"wh-secret"},
Check: checkShape,
})
}

// checkShape asserts the execute-webhook body: content within the 2000
// character limit, balanced fences, and mentions disabled.
// checkShape asserts the execute-webhook body: one embed within
// Discord's limits, balanced fences, and mentions disabled.
func checkShape(
t *testing.T, name string, reqs []providertest.Request,
) {
Expand All @@ -56,14 +56,21 @@ func checkShape(
t.Fatalf("requests = %+v", reqs)
}
body := reqs[0].JSON(t)
content, _ := body["content"].(string)
if strings.TrimSpace(content) == "" {
t.Fatalf("content is empty: %s", reqs[0].Body)
embeds, _ := body["embeds"].([]any)
if len(embeds) != 1 {
t.Fatalf("want one embed: %s", reqs[0].Body)
}
if n := utf8.RuneCountInString(content); n > maxContent {
t.Fatalf("content is %d characters, limit %d", n, maxContent)
embed, _ := embeds[0].(map[string]any)
title, _ := embed["title"].(string)
desc, _ := embed["description"].(string)
if strings.TrimSpace(title) == "" {
t.Fatalf("title is empty: %s", reqs[0].Body)
}
providertest.AssertFencesBalanced(t, content)
if n := utf8.RuneCountInString(desc); n > maxEmbedDescription {
t.Fatalf("description is %d characters, limit %d", n,
maxEmbedDescription)
}
providertest.AssertFencesBalanced(t, desc)
am, _ := body["allowed_mentions"].(map[string]any)
if parse, ok := am["parse"].([]any); !ok || len(parse) != 0 {
t.Fatalf("mentions must be disabled: %s", reqs[0].Body)
Expand Down
31 changes: 31 additions & 0 deletions internal/alert/discord/cut_levels_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
package discord

import (
"fmt"
"strings"
"testing"

"github.com/abahmed/kwatch/internal/delivery/providertest"
)

// TestEmbedCutLevels pins the description at every evidence cut level:
// it fits, and the fix survives. Update with
// go test -run 'TestEmbedCutLevels/level-1$' -update.
func TestEmbedCutLevels(t *testing.T) {
for _, level := range providertest.CutLevels {
t.Run(fmt.Sprintf("level-%d", level), func(t *testing.T) {
e := sendOne(t, providertest.CutLevel(level,
maxEmbedDescription))
if len(e.Description) > maxEmbedDescription ||
embedSize(e) > maxEmbedTotal {
t.Fatalf("embed is %d bytes", embedSize(e))
}
if !strings.Contains(e.Description, providertest.FixCommand) {
t.Fatalf("the fix was cut:\n%s", e.Description)
}
providertest.CompareGolden(t, fmt.Sprintf(
"testdata/cut/level-%d.golden", level),
e.Title+"\n---\n"+e.Description+"\n")
})
}
}
21 changes: 8 additions & 13 deletions internal/alert/discord/discord.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ import (
"net/url"
"strings"

"github.com/abahmed/kwatch/internal/alert/safetext"
"github.com/abahmed/kwatch/internal/clock"
"github.com/abahmed/kwatch/internal/delivery/transport"
"github.com/abahmed/kwatch/internal/notification"
Expand Down Expand Up @@ -172,9 +171,8 @@ func (d *Discord) verifyURL() string {
return target
}

// SendIncident posts the incident narrative as the message content, with
// the workload's last output as a code block. The content is cut to
// Discord's 2000-character limit; bytes never undercount characters.
// SendIncident posts the incident as one embed (see incidentEmbed).
// Plain notices and the digest go through SendMessage as content.
func (d *Discord) SendIncident(
ctx context.Context,
m notification.Message,
Expand All @@ -184,20 +182,15 @@ func (d *Discord) SendIncident(
err := d.execute(
&discordgo.WebhookParams{
AllowedMentions: noMentions(),
Content: incidentContent(m),
Embeds: []*discordgo.MessageEmbed{
incidentEmbed(m, d.clusterName, d.clockSource.Now()),
},
},
discordgo.WithContext(ctx),
)
return wrapDiscordRateLimit(err)
}

// incidentContent is the Note plus the last output, mention-neutralized
// and bounded.
func incidentContent(m notification.Message) string {
return safetext.RichWithOutput(m,
notification.MarkdownDialect(nil, "\n"), "\n", maxContent)
}

// SendMessage sends text using the caller's cancellation context.
func (d *Discord) SendMessage(
ctx context.Context,
Expand All @@ -207,7 +200,9 @@ func (d *Discord) SendMessage(
err := d.execute(
&discordgo.WebhookParams{
AllowedMentions: noMentions(),
Content: msg,
// The delivery bound is the embed's; plain content has its
// own, smaller limit.
Content: notification.Truncate(msg, maxContent),
},
discordgo.WithContext(ctx),
)
Expand Down
82 changes: 82 additions & 0 deletions internal/alert/discord/embed.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
package discord

import (
"strings"
"time"

discordgo "github.com/bwmarrin/discordgo"

"github.com/abahmed/kwatch/internal/alert/safetext"
"github.com/abahmed/kwatch/internal/notification"
)

// Discord embed limits. The whole message (title, description, footer)
// must stay within maxEmbedTotal. The narrative goes in the description,
// never in one field per section: fields hold only 1024 characters each.
const (
maxEmbedTitle = 256
maxEmbedDescription = 4096
maxEmbedFooter = 2048
maxEmbedTotal = 6000
)

// Embed colours by status, the brand's severity tokens.
const (
colorCritical = 0xB42318
colorWarning = 0x8A5300
colorResolved = 0x1E6B3A
colorInfo = 0x5B6270
)

func statusColor(s notification.Status) int {
switch s {
case notification.StatusCritical:
return colorCritical
case notification.StatusWarning, notification.StatusFlapping:
return colorWarning
case notification.StatusResolved:
return colorResolved
}
return colorInfo
}

// discordDialect writes Markdown with mentions neutralized, so a pod's
// log line can never ping a server even if allowed mentions change.
func discordDialect() notification.Dialect {
return notification.MarkdownDialect(notification.NeutralizeMentions, "\n")
}

// incidentEmbed lays the incident out as one embed: the headline as the
// title, the cause, proof, log quote and fix in the description (a Pro
// fix is pinned, so the proof is cut first and the fix never), and the
// cluster and time in the footer.
func incidentEmbed(
m notification.Message, cluster string, now time.Time,
) *discordgo.MessageEmbed {
title := notification.Truncate(
notification.NeutralizeMentions(m.ShortText()), maxEmbedTitle)
footer := "kwatch"
if cluster != "" {
footer += " · " + notification.NeutralizeMentions(cluster)
}
footer = notification.Truncate(footer, maxEmbedFooter)
room := min(maxEmbedDescription,
maxEmbedTotal-len(title)-len(footer))
return &discordgo.MessageEmbed{
Title: title,
Description: description(m, room),
Color: statusColor(m.Status),
Footer: &discordgo.MessageEmbedFooter{Text: footer},
Timestamp: now.UTC().Format(time.RFC3339),
}
}

// description is the narrative without its headline (the title says it)
// plus the output, within limit bytes.
func description(m notification.Message, limit int) string {
if len(m.Doc) > 1 {
m.Doc = m.Doc[1:]
}
text := safetext.RichWithOutput(m, discordDialect(), "\n", limit)
return strings.TrimSpace(text)
}
12 changes: 6 additions & 6 deletions internal/alert/discord/fence_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@ import (
"github.com/abahmed/kwatch/internal/delivery/providertest"
)

func TestIncidentContentFenceSurvivesTruncation(t *testing.T) {
func TestEmbedDescriptionFenceSurvivesTruncation(t *testing.T) {
m := providertest.Announce()
m.Output = []string{"```@everyone", strings.Repeat("x", maxContent)}
content := incidentContent(m)
assert.LessOrEqual(t, len(content), maxContent)
assert.True(t, strings.HasSuffix(content, "\n````"), content[len(content)-20:])
assert.NotContains(t, content, "@everyone")
m.Output = []string{"```@everyone", strings.Repeat("x", maxEmbedTotal)}
got := description(m, maxEmbedDescription)
assert.LessOrEqual(t, len(got), maxEmbedDescription)
assert.True(t, strings.HasSuffix(got, "\n````"), got[len(got)-20:])
assert.NotContains(t, got, "@everyone")
}
19 changes: 19 additions & 0 deletions internal/alert/discord/fix_fitting_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
package discord

import (
"strings"
"testing"

"github.com/abahmed/kwatch/internal/delivery/providertest"
)

func TestEmbedKeepsFixWithinLimit(t *testing.T) {
e := sendOne(t, providertest.ProFix())
if len(e.Description) > maxEmbedDescription ||
embedSize(e) > maxEmbedTotal {
t.Fatalf("embed is %d bytes", embedSize(e))
}
if !strings.Contains(e.Description, providertest.FixCommand) {
t.Fatalf("the fix was cut:\n%s", e.Description)
}
}
Loading
0