8000 fix false alerts and dashboard usability by abahmed · Pull Request #583 · abahmed/kwatch · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,9 @@ narrowing the watch list and turning off the noisy reasons.
kwatch always watches every supported resource. `namespaces`, `namespaceSelector`,
`reasons` and `silences` only filter what is **delivered**: an incident is
delivered while at least one finding it explains is in scope and not silenced.
Objects without a namespace (Nodes, PersistentVolumes, webhooks, CRDs,
PriorityClasses) are always inside the namespace filters, because their
failure reaches every namespace; use `reasons` or `silences` to quiet them.
State lives on a small disk (a PVC), not in ConfigMaps.

#### 🔽 Filter by namespace
Expand Down
4 changes: 2 additions & 2 deletions docs/kubernetes-coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ Health is taken from the severity each detector assigns: warning is degraded, cr
| Exit.Killed | varies | `ContainerExitKilled` | `exit_codes` |
| Exit.NotExecutable | varies | `ContainerExitNotExecutable` | `exit_codes` |
| Exit.Segfault | varies | `ContainerExitSegfault` | `exit_codes` |
| Failed | degraded | `PhaseFailed`, `PodFailed` | `generic`, `pod_failed` |
| Failed | varies | `PhaseFailed`, `PodFailed` | `generic`, `pod_failed` |
| Filesystem.Usage | varies | `NodeFilesystemUsageCritical`, `NodeFilesystemUsageHigh` | `usage` |
| FlowControl | varies | `APIPriorityAndFairnessFailure` | `custom` |
| Heartbeat.Stale | degraded | `NodeHeartbeatStale` | `node_health` |
Expand Down Expand Up @@ -240,7 +240,7 @@ Health is taken from the severity each detector assigns: warning is degraded, cr
| Scaling.Disabled | varies | `ScalingDisabled` | - |
| Scaling.Error | varies | `HPAScalingError` | `hpa` |
| Scaling.InvalidSelector | varies | `HPAInvalidSelector` | `hpa` |
| Scaling.MaxedOut | degraded | `HPAMaxedOut` | `hpa` |
| Scaling.MaxedOut | varies | `HPAMaxedOut` | `hpa` |
| Scaling.NoMetrics | varies | `FailedComputeMetricsReplicas`, `FailedGetResourceMetric` | `hpa` |
| Scaling.TargetMissing | degraded | `FailedGetScale`, `HPATargetMissing` | `hpa`, `hpa_target` |
| Scaling.UpdateFailed | varies | `FailedUpdateScale` | `hpa` |
Expand Down
49 changes: 44 additions & 5 deletions docs/production-operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -378,13 +378,14 @@ has another schema version (older or newer) or bbolt cannot read it, Kwatch
renames it to `state.db.corrupt.<UTC time>` (the newest two copies are kept,
older ones are deleted; a copy larger than a quarter of the volume limit is
deleted at once), creates a fresh store, and continues: it starts cold and
sends one startup summary. After a schema change (not for an unreadable
file) the delivery outbox and the thread IDs are copied into the fresh file,
so queued pages and resolves are still sent and chat and ticket threads
continue. Their records carry their own version: a record in a layout the new
sends one startup summary. The delivery outbox and the thread IDs are copied into the
fresh file after a schema change and, best effort, after an unreadable or
corrupt file too (records and buckets that still read are carried, the
rest are skipped; a file bbolt cannot open carries nothing), so queued
pages and resolves are still sent and chat and ticket threads continue. Their records carry their own version: a record in a layout the new
release does not know is dropped and counted by the outbox drop metric.
Everything else is rebuilt from the cluster, so open incidents are
re-announced once. The log line `state store reset` shows `carriedRecords`. A locked or otherwise unknown open error does not reset; startup
re-announced once. The log line `state store reset` shows `carriedRecords` and `skippedRecords`. A locked or otherwise unknown open error does not reset; startup
fails instead. The event is logged with the old
version and reason and appears on `/health` as the degraded `state-store`
component with reason `storage_reset`; it does not affect readiness. The
Expand Down Expand Up @@ -530,6 +531,44 @@ rules, a few kilobytes) in the state file, and `/status` shows, per rule,
confidence, a tier or a message. Incidents superseded by a revised cause
are not counted.

## Auditing incidents from the logs

Every lifecycle step is one default-level log line with
`component="incident"`, the incident ID, its root and its tier, so what
kwatch decided can be followed from the logs alone:

| Line | `operation` | Extra fields |
|:--|:--|:--|
| `incident opened` | `open` | `reason`, `sendsAt` |
| `incident resolved` | `resolve` | `reason`, `lasted`, `quiet` |
| `incident reopened` | `reopen` | `times` |
| `incident flapping` | `flap` | `cycles`, `what` |
| `incident page held: failing again` | `hold` | `times` |
| `incident tier changed` | `tier` | `from` |

Each incident keeps its last 50 timeline events. Past that, an event
already reported that a later event repeats (same text, same member) is
folded into the later one, which counts it (`repeats` in the dashboard
API), so a failure that keeps coming back cannot push out the start of
the story; only without such repeats do the oldest events go.

An announcement the configured scope drops (namespaces, reasons,
silences) is logged once as `incident not announced: out of scope`
(`component="pipeline"`), so a filtered-out failure leaves a trace.

The dashboard logs sign-in, sign-out and every failed request (`401`,
`403`, `429`, `5xx`) at the default level; successful reads, which an
open page makes every few seconds, only at `-v=2`.

A kind counts as synced for the detectors only once its first list is
in the model, so the first seconds after a start do not report listed
objects as missing. A detector that needs a kind not synced yet keeps
its earlier findings instead of clearing them. A custom-kind run that
finished failed (one recording `status.finishedAt` or
`status.completionTime`, such as an Argo Workflow) goes to the digest a
day after it ended; a long-lived object in a failed phase stays a
warning.

## Delivery guarantee

Delivery is at least once, not exactly once. A delivery is written to the
Expand Down
3 changes: 2 additions & 1 deletion docs/providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,8 @@ In plain words: the same options work for all providers, not just Slack.
revoked token — is given up on immediately and goes straight to the dead-letter queue, so
it cannot hold up the alerts queued behind it. The retry options are
`maxAttempts` (default 3, 1 to 20), `delay` (default `1s`), `maxBackoff`
(cap on the growing wait, default `30s`), `jitterEnabled` (default `false`)
(cap on the growing wait, default `30s`), `jitterEnabled` (default `true`,
so many clusters alerting into one provider do not retry in lockstep)
and `jitterFactor` (0 to 1, default `0.25`, used only when jitter is on).
Durations are strings such as `5s`.
- **`fallback`** — if this provider fails for good, hand the alert to another provider
Expand Down
9 changes: 7 additions & 2 deletions internal/app/pipeline_build.go
10BC8
Original file line number Diff line number Diff line change
Expand Up @@ -123,13 +123,18 @@ func (p *pipelineBuild) synced(kind inventory.Kind) bool {
// message and the audit entry goes to the decision log's worker.
func (p *pipelineBuild) newEngine() (*pipeline.Engine, error) {
deps := p.deps
// Detectors and the engine read sync through the gate: a kind is
// synced for them once its first list is in the model, not as soon
// as the source has handed it over.
gate := pipeline.NewSyncGate(p.synced)
return pipeline.NewEngine(pipeline.Dependencies{
Model: p.model,
Detectors: newDetectorRegistry(p.synced, deps.entitlements,
Detectors: newDetectorRegistry(gate.Synced, deps.entitlements,
p.withheld),
Withheld: p.withheld,
Incidents: p.incidents,
Synced: p.synced,
Synced: gate.Synced,
SyncGate: gate,
Writer: p.newWriter(),
Sink: p.recordDecision,
Clock: pipelineClock{deps.clients.Clock},
Expand Down
4 changes: 3 additions & 1 deletion internal/config/runtime_providers.go
Original file line number Diff line number Diff line change
Expand Up @@ -118,9 +118,11 @@ func runtimeStringList(value interface{}) []string {
}

func compileRetry(settings map[string]interface{}) RetryPolicy {
// Jitter is on by default: many kwatch instances alerting into one
// shared provider would otherwise retry in lockstep after a blip.
policy := RetryPolicy{
MaxAttempts: 3, Delay: time.Second,
MaxBackoff: 30 * time.Second, JitterFactor: 0.25,
MaxBackoff: 30 * time.Second, JitterEnabled: true, JitterFactor: 0.25,
}
values, ok := settings["retry"].(map[string]interface{})
if !ok {
Expand Down
16 changes: 16 additions & 0 deletions internal/config/runtime_retry_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
package config

import "testing"

// Jitter is on unless a provider turns it off: instances alerting into
// one shared provider must not retry in lockstep after a blip.
func TestCompileRetryJittersByDefault(t *testing.T) {
if got := compileRetry(map[string]interface{}{}); !got.JitterEnabled {
t.Errorf("default policy %+v has no jitter", got)
}
off := compileRetry(map[string]interface{}{
"retry": map[string]interface{}{"jitterEnabled": false}})
if off.JitterEnabled {
t.Errorf("jitterEnabled=false is not honored: %+v", off)
}
}
30 changes: 20 additions & 10 deletions internal/dashboard/auth.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,18 +43,27 @@ func NewAuth(token, password string, shared bool, now clock.Clock) *Auth {
}

// Middleware lets a request through with a valid bearer token or
// session cookie. A request that presented a wrong credential counts
// against the client's failure limit; one that presented none does not,
// so a signed-out browser asking whether it is signed in is not a guess.
// session cookie. A bearer token can be guessed: a wrong one counts
// against the client's failure limit, and past the limit even a right
// one is refused, so a guess that happens to be right cannot get in. A
// session cookie cannot be guessed (its ID is random and only a sign-in
// makes one): a valid one is let through even while the address is
// limited, so guesses from a shared address do not lock out people
// already signed in, and an invalid one does not count, because after a
// restart every open tab presents one on each poll.
func (a *Auth) Middleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ok, guess := a.check(r)
if ok && !guess {
next.ServeHTTP(w, r)
return
}
if !a.allowed(r) {
writeError(w, http.StatusTooManyRequests, "too many requests")
return
}
ok, presented := a.check(r)
if !ok {
if presented {
if guess {
a.recordFailure(r)
}
writeError(w, http.StatusUnauthorized, "unauthorized")
Expand All @@ -64,17 +73,18 @@ func (a *Auth) Middleware(next http.Handler) http.Handler {
})
}

// check reports whether r carries a valid credential and whether it
// carried one at all.
func (a *Auth) check(r *http.Request) (ok, presented bool) {
// check reports whether r carries a valid credential and whether a
// failed one counts as a guess: a bearer token does, a session cookie
// (random, so never guessed) and no credential at all do not.
func (a *Auth) check(r *http.Request) (ok, guess bool) {
if token, has := bearerToken(r); has {
return a.validToken(token), true
}
if c, err := r.Cookie(sessionCookie); err == nil && c.Value != "" {
if a.password == "" {
return false, true
return false, false
}
return a.sessions.valid(c.Value), true
return a.sessions.valid(c.Value), false
}
return false, false
}
Expand Down
106 changes: 106 additions & 0 deletions internal/dashboard/auth_session_limit_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
package dashboard

import (
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"

"github.com/abahmed/kwatch/internal/clock"
"github.com/abahmed/kwatch/internal/rootcause"
)

func cookieRequest(value string) *http.Request {
req := httptest.NewRequest("GET", "/api/incidents", nil)
req.AddCookie(&http.Cookie{Name: sessionCookie, Value: value})
return req
}

// After a restart every open tab presents a session that is no longer
// valid on each poll. Those are not guesses: they must not use up the
// address's failure limit and lock out the next sign-in.
func TestAuthStaleSessionsDoNotCountAsGuesses(t *testing.T) {
auth := NewAuth("secret-token", "pw", false, clock.RealClock{})
handler := auth.Middleware(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {}))
for i := 0; i < 30; i++ {
w := httptest.NewRecorder()
handler.ServeHTTP(w, cookieRequest("stale-session"))
if w.Code != http.StatusUnauthorized {
t.Fatalf("request %d: status = %d, want 401", i, w.Code)
}
}
if !auth.allowed(cookieRequest("")) {
t.Fatal("stale sessions used up the failure limit")
}
}

// Guesses from a shared address lock out further guesses, not the
// people already signed in from it.
func TestAuthLimitedAddressKeepsValidSessions(t *testing.T) {
auth := NewAuth("secret-token", "pw", false, clock.RealClock{})
handler := auth.Middleware(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {}))
for i := 0; i < 11; i++ {
req := httptest.NewRequest("GET", "/api/incidents", nil)
req.Header.Set("Authorization", "Bearer wrong-token")
handler.ServeHTTP(httptest.NewRecorder(), req)
}
session, _, err := auth.sessions.issue()
if err != nil {
t.Fatal(err)
}

w := httptest.NewRecorder()
handler.ServeHTTP(w, cookieRequest(session))

if w.Code != http.StatusOK {
t.Errorf("signed-in session on a limited address: status = %d",
w.Code)
}
}

// Search looks at the cause as the API shows it, redacted, so a search
// cannot tell what a redacted secret contains.
func TestSearchDoesNotMatchRedactedText(t *testing.T) {
p := testIncident("a", "shop", 0)
p.Cause = &rootcause.CauseRecord{
Summary: "login fails: password=hunter2 rejected"}
s := newTestServer(p)

var list ListResponse
body := get(t, s, "/api/incidents?state=active&q=hunter2").Body.Bytes()
if err := json.Unmarshal(body, &list); err != nil {
t.Fatal(err)
}
if list.Total != 0 {
t.Errorf("search matched redacted text: %+v", list.Incidents)
}
}

// A list row carries a short cause, so a long one cannot push incidents
// out of the response; the detail carries it whole.
func TestListCauseIsShortAndDetailCauseWhole(t *testing.T) {
p := testIncident("a", "shop", 0)
p.Cause = &rootcause.CauseRecord{Summary: strings.Repeat("x", 500)}
s := newTestServer(p)

var list ListResponse
if err := json.Unmarshal(
get(t, s, "/api/incidents?state=active").Body.Bytes(), &list); err != nil {
t.Fatal(err)
}
var d IncidentDetail
if err := json.Unmarshal(
get(t, s, "/api/incidents/a").Body.Bytes(), &d); err != nil {
t.Fatal(err)
}
if n := len([]rune(list.Incidents[0].RootCause)); n != maxListCause {
t.Errorf("list cause has %d characters, want %d", n, maxListCause)
}
if len(d.RootCause) != 500 {
t.Errorf("detail cause has %d characters, want 500",
len(d.RootCause))
}
}
30 changes: 23 additions & 7 deletions internal/dashboard/convert.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,29 @@ func toListItem(p incident.Incident) IncidentListItem {
RepeatCount: p.RepeatCount,
AckedBy: ackedBy(p),
ResolvedAt: p.Resolved,
RootCause: shortCause(redact.Evidence(causeText(p))),
}
}

// maxListCause bounds the cause a list item carries: one line on a row,
// and a long cause must not push incidents out of the list's response
// budget. The detail carries it whole.
const maxListCause = 200

func shortCause(cause string) string {
runes := []rune(cause)
if len(runes) <= maxListCause {
return cause
}
return strings.TrimSpace(string(runes[:maxListCause-1])) + "…"
}

func toDetail(p incident.Incident) IncidentDetail {
steps, commands, text := toSteps(p.Steps)
item := toListItem(p)
item.RootCause = redact.Evidence(causeText(p))
return IncidentDetail{
IncidentListItem: toListItem(p),
RootCause: redact.Evidence(causeText(p)),
IncidentListItem: item,
SuggestedFix: text,
FixSteps: steps,
FixCommands: commands,
Expand Down Expand Up @@ -119,11 +134,12 @@ func toTimeline(events []incident.Event) []Event {
for _, e := range events {
kind, reason := classifyEvent(e.Text)
out = append(out, Event{
At: e.At,
Kind: kind,
Reason: reason,
Text: redact.Evidence(e.Text),
Entity: entityIDString(e.Entity),
At: e.At,
Kind: kind,
Reason: reason,
Text: redact.Evidence(e.Text),
Entity: entityIDString(e.Entity),
Repeats: e.Repeats,
})
}
return out
Expand Down
Loading
0