8000 🧭 Teach the engine about 300 blind failure scenarios by abahmed · Pull Request #619 · abahmed/kwatch · GitHub
[go: up one dir, main page]

Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
902af29
🧪 Add labelled scenarios for gaps in failure coverage
abahmed Nov 19, 2026
a71dfdf
🧭 Teach the engine thirteen blind-scenario gaps
abahmed Nov 19, 2026
e96ed65
🧭 Teach the engine thirteen blind-scenario gaps (lint)
abahmed Nov 19, 2026
66dd5b6
🧭 Keep the hop table within lint limits
abahmed Nov 19, 2026
7d18b11
✅ Cover the new rows and refresh the coverage page
abahmed Nov 19, 2026
63d6a75
✏️ Say a node is low on process IDs without predicting
abahmed Nov 19, 2026
284fda3
✏️ Apply designer wording to the gap alerts
abahmed Nov 19, 2026
10d965a
🧹 Keep the root steps within the complexity limit
abahmed Nov 19, 2026
2591d96
🧪 Add labelled blind scenarios for gap rounds 2 and 3
abahmed Nov 19, 2026
21d2bac
🧪 Add labelled blind scenarios for gap rounds 4a and 4b
abahmed Nov 19, 2026
5aa105f
🏷️ Refine labels of the scheduler lease, stale endpoint and schedulin…
abahmed Nov 19, 2026
5abf7fc
🧪 List the scoped quota's PriorityClass in its scenario
abahmed Nov 19, 2026
880eefa
🧪 Add labelled blind scenarios for gap rounds 5a and 5b
abahmed Nov 19, 2026
1a53123
🧪 Add labelled blind scenarios for gap rounds 6a, 6b, 7a and 7b
abahmed Nov 19, 2026
a8ec950
🏷️ Spell snapshot roots group-qualified and relabel the drained Job
abahmed Nov 19, 2026
ea6c601
🧭 Name the object a scheduling verdict points at
abahmed Nov 19, 2026
e4a840f
🧭 Blame the node when its replicas fail with a node-local error and t…
abahmed Nov 19, 2026
d94aefb
🧭 Let a node-local error split replicas that share the node
abahmed Nov 19, 2026
03e922f
🧭 Let the cause behind the DNS servers stand for the DNS outage
abahmed Nov 19, 2026
71caabf
🧪 Refresh goldens for the DNS and node-local rules
abahmed Nov 19, 2026
282c8f2
🧭 Blame the node when one replica fails where its siblings run
abahmed Nov 19, 2026
9bf7513
🛰️ Report a control-plane service that keeps going down and tie its p…
abahmed Nov 19, 2026
235c85f
✅ Register the scheduler-named mode and cover its row
abahmed Nov 19, 2026
90c79a2
🔕 Hold quiet scenarios: stale evictions and a silent fleet stay quiet
abahmed Nov 21, 2026
af53bf6
📈 Report a replica count that keeps flipping and name the autoscaler
abahmed Nov 21, 2026
1590ba1
🛡️ Notify when a disruption budget selects no pods
abahmed Nov 21, 2026
0180927
🩺 Put sporadic readiness failures over an hour in the digest
abahmed Nov 21, 2026
d808444
🧭 Send an ambiguous autoscaler selector to the digest
abahmed Nov 21, 2026
9fb71df
🧪 Give running Job pods a Ready condition in a drain scenario
abahmed Nov 21, 2026
95cc688
🐛 Word and categorize the control-plane component pod rows
abahmed Nov 21, 2026
a3c4c81
🌐 Blame a changed DNS object for failing lookups
abahmed Nov 21, 2026
0fd7dc3
💾 Blame storage classes, pinning claims and mount errors for storage …
abahmed Nov 21, 2026
a4bc211
💾 Cover storage rules with fixtures and keep mount errors on the claim
abahmed Nov 21, 2026
3b277a2
🔐 Name the quota or Pod Security namespace behind refused creates
abahmed Nov 21, 2026
d58a96e
🔑 Blame the registry, role, account or join token behind refused calls
abahmed Nov 21, 2026
28fe69d
🧭 Read RBAC bindings and drop the Secret value read
abahmed Nov 21, 2026
c302d1d
🏷️ Relabel the expired join token case to what kwatch can see
abahmed Nov 21, 2026
7d09adc
🧪 Regenerate stale goldens and fixtures
abahmed Nov 21, 2026
0bc5014
🐛 Leave pods that followed their node out of workload readiness flapping
abahmed Nov 21, 2026
867bc20
🧪 Regenerate the other stale fixtures
abahmed Nov 21, 2026
f5a25c2
🐛 Page a silent fleet at once when its pods are affected
abahmed Nov 21, 2026
017b1b8
🌐 Blame the changed DNS object over the name its lookups asked for
abahmed Nov 21, 2026
d5ff1bc
✏️ Say in plain words what each scheduler reason is wrong with
abahmed Nov 21, 2026
6de2640
🛡️ Hedge a DNS cause while the DNS configuration is unreadable
abahmed Nov 21, 2026
e808b9c
🧪 Regenerate goldens for the reworded and restored alerts
abahmed Nov 21, 2026
13e80ad
🧪 Build Ready Job pods and aged evictions in the scenario generators
abahmed Nov 21, 2026
83abcd9
🏷️ Page when a lost node strands a Job replacement
abahmed Nov 21, 2026
cd5af2f
🔎 Read a StatefulSet pod's stable name as its governing Service and k…
abahmed Nov 21, 2026
afdd310
🧭 Blame a release whose new pods no node can take
abahmed Nov 21, 2026
bb108c5
🔕 Send a resize error, a skipped allocatable enforcement and a load b…
abahmed Nov 21, 2026
16797e3
🏷️ Name the bare pod that holds a StatefulSet's ordinal instead of th…
abahmed Nov 21, 2026
9ab8b14
🧭 Blame a headless Service that hides not-ready peers
abahmed Nov 21, 2026
3aa0f1a
🧭 Blame the node whose kubelet refuses a pod for want of room
abahmed Nov 21, 2026
9ac66fb
🧭 Blame a node agent DaemonSet that has no pod anywhere when new pods…
abahmed Nov 8000 21, 2026
1931170
🧭 Blame the node that has no room for a deferred resize
abahmed Nov 21, 2026
39ee8ad
🧭 Blame a cordoned node when the rest have no room for waiting pods
abahmed Nov 21, 2026
2a94cfc
🔕 Send a kubelet serving certificate waiting for approval to the digest
abahmed Nov 21, 2026
006b232
🧭 Blame a Service whose declaration cannot carry the call
abahmed Nov 21, 2026
a9ff656
🧭 Blame the policy that isolates a webhook backend and the finalizer …
abahmed Nov 21, 2026
a41482f
🧭 Blame the DNS server workload when lookups fail while its pods run
abahmed Nov 21, 2026
82233d0
💾 Name the missing, held or restoring storage object behind a stuck v…
abahmed Nov 21, 2026
de0438b
🧯 Join nodes lost together and late failures to the incident that exp…
abahmed Nov 21, 2026
239712b
🔇 Hold unfelt node conditions and restore waits for the digest, and s…
abahmed Nov 21, 2026
ff4442c
🧱 Fold control-plane failures that open together into one incident an…
abahmed Nov 21, 2026
7145b6b
🧠 Blame the workload a node's system OOM names once it takes a second…
abahmed Nov 22, 2026
034c919
🔎 Blame the workload that holds the resolver address failing lookups …
abahmed Nov 22, 2026
d4ee816
🧭 Blame a removed node for pods stuck terminating on it and give the …
abahmed Nov 22, 2026
2268300
💽 Let the node that evicted a pod explain the exit its container reports
abahmed Nov 22, 2026
81a3dbe
🧪 Regenerate message goldens and stop a node group naming its own nod…
abahmed Nov 22, 2026
46b298f
🧪 Make the replay judge tolerate a hedge on an ambiguous label like t…
abahmed Nov 22, 2026
4e2698d
🩹 Stop naming a trigger as the fix when the fault is the spread rule,…
abahmed Nov 22, 2026
92f8863
✏️ Apply the reviewed wording changes and name the registry that no l…
abahmed Nov 22, 2026
93c5be8
🧹 Find DNS and control-plane pods by relations, merge duplicate rules…
abahmed Nov 22, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
Empty file added base-fails.txt
Empty file.
8 changes: 6 additions & 2 deletions 8 docs/kubernetes-coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,9 @@ Health is taken from the severity each detector assigns: warning is degraded, cr
| Missing.ConfigMap | varies | `ProjectedConfigMapMissing` | `config` |
| Missing.Secret | varies | `ProjectedSecretMissing` | `config` |
| Missing.ServiceAccount | varies | `ServiceAccountMissing` | `config` |
| NameTaken | degraded | `PodNameTaken` | `pod_name_taken` |
| Network.CNINotReady | varies | `NodeCNINotReady` | `node_health` |
| Network.Conflict | varies | `NodePodNetworkConflict` | `node_health` |
| Network.IPExhausted | varies | `NodePodIPExhausted` | `node_health` |
| NetworkErrors | varies | `NodeNetworkErrors` | `resources` |
| NetworkPolicy | degraded | `RestrictiveNetworkPolicy` | `network` |
Expand All @@ -215,6 +217,7 @@ Health is taken from the severity each detector assigns: warning is degraded, cr
| OOMKilled | failing | `OOMKILLED`, `OOMKilled` | `container`, `container_crashloop_run` |
| PIDPressure | varies | `PIDPressure` | `condition_rules` |
| PLEGSlow | degraded | `NodePLEGSlow` | `kubelet_health` |
| PausedEdit | degraded | `DeploymentPausedEdit` | `workload_paused` |
| Pending | degraded | `PhasePending`, `PodPending` | `generic`, `pod` |
| PortMismatch | failing | `IngressBackendPortMissing`, `ServicePortMismatch` | `ingress_ports`, `service_ports` |
| Preempted | degraded | `PodPreempted` | `pod_preempted` |
Expand Down Expand Up @@ -242,6 +245,7 @@ Health is taken from the severity each detector assigns: warning is degraded, cr
| Rollout.Stuck | degraded | `StatefulSetRolloutStuck` | `rollout` |
| RolloutStuck | varies | `ProgressDeadlineExceeded` | `condition_rules` |
| RuntimeErrors | varies | `NodeRuntimeErrors` | `resources` |
| Scale.Flapping | degraded | `ReplicasFlapping` | `replicas_flapping` |
| ScaledToZero | varies | `ScaledToZeroRouted` | `scaled_to_zero` |
| Scaling.Disabled | varies | `ScalingDisabled` | - |
| Scaling.Error | varies | `HPAScalingError` | `hpa` |
Expand Down Expand Up @@ -279,8 +283,8 @@ Health is taken from the severity each detector assigns: warning is degraded, cr
| VolumeFull | varies | `VolumeUsageHigh` | `usage` |
| VolumeFull.Inodes | varies | `VolumeInodesHigh` | `usage` |
| Waiting.Unknown | varies | `UnknownWaiting.` | - |
| Webhook.BackendMissing | varies | `WebhookBackendNotFound` | `policy` |
| Webhook.NoEndpoints | varies | `WebhookNoEndpoints` | `policy` |
| Webhook.BackendMissing | degraded | `WebhookBackendNotFound` | `conversion_backend`, `policy` |
| Webhook.NoEndpoints | degraded | `WebhookNoEndpoints` | `conversion_backend`, `policy` |
| Webhook.Rejecting | degraded | `WebhookRejecting` | `webhook_calls` |
| Webhook.Slow | varies | `WebhookSlow` | `webhook_calls` |

Expand Down
2 changes: 1 addition & 1 deletion internal/detection/detectors/budget.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ func budgetSelectsNothing(
}
return detection.Finding{
Reason: reasons.PdbSelectsNothing, Severity: detection.Warning,
Since: since, Advisory: true,
Since: since,
Summary: "Disruption budget selects no pods; it protects nothing",
Evidence: []detection.Evidence{{
Label: "selector", Value: text(e, kube.AttrSelector),
Expand Down
2 changes: 1 addition & 1 deletion internal/detection/detectors/budget_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ func TestBudgetReportsSelectingNothing(t *testing.T) {
eval := evaluate(Budget{}, m, t0.Add(11*time.Minute), id, nil)
require.Len(t, eval.Findings, 1)
assert.Equal(t, "Budget.SelectsNothing", string(eval.Findings[0].Mode))
assert.True(t, eval.Findings[0].Advisory)
assert.False(t, eval.Findings[0].Advisory)
}

func TestBudgetSelectingPodsIsFine(t *testing.T) {
Expand Down
5 changes: 3 additions & 2 deletions internal/detection/detectors/condition_rules.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ func registerNodeAndWorkload() {
nodePressureCheck("DiskPressure", reasons.DiskPressure,
"Node is low on disk space"),
nodePressureCheck("PIDPressure", reasons.PIDPressure,
"Node is running out of process IDs"),
"Node is low on process IDs"),
conditionCheck{cond: "NetworkUnavailable", status: "True",
finding: reasons.NetworkUnavailable, severity: detection.Critical,
waitsForYoungNode: true,
Expand Down Expand Up @@ -85,7 +85,8 @@ func registerClusterObjects() {
"its node"},
conditionCheck{cond: "PodResizeInProgress", status: "True",
reason: "Error", finding: reasons.PodResizeError,
severity: detection.Warning, message: quoteMessageIfAny,
// The pod keeps running at its old size.
severity: detection.Info, message: quoteMessageIfAny,
summary: "Pod resize failed to apply"},
)
}
Expand Down
4 changes: 2 additions & 2 deletions internal/detection/detectors/condition_table_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ var condCases = []condCase{
{name: "pid pressure", kind: kube.KindNode,
cond: "PIDPressure", status: "True", age: 2 * time.Minute,
want: []wantFinding{{reasons.PIDPressure, detection.Critical, 0,
"Node is running out of process IDs", nil, true}}},
"Node is low on process IDs", nil, true}}},
{name: "pressure false", kind: kube.KindNode,
cond: "DiskPressure", status: "False", age: 2 * time.Minute},
{name: "network unavailable at once", kind: kube.KindNode,
Expand Down Expand Up @@ -184,7 +184,7 @@ var condCases = []condCase{
{name: "pod resize error", kind: kube.KindPod,
cond: "PodResizeInProgress", status: "True", reason: "Error",
message: "kubelet said no",
want: []wantFinding{{reasons.PodResizeError, detection.Warning, 0,
want: []wantFinding{{reasons.PodResizeError, detection.Info, 0,
"Pod resize failed to apply", msg("kubelet said no"), false}}},
}

Expand Down
6 changes: 6 additions & 0 deletions internal/detection/detectors/container.go
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,12 @@ func terminatedFinding(
// The Job succeeded on a retry: this run is history, even an OOM.
return nil
}
if pod, ok := owningPod(ctx, e); ok &&
text(pod, kube.AttrReason) == reasons.Evicted &&
evictionOver(ctx, terminatedAt(e)) {
// The eviction is over; its exit is history, as the pod's is.
return nil
}
if reason != reasons.OOMKilled && stoppedOnPurpose(ctx, e, code) {
return nil
}
Expand Down
6 changes: 4 additions & 2 deletions internal/detection/detectors/controlplane.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,13 @@ func (ClusterService) Detect(
return nil
}
if ok, _ := healthy.Value.AsBool(); ok {
return append(apiLatency(ctx, e), errorRates(ctx, e)...)
return append(append(apiLatency(ctx, e), errorRates(ctx, e)...),
recurringOutage(ctx, e, false, healthy.Since)...)
}
if !sustained(ctx, "api-unhealthy", healthy.Since, DefaultProbeFailing) {
return nil
return recurringOutage(ctx, e, true, healthy.Since)
}
recurringOutage(ctx, e, true, healthy.Since)
reason, what := serviceReason(e.ID)
return []detection.Finding{{
Reason: reason, Severity: detection.Critical, Since: healthy.Since,
Expand Down
81 changes: 81 additions & 0 deletions internal/detection/detectors/controlplane_episodes.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
package detectors

import (
"strconv"
"time"

"github.com/abahmed/kwatch/internal/detection"
"github.com/abahmed/kwatch/internal/inventory"
)

const (
// outageEpisodes is how many separate outages of one control plane
// service, each too short to be a finding alone, make a pattern.
outageEpisodes = 3
// outageWindow is how far back the first of those outages may lie.
outageWindow = 15 * time.Minute
// outageRapid is the span of those outages below which the service
// is in a restart loop.
outageRapid = 5 * time.Minute
)

// episodeKey names the tracked start of the n-th latest outage.
func episodeKey(n int) string { return "outage-start/" + strconv.Itoa(n) }

// outageStarts returns the starts of the latest outages, newest first,
// and keeps them remembered. A start is zero when none is known.
func outageStarts(ctx detection.Context) [outageEpisodes]time.Time {
var starts [outageEpisodes]time.Time
for n := range starts {
prev, found := ctx.Track(episodeKey(n), time.Time{})
if found {
starts[n] = prev
ctx.Track(episodeKey(n), prev)
}
}
return starts
}

// recurringOutage notes a new outage when the probe turns unhealthy and
// returns a finding while the latest outageEpisodes outages all lie
// within outageWindow: a service that goes down and comes back again
// and again never stays down long enough for the sustained check, yet
// it is failing. since is when the current unhealthy state began.
func recurringOutage(
ctx detection.Context, e inventory.Entity, unhealthy bool,
since time.Time,
) []detection.Finding {
starts := outageStarts(ctx)
if unhealthy {
if !ctx.Ongoing("outage-ongoing") {
for n := outageEpisodes - 1; n > 0; n-- {
starts[n] = starts[n-1]
}
starts[0] = since
for n, at := range starts {
ctx.Track(episodeKey(n), at)
}
}
ctx.Onset("outage-ongoing", ctx.Now)
}
oldest := starts[outageEpisodes-1]
if oldest.IsZero() || ctx.Now.Sub(oldest) > outageWindow {
return nil
}
ctx.RecheckAfter(outageWindow - ctx.Now.Sub(oldest))
reason, what := serviceReason(e.ID)
// Outages that come within minutes of each other are a restart
// loop; spread over a quarter of an hour they are a warning.
severity := detection.Warning
if starts[0].Sub(oldest) <= outageRapid {
severity = detection.Critical
}
return []detection.Finding{{
Reason: reason, Severity: severity, Since: oldest,
Summary: what + " has been down " + strconv.Itoa(outageEpisodes) +
" times in the last " +
strconv.Itoa(int(outageWindow.Minutes())) +
" min",
Evidence: errorEvidence(e),
}}
}
51 changes: 51 additions & 0 deletions internal/detection/detectors/conversion_backend.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
package detectors

import (
"github.com/abahmed/kwatch/internal/detection"
"github.com/abahmed/kwatch/internal/detection/reasons"
"github.com/abahmed/kwatch/internal/inventory"
"github.com/abahmed/kwatch/internal/inventory/kube"
)

// conversionBackendFindings reports a CustomResourceDefinition whose
// conversion webhook calls a Service that does not exist or has no ready
// endpoint. Every read of a resource that needs converting between the
// stored and the requested version then fails, whatever the client is.
// The findings are the admission webhooks' (WebhookBackendNotFound,
// WebhookNoEndpoints), at Warning: they break the reads of one kind, not
// every create in the cluster.
func conversionBackendFindings(
ctx detection.Context, e inventory.Entity,
) []detection.Finding {
if ctx.Model == nil || !ctx.Synced(kube.KindService) {
return nil
}
var out []detection.Finding
for _, service := range ctx.Model.Related(
e.ID, inventory.Serves, inventory.Outgoing,
) {
if !ctx.Model.Exists(service) {
out = append(out, detection.Finding{
Reason: reasons.WebhookBackendNotFound,
Severity: detection.Warning,
Summary: "Conversion webhook calls Service " +
service.Name + ", which does not exist",
Evidence: []detection.Evidence{{
Label: "service", Value: service.Name + " is gone"}},
})
continue
}
ready, known := readyEndpoints(ctx, service)
if !known || ready > 0 || backendsAreStarting(ctx, service) {
continue
}
out = append(out, detection.Finding{
Reason: reasons.WebhookNoEndpoints, Severity: detection.Warning,
Summary: "Conversion webhook backend " + service.Name +
" has no ready pods",
Evidence: []detection.Evidence{{
Label: "service", Value: service.Name + " has no ready pods"}},
})
}
return out
}
29 changes: 29 additions & 0 deletions internal/detection/detectors/custom.go
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,9 @@ type failingCondition struct {
// unknown marks a controller that cannot tell yet, not one that
// reports a failure.
unknown bool
// unclaimed marks a Gateway API object whose Accepted or Programmed
// condition no controller has set: it waits for a controller.
unclaimed bool
}

// failingConditions lists the conditions that are in their bad status
Expand All @@ -99,6 +102,7 @@ func failingConditions(e inventory.Entity) []failingCondition {
}
failing = append(failing, failingCondition{
label: label, since: at, unknown: rule.Bad == "Unknown",
unclaimed: rule.Bad == "Unknown" && gatewayAcceptance(rule.Type),
message: text(e, kube.ConditionKey(rule.Type)+
kube.AttrConditionMessage),
})
Expand All @@ -122,12 +126,37 @@ func failingGrace(failing []failingCondition) time.Duration {
return DefaultCustomFailing
}
}
if unclaimed(failing) {
return DefaultGatewayUnclaimed
}
if len(failing) == 0 {
return DefaultCustomFailing
}
return DefaultGenerationGrace
}

// DefaultGatewayUnclaimed is how long a Gateway API object may wait for a
// controller to accept it. A controller sets Accepted within seconds of
// seeing the object; one that is still Unknown after this has none.
const DefaultGatewayUnclaimed = 2 * time.Minute

// gatewayAcceptance reports the condition types a Gateway API controller
// sets first when it takes an object over.
func gatewayAcceptance(conditionType string) bool {
return conditionType == "Accepted" || conditionType == "Programmed"
}

// unclaimed reports failing conditions that are all a Gateway API
// object waiting for a controller.
func unclaimed(failing []failingCondition) bool {
for _, c := range failing {
if !c.unclaimed {
return false
}
}
return len(failing) > 0
}

// failingSince is when the first of the conditions began, or zero. A
// condition without a time is ignored, so the result does not depend on
// their order.
Expand Down
9 changes: 7 additions & 2 deletions internal/detection/detectors/custom_kinds.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,8 @@ func kindFindings(
return routeBackends(ctx, e)
}
if e.ID.Kind == kindCRD {
return conditionFindings(ctx, e)
return append(conditionFindings(ctx, e),
conversionBackendFindings(ctx, e)...)
}
var f detection.Finding
var ok bool
Expand Down Expand Up @@ -82,13 +83,17 @@ func certificateRequest(
}
approved, _, approvedAt := condition(e, "Approved")
grace := DefaultCertificateIssue
severity := detection.Warning
since := approvedAt
summary := "Certificate request was approved but no certificate " +
"was issued; no signer handles " + signer
if approved != "True" {
if signer != kubeletServingSigner {
return detection.Finding{}, false
}
// The kubelet keeps serving with its own certificate; only the
// clients that verify it fail, so it is a digest line.
severity = detection.Info
grace, since = DefaultServingApproval, clusterSince(e,
valueSince(e, kube.AttrCertificateIssued))
summary = "Kubelet serving certificate request is waiting for " +
Expand All @@ -98,7 +103,7 @@ func certificateRequest(
return detection.Finding{}, false
}
return detection.Finding{
Reason: reasons.CertificateNotIssued, Severity: detection.Warning,
Reason: reasons.CertificateNotIssued, Severity: severity,
Health: detection.Failing, Since: since, Summary: summary,
Evidence: evidence,
}, true
Expand Down
25 changes: 25 additions & 0 deletions internal/detection/detectors/deletion_pinned.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
package detectors

import (
"github.com/abahmed/kwatch/internal/detection"
"github.com/abahmed/kwatch/internal/inventory"
"github.com/abahmed/kwatch/internal/inventory/kube"
)

// heldByBoundClaim reports a PersistentVolume whose deletion waits for
// a claim that is still bound and not being deleted.
func heldByBoundClaim(ctx detection.Context, e inventory.Entity) bool {
if e.ID.Kind != kube.KindPV || ctx.Model == nil {
return false
}
for _, id := range ctx.Model.Related(
e.ID, inventory.References, inventory.Incoming) {
claim, ok := ctx.Model.Entity(id)
if id.Kind == kube.KindPVC && ok &&
text(claim, kube.AttrPhase) == "Bound" &&
!flag(claim, kube.AttrDeleting) {
return true
}
}
return false
}
3 changes: 2 additions & 1 deletion internal/detection/detectors/event.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,8 @@ func (Event) Detect(
overcomeAtStart(e, note) || partOfLivenessKill(ctx, e, note) ||
recoveredBefore(e, note) || jobCreateOvercome(ctx, e, note) ||
bootNoise(ctx, e, note) ||
daemonNodeNoise(ctx, e, note) || staleNodeNote(e, note) {
daemonNodeNoise(ctx, e, note) || staleNodeNote(e, note) ||
conflictNamesPod(ctx, note) {
continue
}
// A note exactly as old as its life still counts, so the
Expand Down
4 changes: 4 additions & 0 deletions 4 internal/detection/detectors/event_kubelet.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,10 @@ var sandboxClasses = []struct {
"network plugin is not ready", "cni plugin not initialized",
"cni config uninitialized", "networkpluginnotready",
}},
{detection.ModeNetworkConflict, []string{
"already exists for an interface", "address already in use",
"address already allocated", "route already exists",
}},
}

// eventMode names the failure mode an event's message makes more precise
Expand Down
Loading
Loading
0