8000 🧭 Teach the engine about 300 blind failure scenarios by abahmed · Pull Request #619 · abahmed/kwatch · GitHub
[go: up one dir, main page]

Skip to content

🧭 Teach the engine about 300 blind failure scenarios - #619

Open
abahmed wants to merge 73 commits into
mainfrom
scenario-gaps-round
Open

abahmed wants to merge 73 commits into
mainfrom
scenario-gaps-round

Conversation

@abahmed
@abahmed abahmed commented Nov 19, 2026 •
Copy link
Copy Markdown
Owner

Status: work in progress. CI is red on purpose until every new scenario passes. Engine fixes land area by area (below). Don't merge yet.

What this PR does

Adds about 300 blind, labelled failure scenarios to the alert-quality corpus, then teaches the engine to name the right root cause for them. Each scenario was written from a real Kubernetes failure without reading engine code. The label comes from Kubernetes behaviour, never from kwatch's output.

Where the scenarios come from

Round Source Scenarios
1 Kubernetes failure reasons, kubernetes-mixin alerts, k8sgpt analyzers, Chaos Mesh, AIOpsLab fault list 31
2 Common failures from Stack Overflow, Kubernetes issues and docs 49
3 node-problem-detector rules, controller event reasons, troubleshooting docs, admission 41
4A / 4B Kubernetes e2e and kubelet source; public outage postmortems (no company names) 34 / 21
5A / 5B Generic parts of managed-Kubernetes guides; community forums and Helm chart issues 20 / 24
6A / 6B Upgrade breakages (changelog 1.20–1.34); CSI, CNI, EndpointSlice and dual-stack plumbing 10 / 19
7A / 7B Workload patterns; RBAC, ServiceAccounts, encryption, admission and NetworkPolicy 7 / 12

Every candidate was checked against the existing scenarios for duplicates, and rounds 6–7 were scored for how common, impactful, distinct and visible each failure is. Sources are Apache-2.0 / CC BY 4.0 or used as ideas only. No text was copied.

Engine rules added so far

  • Round 1 (13 misses): volume bound elsewhere, pull Secret of the wrong type, missing PriorityClass, Gateway class not accepted, CNI address conflict, finalizer-stuck pods, CRD conversion backend gone, paused Deployment with a pending edit, NotReady ordering. Full detail is in the first commits.
  • Scheduler names the object: a scheduling verdict roots at the object behind it (PriorityClass with preemptionPolicy Never, ReadWriteOncePod holder, oversized claim, scaled-down affinity target, pinned node, node missing a scarce resource).
  • Node-local failure: a node is blamed when one workload's replicas fail with a node-local error there while same-ReplicaSet siblings run elsewhere.
  • DNS chain: the cause behind the DNS servers (Corefile edit, memory limit) stands for the DNS outage.
  • Control plane: a service that goes down repeatedly is reported, and its kube-system pod is tied to it.
  • Lifecycle:
    • old evictions and a silent fleet stay quiet
    • replica flapping names the autoscaler (digest)
    • a PDB that selects no pods notifies
    • sporadic readiness failures go to the digest
    • an ambiguous autoscaler selector goes to the digest

Progress by area

Area State
Nodes and control plane partly fixed
Workloads and lifecycle partly fixed
DNS and network in progress
Storage and CSI in progress
Access, admission and quota queued

Failing scenarios: 94 at the last count, down from about 160.

Fixture fixes (unrealistic data, not label changes)

  • Added the PriorityClass the pods name (quota-scoped-to-priority-class-exhausted).
  • Added the RuntimeClass the other replicas use (node-runtime-class-handler-missing).
  • Added the DaemonSet matchFields pin real DaemonSet pods carry.
  • Gave running Job pods a Ready condition (4 Job scenarios).
  • Aged the evictions in evicted-pods-linger-after-recovery-quiet as its description says.
  • Spelled snapshot ids group-qualified (kwatch's id for custom resources).

Label changes (Kubernetes reason)

Round 1 label changes are listed in the first section of the earlier description (nodeSelector/anti-affinity root at the scheduling constraint, HPA without CPU request to the digest, NoExecute taint with ready replicas stays quiet, NotReady nodes page). Later label changes were made by review before the scenarios were pushed:

  • The scheduler-lease root stays etcd, with the scheduler allowed as a revision.
  • A stale EndpointSlice is unknown.
  • A Job failing from drains roots at the Job.

New user-facing wording (designer review pending for the newer lines)

  • Round 1 lines were reviewed and applied.
  • Newer lines:
    • "went down 3 times in 15 minutes"
    • "is not running" ( 8000 control-plane pods)
    • "keeps changing the replica count"
    • "is named by the scheduler's verdict"
    • "Replicas of X changed N times in 30m"

Before merge

  • All gates pass, make verify, and ratchet.txt and SCORECARD.md regenerated.
  • A final review for duplicate scenarios and wrong labels.
  • Designer review of new wording.
  • A check that holding reports while every node is silent still pages a real cluster-wide outage within 2m.

…top blaming DNS servers when config is unreadable
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0