8000 fix: keep a resolve no louder than its incident for ntfy, Gotify and Pushover by abahmed · Pull Request #592 · abahmed/kwatch · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion deploy/provider-catalog.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -35,15 +35,18 @@ googlechat|Google Chat|webhook|string|true|true|url||Webhook URL
gotify|Gotify|url|string|true|false|url||Gotify server URL
gotify|Gotify|token|string|true|true|||App token
gotify|Gotify|priority|integer|false|false|integer||Priority (optional)
gotify|Gotify|resolvedPriority|integer|false|false|integer||Priority 0-10 of a resolve (optional, default: the priority, capped at 5)
gotify|Gotify|title|string|false|false|||Custom title
ntfy|ntfy|topic|string|true|true|||Topic to publish to
ntfy|ntfy|url|string|false|false|url|https://ntfy.sh|Server URL (default: https://ntfy.sh)
ntfy|ntfy|token|string|false|true|||Optional auth token
ntfy|ntfy|title|string|false|false|||Custom title
ntfy|ntfy|priority|integer|false|false|integer|4|Priority 1-5 (default: 4)
ntfy|ntfy|priority|integer|false|false|integer||Priority 1-5 (optional, ntfy uses 3 when unset)
ntfy|ntfy|resolvedPriority|integer|false|false|integer||Priority 1-5 of a resolve (optional, default: the priority, capped at 3)
pushover|Pushover|token|string|true|true|||Application token
pushover|Pushover|user|string|true|true|||User or group key
pushover|Pushover|priority|integer|false|false|integer||Priority from -2 to 2 (optional)
pushover|Pushover|resolvedPriority|integer|false|false|integer||Priority from -2 to 1 of a resolve (optional, default: the priority, capped at 0)
pushover|Pushover|retry|integer|false|false|integer||Emergency retry interval in seconds
pushover|Pushover|expire|integer|false|false|integer||Emergency expiration in seconds
pushover|Pushover|title|string|false|false|||Custom title
Expand Down
9 changes: 6 additions & 3 deletions docs/providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ anyone. Sensu Go records them as a passing (OK) check.
| 💬 Chat | Slack, Discord, Microsoft Teams, Teams Workflow, Mattermost, Rocket.Chat, Google Chat, Webex, Matrix, Telegram, Zulip, Feishu, DingTalk, WeCom, Flock, Threema, Signal | The Note as the message text, escaped for the chat's markup, with recent application output as a code or quoted block when there is any. Broadcast mentions such as `@channel` are neutralized. Matrix sends the Note as `body` and HTML-escaped as `formatted_body`. Slack with a bot token announces each incident as one root message holding the whole Note and its output, keeps every update and the resolve in its thread, and edits the root to the same announcement under the new status marker when the status changes (a thread reply that is not a status change leaves the root alone; after a restart the root cannot be rebuilt, so only the thread grows). Application output is fenced with a code fence longer than any run of backticks in it, so it cannot close the block; it is cut before the closing fence is added. WeCom `<@userid>` and Zulip `@*group*` mentions in workload text are neutralized too, and so is Telegram's `@username` (a zero-width space follows the `@`). Microsoft Teams shows the incident title as escaped text, so a title taken from the cluster never becomes a link or a mention. |
| 🚨 Paging | PagerDuty, Opsgenie, Splunk On-Call, GoAlert, ilert, incident.io, Squadcast, Zenduty, SIGNL4, Alerta, Datadog, New Relic, AWS SNS, Splunk HEC, Sensu Go | The Short as the alert title or summary, the Note (plus output) as details, the incident id as the dedup or alert key, and the provider's resolve or close action when the incident resolves. Severity follows the incident severity: only a page-tier incident gets the priority that pages (for ilert the configured `priority`); a notify or low tier incident is sent at a lower one (ilert `LOW`, Opsgenie P3 or P4). |
| 📋 Issue trackers | GitHub, GitLab, Gitea, Jira, ClickUp | One issue per incident: the Short as title and the Note (plus output and the cluster name) as body. Updates add a comment. On resolve kwatch comments and closes the issue (GitHub, GitLab, Gitea; Jira moves it through `closeTransition` and ClickUp to `closeStatus` when you set one, otherwise they only comment, because workflows are defined per project or list). A resolved incident that fails again within its reopen window comments on the same issue and reopens it on GitHub, GitLab and Gitea. Jira (`reopenTransition`) and ClickUp (`reopenStatus`) reopen only when you configure both the close and the reopen name; otherwise the mapping is forgotten at resolve and a recurrence opens a new issue. Pod-controlled text in the Note has its `@mentions` neutralized, and Jira wiki markup in it is escaped. A GitHub 403 that says the rate limit was hit is retried like a 429. |
| 📱 SMS and push | Twilio, Plivo, Vonage, MessageBird, Pushover, Pushbullet, Gotify, ntfy, IFTTT, Home Assistant | The Short. A resolve is sent at normal priority. IFTTT also sends the Note as `value2` and the status as `value3`. |
| 📱 SMS and push | Twilio, Plivo, Vonage, MessageBird, Pushover, Pushbullet, Gotify, ntfy, IFTTT, Home Assistant | The Short. A resolve is never louder than the incident: for Pushover, Gotify and ntfy it is sent at the configured priority capped at normal, or at `resolvedPriority` when set. IFTTT also sends the Note as `value2` and the status as `value3`. |
| 📧 Email | Email (SMTP), AWS SES, SendGrid, Mailgun, Resend | Subject is the Short on one line; the body is the Note followed by recent output as a `> ` quoted block. |
| 🔗 Structured | Custom Webhook, n8n, Zapier | The whole message as JSON (fields below). |

Expand Down Expand Up @@ -297,6 +297,7 @@ Opsgenie creates alerts asynchronously, so an update or a close sent right after
| `alert.gotify.url` | 🔗 Gotify server URL |
| `alert.gotify.token` | 🔑 App token |
| `alert.gotify.priority` | 🎚️ Priority (optional) |
| `alert.gotify.resolvedPriority` | 🔕 Priority 0-10 of a resolve (optional). Default: the priority, capped at 5; with no priority the server default applies |
| `alert.gotify.title` | ✏️ Custom title |

```yaml
Expand All @@ -313,7 +314,8 @@ alert:
| `alert.ntfy.topic` | 📢 Topic to publish to. kwatch publishes JSON to the server root with the topic in the body, as ntfy documents |
| `alert.ntfy.url` | 🔗 Server URL (default: `https://ntfy.sh`) |
| `alert.ntfy.token` | 🔑 Optional auth token |
| `alert.ntfy.priority` | 🎚️ Priority 1-5 (default: 4) |
| `alert.ntfy.priority` | 🎚️ Priority 1-5 (optional; ntfy uses 3 when unset) |
| `alert.ntfy.resolvedPriority` | 🔕 Priority 1-5 of a resolve (optional). Default: the priority, capped at 3 |
| `alert.ntfy.title` | ✏️ Custom title |

```yaml
Expand All @@ -328,7 +330,8 @@ alert:
|:---|---|
| `alert.pushover.token` | 🔑 Application token |
| `alert.pushover.user` | 👤 User or group key |
| `alert.pushover.priority` | 🎚️ Priority from -2 to 2 (optional). It applies to the announcement of an incident only; priority 2 (emergency) only to a page-tier announcement, other tiers are held at 1. Updates, summaries, digests, startup and plain messages are normal priority. A resolve cancels the emergency receipt first, so the alarm stops repeating, and then sends the normal-priority "resolved" push; if the cancel fails transiently nothing is sent and the whole resolve is retried |
| `alert.pushover.priority` | 🎚️ Priority from -2 to 2 (optional). It applies to the announcement of an incident only; priority 2 (emergency) only to a page-tier announcement, other tiers are held at 1. Updates, summaries, digests, startup and plain messages are normal priority. A resolve cancels the emergency receipt first, so the alarm stops repeating, and then sends the "resolved" push at the priority capped at normal; if the cancel fails transiently nothing is sent and the whole resolve is retried |
| `alert.pushover.resolvedPriority` | 🔕 Priority from -2 to 1 of the "resolved" push (optional); 2 is not allowed. Default: the priority, capped at 0 |
| `alert.pushover.retry` | ⏱️ Emergency retry interval in seconds |
| `alert.pushover.expire` | ⌛ Emergency expiration in seconds |
| `alert.pushover.title` | ✏️ Custom title |
Expand Down
10 changes: 1 addition & 9 deletions internal/alert/clickup/clickup.go
Original file line number Diff line number Diff line change
Expand Up @@ -61,15 +61,7 @@ func NewClickup(
return nil
}

priority := 0
switch v := config["priority"].(type) {
case float64:
priority = int(v)
case int:
priority = v
case int64:
priority = int(v)
}
priority, _ := transport.IntSetting(config["priority"])

// Status names are defined per list, so there is no safe default:
// without closeStatus a resolve only comments.
Expand Down
12 changes: 8 additions & 4 deletions internal/alert/gotify/contract_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,13 @@ func checkShape(
if s, _ := body["message"].(string); s == "" {
t.Fatalf("%s: message is empty", name)
}
// Resolves use the server default (priority omitted).
_, has := body["priority"]
if (name == "resolve") == has {
t.Fatalf("%s: priority = %v", name, body["priority"])
// A resolve is capped at the normal level (5).
want := 8.0
if name == "resolve" {
want = gotifyNormalPriority
}
if body["priority"] != want {
t.Fatalf("%s: priority = %v, want %v", name,
body["priority"], want)
}
}
66 changes: 50 additions & 16 deletions internal/alert/gotify/gotify.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ const gotifyAPIURL = "/message"
type gotifyPayload struct {
Title string `json:"title,omitempty"`
Message string `json:"message"`
Priority int `json:"priority,omitempty"`
Priority *int `json:"priority,omitempty"`
}

type Gotify struct {
Expand All @@ -25,6 +25,8 @@ type Gotify struct {
token string
title string
priority int
// resolved is the explicit resolve priority; nil derives it.
resolved *int

clusterName string
}
Expand Down Expand Up @@ -56,15 +58,8 @@ func NewGotify(

title, _ := config["title"].(string)

priority := 0
switch v := config["priority"].(type) {
case float64:
priority = int(v)
case int:
priority = v
case int64:
priority = int(v)
}
priority, _ := transport.IntSetting(config["priority"])
resolved := resolvedPriority(config["resolvedPriority"])

klog.InfoS("initializing gotify",
"url", transport.LogURL(server),
Expand All @@ -76,6 +71,7 @@ func NewGotify(
token: token,
title: title,
priority: priority,
resolved: resolved,
clusterName: clusterName,
}
}
Expand All @@ -86,24 +82,62 @@ func (g *Gotify) Name() string {
}

// SendIncident sends the incident's one-line lead as the push message.
// Open incidents use the configured priority; a resolve uses the server
// default priority.
// Open incidents use the configured priority; a re 10000 solve uses it capped at
// gotifyNormalPriority, or resolvedPriority when set. With no priority
// configured the server default applies.
func (g *Gotify) SendIncident(
ctx context.Context, m notification.Message,
) error {
priority := g.priority
priority := nonZero(g.priority)
if m.Resolved() {
priority = 0
priority = g.resolvePriority()
}
return g.send(ctx, m.ShortText(), priority)
}

// gotifyNormalPriority is the most a resolve uses unless
// resolvedPriority says otherwise.
const gotifyNormalPriority = 5

// resolvePriority returns nil (omitted, server default) only when
// neither priority nor resolvedPriority is set; an explicit 0 is sent.
func (g *Gotify) resolvePriority() *int {
if g.priority == 0 && g.resolved == nil {
return nil
}
v := transport.ResolvePriority(
&g.priority, g.resolved, gotifyNormalPriority)
return &v
}

// nonZero maps an unset (zero) priority to nil so it stays omitted.
func nonZero(v int) *int {
if v == 0 {
return nil
}
return &v
}

// resolvedPriority reads resolvedPriority; a value outside Gotify's
// 0-10 range is ignored with a log line.
func resolvedPriority(value interface{}) *int {
p := transport.IntSettingPtr(value)
if p != nil && (*p < 0 || *p > 10) {
klog.InfoS("ignoring gotify resolvedPriority outside 0-10",
"resolvedPriority", *p)
return nil
}
return p
}

// SendMessage sends text message to the provider
func (g *Gotify) SendMessage(ctx context.Context, msg string) error {
return g.send(ctx, msg, g.priority)
return g.send(ctx, msg, nonZero(g.priority))
}

func (g *Gotify) send(ctx context.Context, msg string, priority int) error {
func (g *Gotify) send(
ctx context.Context, msg string, priority *int,
) error {
payload := gotifyPayload{
Title: g.title,
Message: msg,
Expand Down
10 changes: 6 additions & 4 deletions internal/alert/gotify/incident_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,12 @@ func TestSendIncidentSendsShortWithStatusPriority(t *testing.T) {
t.Fatalf("payload = %v", body)
}
providertest.AssertOneLeadingEmoji(t, message)
_, hasPriority := body["priority"]
if tc.Message.Resolved() == hasPriority {
t.Fatalf("priority = %v, resolved = %v",
body["priority"], tc.Message.Resolved())
want := 8.0
if tc.Message.Resolved() {
want = gotifyNormalPriority
}
if body["priority"] != want {
t.Fatalf("priority = %v, want %v", body["priority"], want)
}
})
}
Expand Down
49 changes: 49 additions & 0 deletions internal/alert/gotify/resolved_priority_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
package gotify

import (
"context"
"testing"

"github.com/abahmed/kwatch/internal/delivery/providertest"
)

func TestResolvePriorityRules(t *testing.T) {
cases := []struct {
name string
config map[string]interface{}
want interface{}
}{
{"unset stays omitted", map[string]interface{}{}, nil},
{"high is capped", map[string]interface{}{"priority": 8}, 5.0},
{"low stays low", map[string]interface{}{"priority": 2}, 2.0},
{"override wins over cap", map[string]interface{}{
"priority": 8, "resolvedPriority": 7}, 7.0},
{"override wins over low", map[string]interface{}{
"priority": 2, "resolvedPriority": 4}, 4.0},
{"override without priority", map[string]interface{}{
"resolvedPriority": 3}, 3.0},
{"explicit zero is sent", map[string]interface{}{
"priority": 8, "resolvedPriority": 0}, 0.0},
{"explicit zero without priority", map[string]interface{}{
"resolvedPriority": 0}, 0.0},
{"invalid override ignored", map[string]interface{}{
"priority": 8, "resolvedPriority": 11}, 5.0},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
rec := providertest.NewRecorder(t)
tc.config["url"] = rec.URL()
tc.config["token"] = "t"
c := NewGotify(tc.config, "dev", rec.Dependencies())
err := c.SendIncident(
context.Background(), providertest.Resolve())
if err != nil {
t.Fatalf("SendIncident() error = %v", err)
}
got := rec.Last(t).JSON(t)["priority"]
if got != tc.want {
t.Fatalf("priority = %v, want %v", got, tc.want)
}
})
}
}
1 change: 1 addition & 0 deletions internal/alert/gotify/testdata/contract/resolve.golden
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,6 @@ X-Gotify-Key: <redacted>

{
"message": "✅ payments in shop is healthy again.",
"priority": 5,
"title": "kwatch"
}
43 changes: 31 additions & 12 deletions internal/alert/ntfy/ntfy.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ type Ntfy struct {
token string
title string
priority int
// resolved is the explicit resolve priority; nil derives it.
resolved *int

clusterName string
}
Expand Down Expand Up @@ -58,15 +60,8 @@ func NewNtfy(
token, _ := config["token"].(string)
title, _ := config["title"].(string)

priority := 0
switch v := config["priority"].(type) {
case float64:
priority = int(v)
case int:
priority = v
case int64:
priority = int(v)
}
priority, _ := transport.IntSetting(config["priority"])
resolved := resolvedPriority(config["resolvedPriority"])

klog.InfoS("initializing ntfy",
"url", transport.LogURL(server),
Expand All @@ -82,6 +77,7 @@ func NewNtfy(
token: token,
title: title,
priority: priority,
resolved: resolved,
clusterName: clusterName,
}
}
Expand All @@ -91,19 +87,42 @@ func (n *Ntfy) Name() string {
return "Ntfy"
}

// ntfyDefaultPriority is ntfy's normal priority, used for resolves.
// ntfyDefaultPriority is ntfy's normal priority, the most a resolve
// uses unless resolvedPriority says otherwise.
const ntfyDefaultPriority = 3

// configured returns the alert priority, or nil when unset.
func (n *Ntfy) configured() *int {
if n.priority == 0 {
return nil
}
return &n.priority
}

// resolvedPriority reads resolvedPriority; a value outside ntfy's 1-5
// range is ignored with a log line.
func resolvedPriority(value interface{}) *int {
p := transport.IntSettingPtr(value)
if p != nil && (*p < 1 || *p > 5) {
klog.InfoS("ignoring ntfy resolvedPriority outside 1-5",
"resolvedPriority", *p)
return nil
}
return p
}

// SendIncident sends the incident's one-line lead as the message. Tags
// are plain words, never ntfy emoji shortcodes, so the status marker at
// the start of the lead stays the only emoji. A resolve is sent at the
// normal priority.
// normal priority, or lower when the configured priority is lower, unless
// resolvedPriority is set.
func (n *Ntfy) SendIncident(
ctx context.Context, m notification.Message,
) error {
priority := n.priority
if m.Resolved() {
priority = ntfyDefaultPriority
priority = transport.ResolvePriority(
n.configured(), n.resolved, ntfyDefaultPriority)
}
return n.send(ctx, ntfyPayload{
Title: n.title, Message: m.ShortText(), Priority: priority,
Expand Down
Loading
3A4B
0