Alerts

Alert rules are created and managed from the TUI (Alerts view, press n). Rules are stored in the database and evaluated by the daemon on every evaluation cycle.

Alert Types

Threshold

Fires when a metric average exceeds a value for a sustained duration.

Predictive

Uses linear regression to predict when a metric will breach a threshold within a given timeframe.

• disk.used_pct — predicts disk fill within 24h, 3d, or 7d

Variance

Fires when memory usage changes exceed a delta threshold a certain number of times within a window. Useful for detecting memory thrashing or crash loops.

• memory.variance — counts memory usage spikes exceeding N% within a time window

Debouncing

The alert engine won't re-fire the same rule if an unacknowledged alert exists within 3x the evaluation interval. This prevents alert storms for persistent conditions.

Notification Channels

Alerts can be delivered to any combination of five notification methods. All are configured in the TOML config file under [alerts].

Webhook

HTTP POST with JSON payload. Supports custom headers.

[[alerts.webhooks]]
url = "https://hooks.example.com/alert"
# headers = { "X-Auth" = "secret" }

ntfy

Push notifications via ntfy.sh or a self-hosted server. Alert severity maps to ntfy priority.

[[alerts.ntfy]]
url = "https://ntfy.sh"
topic = "bewitch-alerts"
# token = ""  # optional auth

Email (SMTP)

Send email alerts via SMTP with STARTTLS or implicit TLS.

[[alerts.email]]
smtp_host = "smtp.example.com"
smtp_port = 587
username = "alerts@example.com"
password = "app-password"
from = "alerts@example.com"
to = ["admin@example.com", "ops@example.com"]
starttls = true  # false for implicit TLS (port 465)

Gotify

Push notifications to a self-hosted Gotify server.

[[alerts.gotify]]
url = "https://gotify.example.com"
token = "AxxxxxxxxxxxxxxR"  # application token
priority = 0  # 0 = auto-map from severity (warning=5, critical=8)

Command

Execute an arbitrary shell command with alert details as environment variables.

[[alerts.commands]]
cmd = "/usr/local/bin/alert-handler"

Available environment variables:

Commands run with a 10-second timeout.

Testing Notifications

Press t on the Alerts view to send a test notification through all configured channels. This triggers the POST /api/test-notifications endpoint which sends synchronously (blocks until all channels respond).

Managing Rules via API

Rules can also be managed programmatically. See the API Reference for the alert-rules endpoints.