Chapter 19 of 26
Notification Routes and Delivery Log
Overview
In PatchMon, a route (labelled Event Rule in the UI) connects one or more event types, and optionally a severity floor, a host scope, or a match rule, to a destination. When an event fires, the notifications engine evaluates every enabled route; each matching route produces a delivery to its destination.
Routes handle fan-out: one host_down event can notify your on-call ntfy topic, post to a Discord #alerts channel, and write an internal alert record, all from a single emit.
Both routes and the Delivery Log live under Reporting in the main navigation:
- Reporting → Event Rules: create, edit, and disable routes.
- Reporting → Delivery Log: every outbound delivery attempt, sent or failed.
Permissions
| Action | Permission |
|---|---|
| Create / edit / disable / delete routes | can_manage_notifications |
| Read the delivery log | can_view_notification_logs |
Admins and superadmins bypass these checks.
Creating a route
- Go to Reporting → Event Rules.
- Click Add event rule. (Disabled until at least one destination exists. Create one first under Notification Destinations.)
- Fill in the modal:
| Field | Notes |
|---|---|
| Destination | Required. Pick from the list of configured destinations. You can only route to enabled destinations; disabled destinations are skipped at delivery time. |
| Events | Tick All events to match every event type, or tick individual events. Selecting every individual event collapses back to "All events". |
| Minimum severity | Floor for the route. Events below this severity are ignored. Order is informational < warning < error < critical. |
| Host groups | Optional. If any are selected, only events whose host is a member of at least one of the groups match. Leave empty for "any host". |
| Individual hosts | Optional. If any are selected, only events for those specific hosts match. Leave empty for "any host". |
| Enabled | On by default. Turn off to keep the rule for later without it firing. |
- Click Add.
Event type reference
Pick from the same set documented in Alerts Overview. host_down, host_recovered, patch_run_completed, ssh_session_started, and so on. You can also select high-volume or low-volume events like user_login and account_locked to route sign-in telemetry.
Host group and host filters combined
If both host groups and individual hosts are set, the event must satisfy both filters. In practice you usually pick one or the other, not both.
Events without a host context (e.g. server_update, user_created) are filtered out by any host scope you add. Leave both scope fields empty to match those as well.
Severity, delay, and lifecycle
Per-type Alert delay in Alert Lifecycle applies before the route fan-out: if the event has a configured alert_delay_seconds, PatchMon enqueues the delivery with that delay. If a counterpart event fires within the window (for example, host_recovered while a delayed host_down is queued), the delayed notification is cancelled. Counterpart mapping:
| Delayed event | Cancelled by |
|---|---|
host_down |
host_recovered |
container_stopped |
container_started |
host_security_updates_exceeded |
host_security_updates_resolved |
host_pending_updates_exceeded |
host_pending_updates_resolved |
Editing and deleting routes
Each row in Event Rules has Edit and Delete buttons.
- Edit reopens the modal with the saved values. Save to update; the new criteria take effect for the next matching event.
- Delete removes the rule entirely. Deliveries already enqueued finish, but no new deliveries are produced.
Disabled routes are displayed with a muted Disabled badge and do not receive deliveries. Disable is the safer option if you want to pause a rule temporarily.
How matching works
For each outgoing event, the server:
- Looks up all routes whose
event_typesinclude the event type (or the wildcard*). - Drops routes whose destination is disabled.
- Drops routes whose
min_severityis above the event's severity. - For each remaining route, applies the host-group and host-ID filters.
- Deduplicates: events that repeat within a 2-minute window for the same destination are collapsed into one delivery. The fingerprint key is
event_type + reference_id + destination_id + 2-minute bucket. - Rate-limits: each destination is capped at 60 deliveries per minute. Deliveries over the cap are dropped with a warning in the server log.
- Enqueues an asynq task to the
notificationsqueue withMaxRetry=5.
The queue worker then dispatches the delivery according to the destination's channel type (SMTP send, HTTP POST, ntfy publish, or internal alert write).
The Delivery Log
The Delivery Log tab shows every outbound notification attempt with its result. Use it when a destination is not receiving messages, a webhook recipient reports errors, or you want an audit trail of what went where.
Columns
| Column | Meaning |
|---|---|
| Time | When the delivery was processed, shown as a relative time ("5m ago"). Hover for the exact timestamp. |
| Status | sent for success (green), anything else for failure (red). |
| Event | The event type that produced the delivery (e.g. host_down, patch_run_failed). |
| Destination | The destination display name at the time of delivery. Shows the UUID if the destination has been deleted. |
| Reference | reference_type:reference_id, clickable for host, patch_run, and alert references so you can jump to the source. |
| Error | Error message returned by the delivery attempt. Empty for successful deliveries. |
Pagination
The log is paginated at 50 rows per page. Use the left and right arrows at the bottom to move through history. The most recent deliveries are on page 1.
Use the Refresh log button in the page header to pull the latest entries without navigating away.
Retries
The notifications worker retries failed deliveries up to 5 times with exponential back-off (handled by asynq). Each attempt is recorded on the same delivery log row: the attempt_count field increments, and the row is upserted with the latest status and error_message. The provider message ID (e.g. SMTP queue ID, webhook Message-ID) is captured in provider_message_id when the remote end returns one.
If all five retries fail, the delivery row stays at the last failed state. There is no automatic escalation; diagnose the failure from the Error column.
Common failure reasons
| Error (excerpt) | Likely cause |
|---|---|
connect: connection refused / i/o timeout |
Destination host is unreachable from the PatchMon server. Check firewall / network. |
authentication failed / 535 5.7.8 |
Wrong SMTP credentials or token. Re-edit the destination and re-enter. |
400 Bad Request from Discord/Slack webhook |
Webhook URL is wrong, revoked, or the rich payload is malformed for a customised Slack app. |
403 Forbidden from ntfy |
Topic requires auth you have not provided, or token is expired. |
destination disabled |
Someone disabled the destination between enqueue and delivery. Re-enable and re-trigger. |
If an expected entry is missing, check that the route is enabled, the destination is enabled, the event passed the severity and scope filters, and the alert type itself is enabled in Alert Lifecycle.
Deduplication and rate-limiting in the log
Duplicates suppressed by the 2-minute dedup window do not appear in the delivery log; they are silently skipped before a delivery task is created. Rate-limited deliveries are also skipped silently (a warning goes to the server log, not the delivery log). If a destination suddenly stops receiving events, check:
- The destination is enabled.
- No route has been deleted.
- The per-minute rate cap is not being exceeded upstream. 60 messages/minute is per-destination.
App links in notifications
Every notification includes an app_link in its metadata pointing back to the most relevant page in PatchMon:
| Reference type | Link |
|---|---|
patch_run |
/patching/runs/<id> |
host |
/hosts/<id> |
alert |
/hosts/<host_id> if known, otherwise / |
user |
/settings/users |
test |
/reporting |
Formatters for each channel render this as a clickable button (Discord/Slack rich embeds), an <a> tag (email), or a Click action (ntfy).