Running a Patch

This page walks you through starting a patch run from the PatchMon web UI, from the initial click through dry-run validation, approval and live log streaming, to the final "patched" state. Everything here happens in the browser against a logged-in session.

Assumes you have the patching module enabled and the can_manage_patching permission. If the action buttons are missing, see Patching Overview for the permission matrix.

Starting Points

There are three entry points into a patch run:

All three funnel into the same Patch Wizard, a single modal used everywhere patching is initiated, so the mental model is identical regardless of where you started from.

The Patch Wizard

The wizard has a fixed six-step sequence, but it auto-skips steps that have no decision to make for your starting point. Unused steps are shown muted in the step indicator so you always see the full mental model.

Navigation is forward-only through the wizard; the Back button steps through enabled steps and skips the hidden ones.

Flow A: Patch All on a Single Host

This is the simplest case: update every out-of-date package on one host.

1. Open Hosts → select your host → Patching tab.
2. Click Patch all. The wizard opens at the Timing step (Hosts, Packages, Validate and Approval are all skipped for patch_all).
3. Review the effective patch policy shown on the Timing step. If the host has a delayed or fixed-time policy attached, the wizard tells you when the run will actually start (for example "Runs at 03:00 Europe/London").
4. If you need to bypass the delay, tick Run immediately. This sets schedule_override=immediate on the trigger call and fires the run as soon as the worker dequeues it.
5. Click Next to advance to Submit.
6. On Submit, read the per-host summary (host name, patch type, effective run time) and click Queue & patch.

What happens next on the server:

• A patch_runs row is inserted with patch_type=patch_all, dry_run=false, and the policy snapshot.
• A run_patch task is enqueued on the patching asynq queue.
• If the policy introduces a delay, the task is scheduled for the future and the run is shown as scheduled in Runs & History. Otherwise it goes straight to queued.
• The browser deep-links into the Run Detail page so you can watch the live terminal.

Note: On Linux and FreeBSD, patch_all cannot be dry-run. The agent's bulk-upgrade path (apt-get --with-new-pkgs upgrade, dnf upgrade, pkg upgrade, pacman -Syu, apk upgrade) does not support a reliable simulation mode. On Alpine a real patch_all still runs apk upgrade --simulate first and stops before changing anything if the plan would downgrade or replace a package, or remove one listed in /etc/apk/world. If you want a dry-run, patch specific packages instead. Windows hosts can validate a Patch all; see Windows patching.

Flow B: Patch a Specific Package with Dry-Run

This is the richer flow and the one to use for anything security-sensitive. The dry-run runs first, you review the transaction, then you approve.

1. Open Patching → click into the package from a host, or start from Packages → package name → Patch this package.

2. The wizard opens at the Validate step (Hosts and Packages steps may be visible for fleet rollouts).

3. Click Run dry-run. The browser calls POST /patching/trigger with dry_run=true. For each target host, the server:

   • Creates a patch_runs row in pending_validation status.
   • Sends the run_patch command to the agent with DryRun=true.
   • The agent runs the package-manager simulation step (for example apt-get -s install <packages>).

4. The wizard polls each run until it terminates. While you wait you see:

   • The per-host status badge (pending validation → running → validated).
   • A live terminal excerpt for the currently running host.

   If the server has not dispatched the dry-run to the agent within 30 seconds, the host is shown as Offline; it will patch without validation, by approving that pending validation, once the host is back. Once the server has dispatched it to the agent, the wizard waits for the result, which matters on Windows where a Windows Update search can take several minutes. If the agent drops its connection meanwhile, the host shows Agent reconnecting until it returns. After 25 minutes the host is shown as Still running.

   You can close the wizard or choose Continue without waiting at any point. Dry-runs already sent carry on and their results appear in Runs & History. Hosts still validating are left out when you submit; patch them from Runs & History once validation finishes. If you change the package selection after validating, a validated host is patched with the new selection without validation, and an offline host is left out until you cancel or delete its old validation in Runs & History.

5. When the dry-run completes, the run transitions to validated and the wizard shows:

   • The final Packages affected list (what would be installed, always a superset of the original request because of dependency resolution).
   • An Extra deps badge if dependency resolution pulled in packages you didn't originally ask for.
   • The captured stdout/stderr for the simulation.

6. Review the output. If the transaction looks correct, click Next to the Timing step, choose Run immediately or leave the policy delay, and then Approve & Patch on the Submit step.

7. The browser calls POST /patching/runs/{validationId}/approve. The server:

   • Marks the validation run approved (terminal state; the row and output are preserved for audit).
   • Creates a new patch_runs row with dry_run=false, linked to the validation via validation_run_id.
   • Enqueues the real run_patch task with the effective policy delay.
   • Returns the new run ID; the UI deep-links you into its Run Detail page if it's going to start immediately.

Retrying a stuck validation

If the agent was offline when you triggered the dry-run, the run will sit in pending_validation until it comes back. You have two options:

• Retry Validation: re-queues the same dry-run task. Call this once the agent is back online. Works via POST /patching/runs/{id}/retry-validation and is only available for patch_package runs.
• Skip & Patch: bypasses the dry-run entirely and queues the real patch run directly. Use this when you're confident about the change and the host won't come back soon. The UI labels the button amber to make it clear you're skipping a safety step.

Both options are available from the Runs & History table, from the Run Detail page, and inline on a per-row basis.

Submitting a patch_all run for approval

patch_all can't be dry-run, but you can still route it through an approval gate. In the wizard's Approval step tick Submit for approval. The run is created in pending_approval status with no execution task enqueued. A second reviewer with can_manage_patching can then open Runs & History, click Approve on that row, and the server builds the real execution run the same way as a validated approval. Until that happens the run sits in the DB and can also be deleted.

Flow C: Patch a Package Across the Fleet

Same as Flow B, but starting from the Packages page. The wizard discovers which hosts have the package out of date (only those show up in the Hosts step) and validates each in parallel.

1. Navigate to Packages → click the package name → Patch this package.
2. The wizard opens at Hosts. Tick the hosts you want to patch, or Select all.
3. Click Next to Validate. The wizard fans out dry-runs across the selected hosts with a bounded concurrency pool (5 at a time by default) to avoid hammering the queue.
4. When every host has reached a terminal validation state, you see a per-host results table. Hosts with failed dry-runs are flagged so you can exclude them from the approval step or retry them.
5. Timing lets you pick a per-host policy override, useful when your fleet mixes policies.
6. Submit fires POST /patching/runs/{id}/approve once per validated run. The UI tracks failures in a bulk-approve result banner when you come back to Runs & History.

Watching a Run: The Run Detail Page

Once a run is fired and not delayed, the UI redirects you into /patching/runs/{id}. The page has:

• A header with the host name, status badge, Awaiting inventory report pill (post-patch), and the primary action buttons for the current state (Approve & Patch, Retry Validation, Skip & Patch, Stop Run).
• A left-side Run summary card: host, type, initiated by, approved by, started / completed timestamps, link to the validation run if any, patch policy in effect, and packages affected. A package name links to its Package Detail page while the package is in the inventory. Names that are no longer in it, such as a Windows update once it is installed, show as plain text.
• A right-side Shell output terminal for the live log stream.

Live log streaming

While the run is running, the Run Detail page opens a WebSocket connection to /api/v1/patching/runs/{id}/stream. The server's in-process patchstream hub fans out agent-published events to every connected browser:

• snapshot: sent once when you connect. Contains the current stage and whatever shell_output is already persisted, so the terminal pane is primed even if you arrived mid-run.
• chunk: a short piece of stdout/stderr from the agent, appended to the terminal as it arrives.
• done: a terminal stage (completed, failed, cancelled, validated, or dry_run_completed). The socket closes and the UI refetches run metadata to show the final state.

The header shows a pulsing green Live pill while the WebSocket is open. If you scroll up in the terminal to read earlier output, the UI stops auto-scrolling; scroll back to the bottom and it resumes.

If you arrive on a run that's already in a terminal state, the server sends a single snapshot message containing the persisted shell_output plus a synthetic done message, then closes the connection. The database is the source of truth, so you see exactly the same terminal contents as everyone else.

Copying output

When the run is not queued or running, a Copy output button appears above the terminal. It copies the full shell output to your clipboard. Use this for incident reports or to paste into a ticket.

The terminal normalises carriage returns. apt-get and dpkg use \r to overwrite progress bars on a single line, which would be invisible in a scrollback view. The UI converts \r to \n so every progress update becomes its own readable line.

Stopping a Running Patch

You can stop a run while it is in the running state. This is a hard stop with no graceful "let it finish the current package" behaviour.

1. On the Run Detail page, click Stop Run in the header.
2. Confirm in the dialog. The warning "Partially-installed packages may leave the host in an intermediate state" is there for a reason. Interrupting apt or dnf mid-transaction can leave dpkg or rpmdb needing manual repair.
3. The browser calls POST /patching/runs/{id}/stop. The server looks up the agent in the agentregistry, sends a patch_run_stop WebSocket message, and returns 202 Accepted.
4. The agent cancels the subprocess via SIGINT, collects whatever output it has, and reports a terminal cancelled stage back to the server.
5. The live stream closes and the run's final status is cancelled.

When Stop Run is not available

• The run is not in the running state (no subprocess to interrupt). For queued, scheduled, pending_validation, pending_approval, or validated, use Delete in Runs & History instead. That also removes any scheduled task from the asynq queue.
• The agent is not currently connected. The UI returns a 409 with "Agent is not currently connected". Wait for the agent to reconnect, or stop the agent's systemd service if you need the patch process killed at the OS level.

Post-Patch: The Awaiting Inventory Report Pill

When a patch_all or non-dry-run patch_package completes, the server sets awaiting_post_patch_report_run_id on the host. The Run Detail page shows an Awaiting inventory report pill next to the status badge, and the polling interval keeps ticking every 3 seconds.

The agent's next scheduled inventory report (usually within 60 minutes, sooner if triggered manually) updates the host's package list. The server clears the awaiting flag, and:

• If the host's last_update timestamp is newer than the run's completed_at, the pill changes to New report received.
• Otherwise the pill disappears. Absence is the soft success signal.

This is how you know the packages the patch run installed are now reflected in the Package Inventory view, not just that the apt command returned exit 0.

Troubleshooting Common Cases

The run sits in queued forever

The asynq worker picked up the task but cannot reach the agent, or the agent never received the WebSocket command.

• Check the agent's connection status on the Host Detail page (green Connected pill).
• Check the server logs for patching: entries around the task ID (patch-run-<run-id>).
• If the agent has been offline and just reconnected, wait up to 30 seconds for the WebSocket re-handshake.

"package manager ... not supported for patching"

The agent rejected the run because the host's detected package manager is not in the supported list (apt, dnf, yum, pkg, pacman, apk). Alpine hosts show this error until their agent is updated to 2.1.4 or later. The run immediately transitions to failed with the message in error_message.

Alpine: "/etc/apk/world requires ... but ... is installed"

A version constraint in /etc/apk/world no longer matches the installed package, so apk refuses every transaction. Nothing was changed. Edit that line in /etc/apk/world as the message suggests (pin it to the installed version, or remove the version), then run the patch again. apk fix fails the same way, so it does not help here.

Alpine: "apk plans changes PatchMon does not apply automatically"

The simulated plan contained a step the agent does not apply on its own, so it stopped before changing anything. The message lists the steps and the likely cause of each:

• Downgrading: usually a repository, often a tagged one such as @edge, was removed or changed in /etc/apk/repositories while packages from it are still installed. Restore the repository or deal with those packages by hand.
• Purging a package listed in /etc/apk/world: usually a conflict (!name) or a changed constraint in /etc/apk/world, or the package was renamed or replaced by another. Packages that are not in world are removed as normal clean-up and do not stop the run.
• Replacing: a repository now serves a different build of a version that is already installed.
• Anything else: apk printed a step this agent does not recognise.

Run apk upgrade --simulate on the host to see the whole plan.

Alpine: "another apk process held the package database"

Another apk command held the database lock (/lib/apk/db/lock) for more than 30 seconds, for example an install someone was running by hand. Wait for it to finish and run the patch again.

The run completes but the package inventory hasn't updated

The agent hasn't sent its post-patch inventory report yet. The Awaiting inventory report pill will flip automatically once it arrives. If it doesn't arrive within an hour, force a report from the agent CLI:

sudo patchmon-agent report

Approve returns 400 "Only validated... runs can be approved"

Another operator already approved or deleted the run while you were looking at it. Reload Runs & History; the row will now be in approved or gone.

"A patch run is already queued or in progress on this host"

PatchMon allows one queued, validating or running patch run per host at a time, because package managers lock while they work and a second run would fail on the lock. The blocking run is listed under the Active status filter in Runs & History. Wait for it to finish, or stop or delete it, then start the new one; in a multi-host wizard, busy hosts are reported as failed and the other hosts go ahead. Runs waiting for a policy window show as Scheduled and do not block the host. If the host is still busy when the window arrives, the scheduled run is cancelled with the reason "Skipped: another patch run was in progress on this host when the scheduled window arrived". Runs that are not dispatched to the agent within 24 hours of becoming due (for example because the host stayed offline) are marked Timed out by the cleanup sweep, so they never block a host indefinitely.

Related Documentation

• Patching Overview: the three core concepts, module gates, and supported OS coverage.
• Patch Policies and Scheduling: control when approved runs actually fire.
• Patch History and Live Logs: the Runs & History table, filtering, and deeper dive into the live log stream.
• Managing the PatchMon Agent: agent CLI, service management, and troubleshooting connection issues.