Agent Troubleshooting

Overview

This page is a quick-reference decision tree for the most common PatchMon agent symptoms. Each entry points to detailed steps in Managing the PatchMon Agent: Common Troubleshooting or another relevant section. Use this page to triage, then follow the linked section for the full fix.

If the problem is on the server side (container crashes, database migrations, CORS, reverse proxy), go to Server Troubleshooting instead.

The 30-Second Triage

Before anything else, run these three commands on the affected host:

sudo patchmon-agent diagnostics    # full picture in one shot
sudo systemctl status patchmon-agent    # (or: rc-service patchmon-agent status)
sudo journalctl -u patchmon-agent -n 50 --no-pager    # (or: tail -n 50 /etc/patchmon/logs/patchmon-agent.log)

On Windows (elevated PowerShell):

patchmon-agent diagnostics
Get-Service PatchMonAgent
Get-Content 'C:\ProgramData\PatchMon\patchmon-agent.log' -Tail 50

The diagnostics output includes system info, configuration status, network reachability, credential validity, and the last 10 log lines. In most cases it tells you immediately which part is broken. Full details: Managing the PatchMon Agent: diagnostics.

Decision Tree

Symptom → Likely Cause → Where to Look

Host Shows Pending

The host record was created but the agent has not sent its first report.

Quick checks:

sudo systemctl status patchmon-agent
sudo patchmon-agent ping
sudo patchmon-agent report      # force an immediate report

If ping fails, see Cannot reach server: DNS, TLS, or firewall.

If ping succeeds but the host stays Pending, the agent has not pushed a report yet. Run sudo patchmon-agent report and watch journalctl -u patchmon-agent -f for errors during report upload. A common cause is 413 Request Entity Too Large. See Server Troubleshooting: Upload/Body-Limit Errors.

Full details: Managing the PatchMon Agent: Agent Shows "Pending" in PatchMon.

Host WS Pill is Red

The host sent at least one report in the past, but its WebSocket has been disconnected for longer than the host_down threshold (default 30 seconds, configurable in Reporting → Alert Lifecycle). The host may still be alive — check the Reporting pill: if it's green, the agent is pushing HTTP reports normally and only the real-time control channel is unavailable.

Quick checks:

sudo systemctl is-active patchmon-agent    # should print "active"
sudo journalctl -u patchmon-agent -n 50 --no-pager

Common causes:

• Service stopped or crashed: sudo systemctl restart patchmon-agent and watch the logs for the underlying error.
• Reverse proxy not forwarding WebSocket upgrade headers: see Server Troubleshooting: Agent Can't Connect Over WebSocket.
• NAT / load balancer timing out idle connections: raise the proxy's idle timeout to at least 65 s. The agent sends WebSocket pings every 30 s.
• Temporary network blip: the agent auto-reconnects with exponential backoff. Wait 60 s and re-check. The WS pill goes amber for the grace window, then red.

Full details: Managing the PatchMon Agent: Agent's WS Pill is Red in PatchMon.

Host Reporting Pill is Stale

The agent hasn't pushed an HTTP report within its update interval and the WebSocket is also disconnected. This is the strongest indicator that the host is genuinely unreachable (as opposed to just losing the real-time channel).

Quick checks:

# From the affected host (if you can reach it):
sudo systemctl status patchmon-agent
sudo patchmon-agent ping
sudo patchmon-agent report       # force an immediate report

# From another host:
ping <host-ip>
ssh <host>                       # confirm host is alive

Common causes:

• Host is genuinely down (powered off, kernel panic, hardware fault). Check console / hypervisor.
• Network partition between the host and the PatchMon server. Verify outbound HTTPS to the server URL still works.
• Agent service stopped without WebSocket disconnect notice (e.g. host was suspended). sudo systemctl restart patchmon-agent once it's reachable.

If the host is online but only the Reporting pill is red while WS is also red, run sudo patchmon-agent report to push a fresh report and the pill should flip back to green.

Agent Won't Start

The service exits immediately or never stays running.

Diagnose by running the agent in the foreground with debug logging:

sudo systemctl stop patchmon-agent
sudo patchmon-agent serve --log-level debug
# (Ctrl+C to stop; then: sudo systemctl start patchmon-agent)

Common causes:

• Broken config.yml: YAML syntax error. Validate with yq e . /etc/patchmon/config.yml or reinstall to regenerate defaults.
• Missing credentials.yml: see Credentials file missing or corrupt.
• Invalid patchmon_server URL: must start with http:// or https://, no trailing slash.
• Port conflict (rare): the agent itself doesn't listen on any port, but if you've enabled the SSH-proxy integration with a static local port, check with ss -lntp | grep patchmon.
• Wrong architecture binary: file /usr/local/bin/patchmon-agent and compare against uname -m. If they disagree, reinstall from the UI so the correct binary is fetched.

Full details: Managing the PatchMon Agent: Testing and Diagnostics and Debugging a Problem.

Cannot Reach Server: DNS

Log or ping output contains no such host, dial tcp: lookup <hostname>, or server misbehaving.

Diagnose:

nslookup patchmon.example.com
# or
dig +short patchmon.example.com

Fix:

• If DNS resolution fails, fix /etc/resolv.conf or your internal DNS records.
• Inside containers (Docker/LXC/Kubernetes): make sure the container's DNS resolver can reach your internal DNS.
• If you use a /etc/hosts override, check the entry matches the URL in /etc/patchmon/config.yml.

Full details: Managing the PatchMon Agent: "Connectivity Test Failed".

Cannot Reach Server: TLS

Log contains x509: certificate signed by unknown authority, tls: failed to verify certificate, or (on Windows) Could not establish trust relationship for the SSL/TLS secure channel.

Preferred fix: install the CA into the system trust store:

• Debian/Ubuntu: copy CA to /usr/local/share/ca-certificates/ (must have .crt extension) and run sudo update-ca-certificates.
• RHEL/Rocky/Fedora: copy CA to /etc/pki/ca-trust/source/anchors/ and run sudo update-ca-trust.
• Alpine: sudo apk add ca-certificates, then same as Debian.
• Windows: import via certlm.msc into Local Computer → Trusted Root Certification Authorities.

Quick bypass (lab only): set skip_ssl_verify: true in /etc/patchmon/config.yml and restart the service, or set env var PATCHMON_SKIP_SSL_VERIFY=true.

Do not enable skip_ssl_verify in production. It disables TLS verification entirely. See the warning in Managing the PatchMon Agent: SSL Certificate Errors.

Full details: Managing the PatchMon Agent: SSL Certificate Errors.

Cannot Reach Server: Firewall

ping reports "connectivity test failed" but DNS and TLS are fine. Typically i/o timeout or connection refused.

Diagnose:

# TCP reachability on 443
timeout 5 bash -c "</dev/tcp/patchmon.example.com/443" && echo open || echo blocked
# or:
nc -vz patchmon.example.com 443

Fix:

• Open outbound TCP/443 (or whatever port your reverse proxy uses) to the PatchMon server on the host's / network's egress firewall.

• If the host is behind an outbound HTTP proxy, set HTTPS_PROXY and HTTP_PROXY in the agent's environment. On systemd, add them as Environment= lines to a drop-in at /etc/systemd/system/patchmon-agent.service.d/proxy.conf:

  [Service]
  Environment="HTTPS_PROXY=http://proxy.corp:3128"
  Environment="HTTP_PROXY=http://proxy.corp:3128"
  Environment="NO_PROXY=localhost,127.0.0.1"
  

  then sudo systemctl daemon-reload && sudo systemctl restart patchmon-agent.

Full details: Managing the PatchMon Agent: "Connectivity Test Failed".

Report Interval Not Updating

You changed the report interval in Settings → Agent Updates, but the affected host still reports on the old schedule.

Background: The server pushes update_interval changes to connected agents via the WebSocket. If the WebSocket is down, or the change was made just before a restart, the agent may be running on its cached value in /etc/patchmon/config.yml.

Fix:

# 1. Verify the WebSocket is up (host shows as Online in the UI)
# 2. Restart the agent so it fetches fresh values on startup
sudo systemctl restart patchmon-agent

# 3. Check the value the agent sees
sudo patchmon-agent config show | grep -i interval

The agent syncs update_interval, docker_enabled, and compliance_enabled from the server on startup and on every WebSocket-pushed setting change, then writes the result back to config.yml. No restart is normally required for runtime sync. A restart is only needed for the first startup sync.

Full details: Managing the PatchMon Agent: Configuration Management (see "When do changes require a restart?" table).

Auto-Update Failing

The agent log contains update failed, hash mismatch, binary verification failed, or update was performed X ago, skipping.

Diagnose:

sudo patchmon-agent --version
sudo patchmon-agent check-version
ls -la /etc/patchmon/.last_update_timestamp
ls -la /usr/local/bin/patchmon-agent.backup.*

Common causes:

• Update loop cooldown: the agent refuses to re-update within 5 minutes of the last attempt. Wait 5 minutes.
• Server has not shipped a new binary: verify on the server that the file in AGENT_BINARIES_DIR (or AGENTS_DIR) matches the expected architecture.
• Hash mismatch: the server must send a SHA-256 hash of the binary. If it doesn't, the agent refuses to install (this is mandatory for security, not a bug). Update your server to a release that provides hashes.
• SSL error during download: treat as Cannot reach server: TLS. skip_ssl_verify is explicitly blocked for binary downloads in production, so fix the certificate trust instead.
• Auto-update disabled: patchmon-agent check-version will say Auto-update disabled by server administrator. Enable it at Settings → Agent Updates → Master auto-update and the per-host toggle on the host detail page.

Manual force:

sudo patchmon-agent update-agent

Full details: Managing the PatchMon Agent: Agent Updates and Agent Not Auto-Updating.

Credentials File Missing or Corrupt

Log contains credentials file not found, failed to load credentials, or API credentials are missing.

Diagnose:

ls -la /etc/patchmon/credentials.yml
# expected: -rw------- 1 root root  ~120  <date>  /etc/patchmon/credentials.yml

sudo cat /etc/patchmon/credentials.yml
# expected:
#   api_id: "patchmon_abc123"
#   api_key: "<64 hex chars>"

If the file is missing:

# Reconfigure using the credentials shown at enrolment time.
# If you don't have them, regenerate from the UI:
#   Hosts -> <host> -> Show Credentials -> Regenerate
sudo patchmon-agent config set-api <API_ID> <API_KEY> <SERVER_URL>

If the file is present but permissions are wrong:

sudo chmod 600 /etc/patchmon/credentials.yml
sudo chown root:root /etc/patchmon/credentials.yml

If the YAML is malformed (e.g. manually edited and broken), either restore the latest backup (ls -la /etc/patchmon/credentials.yml.backup.*) or re-run config set-api.

Full details: Managing the PatchMon Agent: "Credentials File Not Found" and config set-api.

Logs Not Being Written

The agent is running but /etc/patchmon/logs/patchmon-agent.log is empty or missing.

Diagnose:

# Does the directory exist and is it writable by root?
ls -la /etc/patchmon/logs/
# expected: drwx------ 2 root root ...

# Check disk space
df -h /etc/patchmon

Common causes:

• Log directory missing: sudo mkdir -p /etc/patchmon/logs && sudo chmod 700 /etc/patchmon/logs.
• log_file points elsewhere: run sudo patchmon-agent config show and check the path.
• Disk full: df -h. Free space or change log_file to a different partition.
• Running as the wrong user (non-standard install): the agent must run as root. See Permission denied.
• systemd-only logging: on systemd, the agent logs to both the journal and the file. If the file is empty but journalctl -u patchmon-agent has entries, the file may be rotating correctly. Check for .log.gz backups.

The agent uses built-in log rotation (10 MB per file, 5 backups, 14-day retention). You do not need logrotate.

Full details: Managing the PatchMon Agent: Viewing Logs.

Service Keeps Restarting

systemctl status patchmon-agent shows activating (auto-restart) repeatedly, or the process count climbs.

Diagnose:

sudo systemctl status patchmon-agent
sudo journalctl -u patchmon-agent --since "15 minutes ago" --no-pager

Common causes:

• Invalid config.yml (YAML syntax error): the agent refuses to start; systemd restarts it every 10 s.
• Invalid credentials: the agent logs "invalid API credentials" and exits.
• Server unreachable for long: the agent will keep trying (it does not exit on network errors in normal operation), so constant restarts point at one of the two above.
• Binary crash: check dmesg for OOM kills or segfaults: sudo dmesg -T | grep -i patchmon.

The systemd unit uses Restart=always with RestartSec=10, which is correct for production but masks crash loops. Disable auto-restart temporarily to see the real error:

sudo systemctl edit patchmon-agent --force --full
# change: Restart=always  ->  Restart=no
sudo systemctl daemon-reload
sudo systemctl start patchmon-agent
# read the single run's output, then revert:
sudo systemctl edit patchmon-agent --force --full    # restore Restart=always
sudo systemctl daemon-reload

Full details: Managing the PatchMon Agent: Service Keeps Restarting.

Permission Denied

Any agent command exits immediately with permission denied or this script must be run as root.

Fix:

• Linux / FreeBSD: use sudo. The agent reads package databases, writes to /etc/patchmon/, and manages the system service; none of which work without root.
• Windows: open PowerShell with Run as Administrator. A non-elevated PowerShell cannot read installed-package inventory or manage the PatchMonAgent service.

If you are root and still see permission errors, check file ownership:

ls -la /etc/patchmon/config.yml /etc/patchmon/credentials.yml /usr/local/bin/patchmon-agent
# All three should be owned by root.

Full details: Managing the PatchMon Agent: "Permission Denied" Errors.

Host ID Mismatch

The host page shows a Host ID mismatch warning, and the agent log contains config.yml names a different host than these credentials belong to.

The agent's config.yml holds a host_id for one host, but its credentials (credentials.yml) belong to another. Reports are still accepted and land on the host the credentials belong to; nothing is blocked. Common causes:

• config.yml was copied from another machine, or edited by hand.
• config set-api was run with another host's credentials by an agent older than 2.1.4.

Check which host the credentials belong to, and what the file says:

sudo patchmon-agent config show     # Host ID and API ID

Fix, once you are sure the credentials on this machine are the right ones:

sudo patchmon-agent config sync-host-id

This replaces host_id with the server's value. Alternatively, reinstall the agent with the install command from the correct host's page. The warning clears on the agent's next check-in. If the credentials are the wrong ones instead, run config set-api with this host's credentials.

The agent never fixes this on its own, because a mismatch nearly always means a copied or edited file that someone should look at.

Possible Cloned Machine

The host page warns that two machines appear to be reporting with this host's credentials. The server log contains possible cloned agent: two machines are reporting with this host's credentials.

This happens when a VM or container that already had the agent installed is cloned, or made into a template and deployed. Every copy carries the same credentials.yml and host_id, so they all report as one host and its data alternates between them. PatchMon notices when the machine ID and hostname reported for the host keep switching between two values within 24 hours.

Fix: give each copy its own host record.

1. On each clone (not the original), remove the copied agent identity and enrol it as a new host:
   • With an auto-enrolment script, rerun it with FORCE_NEW_HOST=true on the clone (for the Proxmox script, FORCE_NEW_HOST=<vmid>). This works even while the clone's agent is running: the script moves the copied config.yml, credentials.yml and their backups into /etc/patchmon/pre-force-new-<timestamp>/, then enrols and installs a new host. See the Auto-Enrollment chapter of the API and integrations guide.
   • Otherwise, add a new host in PatchMon, remove /etc/patchmon/config.yml and /etc/patchmon/credentials.yml on the clone, and run the new host's install command.
2. On the original host's page, dismiss the warning (needs can_manage_hosts). Alternatively, regenerate the original host's credentials and apply them to the original machine; that also clears the warning and locks out any copy you missed.

If the machines keep alternating after you dismiss the warning, it comes back.

Limits: detection relies on the machine ID and hostname. Two copies with the same hostname and the same machine ID cannot be told apart. Agents older than 2.1.4 only send what detection needs in full reports, not in every check-in, so the warning can take longer to appear for them.

Template hygiene: never turn a machine into a template, golden image or snapshot for cloning while /etc/patchmon is present. Remove the agent first (see Uninstalling the PatchMon Agent), or install the agent after cloning.

Config Version Out of Date

The host page shows the config version alongside the agent version. Agents from 2.1.4 onwards set config_version in config.yml to their own version every time they load the file, so the two normally match from the first report after an upgrade.

To see what the agent itself has, run sudo patchmon-agent config show and compare Agent Version with Config Version.

Escalation

If the decision tree above does not match your symptom, collect the following before asking for help:

# Everything in one command
sudo patchmon-agent diagnostics > /tmp/patchmon-diag.txt 2>&1
sudo journalctl -u patchmon-agent --since "1 hour ago" --no-pager \
  > /tmp/patchmon-journal.txt 2>&1
sudo tail -n 200 /etc/patchmon/logs/patchmon-agent.log \
  > /tmp/patchmon-log.txt 2>&1
sudo patchmon-agent config show > /tmp/patchmon-config.txt 2>&1
sudo patchmon-agent --version >> /tmp/patchmon-config.txt
uname -a >> /tmp/patchmon-config.txt

Redact the api_id before sharing the bundle publicly. The full api_id is sensitive even though the key hash is not shown by config show.

See Also

• Managing the PatchMon Agent: full CLI, service, log, and removal reference.
• Managing the PatchMon Agent: Common Troubleshooting: the detailed counterpart to this decision tree.
• Agent Configuration Reference (config.yml): every config parameter.
• Installing the PatchMon Agent: enrolment walkthrough.
• Uninstalling the PatchMon Agent: removal walkthrough.
• Server Troubleshooting: for server-side issues.