Server Troubleshooting

Overview

This page covers diagnosing and fixing PatchMon 2.0 server problems on a Docker Compose deployment. It covers container startup failures, database and Redis issues, CORS, WebSocket, reverse-proxy, and admin recovery scenarios.

If the problem is on the agent side (won't start, can't reach the server, credentials file missing), jump to Agent Troubleshooting instead.

Reference Architecture

A stock docker-compose.yml deployment runs four containers on the patchmon-internal bridge network:

The server container embeds the Go HTTP server, the frontend, the queue worker, and the migration runner. No separate migration job is needed. In front of it you typically run Nginx / Traefik / Caddy / Cloudflare that terminates TLS and forwards to server:3000, or to whichever port you set PORT to.

Gathering Diagnostic Info

Before attempting any fix, capture the current state of the stack. These commands are safe and non-destructive.

If the server logs look empty, check ENABLE_LOGGING. It defaults to true from 2.0.3, but an explicit false silences the server completely, and false was the default on 2.0.2 and earlier. With it off, docker compose logs server shows container lifecycle output and nothing from PatchMon itself, so do not read anything into the silence.

# In the directory where your docker-compose.yml lives

# 1. Show container state (Up / Restarting / Exit)
docker compose ps

# 2. Server logs (last 200 lines)
docker compose logs --tail 200 server

# 3. Everything (server + database + redis + guacd)
docker compose logs --tail 200 --timestamps

# 4. Follow logs live
docker compose logs -f server

# 5. Health check (from the Docker host)
curl http://localhost:3000/health
# expected: 200 OK, body "healthy"

# 6. Health check as JSON
curl -H 'Accept: application/json' http://localhost:3000/health
# expected: {"status":"healthy","database":"healthy","redis":"healthy"}

# 7. Resource usage
docker stats --no-stream

The health endpoint at /health is public and unauthenticated and reports the status of both the Postgres connection and the Redis connection. A 503 Service Unavailable from /health means at least one dependency is down.

Check the values of your .env without leaking secrets:

# Show keys only (no values) -- safe to share in a bug report
grep -v '^#' .env | grep '=' | cut -d= -f1 | sort

Check what the server sees at runtime (Settings UI → Server tab):

• Log into PatchMon as a superadmin.
• Navigate to Settings → Server.
• The "Effective value" column shows the resolved value (env → database → default) and "Source" tells you which layer won.
• "Conflict" flags any setting defined in both .env and the database. The env always wins, but it is worth resolving.

1. Container Won't Start / Crashes on Boot

Symptoms

• docker compose ps shows server in state Restarting or Exited (1).
• docker compose logs server shows an error immediately and the container loops.

Diagnose

docker compose logs --tail 100 server

Look at the first 30 lines. The crash cause is almost always logged right there.

Common Causes and Fixes

Escape Hatch: Start a Shell in the Server Image

If the container crashes too fast to inspect, run it with an override command:

docker compose run --rm --entrypoint /bin/sh server

From the resulting shell you can env | grep -E 'DATABASE_URL|REDIS|JWT' and test connectivity with nc -zv database 5432 and redis-cli -h redis -a "$REDIS_PASSWORD" ping.

2. Database Migration Errors at Boot

Symptoms

[fatal] migrations failed: Dirty database version N. Fix and force version.

or

[fatal] migrations failed: migration file XXXX is corrupted

Background

PatchMon uses golang-migrate with embedded SQL files. On every server start, the server runs pending migrations before opening its HTTP listener. A dirty state means a previous migration started and crashed mid-way. The schema_migrations table records the version but the dirty column is true.

From v2.1.1 onwards the server prints the recovery SQL for you, naming the database and the exact statement to run, so in most cases you can follow that block instead of working it out by hand:

[migrate] Migration 42 did not complete on database "patchmon_db", so it is marked dirty and
[migrate] no further migrations will run against it until that marker is cleared.
...
[migrate]   UPDATE schema_migrations SET version = 41, dirty = false;

One exception: if the dirty version is 1, there is no earlier version to rewind to. Run DELETE FROM schema_migrations; instead, which clears the migration marker only and touches no application data, then let the server re-run migrations from the beginning. The server prints this variant automatically.

Fix: Stuck on Dirty

When the server logs report Dirty database version N. Fix and force version., the simplest path is to connect directly to Postgres, confirm whether the migration's actual work landed, and either mark the version clean or rewind one step so the migration re-runs. PatchMon's migrations are written to be idempotent, so re-running a clean migration is safe.

The example below uses the v2.0.2 dirty-30 case (migration 000030_v1-5-0_compliance_scan_dedup, which adds the partial unique index idx_compliance_scans_host_profile_completed). Substitute the version number from your own log line.

1. Connect to the database

Community script (Proxmox LXC, bare-metal Postgres):

sudo -u postgres psql -d patchmon_db

Docker:

docker compose exec database psql -U patchmon_user -d patchmon_db

(Use whatever POSTGRES_USER / POSTGRES_DB you have set in .env. The defaults are patchmon_user / patchmon_db. Note the compose service is named database, not postgres.)

2. Check what actually migrated

-- Current migration state. Should show version=30, dirty=t
SELECT * FROM schema_migrations;

-- Did migration 30 finish creating its index?
SELECT indexname FROM pg_indexes
WHERE indexname = 'idx_compliance_scans_host_profile_completed';

3. Pick one of these

A. Index exists. Migration 30's work is already done. This is the most common case, and the failure was usually a connection blip after the DDL had already committed. Mark the row clean and let migrations continue from 31:

UPDATE schema_migrations SET dirty = false WHERE version = 30;

B. Index does NOT exist. Migration 30 died before the CREATE INDEX ran. Roll the marker back to 29 and let PatchMon re-run 30 cleanly on the next boot:

UPDATE schema_migrations SET dirty = false, version = 29;

4. Restart PatchMon

After updating schema_migrations, exit psql and restart:

• Community script (LXC): reboot the container, or sudo systemctl restart patchmon-server and tail the log with sudo journalctl -u patchmon-server -f.
• Docker: docker compose down && docker compose up -d, then docker compose logs -f server.

You should see migrations advance through 31, 32, 33, then server starting.

Fix: Stuck on Dirty 42 After Upgrading to v2.1.0

v2.1.0 shipped a migration (000042) that failed on a small number of installs with:

cannot set path in scalar (22023)

This happens when the host_down row in alert_config stores its metadata as the JSON value null rather than an empty object, which the migration did not allow for. The migration rolls back cleanly when it fails, so nothing is half-applied, but the database is left dirty at 42 and the server crash-loops.

Upgrade to v2.1.1 or later first. The migration is fixed there and handles that value correctly. Rewinding on v2.1.0 will just fail the same way on the next boot.

Once you are on the fixed image, connect to the database as shown above and rewind one step:

UPDATE schema_migrations SET version = 41, dirty = false;

Then restart. Migration 42 re-runs and succeeds.

Only mark a migration clean after you've verified the schema is actually consistent. Forcing onto an inconsistent schema hides the problem until the next migration.

Fix: Run Migrations Manually

The server runs migrations automatically at startup, so you normally never need to run them by hand.

The Docker image does not ship a separate migration tool. Migrations are embedded in the server binary and there is no migrate command inside the container, so drive them by restarting the server and reading its logs, and inspect or adjust state with SQL as shown above:

-- Show current version
SELECT * FROM schema_migrations;

If you are building from source, make build-migrate in server-source-code/ produces a standalone migrate CLI supporting up, down, force VERSION, and version. It needs DATABASE_URL set and is not part of any released image.

3. "CORS Error" in the Browser

Symptoms

• Browser DevTools network panel shows OPTIONS /api/v1/... returning 403/404 with No 'Access-Control-Allow-Origin' header.
• Login or wizard requests fail silently.

Cause

CORS_ORIGIN does not match the URL the user's browser is hitting. CORS_ORIGIN accepts a single origin or a comma-separated list of origins (no spaces between entries), and each entry must be the exact origin the browser uses (protocol + host + port, no path, no trailing slash). Common mismatches:

• .env has CORS_ORIGIN=http://localhost:3000 but users access PatchMon at https://patchmon.example.com.
• CORS_ORIGIN=https://patchmon.example.com but users access via https://patchmon.example.com:8443.
• Trailing slash in CORS_ORIGIN (https://patchmon.example.com/). Strip the slash.
• PatchMon is reached from more than one URL (e.g. an external domain and an internal LAN address) but only one is listed.

Fix

Set CORS_ORIGIN to the exact origin the browser uses (protocol + host + port, no path, no trailing slash):

# .env
CORS_ORIGIN=https://patchmon.example.com

For multiple allowed origins (e.g. staging and production share a database during migration, or users reach PatchMon on both an external and internal URL), comma-separate them with no spaces between entries:

CORS_ORIGIN=https://patchmon.example.com,https://staging.patchmon.example.com

Then restart the server:

docker compose restart server

Requires a server restart. The Settings UI flags CORS_ORIGIN as "Requires a server restart to take effect".

Alternative: Settings UI

You can also set CORS_ORIGIN in Settings → Server → CORS_ORIGIN via the UI. If both the env var and the UI value are set, the env wins and the Settings UI shows a "Conflict" flag. Pick one source of truth.

4. Agent Can't Connect Over WebSocket

Symptoms

• Agents check in via HTTP reports (the Reporting pill is green) but the WS pill is red in the Hosts list.
• Agent log shows repeated websocket: bad handshake or reconnection loops.
• The "Waiting for Connection" screen after enrolment gets past Waiting to Connected slowly or never.

Cause

The agent opens a WebSocket at GET /api/v1/agents/ws with Upgrade: websocket / Connection: Upgrade. If your reverse proxy is not forwarding those headers, the upgrade handshake fails and the connection falls back to HTTP, which the agent then drops.

Fix: Nginx

location / {
    proxy_pass http://127.0.0.1:3000;
    proxy_http_version 1.1;

    # Required for WebSocket upgrade
    proxy_set_header Upgrade    $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    # Don't time out long-lived connections (WS, SSE, patching streams)
    proxy_read_timeout 86400s;
    proxy_send_timeout 86400s;
}

# Top of the config:
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

Fix: Traefik

Traefik forwards the required headers by default. You normally do not need extra config. If you have a custom headers middleware, make sure it does not strip Upgrade / Connection.

Fix: Caddy

patchmon.example.com {
    reverse_proxy 127.0.0.1:3000
    # Caddy auto-handles WebSocket upgrades -- nothing extra needed.
}

Fix: Cloudflare

Cloudflare's free tier supports WebSocket but check the dashboard: Network → WebSockets → On.

Verify the Fix

From the server host (not the agent host), this should return 101 Switching Protocols:

curl -i -N \
  -H "Connection: Upgrade" \
  -H "Upgrade: websocket" \
  -H "Sec-WebSocket-Version: 13" \
  -H "Sec-WebSocket-Key: $(openssl rand -base64 16)" \
  https://patchmon.example.com/api/v1/agents/ws?apiId=dummy

If you see 101, the proxy is forwarding the handshake. A 4xx back from /api/v1/agents/ws is fine here. The agent uses real credentials; the point is that the proxy did not interfere with the upgrade attempt.

X-Forwarded-Proto Matters

Set X-Forwarded-Proto $scheme (Nginx) or equivalent in other proxies. PatchMon uses the TRUST_PROXY=true env setting to read this header when present. Without it, the server may think it is serving over HTTP and emit Secure cookies that the browser then refuses to send.

5. 502 Bad Gateway from the Reverse Proxy

Symptoms

• Browser shows 502 Bad Gateway or 503 Service Unavailable.
• Nginx log shows upstream prematurely closed connection or connect() failed (111: Connection refused).

Diagnose

# 1. Is the server container up?
docker compose ps server

# 2. Is it healthy?
curl http://localhost:3000/health

# 3. Can the reverse proxy reach the container?
#    From the reverse-proxy host:
curl -v http://<server-host>:3000/health

Common Fixes

• Container not running: docker compose up -d server.
• Health check failing: follow 1. Container Won't Start above.
• Wrong upstream port: the default is 3000. If you changed PORT in .env, update the reverse-proxy config to match.
• Firewall blocking 3000: if the reverse proxy is on a different host, open TCP/3000 between them, or bind the container on a UNIX socket / private interface.
• Healthcheck grace too short: on slow storage, Postgres may take 20-60 s to become healthy on first boot. Increase start_period on the database and redis healthcheck, or simply wait.

6. Redis Connection Refused / NOAUTH

Symptoms

redis: dial tcp redis:6379: connect: connection refused

or

redis: NOAUTH Authentication required.

Cause

• Connection refused: the redis container is not running or crashed.
• NOAUTH: the server is connecting with no password (or the wrong one) while Redis requires one.

Fix

The docker-compose.yml starts Redis with:

redis:
  command: redis-server --requirepass ${REDIS_PASSWORD}

Both the server and Redis read REDIS_PASSWORD from the same .env. If you change it, restart both:

docker compose up -d redis server

Verify from inside the network:

docker compose exec redis redis-cli -a "$REDIS_PASSWORD" ping
# expected: PONG

Check the server sees the right password:

docker compose exec server env | grep REDIS_

If you set REDIS_PASSWORD with special shell characters ($, !, space), Docker Compose may mis-quote them. Stick to alphanumerics + -_ for the password or wrap it in single quotes in .env.

7. Upload / Body-Limit Errors

Symptoms

• Agent reports fail with HTTP 413 Request Entity Too Large.
• Web UI pages showing very large tables (huge Docker inventory, thousands of packages) return 413 on save.

Cause

PatchMon caps JSON request bodies to protect against memory exhaustion. Two separate limits apply:

Both are integers in megabytes. A host with thousands of packages + Docker + compliance data can breach the 5 MB default; bump this env var on those installations.

Fix

Raise the limits in .env:

JSON_BODY_LIMIT=10
AGENT_UPDATE_BODY_LIMIT=8

Restart:

docker compose restart server

If you are fronting PatchMon with Nginx, also raise its limit, otherwise Nginx 413s before the server sees the body:

client_max_body_size 16m;

8. Slow Queries / Database Pressure

Symptoms

• API responses slow down as the number of hosts grows.
• Postgres container CPU usage sustained near 100%.
• Server logs show context deadline exceeded on database calls.

Diagnose

# Postgres activity
docker compose exec database psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" \
  -c "SELECT pid, state, wait_event, query_start, substr(query, 1, 120) FROM pg_stat_activity WHERE state != 'idle';"

# Check pool stats (from the server's perspective, via health+metrics if enabled)
curl -H 'Accept: application/json' http://localhost:3000/health

Common Fixes

Tune the pool and related timeouts in .env:

For the full env-var reference, see the Configuration section in the main admin docs.

If the problem persists, enable the request logger (ENABLE_LOGGING=true, LOG_LEVEL=debug) for a short window to catch which endpoint is hot.

9. Admin User Locked Out

Situation

You are the only superadmin and you:

• Forgot your password and disabled "Forgot password" flows, or
• Lost access to your TFA device and backup codes, or
• Enabled OIDC_DISABLE_LOCAL_AUTH=true and your IdP is now broken.

No Built-in CLI Reset

PatchMon does not ship a CLI subcommand for resetting admin passwords or clearing TFA. The server binary is a single daemon (patchmon-server) with no subcommands, and the separate migrate binary only supports up, down, force V, and version. The intentional design is that all user management goes through the web UI.

The workaround is direct database modification.

Workaround: Reset the Password via psql

Step 1: generate a bcrypt hash of the password you want to set. PatchMon stores passwords with bcrypt cost 10 (matching the legacy Node.js implementation). Any language works:

# With htpasswd (from apache2-utils / httpd-tools)
htpasswd -bnBC 10 "" 'new-password-here' | tr -d ':\n'
# prints something like: $2y$10$...

# Or with Python (bcrypt library)
python3 -c 'import bcrypt; print(bcrypt.hashpw(b"new-password-here", bcrypt.gensalt(10)).decode())'

Note: bcrypt produces hashes starting with $2a$10$, $2b$10$, or $2y$10$: all three are compatible.

Step 2: connect to Postgres and update the user row:

docker compose exec database psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"

-- See what accounts exist
SELECT id, username, email, role, is_active FROM users ORDER BY created_at;

-- Reset the password hash. Replace <bcrypt-hash> with the output from step 1.
-- Keep the single quotes and escape the hash's $ signs if your shell interprets them.
UPDATE users
SET password_hash = '<bcrypt-hash>', updated_at = NOW()
WHERE username = 'admin';

-- Confirm
SELECT username, role, is_active, updated_at FROM users WHERE username = 'admin';

\q

Step 3: log in with the new password and immediately rotate the credentials through the UI (Profile → Change password) to confirm the flow works, and re-enrol TFA if needed.

Workaround: Unlock a Locked-Out Account

After too many failed logins, accounts are locked for LOCKOUT_DURATION_MINUTES (default 15). To clear the lock immediately:

UPDATE users
SET failed_login_attempts = 0, locked_until = NULL
WHERE username = 'admin';

Workaround: Disable TFA on an Account

UPDATE users
SET tfa_enabled = false, tfa_secret = NULL, tfa_backup_codes = NULL
WHERE username = 'admin';

The user can re-enrol TFA after logging in.

Workaround: Re-enable Local Login When OIDC is Broken

If OIDC_DISABLE_LOCAL_AUTH=true is blocking you, edit .env:

OIDC_DISABLE_LOCAL_AUTH=false

then:

docker compose restart server

Log in with local credentials, fix the OIDC config, and flip it back.

Always take a database backup before running UPDATE statements. docker compose exec database pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB" > patchmon-backup-$(date +%F).sql.

10. Quick-Reference: When to Restart What

See Also

• Agent Troubleshooting: decision tree for agent-side issues.
• Managing the PatchMon Agent: CLI, service, logs, update, and removal.
• Installing the PatchMon Agent: enrolment walkthrough.
• Agent Configuration Reference (config.yml): every agent config parameter.