05

High availability

HA in Proxmox is not magic replication: it is a promise that if a node dies, its HA guests are restarted on another node, from the same shared (or replicated) disk. The hard part is not starting them; it is being sure the old copy is gone first. Two daemons do the work: pve-ha-crm (one active cluster-wide manager) and pve-ha-lrm (one per node, holding a lock and feeding a watchdog).

HA failover simulator

failure
scheduler
pve1CRM master
online
32c / 128G · click ▶ to fail
  • vm:100 db01
    started
  • vm:101 web01
    started
  • vm:200 scratchno HA
    started
pve2
online
16c / 64G · lrm: active
  • vm:102 web02
    started
  • ct:103 dns
    started
pve3
online
32c / 128G · lrm: active
  • vm:104 app
    started
pve4
online
32c / 128G · lrm: active
  • vm:105 ci
    started
watchdog 60slock expiry
0 st = 0 s210 s
  1. Click a node to choose the victim, pick a failure type, then press ▶.
the reasoning

Fencing: why recovery waits two minutes

From the rest of the cluster, a powered-off node and a node with a dead network cable look identical: silent. If the silent node is actually alive and still writing to a Ceph RBD or an iSCSI LUN, starting the same VM elsewhere means two kernels writing one filesystem: guaranteed corruption. Proxmox solves this with self-fencing: each LRM holds a lock in /etc/pve and feeds a watchdog through watchdog-mux. A node that loses quorum can't renew its lock, stops feeding the watchdog, and is hard-reset after 60 s. The CRM waits until that lock has expired (longer than the watchdog timeout), so by the time it can take the lock, the node has already reset itself. No out-of-band power switch needed.
Watchdog
By default the kernel's softdog module. For more certainty use a hardware watchdog (IPMI, iTCO) via WATCHDOG_MODULE=ipmi_watchdog in /etc/default/pve-ha-manager. Only nodes with active HA services arm it.
Requirements
At least 3 nodes (or 2 + QDevice) for quorum, guest disks on shared storage (Ceph, NFS, iSCSI) or ZFS replication (then you may lose the changes since the last sync), and a reliable, redundant corosync network: a flapping link looks like a dead node and gets nodes fenced.
PVE 9: groups became rules
HA groups were replaced by HA rules. Node affinity rules say which nodes a resource may or should run on, with priorities (and strict). Resource affinity rules keep resources together (positive) or apart (negative). Existing groups are migrated automatically after the upgrade.
rules + CRS

Where recovered services go

For each service in recovery the CRM first filters nodes by rules: the highest-priority online nodes of its node affinity rule (strict rules never fall back to other nodes), minus nodes that violate a negative resource affinity. Among the remaining candidates the Cluster Resource Scheduler decides: basic picks the node with the fewest HA services, static weighs the configured CPU and memory of the services against each node's capacity. Switch the simulator between both and fail pve2 to see a different choice for the small services.
Meaning
startedCRM wants it running; LRM starts it if needed
fenceits node is gone; wait until the node is provably fenced
recoverynode fenced; looking for (or waiting for) a new node
migrate / relocatemoving to another node, online (VM) or stop-start (CT, relocate)
freezedo not touch: node is rebooting or updating packages
errortoo many failed start/relocate attempts; needs an admin (set to disabled, fix, re-enable)
shutdown policy

Planned maintenance is a different path

Planned shutdown: what happens to HA guests?

datacenter.cfg → ha: shutdown_policy=
  1. 1reboot pve1
    pve1
    vm:100 started
    vm:101 started
    pve2/3
    —

    You run "reboot" on pve1 (or click it in the GUI).

  2. 2LRM asks CRM to freeze
    pve1
    vm:100 freeze
    vm:101 freeze
    pve2/3
    —

    Services are stopped and put into 'freeze': the CRM will not recover or move them.

  3. 3pve1 reboots (≈2 min)
    pve1
    vm:100 freeze
    vm:101 freeze
    pve2/3
    —

    Downtime = guest shutdown + node reboot + guest boot.

  4. 4pve1 back
    pve1
    vm:100 started
    vm:101 started
    pve2/3
    —

    Unfrozen and started again on the same node.

conditional (the default) behaves like freeze on reboot and like failover on shutdown: a reboot is short, so waiting is cheaper than moving; a poweroff may last hours.

cli

Configuration

resources and rules
# make guests HA-managed (stored in /etc/pve/ha/resources.cfg)
ha-manager add vm:100 --state started --max_restart 1 --max_relocate 1
ha-manager add ct:103 --state started

# PVE 9: placement is expressed as HA rules (/etc/pve/ha/rules.cfg)
ha-manager rules add node-affinity db-prefers-pve1 \
    --resources vm:100 --nodes pve1:2,pve2:1 --strict 0
ha-manager rules add node-affinity app-only-pve3-4 \
    --resources vm:104 --nodes pve3,pve4 --strict 1
ha-manager rules add resource-affinity web-apart \
    --resources vm:101,vm:102 --affinity negative

ha-manager status            # quorum, master, per-node LRM state, per-service state
ha-manager crm-command node-maintenance enable pve2   # drain a node
rules.cfg
# /etc/pve/ha/rules.cfg
node-affinity: db-prefers-pve1
	resources vm:100
	nodes pve1:2,pve2:1
	strict 0

node-affinity: app-only-pve3-4
	resources vm:104
	nodes pve3,pve4
	strict 1

resource-affinity: web-apart
	resources vm:101,vm:102
	affinity negative
cluster-wide HA settings
# /etc/pve/datacenter.cfg
ha: shutdown_policy=conditional
crs: ha=static,ha-rebalance-on-start=1
ha-manager status
$ ha-manager status
quorum OK
master pve1 (active, Sat Oct  3 14:02:11 2026)
lrm pve1 (active, Sat Oct  3 14:02:12 2026)
lrm pve2 (active, Sat Oct  3 14:02:15 2026)
lrm pve3 (idle, Sat Oct  3 14:02:13 2026)
service ct:103 (pve2, started)
service vm:100 (pve1, started)
service vm:101 (pve1, started)
  • why it mattersHA restarts guests; it does not keep a second copy running. Expect a couple of minutes of downtime and a crash-consistent disk, like after a power cut.
  • why it mattersFencing is the whole point. The watchdog guarantees an isolated node kills itself before anyone else starts its guests, so shared disks never see two writers.
  • why it mattersEverything rests on quorum. A bad corosync network turns HA from a safety net into a source of surprise reboots.