Cluster & quorum

A Proxmox cluster is a group of nodes that share one configuration: /etc/pve. Corosync carries every change to every node in a single agreed order, and votequorum decides which nodes are allowed to change anything. The rule is blunt: a group of nodes may write only if it holds more than half of all votes. Two halves can never both have a majority, so two halves can never both act.

Quorum simulator

nodes

click a node to

QDeviceredundant link1link0 switch dies

expected votes 4quorum = ⌊4/2⌋+1 = 3

pve1pve2pve3pve4

QUORATE

votes 4 = 4 ≥ quorum 3

✓ /etc/pve is writable: config changes, start/create guests
✓ HA manager may recover services from nodes outside this partition (after fencing)

pvecm status as seen from pve1 (hover a node)

Votequorum information
----------------------
Expected votes:   4
Highest expected: 4
Total votes:      4
Quorum:           3
Flags:            Quorate

Try it: split the ring, power nodes off, kill link0 with and without a redundant link1, or give one node 2 votes. A partition is quorate only with more than half of all expected votes, so at most one partition can ever be quorate, unless someone forces it.

the reasoning

Why “more than half” and not “whoever is alive”

A node can't distinguish “my peers crashed” from “the cable between us is cut”. In both cases it simply stops hearing them. If every group that still runs were allowed to act, a broken switch would turn one cluster into two, each starting the same VMs against the same shared disk. Majority is the cheapest rule that makes that impossible without any node needing to know what the others see. The price: with an even split, nobody gets a majority, and with two nodes, losing either one stops changes. That is what a QDevice is for.

QDevice: an outside vote, not a node

corosync-qnetd runs on a host outside the cluster; every node runs corosync-qdevice and talks to it over TCP (port 5403). On a split it hands its vote to one partition only. With an even node count Proxmox uses the ffsplit algorithm (1 vote, larger partition wins, tie goes to the lowest node id). It needs no low latency, so it can live in another building.

Corosync is latency-sensitive

Totem passes a token around all nodes; if it isn't seen within the token timeout, membership is recomputed. Keep corosync under about 5 ms (LAN latency, not WAN), on its own physical network. A saturated storage or backup link causes token loss, node “fencing storms” with HA, and phantom quorum loss. Give it a second link1 as a fallback; knet fails over between links on its own.

/etc/pve

pmxcfs: the cluster filesystem that is really a replicated database

/etc/pve is a FUSE mount provided by pmxcfs (the pve-cluster service). Its content lives in RAM and is persisted to SQLite at /var/lib/pve-cluster/config.db on every node. It is built for small config files: guest configs, storage.cfg, user.cfg, firewall rules, HA state, certificates. Because every write is a totem message, all nodes see the same files in the same order, and locks taken there (HA, migrations) are cluster-wide.

pmxcfs: one file, three copies, one order

cut pve3 off

pve1rw · quorate

/etc/pve/qemu-server/100.conf

name: web01
cores: 2
memory: 4096
net0: virtio,bridge=vmbr0

applied up to #0

pve2rw · quorate

/etc/pve/qemu-server/100.conf

name: web01
cores: 2
memory: 4096
net0: virtio,bridge=vmbr0

applied up to #0

pve3rw · quorate

/etc/pve/qemu-server/100.conf

name: web01
cores: 2
memory: 4096
net0: virtio,bridge=vmbr0

applied up to #0

Every write to /etc/pve becomes a corosync message. Totem stamps it with a global sequence number, and every node applies messages in exactly that order, even its own. Hit concurrent writes: pve1 and pve3 each set a different memory size at the same instant. pve3 sees its own packet first but must wait for the lower-numbered one, so all three apply both writes in the same order and converge to the same value. Cut pve3 off and its writes fail: a node without quorum cannot change the shared truth. When it rejoins, it resyncs.

cli

Building and inspecting a cluster

create, join, inspect

# on the first node
pvecm create prod --link0 10.10.10.1 --link1 192.168.1.11

# on every other node (it must not contain guests yet)
pvecm add 10.10.10.1 --link0 10.10.10.2 --link1 192.168.1.12

pvecm status        # votequorum view: expected votes, total, quorum, flags
pvecm nodes         # membership with node ids and votes
corosync-cfgtool -n # per-link knet status to every peer

add a QDevice (even-sized clusters)

# external host (any Debian box, VM outside the cluster, even a Raspberry Pi)
apt install corosync-qnetd

# on ALL cluster nodes
apt install corosync-qdevice

# on one cluster node: sets up TLS certs and adds the device to corosync.conf
pvecm qdevice setup 10.10.10.50

pvecm status   # now shows "Qdevice" in Flags and an extra vote

/etc/pve/corosync.conf

# /etc/pve/corosync.conf  (edit a copy, bump config_version, then move it in place)
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.10.10.1     # link0: dedicated corosync network
    ring1_addr: 192.168.1.11   # link1: fallback on another NIC/switch
  }
  node {
    name: pve2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.10.10.2
    ring1_addr: 192.168.1.12
  }
  # ... one block per node
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: prod
  config_version: 7
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
  ip_version: ipv4-6
  link_mode: passive          # knet: use the best working link, fail over on loss
  secauth: on
  version: 2
}

the dangerous override

# Last resort, on a node you KNOW is the only one alive:
pvecm expected 1
# votequorum now thinks 1 vote is a majority -> /etc/pve becomes writable.
# If the "dead" nodes are actually alive on the other side of a broken switch
# and someone does the same there, you have two clusters editing one config
# and possibly starting the same VMs on shared storage.

	Quorate	Not quorate
/etc/pve	read-write	read-only (writes: Permission denied)
start / create guests	yes	no
running guests	keep running	keep running; HA nodes self-fence
web UI / API	full	reads work, changes fail

why it mattersQuorum is a strict majority of expected votes. It guarantees that at most one part of a split cluster can change configuration or start guests.
why it mattersEven clusters, and two-node clusters most of all, need a tie-breaker. A QDevice adds one vote from outside without becoming a full node.
why it mattersCorosync needs low, stable latency and its own redundant links. A flapping corosync network is the most common root cause of “random” cluster and HA trouble.