The stack

Proxmox VE is not a monolithic hypervisor. It's Debian with a custom kernel, the two virtualization engines Linux already has (KVM/QEMU for VMs, LXC for containers), a small replicated file system for configuration, and a handful of Perl daemons that expose all of it as one REST API. Once you can place each piece, almost every Proxmox behaviour, from "why is /etc/pve read-only?" to "why did that VM start on pve2?", becomes obvious.

The Proxmox VE stack: click any layer

bottom = hardware, top = what you touch

you

services

cluster fs

engines

metal

Proxmox Cluster File System

/etc/pve is not a normal directory: it's a FUSE file system served by pmxcfs. All content is held in RAM, persisted to the SQLite database /var/lib/pve-cluster/config.db, and every write is replicated to all cluster nodes through corosync's ordered group messaging, so each node applies writes in the same order. It is meant for small config files (tens of MB in total), not data.

Why: This is the cluster's single source of truth. A VM belongs to the node whose directory holds its config (nodes/pve1/qemu-server/100.conf), and migrating is literally renaming that file into another node's folder. If a node loses quorum, /etc/pve turns read-only, which stops two halves of a split cluster from both changing the same guest.

Files

/etc/pve/nodes/<node>/qemu-server/<vmid>.conf
/etc/pve/qemu-server → nodes/<this node>/qemu-server (symlink)
/etc/pve/storage.cfg, datacenter.cfg, user.cfg, jobs.cfg
/etc/pve/corosync.conf, ha/resources.cfg
/etc/pve/priv/ (authkey, token.cfg, shadow.cfg; root only)
/var/lib/pve-cluster/config.db (the SQLite backing store)

pmxcfs (pve-cluster.service)corosync

mount | grep /etc/pve
ls -l /etc/pve/qemu-server
pvecm status

one click, every layer

Follow a Start button through the stack

The web UI has no privileged shortcut. Clicking Start is an HTTPS call that crosses a privilege boundary, reads a config file from the cluster file system, asks a storage plugin for a block device and finally forks an ordinary Linux process. Step through it.

Trace: what happens when you click "Start" on VM 100

Press Play or Step to follow one click through every layer of the stack. Nothing here is UI-specific: qm start 100 joins at step 4.

# waiting…

the source of truth

Configs are small text files in /etc/pve

Every guest is described by one file. Because /etc/pve is replicated by pmxcfs, every node sees every file, but a guest only runs on the node whose directory contains it. Volume IDs like local-lvm:vm-100-disk-0 keep the file free of node-specific device paths.

a VM, as Proxmox stores it

# /etc/pve/qemu-server/100.conf
# (= /etc/pve/nodes/pve1/qemu-server/100.conf)
agent: 1
boot: order=scsi0;net0
cores: 2
cpu: x86-64-v2-AES
machine: q35
memory: 4096
name: web01
net0: virtio=BC:24:11:5E:20:01,bridge=vmbr0,firewall=1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,discard=on,iothread=1,size=32G
scsihw: virtio-scsi-single

what lives in /etc/pve

/etc/pve/
├── corosync.conf          # cluster membership (copied to /etc/corosync/)
├── datacenter.cfg         # cluster-wide defaults (keyboard, migration net, …)
├── storage.cfg            # every storage definition, visible to all nodes
├── user.cfg               # users, groups, pools, ACLs
├── jobs.cfg               # backup jobs, run by pvescheduler
├── firewall/cluster.fw
├── ha/resources.cfg
├── priv/                  # root-only: authkey.key, token.cfg, shadow.cfg
├── local -> nodes/pve1    # symlinks resolve to *this* node
├── qemu-server -> nodes/pve1/qemu-server
├── lxc -> nodes/pve1/lxc
└── nodes/
    ├── pve1/{qemu-server/100.conf, lxc/101.conf, pve-ssl.pem}
    └── pve2/{qemu-server/200.conf, …}

poking at the layers

# The same operation, three ways
qm start 100                                          # CLI wrapper
pvesh create /nodes/pve1/qemu/100/status/start        # the API, from a shell
curl -X POST -H "Authorization: PVEAPIToken=root@pam!ops=<secret>" \
  https://pve1:8006/api2/json/nodes/pve1/qemu/100/status/start

# See what qemu-server would execute, and the running process
qm showcmd 100 --pretty
ps -o pid,nlwp,rss,cmd -C kvm          # one process per VM, nlwp = threads
systemd-cgls /qemu.slice               # each VM in its own scope

Don't put data in /etc/pve

pmxcfs keeps everything in RAM and replicates every write to every node. It's built for config files (single files are capped at about 1 MiB), not ISOs, logs or scripts that write constantly.

Editing configs by hand is allowed

Editing /etc/pve/qemu-server/100.conf with an editor is supported and takes effect on the next start (or as a pending change). Sections in [brackets] below the main config are snapshots.

why it mattersA VM is a kvm process, a container is a process tree in namespaces. Standard Linux tools (ps, cgroups, ip, journalctl) work on both.
why it matters/etc/pve is the cluster. Ownership, migration and HA all come down to which node directory a config file sits in, and pmxcfs refuses writes without quorum to keep that consistent.
why it mattersEverything is the API. The UI, qm/pct and pvesh call the same handlers, so anything you can click you can script, with the same permission checks.