ZFS

ZFS is a volume manager and a filesystem in one: it pools disks into vdevs, checksums every block, never overwrites live data, and turns that into snapshots, clones, compression and replication for free. Proxmox ships it in the installer and uses it as the zfspool storage type. Getting the pool layout right is the one decision you cannot easily undo.

Pool builder · capacity, redundancy, IOPS

disks6

raw

24 TB

usable ≈

12 TB

efficiency

50%

survives

1 (≤3)

random write IOPS

3×

random read IOPS

6×

pool tank · state ONLINE

click a disk to fail it

mirror-0ONLINE

mirror-1ONLINE

mirror-2ONLINE

All vdevs healthy. The pool stripes data across every top-level vdev.

vdevs are the unit of performance

Why mirrors are the default advice for VM storage

A pool stripes across its top-level vdevs, and each RAIDZ vdev behaves roughly like a single disk for small random I/O, because every block is spread over all its members with parity. VM disks are mostly small random reads and writes. So eight disks as one RAIDZ2 give you about one disk of random IOPS, while the same eight disks as four mirrors give you about four for writes and up to eight for reads.

Mirrors (RAID10)

Best random IOPS, fastest and least stressful resilver (copy from the partner, no parity math), and you can grow the pool two disks at a time. Costs 50% of raw capacity.

RAIDZ

Great capacity for bulk data, backups and ISO stores. For VM zvols, small blocks pay padding: a 16K block on a 5-disk RAIDZ1 with 4K sectors takes 6 sectors, not 5, so the advertised efficiency is not what you get.

dRAID

Distributed RAID with spare capacity spread over every disk, so a rebuild writes to all disks in parallel. Built for big pools (dozens of disks), with a fixed stripe width that makes it a poor fit for small VM blocks.

create a pool and register it as storage

# striped mirrors (RAID10) from four NVMe drives, 4K sectors
zpool create -o ashift=12 tank \
  mirror /dev/disk/by-id/nvme-A /dev/disk/by-id/nvme-B \
  mirror /dev/disk/by-id/nvme-C /dev/disk/by-id/nvme-D

# make it a PVE storage (thin zvols, VMs + CTs)
pvesm add zfspool tank-vm --pool tank --content images,rootdir --sparse 1

# or let the GUI/API do both:  Node → Disks → ZFS → Create
pvesh create /nodes/pve1/disks/zfs --name tank --raidlevel raid10 \
  --devices /dev/nvme0n1,/dev/nvme1n1,/dev/nvme2n1,/dev/nvme3n1 \
  --ashift 12 --compression lz4 --add_storage 1

a degraded pool

$ zpool status tank
  pool: tank
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
config:
        NAME          STATE     READ WRITE CKSUM
        tank          DEGRADED     0     0     0
          mirror-0    DEGRADED     0     0     0
            nvme-A    ONLINE       0     0     0
            nvme-B    UNAVAIL      0     0     0
          mirror-1    ONLINE       0     0     0
            nvme-C    ONLINE       0     0     0
            nvme-D    ONLINE       0     0     0

$ zpool replace tank nvme-B /dev/disk/by-id/nvme-E   # resilver starts

block pointers, txgs, uberblock

Copy-on-write: why snapshots are instant and free (at first)

ZFS writes changes into free space and then updates the pointers above them, all the way to the uberblock, once per transaction group (txg). A snapshot simply keeps an old root alive. It costs nothing when taken; it grows as the live data diverges, because every overwritten block it still references cannot be freed.

Take a snapshot, then overwrite blocks

Overwrite a block:

live tree · txg 1

Writes never overwrite in place: the new block goes to free space, then every parent up to the uberblock is rewritten to point at it. Click a data block to overwrite it.

snapshot-only

shared

freed

allocated blocks on disk

A₁B₁C₁D₁L0₁L1₁uber₁

■ live only■ live + snapshot■ kept alive only by a snapshot

snapshots

None yet. A snapshot just keeps the current uberblock's tree alive.

what local-zfs actually creates

zvols for VMs, datasets for containers

	zvol (VM disk)	dataset / subvol (container)
name	`rpool/data/vm-100-disk-0`	`rpool/data/subvol-101-disk-0`
what it is	a virtual block device at /dev/zvol/…, the guest puts its own filesystem on it	a ZFS filesystem mounted on the host, the CT sees its files directly
block size	volblocksize, fixed at creation: 16K default (OpenZFS 2.2+). Larger = better compression and less RAIDZ padding, worse for small random writes	recordsize, a maximum (128K default): small files use small blocks
size limit	volsize (the virtual disk size)	refquota (the CT disk size)
thin?	only with sparse 1 on the storage (no refreservation)	always: quota, not a reservation
compression	lz4 inherited from the pool: cheap, usually a net speedup	same, and it compresses per file record

memory and devices

ARC, and the extra vdev classes

ZFS caches reads in RAM (the ARC) and can use extra devices for specific jobs. None of the extra devices replace redundancy, and two of them are often misunderstood.

Size the ARC

host RAM128 GiBpool size8 TiB

installer zfs_arc_max

13 GiB

rule of thumb (2 + 1/TiB)

10 GiB

left for guests ≈

111 GiB

ARC

VMs and containers

The ARC is ZFS's read cache in RAM. It does shrink under memory pressure, but not instantly, and RAM promised to VMs is not something you want to compete for. New installs since PVE 8.1 cap it at 10% of RAM, at most 16 GiB. That already covers the documented rule of thumb for this pool.

# /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=13743895347
# apply now without reboot:
echo 13743895347 > /sys/module/zfs/parameters/zfs_arc_max
# rpool as root: also run  update-initramfs -u -k all

SLOG (log vdev)

Holds the ZFS intent log for synchronous writes only (databases, NFS, VMs with sync semantics). It makes fsync fast; it does nothing for async writes and is not a write cache. Needs power-loss protection; mirror it.

L2ARC (cache vdev)

A second-level read cache on SSD. Its index lives in RAM, so it competes with the ARC. Add RAM first; L2ARC only helps when the working set is bigger than RAM but fits the SSD. Losing it is harmless.

special vdev

Stores metadata (and optionally small blocks) on fast SSDs in front of HDDs. It is part of the pool: lose it and you lose the pool, so give it the same redundancy as the data vdevs.

pvesr · asynchronous

Storage replication: ZFS's answer to shared storage

Local ZFS is not shared, so a node failure would strand its VMs. pvesr periodically sends incremental snapshots of a guest's disks to another node, so migration only transfers the latest delta and HA can restart the guest there. The price is the interval: anything written since the last successful run is lost on a crash. Let it run, then pull the plug.

Replicate VM 100 from pve1 to pve2

schedule

clock

02:00

last replicated

never

writes at risk

all

≈ data at risk

–

pve1

source · VM 100 running

rpool/data/vm-100-disk-0

changed since last snapshot: 0 MiB

zfs send -i | ssh | zfs recv

pve2

target · replica only

rpool/data/vm-100-disk-0

timeline

| sync ■ not yet replicated

Every run, pvesr takes a new snapshot of each of the guest's disks, sends only the blocks changed since the last common snapshot (zfs send -i) and then deletes the older snapshot on both sides. The first run is a full send. Migration and HA failover can then use the replica instead of copying the whole disk.

datasets, zvols, snapshots

$ zfs list -o name,used,refer,volsize,volblocksize,compressratio -r rpool/data
NAME                              USED  REFER  VOLSIZE  VOLBLOCK  RATIO
rpool/data                       61.2G    96K        -         -  1.52x
rpool/data/subvol-101-disk-0     1.31G  1.31G        -         -  2.03x
rpool/data/vm-100-disk-0         38.4G  31.0G      64G       16K  1.41x

# snapshots: PVE wraps them (qm snapshot also saves the VM config / RAM state)
qm snapshot 100 before-upgrade
zfs list -t snapshot -o name,used,refer rpool/data/vm-100-disk-0
# rpool/data/vm-100-disk-0@before-upgrade   7.4G  30.9G   ← USED = blocks only it still holds

# the raw ZFS equivalent
zfs snapshot rpool/data/vm-100-disk-0@manual
zfs rollback rpool/data/vm-100-disk-0@manual

pvesr — storage replication

# replicate VM 100 to pve2 every 15 minutes, max 50 MB/s
pvesr create-local-job 100-0 pve2 --schedule "*/15" --rate 50

pvesr list                     # jobs
pvesr status                   # last sync, duration, next run, fail count
pvesr schedule-now 100-0       # run it as soon as possible

# under the hood, per disk, per run:
#   zfs snapshot rpool/data/vm-100-disk-0@__replicate_100-0_<epoch>__
#   zfs send -i @<previous> @<new> | ssh pve2 zfs recv ...
#   then the previous replication snapshot is removed on both sides

why it mattersPool layout decides random IOPS: each vdev is roughly one disk's worth. For VM storage, use striped mirrors; use RAIDZ for capacity workloads like backups. Losing any top-level vdev loses the pool.
why it mattersCopy-on-write makes snapshots instant, but a snapshot pins every block overwritten after it. Old snapshots quietly eat space; watch the USED column and prune them.
why it mattersZFS replication gives local disks a failover story, but it is asynchronous: RPO equals the schedule. Zero-loss failover needs shared storage such as Ceph.