12

Ceph

Ceph turns the local disks of every node into one replicated, self-healing pool. Proxmox runs it hyperconverged: the same nodes run VMs and Ceph daemons, all managed with pveceph and the GUI. A VM disk becomes an RBD image, striped into 4 MiB objects, each hashed to a placement group that CRUSH maps to OSDs on different hosts. No central table says where anything is, so every client can compute it.
MON3 (or 5), odd

Keeps the authoritative cluster maps (OSD map, CRUSH map, auth) and agrees on them via Paxos. Needs a majority: 2 of 3 can vote, 1 of 3 cannot. Odd numbers because a 4th mon adds no extra failure tolerance.

MGR1 active + standby

Metrics, the dashboard, the PG autoscaler and balancer modules. Not in the data path; if the active one dies a standby takes over.

OSDone per disk

One daemon per disk (BlueStore, writing to the raw device). Stores objects, replicates to peers, heartbeats, and does recovery and backfill.

MDSonly for CephFS

Metadata server for the CephFS filesystem (directories, permissions). RBD block images for VMs do not use it.

size 3 · min_size 2 · failure domain host

Break the cluster

A small pool with 32 PGs and 3 copies of each, one copy per host. Click an OSD to kill it, or a host name to kill the whole node. Watch what Ceph does, and especially what it waits to do.

Hyperconverged Ceph simulator

HEALTH_OKt+00:00osd: 12 up, 12 in / 12mon quorum 3/3
monmgr
monmgr·sb
mon
pool vm-pool · 32 placement groupsclick one
32 active+clean
vm-100-disk-0 → 4 MiB objects → PG
ceph -w
  • 00:00 pool vm-pool: size 3, 32 PGs, all active+clean

Try it: fail one OSD and watch its PGs turn degraded. Nothing moves for 10 minutes (a reboot should not trigger a rebalance); then the OSD is marked out, CRUSH maps its PGs elsewhere and backfill copies the data (blue arrows). Fail a whole host, set noout for maintenance, or drop to 3 hosts and compare.

Why wait 10 minutes?
A down OSD is not necessarily a dead disk; it may be a reboot. mon_osd_down_out_interval (600 s) gives it time to return before Ceph starts copying terabytes. Until then PGs run degraded on the remaining copies.
Why min_size 2?
With fewer than min_size copies reachable, a PG stops serving I/O. That feels harsh, but the alternative (min_size 1) acknowledges writes that exist on exactly one disk, and a second failure loses them.
Why only some data moves
CRUSH is a deterministic function of the map. Marking one OSD out only remaps the PGs that used it; the rest of the cluster stays put. That is what lets Ceph scale without a lookup database.
client → primary → replicas → ack

The write path

Reads go to the primary OSD only. Writes are acknowledged only once every OSD in the acting set has persisted them, which is what makes Ceph strongly consistent and why its latency tracks the slowest replica and the network.

One 4 KiB write from VM 100

writereplicatereplicateackackackVM 100librbd on pve1osd.4primary · pve2osd.9replica · pve3osd.13replica · pve4CRUSH(pg 2.1f) → [4, 9, 13]━ public network━ cluster network (replication, recovery)
1. Client computes the placement itself

librbd in QEMU knows the object name (rbd_data.1a2b….0000000000000007 for bytes 28–32 MiB of the disk). It hashes it to PG 2.1f and runs CRUSH on its copy of the cluster map: [osd.4, osd.9, osd.13]. No metadata server, no lookup table, no single bottleneck.

Networks
The public_network carries client and monitor traffic; an optional cluster_network carries replication and recovery so a rebalance does not starve VM I/O. Use at least 10 GbE (25 GbE or more for NVMe OSDs), redundant links, and keep Ceph off the corosync network: recovery traffic can saturate a link and corosync needs low latency.
Placement groups and the autoscaler
PGs are the unit of placement and recovery: too few and data is lumpy and recovery is serial, too many and OSDs waste memory. The PG autoscaler (on by default) sizes pg_num per pool, aiming at roughly 100 PGs per OSD; give it a hint with target_size_ratio for pools that will grow.
replication vs erasure coding

Capacity: raw is not usable

Size 3 stores everything three times. Erasure coding with k data and m coding chunks costs (k+m)/k instead (1.5× for 4+2), at the price of more CPU, more network round trips for small writes, and needing at least k+m hosts. For VM disks, replicated size 3 is the usual choice; EC suits large, colder data. Then leave room to heal: after losing a host, its data must fit on the others.

How much can I actually store?

raw
64 TB
usable (theory)
21.3 TB
plan to fill ≤
13.6 TB
host losses: I/O / data
1 / 2
OSD fill today55%
after losing one host and re-healing on 373%

Dashed lines: nearfull 85% (HEALTH_WARN), backfillfull 90% (no more backfill into that OSD), full 95% (the cluster stops accepting writes). Ratios are per OSD, so one unbalanced OSD hits them first. Stored now: 11.7 TB.

pveceph

Setting it up

from empty nodes to an RBD storage
# on every node (PVE 9.2 defaults to Ceph Tentacle 20.2; --version squid for 19.2)
pveceph install --repository no-subscription

# once: write /etc/pve/ceph.conf (shared by all nodes through pmxcfs)
pveceph init --network 10.10.10.0/24 --cluster-network 10.10.20.0/24

# on three nodes: monitors (+ managers)
pveceph mon create
pveceph mgr create

# on every node, for every empty disk
pveceph osd create /dev/nvme0n1

# a replicated pool for VM disks, registered as PVE storage
pveceph pool create vm-pool --size 3 --min_size 2 --application rbd --add_storages

# optional: CephFS for ISOs / templates
pveceph mds create
pveceph fs create --name cephfs --add-storage

# optional: an erasure-coded pool (k data + m coding chunks)
pveceph pool create ec-pool --erasure-coding k=4,m=2 --add_storages
/etc/pve/ceph.conf
# /etc/pve/ceph.conf  (→ /etc/ceph/ceph.conf)
[global]
    fsid = 5d8f…
    public_network  = 10.10.10.0/24   # clients (QEMU/librbd), mons, OSD front side
    cluster_network = 10.10.20.0/24   # OSD ↔ OSD replication, recovery, backfill
    mon_host = 10.10.10.1 10.10.10.2 10.10.10.3
    osd_pool_default_size = 3
    osd_pool_default_min_size = 2
day-2 operations
ceph -s                         # health, mons in quorum, PG states
ceph osd tree                   # hosts → OSDs, up/down, in/out, weights
ceph osd df tree                # per-OSD fill: watch for the 85% nearfull line
ceph osd pool autoscale-status  # PG counts the autoscaler wants

# where does one object live?
ceph osd map vm-pool rbd_data.1a2b3c4d5e6f.0000000000000007
# osdmap e412 pool 'vm-pool' (2) object 'rbd_data…0007' -> pg 2.8d1f2a1f (2.1f)
#   -> up ([4,9,13], p4) acting ([4,9,13], p4)

# planned maintenance: don't rebalance while a node reboots
ceph osd set noout
reboot
ceph osd unset noout
  • why it mattersPlacement is computed, not looked up: object → hash → PG → CRUSH → OSDs. Any client with the cluster map knows where every object lives, and a map change moves only the PGs it affects.
  • why it matterssize 3 / min_size 2 with failure domain host means one node can die with no data loss and no downtime. Use at least 4 nodes if you want the cluster to heal itself back to 3 copies; with 3 it waits for the node to return.
  • why it mattersBudget usable capacity at about raw ÷ 3, then keep headroom for a host failure under the 85% nearfull line. Fast, redundant networking matters more than anything else for VM latency.