Bridges, VLANs & bonds

A VM's network card is a software device. Its frames travel through a tap device, maybe a firewall detour, a Linux bridge that works like a switch inside the kernel, a bond that spreads traffic over several cables, and only then reach real hardware. Proxmox VE 9 configures all of it in /etc/network/interfaces with ifupdown2. Each toggle below changes which devices a frame passes through and where the VLAN tag gets added.

Packet path: VM 100 → the wire

guest firewall (firewall=1)VLAN-aware vmbr0VM tag=20MTU 9000

uplink:

eth0 (inside VM 100)virtio-netdst|src MACIP… ≤1500
The guest sends a plain Ethernet frame. It never sees the VLAN tag: tag=20 is enforced by the host, so the guest cannot escape its VLAN.
tap100i0tap devicedst|src MACIP… ≤1500
fwbr100i0firewall bridgedst|src MACIP… ≤1500
fwln100i0 ⇄ fwpr100p0veth pairdst|src MACIP… ≤1500
vmbr0Linux bridge, VLAN-awaredst|src MAC802.1Q 20IP… ≤1500
↳ classified into VLAN 20 here; tag pushed on egress to the trunk
bond0802.3ad LACP (mode 4)dst|src MAC802.1Q 20IP… ≤1500
eno1 / eno2physical NICsdst|src MAC802.1Q 20IP… ≤1500
Top-of-rack switchLACP port-channeldst|src MAC802.1Q 20IP… ≤1500

The return path is the mirror image: the switch sends VID 20 tagged, the bridge (or bond0.20) strips the tag and only VM ports in VLAN 20 receive the frame.

Which link does each flow use?

eno1

flow 1flow 2flow 4

eno2

flow 3flow 5flow 6

flows

flow 1 192.168.10.50:51544 → 192.168.10.20:443 · hash mod 2 = 0
flow 2 192.168.10.50:51545 → 192.168.10.20:443 · hash mod 2 = 0
flow 3 192.168.10.51:40112 → 192.168.10.21:5432 · hash mod 2 = 1
flow 4 192.168.10.52:33790 → 10.20.0.7:6379 · hash mod 2 = 0
flow 5 192.168.10.50:58001 → 192.168.10.30:80 · hash mod 2 = 1
flow 6 192.168.10.53:49200 → 192.168.10.20:443 · hash mod 2 = 1

switchThe switch ports must be in one LACP port-channel (or MLAG across two switches). Each flow is hashed to ONE link: a single TCP stream never exceeds one NIC's speed.

/etc/network/interfaces (live)

auto lo
iface lo inet loopback
 
auto eno1
iface eno1 inet manual
 
auto eno2
iface eno2 inet manual
 
auto bond0
iface bond0 inet manual
    bond-slaves eno1 eno2
    bond-miimon 100
    bond-mode 802.3ad
    bond-xmit-hash-policy layer3+4
 
auto vmbr0
iface vmbr0 inet static
    address 192.168.10.11/24
    gateway 192.168.10.1
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
 
source /etc/network/interfaces.d/*

/etc/pve/qemu-server/100.conf

net0: virtio=BC:24:11:5E:7A:01,bridge=vmbr0,firewall=1,tag=20

Apply with ifreload -a (ifupdown2), which is what the GUI's "Apply Configuration" runs; no reboot needed.

one bridge, 4094 networks

Why a VLAN-aware bridge

The traditional way creates a bridge and a sub-interface for every VLAN a VM uses. With bridge-vlan-aware yes, one bridge carries every VLAN: each VM port gets a PVID (its access VLAN) and the uplink is a trunk. Changing a VM's tag becomes one field in its config. Because the host adds and removes the tag, a guest cannot send frames into other VLANs, unless you give it a trunk on purpose (trunks=20;30 on the NIC).

dedicated-server hosters

Single public IP? Routed or NAT

At home or in your own rack, bridging is the default and the best choice. A rented server is different: the hoster's switch usually accepts only your server's MAC address. Pick a mode to see why the default layout breaks there and what the two common fixes look like.

One public server, several VMs

frame from VM 203.0.113.18 as the hoster sees it:

MAC of eno1 (host) · src 203.0.113.18 (VM)

Works: the switch only ever sees the host's MAC

The host is a router. VMs get public IPs from an extra subnet routed to the host's main IP; vmbr0 has no physical port. With proxy_arp the host answers ARP for the VM IPs, and ip_forward moves packets between eno1 and vmbr0.

/etc/network/interfaces

auto eno1
iface eno1 inet static
    address 198.51.100.5/29
    gateway 198.51.100.1
    post-up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp

# extra subnet routed to 198.51.100.5 by the hoster
auto vmbr0
iface vmbr0 inet static
    address 203.0.113.17/28
    bridge-ports none
    bridge-stp off
    bridge-fd 0

when Linux bridges aren't enough

Open vSwitch: the alternative

Proxmox also supports Open vSwitch (OVS) bridges in the same file. Linux bridges with VLAN awareness now cover what most clusters need and are the default; OVS fits when you need OpenFlow, OVS-native LACP (balance-tcp) or a team that already runs OVS. Note that Proxmox SDN and the nftables firewall are built around Linux bridges.

	Linux bridge	Open vSwitch
Package	in the kernel, nothing to install	openvswitch-switch daemon
VLANs	bridge-vlan-aware yes	native, tag= per port
Bonding	kernel bonding driver	OVSBond (balance-slb, balance-tcp)
Guest firewall	direct (nftables) or fwbr (iptables)	always via fwbr
Debugging	ip, bridge, tcpdump	ovs-vsctl, ovs-appctl, ovs-ofctl

/etc/network/interfaces (OVS variant)

# apt install openvswitch-switch   (ifupdown2 understands ovs_* options)
auto bond0
iface bond0 inet manual
    ovs_bridge vmbr0
    ovs_type OVSBond
    ovs_bonds eno1 eno2
    ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast

auto vmbr0
iface vmbr0 inet manual
    ovs_type OVSBridge
    ovs_ports bond0 mgmt

# host IP lives on an internal port, tagged VLAN 10
auto mgmt
iface mgmt inet static
    ovs_type OVSIntPort
    ovs_bridge vmbr0
    ovs_options tag=10
    address 192.168.10.11/24
    gateway 192.168.10.1

management vs corosync vs storage vs migration

Separate the traffic that must never wait

One cluster carries several kinds of traffic with very different needs. Corosync needs almost no bandwidth, but it needs low, steady latency: Proxmox asks for under 5 ms between nodes. Ceph recovery and live migration are the opposite: they use all the bandwidth they get. If they share a link, the bulk traffic decides when corosync's packets leave.

Who shares the wire with corosync?

Ceph is recovering after an OSD failurea 64 GiB VM is live-migrating

bond0 / vmbr0 — 10 Gbit/s3.8 Gbit/s offered

corosync latency

0.26 ms

token / quorum

stable

HA nodes

healthy

Fine while idle. Now turn on a recovery and a migration at the same time.

where each network is chosen

# /etc/pve/corosync.conf (excerpt): two independent links
nodelist {
  node {
    name: pve1
    nodeid: 1
    ring0_addr: 10.10.1.11      # link0: dedicated NIC
    ring1_addr: 192.168.10.11   # link1: fallback over mgmt
  }
}

# /etc/pve/datacenter.cfg: send migrations over their own network
migration: secure,network=10.10.3.0/24

# /etc/pve/ceph.conf (excerpt)
[global]
    public_network  = 10.10.2.0/24
    cluster_network = 10.10.2.0/24

MTU 9000 (jumbo frames)

Larger frames mean fewer packets and fewer interrupts per gigabyte, which helps Ceph and migration traffic. The MTU must match on every hop: NICs, bond, bridge, VLAN interfaces and every switch port. If one hop stays at 1500, small packets pass (ping works, SSH works) and large ones are dropped, so the failure looks random. Corosync gains nothing from jumbo frames; keep its network simple.

test the path

# verify jumbo frames end to end: 9000 - 20 (IP) - 8 (ICMP) = 8972
ping -M do -s 8972 10.10.2.12
# "message too long" or silence = something in the path is still at 1500

Bonds carrying corosync

Proxmox only supports active-backup for a bond that carries corosync. Better still: no bond at all, and two corosync links on two separate NICs, because corosync (kronosnet) switches links by itself.

why it mattersA VM's frame goes eth0 → tap → (fwbr when the iptables firewall is on) → bridge → bond → NIC. When something breaks, check each of these devices in order with ip link and tcpdump.
why it mattersUse VLAN-aware bridges: the host adds and removes 802.1Q tags at the bridge port, guests can't leave their VLAN, and one trunk serves every VLAN. LACP spreads separate flows over the links; a single flow still uses only one link.
why it mattersGive corosync its own quiet link (two, ideally) and move Ceph and migration to fast, separate networks with matching MTU. Shared links turn a storage recovery into HA fencing.