Configuring SR-IOV Network Passthrough for High-Performance VMs in Proxmox VE

Introduction

Virtual machines running on Proxmox VE typically reach the network through a paravirtualized VirtIO NIC bridged to a Linux Bridge or Open vSwitch on the host. This model is flexible, migrates cleanly, and is fast enough for the vast majority of workloads. It is not, however, free. Every packet still crosses a software switch and a paravirtualized driver stack, which consumes host CPU cycles and adds latency that becomes visible under high packet-per-second workloads such as NFV appliances, financial trading systems, high-throughput databases, or dense multi-tenant environments.

SR-IOV (Single Root I/O Virtualization) lets a single physical NIC present multiple lightweight PCIe functions, called Virtual Functions (VFs), directly to the hypervisor. Each VF can be passed through to a guest with hostpci so the VM talks to the network card almost as if it owned the hardware, bypassing the host networking stack entirely. This article walks through enabling SR-IOV on Proxmox VE 8.x, from IOMMU setup to VF creation, VM assignment, and the operational trade-offs you need to understand before putting it into production.

Problem Overview

Administrators running latency-sensitive or high-throughput workloads on Proxmox VE frequently hit a ceiling with standard VirtIO networking. The bridge/tap path forces every frame through the host kernel's networking stack, consuming host CPU for switching, and adding scheduling jitter from the QEMU I/O thread. On a busy hypervisor with many VMs contending for the same bridge, this overhead compounds: aggregate throughput drops, CPU steal time rises inside guests, and tail latency becomes unpredictable.

The direct answer is passthrough. Passing an entire physical NIC to one VM works, but it sacrifices the card for every other guest and for the host's own management traffic. SR-IOV solves this by splitting one physical function (PF) into many virtual functions, each independently assignable, so a single 10/25/40GbE NIC can serve several VMs with near bare-metal performance while the PF stays available to the host or for further VF allocation.

Symptoms

You are likely dealing with a networking-overhead problem that SR-IOV can address if you observe any of the following:

• High %steal or software-interrupt (%si) time inside guest VMs during periods of heavy network I/O.
• Throughput on a 10GbE or faster link plateaus well below line rate despite idle-looking host CPUs at the aggregate level, while a single vCPU handling the VirtIO queue is pegged at 100%.
• Latency-sensitive applications (VoIP, market data feeds, industrial control, NFV data planes) show inconsistent jitter that correlates with host-side bridge or vhost-net activity.
• ethtool -S on the host bridge interface shows rising drop or overrun counters under load.
• Multiple VMs sharing one bridge start competing for CPU time on the host's networking threads, degrading performance for unrelated VMs on the same node.
• Existing full PCI(e) passthrough of the NIC works, but it locks a $2,000+ NIC to a single guest, which does not scale as VM count grows.

Root Cause

The root cause is architectural: standard virtualized networking on Proxmox VE (VirtIO plus a Linux Bridge or Open vSwitch) is a software-mediated path. Every frame is copied and processed by the host kernel, and every VM's network queue is serviced by a host thread (vhost-net or QEMU's own I/O thread). This gives you flexibility — live migration, firewall rules, VLAN tagging at the bridge, bonding — at the cost of CPU cycles and added latency per packet.

SR-IOV moves the data plane out of software and into silicon. The NIC's onboard switch (an embedded switch inside the network controller, sometimes called an "e-switch") handles the L2 forwarding between VFs and the uplink, and each VF exposes its own PCIe configuration space, MAC address, and set of TX/RX queues directly to the guest via passthrough. The host CPU is no longer in the data path at all once the VF is bound into the VM. This is the same principle as PCI(e) passthrough for GPUs, applied to network controllers, but with hardware multiplexing so one physical card can serve many tenants instead of just one.

The trade-off is that you lose most of the flexibility that made the software path convenient in the first place. A VF passed through with hostpci cannot be live-migrated with the standard QEMU migration path, host-side firewall rules and VLAN-aware bridge features no longer apply to that traffic, and the guest needs a compatible VF driver (e.g. ixgbevf, i40evf/iavf, mlx5_core) for the specific silicon in use.

Diagnosis

Before touching any configuration, confirm the hardware and platform actually support SR-IOV end to end. Three things must line up: CPU virtualization extensions (Intel VT-d or AMD-Vi), motherboard/BIOS support for IOMMU, and NIC firmware/driver support for SR-IOV.

Start by checking whether IOMMU is already active:

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi

If this returns nothing, IOMMU is not enabled yet (covered in the Step-by-Step Solution below). Once enabled, confirm the groups are populated:

find /sys/kernel/iommu_groups/ -maxdepth 1 -type d | sort -V

You can also query IOMMU grouping through Proxmox's own PCI API, which is useful when scripting checks across a cluster:

pvesh get /nodes/<nodename>/hardware/pci --pci-class-blacklist ""

Next, confirm the NIC itself advertises SR-IOV capability:

lspci -v -s 01:00.0 | grep -A 3 "Single Root I/O Virtualization"

A card that supports it will report a line such as Capabilities: [160] Single Root I/O Virtualization (SR-IOV) along with the maximum number of VFs it can create (Total VFs). Cards without this capability line cannot do SR-IOV regardless of driver or kernel configuration — this is a hardware/firmware limit, most commonly seen on cheaper consumer-grade NICs.

Finally, check that the kernel driver for the PF exposes the sriov_numvfs control file:

cat /sys/class/net/enp1s0f0/device/sriov_totalvfs
cat /sys/class/net/enp1s0f0/device/sriov_numvfs

If sriov_totalvfs reports a nonzero number, the driver and firmware are ready and you can proceed to enabling IOMMU and creating VFs.

Step-by-Step Solution

1. Enable virtualization extensions in firmware

Reboot the host and enable Intel VT-d (Intel platforms) or AMD-Vi/IOMMU (AMD platforms) in the BIOS/UEFI setup. On many boards this also requires enabling "SR-IOV support" as a separate toggle — check your motherboard manual, since the option is not always grouped with the VT-d/IOMMU setting.

2. Enable IOMMU in the kernel command line

The file you edit depends on the bootloader Proxmox VE is using. Check first with:

proxmox-boot-tool status

If the host boots via systemd-boot (the default for ZFS-on-root and most fresh installs), edit /etc/kernel/cmdline and add the IOMMU parameters on the single existing line, for example:

root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt

Then apply it:

proxmox-boot-tool refresh

If the host boots via GRUB, edit /etc/default/grub instead:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

Then apply it:

update-grub

Use amd_iommu=on in place of intel_iommu=on on AMD platforms. The iommu=pt ("passthrough" mode) parameter is recommended alongside it — it tells the kernel not to bother mapping devices that stay under host control through the IOMMU, which improves performance for those devices without affecting isolation for the ones you do pass through.

3. Load the VFIO modules

Add the following to /etc/modules so they load at boot:

vfio
vfio_iommu_type1
vfio_pci

Then reboot the host and re-run the dmesg | grep -e DMAR -e IOMMU check from the Diagnosis section to confirm IOMMU is now active.

4. Create virtual functions on the physical NIC

With IOMMU confirmed active and the NIC confirmed SR-IOV capable, create VFs on-the-fly to test:

echo 4 > /sys/class/net/enp1s0f0/device/sriov_numvfs

Verify they appeared, both as netdevs and as PCI functions:

ip link show enp1s0f0
lspci | grep -i "Virtual Function"

This sysfs write does not survive a reboot, so once you have confirmed the correct VF count for your workload, persist it as shown in the Configuration Examples section below.

5. Assign a VF to a VM

Each VF now shows up as its own PCI device address (for example 0000:01:10.0). Assign one to a VM with:

qm set 201 -hostpci0 0000:01:10.0,pcie=1

The pcie=1 flag presents the device on a PCIe (rather than legacy PCI) slot inside the guest, which is required for the q35 machine type and recommended for modern guest operating systems. Confirm the VM is using q35:

qm config 201 | grep machine

If no machine line is present, add one:

qm set 201 -machine q35

6. Use Resource Mapping for cluster consistency (recommended)

Rather than hardcoding a bus address that may differ from node to node, define a Resource Mapping under Datacenter → Resource Mappings → Add → PCI Device. This lets you reference the mapped VF by name in the VM configuration and lets Proxmox VE validate device availability and enforce access control for non-root users, which matters once VFs are handed out across a cluster rather than a single host.

7. Install the VF driver inside the guest

The guest OS needs a driver matching the VF's silicon, not the PF's marketing name. Intel X710/XL710 family VFs use iavf (formerly i40evf); Intel 82599/X520/X540 family VFs use ixgbevf; Mellanox ConnectX cards use mlx5_core with SR-IOV support built in. Most current Linux distributions ship these in-kernel; Windows guests need the vendor's VF driver package installed manually.

Commands

Purpose	Command
Check IOMMU is active	dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
List IOMMU groups	find /sys/kernel/iommu_groups/ -maxdepth 1 -type d
Check bootloader in use	proxmox-boot-tool status
Confirm NIC supports SR-IOV	lspci -v -s <bus:dev.fn> | grep -A3 "Single Root I/O Virtualization"
Check max VFs supported	cat /sys/class/net/<iface>/device/sriov_totalvfs
Create VFs (runtime)	echo <N> > /sys/class/net/<iface>/device/sriov_numvfs
List VF PCI functions	lspci | grep -i "Virtual Function"
Assign VF to VM	qm set <vmid> -hostpci0 <pci-addr>,pcie=1
Query PCI devices via API	pvesh get /nodes/<node>/hardware/pci
Inspect VM config	qm config <vmid>

Configuration Examples

Persisting VF count with sysfsutils

Install the helper package and define the VF count so it survives reboots:

apt update
apt install sysfsutils

Add a line to /etc/sysfs.conf (or a dedicated file under /etc/sysfs.d/):

devices/pci0000:00/0000:00:03.0/0000:01:00.0/sriov_numvfs = 4

Substitute the actual PCI path for your card, which you can confirm with:

readlink -f /sys/class/net/enp1s0f0/device

Persisting VF count via /etc/network/interfaces

An alternative that many admins find simpler is a pre-up hook on the physical function's interface stanza:

auto enp1s0f0
iface enp1s0f0 inet manual
    pre-up echo 4 > /sys/class/net/enp1s0f0/device/sriov_numvfs

Kernel command line for systemd-boot hosts

root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt

Kernel command line for GRUB hosts

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

VM configuration snippet (/etc/pve/qemu-server/201.conf)

machine: q35
hostpci0: 0000:01:10.0,pcie=1
net0: virtio=DE:AD:BE:EF:00:01,bridge=vmbr0

Note that net0 is left in place here only as an example of a VM with a mixed setup (management traffic on VirtIO, data-plane traffic on the SR-IOV VF). A pure SR-IOV data-plane VM would typically drop the bridge NIC entirely for that interface and rely solely on hostpci0.

Common Mistakes

• Forgetting the VF count does not survive a reboot. A quick sysfs test works fine until the next maintenance reboot silently removes the VFs and the VM fails to start.
• Skipping the pcie=1 flag on q35 machines. Some guest operating systems will still detect the device without it, but PCIe-specific features (and some Windows driver installers) expect a PCIe topology, not a legacy PCI slot.
• Expecting the VM to live-migrate normally. A VM with a VF attached via hostpci cannot be live-migrated through the standard QEMU migration path; the device must be detached before migration and reattached afterward, or the VM must be excluded from automated migration policies entirely.
• Assuming host-side firewall or VLAN-aware bridge rules still apply. Traffic on a VF bypasses the Proxmox VE firewall and the Linux Bridge entirely — filtering has to move to the guest, the NIC's own e-switch VLAN/ACL controls if supported, or an upstream physical switch.
• Installing the wrong VF driver in the guest. Matching the physical function's driver name to the VF is a common error — the VF often needs a different, lighter driver (ixgbevf, iavf) than the PF driver (ixgbe, i40e).
• Not checking sriov_totalvfs before requesting a VF count. Requesting more VFs than the card supports fails silently or caps at the hardware maximum, leaving fewer VFs than expected without an obvious error.
• Passing through the last available VF needed by the host itself. Some drivers reserve the PF's own queues from the VF pool; over-provisioning VFs can starve the host's own use of that NIC.

Best Practices

• Use Resource Mappings (Datacenter → Resource Mappings) instead of hardcoded bus addresses, especially in clusters where NIC PCI addressing can differ slightly between otherwise identical nodes.
• Reserve a small number of VFs, or a separate physical port, exclusively for host or Ceph/corosync traffic if the same card also serves cluster-critical services — don't let SR-IOV VM traffic and cluster communication compete for the same silicon under load.
• Document which VMs hold VF passthrough devices and exclude them from automated HA migration policies, or build a runbook for detach/reattach around planned maintenance.
• Pin a consistent VF count in your configuration management (Ansible, Terraform) rather than relying on manual echo commands, so a reinstalled or replaced node comes back with identical SR-IOV state.
• Where the NIC and switch support it, configure VLAN tagging at the VF level (many drivers expose per-VF VLAN and spoofchk settings via ip link set enp1s0f0 vf 0 vlan 100) so guest traffic is still segmented even though it bypasses the Proxmox VE bridge.
• Benchmark before and after with a real workload generator (e.g. iperf3, or your application's own load test) rather than assuming SR-IOV is a win — for many workloads the added operational complexity is not worth a marginal throughput gain.
• Keep host NIC firmware and driver versions current; SR-IOV VF stability issues are frequently fixed in firmware updates rather than kernel updates.

FAQ

Does SR-IOV work with live migration?

Not directly. A VF attached through hostpci is tied to the physical hardware of the node it's on, so the standard QEMU live migration path cannot move it. Common patterns are to detach the VF before migrating and reattach afterward, or to keep SR-IOV-dependent VMs pinned to specific nodes and excluded from automated HA relocation.

Can I use SR-IOV and still have redundancy?

Yes, but redundancy has to be built at the guest OS level (e.g. bonding two VFs from separate physical NICs inside the VM) rather than relying on the host bridge's own bonding, since the VF traffic never touches the host bridge.

How many VFs can one NIC create?

It depends entirely on the card and firmware — check sriov_totalvfs for the actual limit. Common enterprise NICs support anywhere from 32 to 128 VFs per port, though practical limits are usually set by how many queues you actually need per guest rather than the hardware ceiling.

Do I still need a Linux Bridge if I'm using SR-IOV?

Only if some VMs on the host still use standard VirtIO networking, or if the host itself needs bridge access to the same physical network. VMs using VF passthrough exclusively do not need a bridge for that interface.

Will this work on AMD platforms?

Yes — the process is the same, substituting amd_iommu=on for intel_iommu=on in the kernel command line. AMD-Vi provides the equivalent IOMMU functionality to Intel VT-d.

What happens if the NIC doesn't list "Single Root I/O Virtualization" in lspci?

That card cannot do SR-IOV, full stop — it's a firmware/silicon capability, not something enabled purely in software. You'd need full PCI(e) passthrough of the whole card to a single VM instead, or standard VirtIO networking.

Conclusion

SR-IOV gives Proxmox VE administrators a way to hand VMs near-native network performance without dedicating an entire physical NIC to a single guest. The setup is mechanical once you understand the pieces: confirm hardware support, enable IOMMU through the correct bootloader file, create and persist virtual functions, and assign them to VMs through hostpci, ideally via Resource Mappings for cluster consistency. The real decision is whether the trade-offs — no live migration for affected VMs, traffic that bypasses the host firewall and bridge, and driver matching inside the guest — are worth it for your workload. For latency-sensitive, high-packet-rate applications, they usually are; for general-purpose VMs, standard VirtIO networking remains the simpler and more operationally flexible default.