Introduction

Proxmox VE ships with native ZFS support baked into the installer, and it's become the default choice for admins who want checksummed storage, snapshots, and replication without bolting on a separate SAN. What catches a lot of people off guard, though, is how ZFS manages memory. Out of the box on Linux, ZFS's Adaptive Replacement Cache (ARC) will happily consume up to half of a host's RAM, and on a hypervisor that's also trying to hand out memory to a dozen KVM guests and a handful of LXC containers, that default is rarely the right one. This article walks through why ARC sizing matters on a Proxmox host, how to recognize when it's misconfigured, and how to tune it safely without starving either your guests or your ZFS read cache.

Problem Overview

The ARC is ZFS's in-memory read cache, roughly analogous to the Linux page cache but implemented independently inside the ZFS module. Unlike the page cache, the ARC does not automatically shrink the instant an application asks for memory — it uses its own reclaim logic and shrinker callbacks, and under certain conditions (memory pressure from cgroups, NUMA imbalances, or simply a host that's already tight on RAM) it can be slow to release memory back to the kernel. On a general-purpose ZFS file server this rarely matters because ZFS is the only heavyweight consumer of RAM. On a Proxmox VE node, ARC competes directly with:

  • KVM/QEMU virtual machines, each with their own fixed or ballooned memory allocation
  • LXC containers, which share the host kernel and are more sensitive to host-level memory pressure
  • Proxmox's own management stack (pvedaemon, pvestatd, pmxcfs, corosync)

When ARC is left at its default ceiling (50% of physical RAM on most distributions, though Proxmox has adjusted this default in recent releases), a host with 64 GB of RAM can end up with 32 GB locked into cache before a single VM is even started. If you then provision VMs assuming most of that RAM is free, you get an OOM situation, or worse, the kernel starts killing processes — sometimes including guest QEMU processes — to reclaim memory.

This is not a theoretical concern. It's one of the most common causes of "mystery" OOM kills reported on the Proxmox community forums, and it typically surfaces weeks or months after a node goes into production, once guest memory allocation creeps up toward the host's real capacity and the ARC ceiling that seemed harmless on day one starts colliding with actual demand. The fix is inexpensive and low-risk, which makes it one of the higher-value tuning changes you can make on a new ZFS-backed Proxmox deployment before it ever becomes a 2 a.m. page.

Symptoms

Misconfigured ARC sizing tends to show up in a handful of recognizable ways:

  • VMs fail to start with errors like kvm: failed to allocate memory or TASK ERROR: start failed: org.qemu.qmp... even though free -h appears to show available memory when ARC usage is subtracted mentally but not by the allocator in time.
  • The Proxmox summary page shows high overall "Memory" usage even with only a few lightly-loaded guests running.
  • The OOM killer in dmesg or journalctl -k targets a kvm or pvestatd process rather than an obviously runaway one.
  • LXC containers become sluggish or get OOM-killed internally under their cgroup memory limit even though the host has "plenty" of RAM according to naive accounting.
  • Backup jobs (vzdump) that read heavily from ZFS-backed storage cause a visible spike in ARC size that doesn't shrink back down for a long time afterward.
  • Migrating a VM onto a node via live migration or HA failover fails or stalls, because the target node's ARC has grown to fill memory that the incoming VM needs to claim.
  • Swap usage climbs on the host even though no single guest process appears to be using excessive memory, because the kernel is reclaiming page cache and swapping guest-adjacent processes before it manages to shrink ARC.

These symptoms are easy to misdiagnose as a "bad VM," a runaway container, or a hardware memory fault, because the actual culprit — ARC — doesn't show up as a discrete process in top or htop. It lives inside the kernel's ZFS module, so admins who only check per-process memory usage will conclude the host "should" have plenty of free RAM and go looking in the wrong place.

Root Cause

The root cause is almost always one of two things, or a combination of both:

  1. No explicit zfs_arc_max configured. The ZFS module falls back to its built-in default, which is a percentage of total system memory. This value was designed for dedicated storage servers, not hypervisors that need to reserve most of their RAM for guest workloads.
  2. Memory accounting confusion. Tools like free -h report ARC-held memory in a way that looks "available" or "cached" depending on kernel version and ZFS module version, which leads admins to under-provision headroom because they believe the cache will be reclaimed instantly the moment a VM asks for memory. In practice, reclaim has latency, and under sudden pressure (e.g., starting several VMs at once after a reboot) that latency causes real allocation failures.

There's a secondary contributing factor worth mentioning: zfs_arc_min. If you set a large minimum without matching headroom planning, ZFS will refuse to shrink below that floor even under pressure, which can make an already tight host worse.

There's also a subtler NUMA-related cause on dual-socket hosts. ZFS's ARC allocation isn't always evenly NUMA-aware in the way guest memory is when the QEMU NUMA policy is configured per-VM. On a two-socket server, it's possible for ARC to lean heavily on one NUMA node's memory while guests pinned to that same node are starved, even though aggregate host memory looks fine. If you run NUMA-pinned VMs on multi-socket hardware, factor that imbalance into your ARC ceiling and consider checking per-node memory with numastat alongside the global ARC figures.

Diagnosis

Before changing anything, quantify what's actually happening. Start with the current ARC size and limits:

arc_summary | head -40

Look specifically at the "ARC size" line versus "Target size" and the configured min/max. On most Proxmox installs without manual tuning you'll see the max sitting at roughly half of total RAM.

Next, check live memory pressure and how much of it is genuinely ARC:

free -h
cat /proc/spl/kstat/zfs/arcstats | grep -E '^(size|c_max|c_min|c)\s'

The size field is the current ARC footprint in bytes; c_max and c_min are the configured ceiling and floor. If size is sitting right at c_max most of the time, ARC is actively using all the memory it's allowed to.

Cross-reference with what the Proxmox stack itself reports:

pvesh get /nodes/$(hostname)/status --output-format json | grep -A3 memory

Finally, check kernel logs for OOM events tied to the time window when problems occurred:

journalctl -k --since "2 days ago" | grep -i "out of memory"

If you see OOM kills correlating with periods of high ARC size and simultaneous VM starts or backup jobs, you've confirmed the diagnosis.

It also helps to build a short timeline before you act. Pull the last few days of ARC size samples if you already have monitoring in place (Proxmox's built-in RRD graphs, or an external Prometheus/Grafana stack scraping node_exporter), and overlay them against your VM start times and backup schedule from /etc/pve/jobs.cfg or the vzdump log at /var/log/vzdump/. In most cases the pattern is obvious once it's laid out: ARC climbs steadily during normal operation, then a backup job or a burst of VM starts creates a demand spike that the ARC ceiling didn't leave room for, and that's the exact moment the OOM killer or allocation failures occur.

Step-by-Step Solution

The fix is to explicitly cap ARC to a value that leaves comfortable headroom for guests and the host itself, then make that cap persistent across reboots.

1. Decide on a target ARC ceiling

A common starting rule of thumb for hypervisor use is to reserve enough RAM for the host OS plus all planned guest allocations, and cap ARC at whatever remains, with a floor around 4–8 GB so ZFS still has a useful cache. For example, on a 128 GB host running 96 GB worth of provisioned VM memory, you might cap ARC at 16 GB and leave the rest as general headroom for the kernel, Proxmox services, and ballooning.

2. Set the limit via the kernel module parameter

Create or edit the ZFS module options file:

echo "options zfs zfs_arc_max=17179869184" > /etc/modprobe.d/zfs.conf

The value is in bytes — 17179869184 bytes is 16 GiB. Adjust to your target. If you also want a floor to prevent excessive shrinking under transient pressure, add a min line as well:

cat >> /etc/modprobe.d/zfs.conf  /sys/module/zfs/parameters/zfs_arc_max

# Live-adjust min ARC size (bytes)
echo <bytes> > /sys/module/zfs/parameters/zfs_arc_min

# Persist across reboots
echo "options zfs zfs_arc_max=<bytes>" > /etc/modprobe.d/zfs.conf
update-initramfs -u -k all

# Watch ARC hit/miss ratio in real time
arcstat 2

Configuration Examples

Example for a 64 GB Proxmox node running mixed VM/LXC workloads, reserving 8 GB ARC ceiling:

options zfs zfs_arc_max=8589934592
options zfs zfs_arc_min=2147483648

Example for a 256 GB storage-heavy node where ZFS performance matters more and guest density is lower, capping ARC at 48 GB:

options zfs zfs_arc_max=51539607552
options zfs zfs_arc_min=8589934592

A useful table for translating common GiB targets to the bytes value zfs_arc_max expects:

Target (GiB)Bytes value
44294967296
88589934592
1617179869184
3234359738368
4851539607552

Common Mistakes

  • Setting zfs_arc_max too low. Capping ARC at 1–2 GB on a busy storage node tanks read performance for VM disk images and can make snapshot/replication jobs noticeably slower.
  • Forgetting to rebuild initramfs. Editing /etc/modprobe.d/zfs.conf alone does nothing until the initramfs is regenerated (or the module is manually reloaded, which isn't practical on a host with imported pools).
  • Setting zfs_arc_min higher than is safe for the workload. A high floor prevents ZFS from releasing memory even when guests genuinely need it, effectively recreating the same OOM risk you were trying to fix.
  • Confusing ARC usage with "wasted" memory. ARC is doing useful work — the goal isn't to minimize it to zero, it's to bound it so it can't crowd out guest memory.
  • Not accounting for balloon driver behavior. If VMs use memory ballooning, available host memory fluctuates, and an ARC ceiling set without margin for ballooning inflation can still leave the host tight during peak guest memory demand.

Best Practices

  • Set an explicit zfs_arc_max on every Proxmox node using ZFS storage — never rely on the distribution default on a hypervisor.
  • Leave at least 10–15% of total RAM as unallocated headroom beyond guest allocations plus ARC max, to absorb host process overhead and transient spikes.
  • Monitor arcstat hit ratio over time; if it's consistently above 95% with room to spare, your cache is sized well. If it drops sharply after lowering the max, you've cut too deep.
  • Re-evaluate ARC sizing whenever you significantly change guest density or add new VMs with large disk images — the right cap isn't a "set once" value.
  • Document the chosen value and the reasoning (total RAM, guest allocation, headroom) in your node's runbook so future capacity planning doesn't second-guess it.
  • For clusters, keep ARC policy consistent across nodes with similar hardware profiles so migrated VMs experience predictable host memory behavior.
  • Factor HA failover scenarios into your headroom math, not just steady-state guest allocation — a node that looks fine day-to-day can still be under-provisioned for the worst case where it absorbs another node's VMs.
  • Wire ARC size into whatever monitoring stack you already run. Proxmox exposes host memory metrics through its API and RRD graphs, and if you run Prometheus, node_exporter's NFS/ZFS-related collectors combined with a dashboard panel for node_zfs_arc_size_bytes makes ARC pressure visible over time instead of something you only discover during an incident.
  • Set an alert threshold, not just a graph. Alerting when ARC size approaches its configured max sustained over several minutes gives you a lead indicator before guest memory contention actually happens.

FAQ

Does lowering ARC max hurt VM disk performance?
It can, if set too aggressively low relative to your working set. Watch the hit ratio with arcstat after changing the value — a meaningful drop in hit ratio combined with slower guest I/O means you've gone too low.

Do I need to reboot for the change to take effect?
No for testing — the sysfs write applies immediately. Yes for a persistent change to survive reboot, since the initramfs must embed the module option.

Is this any different for a single-node setup versus a cluster?
The mechanism is identical; the main difference is that in a cluster you should apply consistent policy across nodes so live migration and HA failover don't move a VM onto a node with a very different memory profile.

What if I use LXC containers instead of VMs?
LXC containers share the host kernel directly, so they're actually more sensitive to ARC-driven memory pressure than VMs, which have their own fixed allocation. The same ARC cap applies and matters just as much, if not more.

Can I automate applying this across many nodes?
Yes — push the same /etc/modprobe.d/zfs.conf content via your configuration management tool of choice and trigger update-initramfs -u -k all as part of the rollout, then schedule a maintenance-window reboot per node.

Will this setting survive a Proxmox version upgrade?
Yes, as long as the file remains at /etc/modprobe.d/zfs.conf and the initramfs is rebuilt after any kernel or ZFS module update. It's good practice to confirm the value with arc_summary after major upgrades, since a kernel/ZFS version bump occasionally resets initramfs state if the update process regenerates images differently than expected.

Should I size ARC differently for all-flash versus spinning-disk pools?
Generally yes, though indirectly. All-flash pools tolerate a smaller ARC better because underlying read latency is already low, so cache misses are less painful. Spinning-disk pools benefit more from a generous ARC because a cache miss means a real seek, so if you have RAM to spare and a disk-backed pool, it's reasonable to lean toward the higher end of your headroom-adjusted range.

Does this apply to Proxmox Backup Server as well as Proxmox VE?
Proxmox Backup Server can also run on ZFS and has its own memory pressure profile, mostly driven by chunk store I/O rather than guest workloads, but the same zfs_arc_max mechanism and the same "don't leave it at the default on a shared-purpose host" logic applies there too.

Conclusion

ARC sizing is one of those Proxmox VE tuning items that's easy to overlook because ZFS "just works" out of the box — until it doesn't, usually during a maintenance window when you're starting several VMs at once and suddenly run into OOM errors that have nothing to do with the VMs themselves. Capping zfs_arc_max explicitly, leaving genuine headroom for guest workloads, and validating the change with arc_summary and arcstat turns ARC from an unpredictable background variable into a known, planned quantity in your capacity calculations. It's a small amount of upfront configuration that removes an entire category of intermittent, hard-to-diagnose memory pressure incidents on ZFS-backed Proxmox hosts.