Introduction
A replication job that silently stops running is one of the more dangerous failure modes in a Proxmox VE cluster, because nothing looks broken until you actually need the replica. The VM keeps running on its original node, pvesr status shows a timestamp that's slowly getting older, and unless someone's watching the GUI's replication tab, that staleness can sit there for weeks. Then a node dies, HA tries to fail the guest over, and you find out the "replica" on the target node is three weeks behind.
This one's for the admins running ZFS-based replication instead of a SAN or Ceph cluster - a setup that's common in three-node homelabs and just as common in small production clusters where a shared storage array wasn't in the budget. I'll walk through why replication jobs fail, how to read the errors pvesr actually gives you, and how to get a cluster back to a state where you trust the numbers in the GUI again.
Problem Overview
Proxmox VE's storage replication (pvesr) exists specifically for clusters that don't have shared storage. Instead of every node reading and writing the same LUN or Ceph pool, each node keeps its own local ZFS pool, and pvesr periodically snapshots the guest's disks and sends the incremental delta to one or more target nodes via zfs send/receive over an SSH tunnel. It's the mechanism that lets you run HA on local-zfs storage at all - without it, HA on local disks is a non-starter, because there's nothing on the other node to fail over to.
The tradeoff is that this isn't synchronous replication. A job runs on a schedule (every 15 minutes by default), and whatever changed since the last successful sync is what you lose if the source node dies mid-interval. That's a known and accepted limitation of the design. What isn't acceptable is a job that's supposed to be running every 15 minutes but has actually been failing silently for a month because a snapshot got deleted out from under it or the target pool ran out of space.
The core problem this article deals with: replication jobs fail more often from environmental drift - a renamed pool, a full target, a network blip during a large initial sync - than from any bug in pvesr itself. Once you know the failure signatures, most of them take under five minutes to fix.
Symptoms
You'll usually notice one of these first:
- The Replication tab in the GUI shows a red X next to a job, or a "Duration" column that hasn't updated in hours.
- pvesr status reports a job with a
fail_countgreater than zero and aerrorfield with actual text in it, not blank. - A live migration that should be near-instant (because the target already has most of the data) instead does a full-disk copy - a strong sign the replica isn't in sync at all.
- HA fails over a guest and it starts from a snapshot that's noticeably behind where the guest actually was - lost writes, an old app version, a database that's rolled back further than you expected.
- Email alerts from
pvesr run --mail 1(if you've got the cron-driven mail flag on) landing in a inbox nobody reads.
None of these on their own tell you the root cause. They just tell you something's wrong, which is exactly why the diagnosis step below matters - the fix is different depending on what actually broke.
Root Cause
In practice, the failures fall into a handful of buckets:
Storage ID mismatch
Replication requires the same storage ID to exist on both the source and target node, and it has to be ZFS (zfspool type in storage.cfg, or a directory-backed ZFS dataset works too in recent PVE releases, but the plain zfs type is what most people use). If someone added local-zfs on node A pointing at rpool/data and then, on node B, named the equivalent pool rpool2 or added it under a different storage ID, replication can't target it. This is the single most common cause I've seen in multi-node clusters that were built up node by node instead of provisioned identically from a template.
Target pool out of space or exceeding quota
ZFS receive needs room for the incoming snapshot before it can prune anything. If the target pool is sitting at 95% capacity, a receive that would normally free old snapshots after landing the new one can fail partway, especially if you've got a quota or refquota set on the target dataset.
Snapshot chain broken
pvesr relies on incremental send based on the last common snapshot between source and target. If someone manually deletes a ZFS snapshot on the target (via zfs destroy, a manual rollback, or a third-party ZFS snapshot tool that doesn't know about pvesr's naming convention) the incremental chain breaks. The next run fails because pvesr can't find a common base, and it won't silently fall back to a full send on its own in most cases - you get an error instead.
Long outage exceeding retention
If a target node is offline for an extended period and the source node's ZFS pool prunes older snapshots for space reasons (or you've got a separate snapshot retention policy running), the common snapshot pvesr needs might no longer exist by the time the target comes back. Same failure mode as above, different cause.
Network path problems
Replication traffic goes over the migration network by default, tunneled through SSH. If that network is saturated - a big backup job running at the same time, a bonded NIC with one leg down and half your expected throughput - large initial syncs can time out. Clock skew between nodes can also cause SSH host key or Corosync-adjacent oddities, though this is rarer than the other four causes.
Guest ID or job ID collisions
Cloning a VM and forgetting to remove the replication config that got copied along with it produces a job that references a job ID already in use, or a target that no longer matches reality. This one usually surfaces as a config error rather than a runtime failure, but it's worth mentioning because it happens after "quick" VM clones more than people expect.
Diagnosis
Start broad, then narrow. First, get the actual list of jobs and their state:
pvesr status
This gives you JobID, target, last sync, next sync, duration, and - critically - a FailCount column. Anything above zero needs attention. If you want it for one specific guest:
pvesr status --guest 100
Next, pull the job's own config to confirm the target node and storage mapping are what you think they are:
pvesr read 100-0
If that matches your expectations, go to the logs. The replication runner logs to the system journal on whichever node the job is scheduled to run on (the source node):
journalctl -u pvesr.service --since "2 hours ago"
You're looking for lines that look like:
100-0: got unexpected replication job error - could not find common ancestor snapshot for zfs volume 'rpool/data/vm-100-disk-0'
or
100-0: error 'storage 'local-zfs' is not available on node 'pve2'' - cannot start job
or
volume 'rpool/data/vm-100-disk-0' does not exist
Each of these maps directly to one of the root causes above - the first two are snapshot chain and storage ID problems respectively, the third usually means the disk was detached or the VMID doesn't actually exist on that storage anymore.
To check space on the target, log into the target node and run:
zfs list -o name,used,avail,refer,mountpoint rpool/data
And to see whether the snapshot chain actually lines up between source and target, compare snapshot lists on both sides:
# on the source node
zfs list -t snapshot -o name,creation -r rpool/data/vm-100-disk-0
# on the target node
zfs list -t snapshot -o name,creation -r rpool/data/vm-100-disk-0
If the target is missing the most recent snapshot that pvesr thinks is the common ancestor, you've confirmed a broken chain - somebody or something pruned a snapshot pvesr still needed.
Step-by-Step Solution
The fix depends on which root cause the diagnosis pointed to, but here's the general order of operations for the most common case - a broken snapshot chain that's forcing every sync attempt to fail.
- Confirm the job is actually broken and not just slow: check
pvesr statusfor a fail_count that's climbing across multiple runs, not a single transient blip. - Disable the job so it stops retrying every 30 minutes while you work on it:
pvesr disable 100-0. - On the target node, remove the orphaned dataset for that guest's disk so the next sync can start clean. Double check the dataset name before running this - it's destructive to whatever's sitting there:
zfs destroy -r rpool/data/vm-100-disk-0. - Re-enable the job:
pvesr enable 100-0. - Force an immediate run instead of waiting for the schedule, so you can watch it happen:
pvesr schedule-now 100-0. - Tail the journal while it runs to confirm a clean full sync:
journalctl -u pvesr.service -f. - Once it completes, verify with
pvesr status --guest 100thatFailCountis back to 0 and the sync timestamp is current.
For a storage ID mismatch, the fix is simpler but requires more care - you need to add the missing storage on the target node with an identical storage ID (same name, same underlying pool structure) via Datacenter → Storage or pvesm add zfspool, matching the source's configuration exactly. Don't just create a similarly-named pool; pvesr matches on storage ID string, not pool contents.
For a target running low on space, either free space on the pool, raise or remove the quota on the target dataset, or move the replication target to a node with more headroom. There's no way around this one - ZFS receive needs the room.
For network-related timeouts on large initial syncs, either run the first sync during a quiet window, apply a rate limit so it doesn't compete with production traffic (counterintuitively, this can make first syncs more reliable even though they take longer), or move replication traffic onto a dedicated network per the configuration example below.
Commands
These are the pvesr subcommands you'll reach for most often when managing or troubleshooting replication:
| Command | What it does |
|---|---|
pvesr create-local-job 100-0 pve2 --schedule "*/15" --rate 20 | Creates a new job for guest 100, replicating to node pve2 every 15 minutes at a 20 MB/s cap. |
pvesr list | Lists every replication job configured cluster-wide. |
pvesr status | Shows current state, last/next sync, and fail counts for all jobs. |
pvesr status --guest 100 | Filters status output to one guest. |
pvesr read 100-0 | Dumps the raw configuration of a single job. |
pvesr update 100-0 --schedule "*/5" | Changes the schedule of an existing job without recreating it. |
pvesr disable 100-0 / pvesr enable 100-0 | Temporarily suspends or resumes a job. |
pvesr schedule-now 100-0 | Triggers an immediate out-of-band run. |
pvesr delete 100-0 --keep 1 | Removes the job config but leaves replicated data on the target intact. |
pvesr run --verbose 1 | Manually invokes the scheduler pass (normally done by systemd) with verbose output - useful for debugging. |
A note on pvesr delete --keep 1: use it when you're planning to recreate the job (say, after fixing a storage mismatch) and don't want to re-transfer the full disk from scratch. Without --keep, the target-side data gets cleaned up along with the job.
Configuration Examples
Creating a job from scratch for VM 100, replicating every 15 minutes to node pve2, capped at 20 MB/s so it doesn't saturate a 1 Gbps link shared with other traffic:
pvesr create-local-job 100-0 pve2 --schedule "*/15" --rate 20 --comment "primary replica"
Adding a second target for the same guest, on a slower schedule, since pvesr supports multiple jobs per guest as long as job numbers differ:
pvesr create-local-job 100-1 pve3 --schedule "*/30" --rate 10 --comment "secondary replica"
Dedicating a separate VLAN or physical link to replication traffic instead of sharing the migration network - set in /etc/pve/datacenter.cfg:
replication: secure,network=10.20.30.0/24
Dropping secure for insecure skips the SSH tunnel's encryption overhead in favor of raw throughput, which only makes sense if the replication network is already isolated and trusted - not something to do on a shared or routed segment.
replication: insecure,network=10.20.30.0/24
An example storage.cfg entry that has to match, name-for-name, between source and target for replication to work at all:
zfspool: local-zfs
pool rpool/data
content images,rootdir
sparse 1
If node A has local-zfs pointing at rpool/data and node B is going to be a replication target, node B needs a storage entry with the exact same ID - local-zfs - even if the underlying pool name differs slightly on disk. Get this wrong and you'll see the "storage is not available on node" error from the diagnosis section above the moment you try to create the job.
Common Mistakes
The same handful of mistakes come up again and again:
- Assuming replication is a backup. It's not versioned in any meaningful way for point-in-time recovery beyond the last couple of snapshots pvesr retains for the incremental chain. If you delete a file inside the guest, replication happily propagates that deletion. Pair it with Proxmox Backup Server for anything you actually need to roll back.
- Manually touching pvesr's ZFS snapshots. Snapshots created by the replication framework follow the naming pattern
__replicate_<vmid>-<jobnum>_<timestamp>__. Running your ownzfs destroyagainst these to "clean up" disk space is exactly what breaks the incremental chain described above. - Renaming storage IDs after replication is already configured. If you rename
local-zfsto something else on one node for consistency reasons, every job pointing at it breaks immediately, and the error message doesn't always make the rename obvious as the cause. - Setting the schedule too aggressive for the disk churn rate. A 1-minute schedule on a VM with a database doing heavy writes can mean the previous sync hasn't finished before the next one is due to start, which queues up and eventually shows as jobs perpetually "running."
- Forgetting replication doesn't survive a full storage migration. Moving a guest's disk to different underlying storage (even ZFS-to-ZFS on the same node) breaks the existing job silently until you notice it's no longer listed under that guest, or it errors on the next scheduled run.
- Combining replication with HA and assuming zero data loss. HA failover restarts the guest from the last successful replica. If a node dies 14 minutes into a 15-minute schedule, you lose those 14 minutes. That's expected behavior, not a bug - but plenty of people don't realize it until it happens.
Best Practices
A few things that consistently save trouble down the line:
Match schedule intervals to how much data loss you can tolerate, not to some default you copied from a forum post. A file server that changes twice a day doesn't need a 5-minute schedule; a busy database VM might need 5 minutes and a dedicated, uncontended network path to hit it reliably.
Give replication its own network segment when you can. Sharing the migration network works fine for small clusters, but once you're doing live migrations and replication syncs at the same time on a single 1 Gbps link, one of the two is going to lose. A dedicated 10G link or VLAN for replication and migration traffic, separate from the corosync network, is worth the extra NIC.
Monitor fail_count, not just job existence. A job that exists and is enabled but has been failing for two weeks looks identical to a healthy one in a casual glance at the GUI unless you actually read the FailCount column or the "Last Sync" timestamp. Wire up an alert - even a basic cron script parsing pvesr status output and mailing on any nonzero fail count is better than nothing.
Test failover on purpose, on a schedule, not just when something breaks for real. Pick a non-critical guest, take its source node down (or just stop the guest and start the replica manually), and confirm the process actually works the way you think it does. The first time you test a failover procedure shouldn't be during an actual outage.
Keep target pools with meaningful headroom - I'd budget for at least 20-25% free space on any node acting as a replication target, more if you're replicating several large guests to the same node. ZFS performance and reliability both degrade as pools fill up, and replication receive operations are exactly the kind of write-heavy workload that suffers first.
Document which guest replicates where. In a three-node cluster with a dozen VMs replicating in different directions for load-balancing reasons, it's easy to lose track of the topology. A simple table in your runbook - guest, source, target(s), schedule - pays for itself the first time you're troubleshooting at 2 a.m.
FAQ
Does storage replication work with Ceph or shared storage?
No. Ceph already provides its own replication at the storage layer, and pvesr is specifically for local, non-shared storage - ZFS. Trying to configure pvesr against a Ceph-backed storage ID isn't supported and won't show up as a valid target.
Can I replicate a guest to more than one node?
Yes, by creating additional jobs with different job numbers for the same VMID, each pointing at a different target node with its own schedule and rate limit.
What happens to replication when I live-migrate a guest to its own replication target?
The direction flips automatically. If VM 100 was replicating from pve1 to pve2 and you migrate it to pve2, the existing replica becomes the live disk and a new job starts replicating back to pve1 - no manual reconfiguration needed.
Why did my job fail with "not enough space" when the pool clearly has room?
Check for a quota or refquota set on the specific dataset, not just the overall pool. zfs get quota,refquota,used,available <dataset> will show if a per-dataset limit is what's actually blocking the receive.
Is there a way to force a full resync instead of relying on the incremental chain?
Deleting the job with pvesr delete (without --keep) and recreating it forces a fresh full sync on the next run, since there's no longer a common snapshot to build an incremental from. Only do this if you've confirmed the incremental approach is genuinely broken - it re-transfers the entire disk.
Conclusion
Storage replication is one of those Proxmox VE features that's invisible right up until the day it matters, which is exactly why it deserves more attention than a "set it up once and forget it" mindset. The failure modes aren't exotic - a renamed storage ID, a snapshot someone shouldn't have touched, a pool that quietly filled up - but they're the kind of thing that only gets caught if someone's actually watching the FailCount column instead of trusting a green checkmark in the GUI.
If you're running HA on local ZFS storage without a SAN behind it, the ten minutes it takes to script a basic pvesr status alert is worth more than almost any other maintenance task on the list. Test the failover path before you need it, keep target pools from filling up, and leave the replication snapshots alone unless pvesr is the one managing them.