Preventing Split-Brain in Two-Node Proxmox VE Clusters with a Corosync QDevice

Introduction

A two-node Proxmox VE cluster is one of the most common topologies in small businesses, branch offices, and homelabs, yet it is also the topology that Proxmox's own documentation warns about most consistently. Corosync, the cluster communication layer underneath Proxmox VE, was designed around the idea that a cluster partition needs a strict majority of votes to remain quorate. With exactly two nodes and two votes, there is no majority to be had: if one node disappears, the survivor cannot prove it is not the one that got isolated, so it fences itself off from cluster-aware operations rather than risk a split-brain scenario.

This article walks through how to eliminate that single point of fragility using a Corosync QDevice (Quorum Device) backed by corosync-qnetd, the officially supported external arbitrator for Proxmox VE clusters. You will learn why two-node clusters lose quorum, how to recognize the symptoms in the wild, how to diagnose the underlying vote math, and how to deploy, verify, and maintain a QDevice in production without introducing a new single point of failure of its own.

Problem Overview

Proxmox VE relies on corosync for cluster membership and on pmxcfs (the /etc/pve cluster filesystem) for configuration replication. Both depend on the cluster holding quorum — a majority of the total configured votes being reachable and in agreement. By default every cluster node carries exactly one vote, and quorum is calculated as floor(total_votes / 2) + 1.

In a three-node cluster this works cleanly: three total votes, quorum requires two, and any single node failure still leaves a quorate majority. In a two-node cluster, three votes never exists — there are only two, and quorum requires both. The moment either node goes offline, whether from a hardware failure, a reboot, or a network partition, the surviving node holds exactly one vote out of two expected. That is not a majority, so the survivor drops out of quorum and Proxmox VE stops allowing changes to the cluster configuration, VM/CT migrations, and — critically — High Availability failover.

This is not specific to Proxmox VE's implementation choices; it is a structural property of any Corosync-based cluster running the votequorum provider over an even, two-member configuration. The knet transport underneath corosync can maintain redundant links between the two nodes to reduce the odds of a false-positive partition, but redundant links only protect against a specific link failing — they do nothing for the case where an entire node is powered off, kernel-panics, or is otherwise legitimately gone. No amount of network redundancy between two nodes can substitute for a third, independent source of truth.

Administrators sometimes work around this by setting pvecm expected 1 on the surviving node during an outage, but that is a manual, reactive, and risky maneuver: if the "dead" node was not actually dead but merely unreachable over the corosync network, you can end up with two independently quorate halves of the cluster running the same VM, corrupting shared storage. A QDevice solves this properly by adding a third, independent vote that only ever sides with one side of a partition.

Symptoms

• The Proxmox VE web UI shows a red cluster status indicator and the message "no quorum" on one or both nodes after a node reboot, NIC failure, or switch outage.
• pvecm status reports Quorate: No even though the node you are running the command on appears healthy.
• Attempts to create, migrate, or delete VMs and containers fail with errors referencing the cluster filesystem, such as error during cfs-locked operation or Cluster not quorate - watchdog will trigger soon.
• /etc/pve becomes read-only on the affected node; changes to /etc/pve/qemu-server/*.conf or /etc/pve/lxc/*.conf silently fail to save.
• If HA is enabled, the ha-manager resource states get stuck in fence and the node's watchdog eventually forces a hard reset, even though the node's own services were fine — only the corosync link to its peer was down.
• Both nodes report themselves as the "surviving" node during a network split, each showing one vote and neither reaching the two-vote threshold, until the link is restored.

Root Cause

The root cause is arithmetic, not a bug: with two configured votes and a quorum requirement of a strict majority, two-node clusters have no mathematically valid way to distinguish "my peer is dead" from "I am the one who got isolated." Corosync deliberately refuses to guess, because guessing wrong means both halves of a split network believe they are authoritative — the classic split-brain failure mode that can lead to two nodes writing to the same shared LUN or the same ZFS pool simultaneously.

Proxmox VE's own pvecm tooling and cluster manager documentation explicitly call out that clusters need either an odd number of full member nodes or an external tie-breaking vote to stay resilient. For any cluster with fewer than three full nodes, that tie-breaker is the QDevice. It runs as an independent daemon — corosync-qnetd — on a host that is not itself a cluster member, and each Proxmox node runs a small client, corosync-qdevice, that asks the arbitrator for its vote during a partition event.

Diagnosis

Before deploying a QDevice, confirm the current vote configuration and quorum state so you have a clear baseline to compare against afterward.

pvecm status

On a healthy two-node cluster without a QDevice, the relevant section looks like this:

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      2
Quorum:           2
Flags:            Quorate

Notice that Quorum equals Total votes — this is the configuration that fails the instant either node drops. You can also confirm cluster membership and per-node vote weight with:

pvecm nodes

If you suspect an active quorum problem right now, check the corosync ring status and journal for link failures:

corosync-cfgtool -s
journalctl -u corosync -b --no-pager | tail -n 100

Look specifically for repeated Retransmit List entries or link: host ... link 0 is down messages — these point to network instability between the nodes rather than an actual node failure, which is precisely the scenario a QDevice is built to arbitrate correctly.

Step-by-Step Solution

1. Choose and prepare the arbitrator host

The QDevice arbitrator must run on a machine that is not a Proxmox VE cluster member and, ideally, is on independent power and network infrastructure from both nodes — a small VM on unrelated infrastructure, a Raspberry Pi, or a lightweight VPS all work well. It only needs to run corosync-qnetd, which is a low-overhead Debian package; 512 MB of RAM and a single vCPU are sufficient.

On the arbitrator host (Debian or Debian-based):

apt update
apt install corosync-qnetd

Confirm the service is active:

systemctl status corosync-qnetd

2. Install the client package on every Proxmox VE node

Run this on both nodes of the cluster:

apt update
apt install corosync-qdevice

3. Ensure passwordless SSH reachability

pvecm qdevice setup connects to the arbitrator over SSH to install the cluster's certificate and configure corosync-qnetd automatically, so the root user needs to be able to reach the arbitrator host over SSH at setup time. Confirm you can log in:

ssh root@<qnetd-ip>

4. Run the QDevice setup from one cluster node

From any single node in the cluster (not from the arbitrator), run:

pvecm qdevice setup <qnetd-ip>

You will be prompted to accept the SSH host key and provide the arbitrator's root password once. The command copies the required corosync configuration, restarts the corosync service cluster-wide, and increases the expected vote count from two to three.

If your cluster has an odd number of full member nodes (relevant when you later grow past two nodes and still want to keep the QDevice temporarily), Proxmox VE requires the --force flag and automatically falls back to the lms (Last-Man-Standing) algorithm instead of the default ffsplit:

pvecm qdevice setup <qnetd-ip> --force

5. Verify the new quorum configuration

pvecm status

You should now see three expected votes instead of two, with the QDevice listed as a membership participant.

Commands

Command	Purpose
apt install corosync-qnetd	Installs the arbitrator daemon on the external host.
apt install corosync-qdevice	Installs the client daemon on each Proxmox VE node.
pvecm qdevice setup <ip>	Configures the cluster to use the specified arbitrator.
pvecm qdevice setup <ip> --force	Forces setup on clusters with an odd node count (uses the lms algorithm).
pvecm qdevice remove	Removes the configured QDevice from the cluster.
pvecm status	Shows current quorum state, expected votes, and QDevice membership.
pvecm nodes	Lists cluster nodes and their vote weight.
pvecm expected <n>	Manually overrides the expected vote count (emergency use only).
corosync-cfgtool -s	Displays the status of each corosync ring/link.
systemctl status corosync-qnetd	Confirms the arbitrator daemon is running on the external host.

Configuration Examples

A healthy pvecm status output on a two-node cluster with an active QDevice looks like this:

Cluster information
-------------------
Name:             prod-cluster
Config Version:   14
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Fri Jul 10 09:14:02 2026
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          1.2a
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1    A,V,NMW 10.10.10.11 (local)
0x00000002          1    A,V,NMW 10.10.10.12
0x00000000          1            Qdevice

The key change from the earlier baseline is Quorum: 2 against Total votes: 3 — the cluster can now lose one full member and remain quorate, because the surviving node plus the QDevice's vote adds up to two, satisfying the majority requirement. The A,V,NMW flags per node mean Alive, Voting, and Not Master Wins (the node is not currently required to hold a special tie-breaking role).

To confirm the corosync configuration file picked up the QDevice definition, inspect the generated stanza:

cat /etc/pve/corosync.conf

quorum {
  provider: corosync_votequorum
  device {
    model: net
    net {
      tls: on
      host: 10.10.10.50
      algorithm: ffsplit
      connect_timeout: 60000
    }
    votes: 1
  }
}

This confirms three things worth checking after any setup: the model is net (the only supported QDevice model in Proxmox VE), TLS is enabled for the arbitrator connection, and the algorithm matches what you expect — ffsplit for the standard two-node case.

Common Mistakes

• Running the QDevice on one of the two cluster nodes. This defeats the entire purpose — if that node goes down, you lose both its cluster vote and the arbitrator vote simultaneously, which is strictly worse than not having a QDevice at all.
• Placing the arbitrator on the same switch, UPS, or power circuit as the cluster nodes. A QDevice is meant to survive failures that take out cluster infrastructure. If a single switch or power event can take down both a node and the arbitrator, the tie-breaker adds no real resilience.
• Forgetting to remove the QDevice before adding a third full cluster node. Once a cluster has three (or any odd number of) full voting members, quorum math is self-sufficient. Leaving a QDevice attached to an odd-sized cluster forces the lms algorithm and unnecessary complexity; remove it first with pvecm qdevice remove, then add the new node.
• Using pvecm expected 1 as a permanent fix instead of a QDevice. This command tells corosync to consider a single vote sufficient for quorum, which permanently removes split-brain protection rather than solving the underlying problem — it should only ever be used as a temporary, manual, and deliberate action during a known and controlled outage.
• Not verifying TLS and SSH reachability before running pvecm qdevice setup. The setup command needs outbound SSH access to the arbitrator at run time; firewalls or security groups that block it mid-setup leave the cluster in a partially configured state.
• Ignoring the QDevice flags in pvecm status. An arbitrator showing NV (Not Voting) or NA (Not Alive) means the tie-breaker is not actually functional, even though the cluster may still report Quorate: Yes under normal conditions — the failure only becomes visible during the next real outage.

Best Practices

• Host the arbitrator on infrastructure that is genuinely independent of both cluster nodes — separate power, separate switch, and ideally a separate physical location or at minimum a separate rack.
• Monitor the arbitrator host itself. A QDevice that silently dies removes your split-brain protection without any obvious symptom until the next real node failure.
• Document and test a full node-failure drill after enabling the QDevice: power off one node deliberately during a maintenance window and confirm the survivor stays quorate and HA resources migrate as expected.
• Keep the corosync-qnetd and corosync-qdevice packages updated alongside routine Proxmox VE updates; version mismatches between the arbitrator and the client daemons can cause connection failures after major corosync version bumps.
• For clusters that expect to grow beyond two nodes eventually, plan the QDevice as a temporary measure and remove it once you reach three full voting members, reverting to native majority quorum.
• Restrict network access to the arbitrator's corosync-qnetd port to only the cluster nodes' IP addresses, using either host-based firewall rules on the arbitrator or a dedicated management VLAN.
• Back up /etc/pve/corosync.conf (or rely on the cluster filesystem replication) before and after QDevice changes so you can quickly diff the configuration if something goes wrong.
• Wire arbitrator health into whatever monitoring stack you already run for the cluster nodes — a simple periodic check of systemctl is-active corosync-qnetd over SSH, or a port check against the corosync-qnetd listener, is enough to catch a silently dead tie-breaker before it matters.
• Avoid stacking multiple unrelated services on the arbitrator host. Keeping it dedicated to corosync-qnetd reduces the chance that an unrelated crash, reboot, or resource exhaustion event takes down your quorum tie-breaker along with something else.

FAQ

Does the QDevice need to run Proxmox VE?

No. The arbitrator only needs corosync-qnetd, which runs on any current Debian release. It should not be a Proxmox VE cluster member.

Can I use a QDevice with a three-or-more-node cluster?

Yes, but it is unnecessary for clusters with an odd number of full voting nodes, since native majority quorum already tolerates a single node failure. Proxmox VE requires the --force flag to set one up in that case, and it switches to the lms algorithm.

What happens if the arbitrator goes offline while both cluster nodes are healthy?

Both nodes still hold their own votes, so total reachable votes drop to two out of three expected — which still satisfies the quorum threshold of two, so the cluster remains fully functional. You simply lose split-brain protection during that window and should treat arbitrator downtime as an operational alert, not just a shrug.

How do I remove a QDevice if I no longer need it?

Run pvecm qdevice remove from any cluster node. This reverts the expected vote count and updates /etc/pve/corosync.conf across the cluster automatically.

Does the QDevice affect VM or container performance?

No. It only participates in corosync quorum decisions and never touches storage, networking, or compute resources for guests.

Is TLS mandatory for the QDevice connection?

Proxmox VE enables TLS by default when you run pvecm qdevice setup, and it should be left enabled — the corosync-qnetd traffic effectively controls cluster quorum decisions, so it must be authenticated and encrypted.

Conclusion

Two-node Proxmox VE clusters are a legitimate and common design, but they are structurally unable to survive a node failure or network partition safely without external help. A Corosync QDevice closes that gap cleanly: it adds a genuine third vote from independent infrastructure, restores proper majority-based quorum math, and lets High Availability and cluster filesystem operations keep working through a single node outage instead of freezing the survivor. The setup itself is a handful of package installs and a single pvecm qdevice setup command, but the payoff — a cluster that fails predictably instead of ambiguously — is worth the twenty minutes it takes to configure and the ongoing discipline of keeping the arbitrator on genuinely independent infrastructure.