Proxmox High Availability for Small Offices: Two-Node Clusters, QDevice Quorum, and ZFS Replication vs Ceph

Backups and HA Answer Different Questions

A backup answers "can we get the data back?" A tested restore answers "how long does that take?" HA answers a third question: "can the guests start again on another machine without anyone driving to the office?"

• Restore only. Your recovery time is the time to get replacement hardware or a spare host running, install Proxmox, and restore every guest in priority order. The test restore tells you the real number. Many one-host offices are right to stop here, with a documented procedure and a spare they have actually tested.
• High availability. When a node fails, the cluster restarts its HA-managed guests on a surviving node. The Proxmox documentation puts detection and failover at typically about two minutes, plus however long the guests take to boot. That is minutes, not hours, but it is not zero: users see a reboot.
• Neither replaces the other. HA does nothing for a deleted file, a corrupted database or ransomware. Keep the backups.

The question to ask first is what an hour of downtime costs the office, and whether a documented restore already meets that number. If it does, a second host is a budget decision, not a technical necessity.

QDevice: The Third Vote

Proxmox supports an external vote called a QDevice. A small daemon, corosync-qdevice, runs on each cluster node and talks to corosync-qnetd on a separate machine. That machine supplies the extra vote, so a two-node cluster has three votes and quorum is two. When one node dies, the survivor plus the QDevice make a majority and the cluster carries on. The Proxmox documentation recommends exactly this for two-node clusters that should provide higher availability.

What the external machine needs

• The corosync-qnetd package, and network access to the cluster. That is the whole list of requirements for the host itself, which is why a small, quiet, low-power machine is enough.
• The corosync-qdevice package installed on every cluster node before you run the setup command (apt install corosync-qdevice).
• TCP port 5403 reachable from the nodes (the qnetd default).
• Root SSH access during setup, ideally key-based, since the setup command copies the cluster's key to it.
• Setup is one command run on a cluster node, with every node online: pvecm qdevice setup <QDEVICE-IP>. Confirm with pvecm status, which should show three expected votes and a quorum of two.

Do not run it on the cluster it arbitrates

It is tempting to make the QDevice a small virtual machine on one of the two hosts, since that costs nothing. Do not. If the third vote lives on a guest of the cluster, losing that host removes a node vote and the third vote at the same moment. You are back to one vote out of three, which is the exact failure the QDevice was added to survive. Put it on separate hardware: a mini PC or single-board computer on its own power supply, or a small machine at another site. The documentation notes that the QDevice link runs over ordinary TCP/IP, can sit outside the cluster LAN, and is not bound by Corosync's latency limits.

What it does not fix

For a cluster with an even node count, the Proxmox documentation says a QDevice has no negative implications: if it fails to work, you are where you would be without one. In a two-node cluster, that means that with the qnetd machine down you fall back to plain two-node behaviour. Both nodes keep quorum while both are up, but if one of them then fails, the survivor loses quorum, and a node running HA guests self-fences (reboots) after about 60 seconds. So the QDevice outage is harmless until a node fails at the same time, which is why you monitor it like any other server. Remove the QDevice before adding or removing cluster nodes. The documentation warns that for clusters with an odd node count a QDevice is close to a single point of failure and discourages it, which is another reason this is a two-node pattern; at three nodes you simply have quorum.

Surviving on one node also means the remaining host runs everything. Size each host so it can carry the whole workload alone, as covered in the sizing guide. A pair of hosts that each run at 70 percent is a pair that cannot fail over.

Replication Is Asynchronous: The Interval Is Your Data-Loss Window

Storage replication runs on a schedule. The Proxmox documentation gives a default of every 15 minutes, a minimum interval of one minute and a maximum of once a week, and it warns that HA is allowed in combination with replication but there may be some data loss between the last synced time and the time the node failed.

Read that as a business decision, not a setting. If the schedule is every 15 minutes, a node that dies just before the next sync can cost you up to about 15 minutes of work in the guests it was running: orders entered, tickets updated, files saved. A one-minute schedule shrinks the window but increases the replication traffic and load, and a job that cannot finish inside its interval will fall behind. Ask the people who own each system how much they can afford to redo, then set the interval per guest to match. A file server and a line-of-business database rarely need the same answer.

Two more behaviors worth knowing. A failed replication job goes into an error state and retries every 30 minutes, which can widen the window quietly, so alert on replication errors. And planned moves are cheap: migrating a guest to the node that already holds its replica transfers only the changes since the last sync, and the replication direction switches automatically.

If the failed node stays down and HA is not managing the guest, the documentation describes moving the guest's configuration to the survivor by hand. HA automates that step; replication supplies the disks it starts from.

Give Corosync Its Own Network

Corosync is latency sensitive. The Proxmox documentation asks for a reliable network with latencies under 5 milliseconds between all nodes, ideally on its own physically separated network, and says a dedicated 1 Gbit NIC is enough in most situations. It supports up to 8 links, and to get reliable redundancy you need at least one more link on a different physical network. The nodes must reach each other on UDP ports 5405 to 5412.

For a small office that means a Corosync link on a dedicated NIC and switch ports that do not carry guest traffic, plus a second link on a different path (another NIC and, where you can, another switch). If the cluster heartbeat shares a link with backups, replication or a busy guest VLAN, a saturated link looks like a dead node and can trigger a fence. Keep it off guest networks. The bond, bridge and VLAN layout that goes around it is in the Proxmox networking guide, and the management-plane lockdown is in the Proxmox hardening guide.

Replication traffic also needs a path. It can share a decent management link in a small office, but give it a schedule and a bandwidth limit rather than letting it compete with the Corosync heartbeat.

Prove It: A Failover Test and the Evidence to Keep

A test restore is the evidence for backups. A documented failover test is the evidence for HA, and insurer questionnaires and the cybersecurity program documentation an Ohio township or fire district keeps under ORC 9.64 both go better with a dated record than with a diagram. Run it in a maintenance window. Pulling the power on a node that runs production guests discards every write made since the last replication sync, so either stop user activity and trigger replication immediately before the test (in the GUI use Schedule now on each replication job, or run pvesr schedule-now <job-id> on the node), or run the test with a non-production guest:

1. Record the starting state: pvecm status (votes and quorum, QDevice shown as alive), ha-manager status, and the last successful replication time for each guest.
2. Simulate the failure. Cutting the power to one node is a truer test than a clean shutdown, because a clean shutdown tells HA what is happening.
3. Time it: from the failure to the guests answering on the surviving node. Compare to your target.
4. Check the data: what is the newest record in each guest, and how does it compare to the replication time? That gap is your measured data loss.
5. Bring the node back, confirm it rejoins, replication resumes and the QDevice stays registered. Test the QDevice failure on its own as well: with the machine off, does the cluster behave as you expect?
6. Write the one-page record: date, what was done, times, data gap, who checked it, and what you changed afterward.

Repeat it after hardware changes, a major Proxmox upgrade, or at least yearly. The first run usually finds something: a guest that was never added to HA, a replication job in error, a firewall that blocked port 5403.