A Proxmox HA cluster installed in production. Failover tested in a test environment. Never tested in production. The first node drops, VMs remain stuck for 45 minutes because the quorum was not configured correctly. This is a real-world scenario, a direct experience highlighting how critical it is not only to install a Proxmox HA cluster but also to verify every single aspect. High availability is not an option; it’s a necessity for any enterprise environment managing mission-critical workloads. If a cluster doesn’t function as expected during an emergency, its value is zero. This guide provides a detailed checklist for installing and verifying a Proxmox HA cluster, based on years of field experience with hundreds of VMs.
Prerequisites: Dedicated Network and Synchronized NTP
Before diving into the actual installation, preparing the basic infrastructure is crucial. Two aspects are vital for the stability and reliability of a Proxmox HA cluster: a dedicated network for cluster traffic and perfect time synchronization across all nodes.
Dedicated Network for Cluster Traffic
The heartbeat and Corosync traffic, essential for inter-node communication and quorum management, must be isolated. Using a separate network ensures that any congestion on the management or VM network does not affect cluster stability. Ideally, each node should have at least two physical network interfaces dedicated to Corosync traffic, configured in a bond (e.g., LACP or balance-rr) for redundancy and throughput. In environments with 300+ VMs, cluster network stability is a priority. Recent studies indicate that 30% of HA cluster split-brains are caused by network issues (ClusterLabs Report 2025).
Synchronized NTP
Time precision is a non-negotiable requirement for proper quorum operation. Even a minimal misalignment between node clocks can lead to quorum issues, causing stalemates or split-brains. Configure all Proxmox nodes to synchronize with reliable NTP servers, preferably internal to your network or public pools like pool.ntp.org. Regularly verify the NTP synchronization status.
Cluster Installation and Quorum Device
Once the prerequisites are prepared, we can proceed with cluster creation and expansion.
Creating the Cluster
The first step is to create the cluster from the first node. This command initializes the cluster and prepares it for adding other nodes.
pvecm create mycluster
Adding Nodes to the Cluster
After creating the cluster, you can add the remaining nodes. Ensure the firewall is correctly configured to allow Corosync communication on UDP ports 5404 and 5405.
pvecm add <node1-ip>
Replace with the IP address of an existing cluster member node. Repeat for all nodes you wish to add.
Verifying Cluster Status
After adding the nodes, check the cluster status to ensure all nodes are online and quorum is established.
pvecm status
Quorum Status and Quorum Device
Quorum is the mechanism that prevents split-brain, ensuring that only one partition of the cluster can operate. In a cluster with an even number of nodes (especially two), a Quorum Device (QDevice) is essential. A QDevice is a lightweight external host (e.g., a Raspberry Pi) that acts as an additional vote, allowing the cluster to maintain quorum if one of the two nodes fails.
corosync-quorumtool -s
This command will show you the quorum status. Ensure that the number of active votes is sufficient to maintain quorum (Generally, (Number_of_Nodes / 2) + 1).
Shared Storage: Ceph or NFS/iSCSI
For high availability, VMs must be able to access their disks from any node in the cluster. This requires shared storage.
Ceph: Distributed Solution
Ceph is a distributed storage solution natively integrated into Proxmox. It offers high availability, scalability, and performance. Installing Ceph on Proxmox is relatively straightforward.
pveceph init --network 10.10.10.0/24 # Initializes Ceph on the current node
pveceph createmon # Creates a Ceph monitor
pveceph createosd /dev/sdb # Adds a disk as an OSD (Object Storage Device)
pveceph createpool vm-storage # Creates a pool for VMs
Remember to configure Ceph on all nodes and have at least 3 OSDs for redundancy. Ceph’s stability is directly proportional to the number of OSDs and the quality of the interconnect network.
NFS/iSCSI: External Storage
Alternatively to Ceph, you can use external storage based on NFS or iSCSI. Ensure that the storage itself is highly available and that Proxmox nodes have redundant connectivity to the storage array. A single point of failure in shared storage would nullify the Proxmox cluster’s HA.
HA Groups and Failover Policies
Proxmox HA Manager automatically handles restarting VMs on a functional node in case of failure. You can define specific HA groups and policies.
Configuring HA Manager
ha-manager status will give you an overview of the current state of HA resources. You can define which VMs or containers should be managed by HA and with what policies (e.g., max_relocate, max_restart).
ha-manager status
HA Groups allow you to group VMs that must reside on the same node or different nodes, preventing a failover from moving all critical VMs to a single host, thereby overloading it. This is particularly useful in environments with 2,000+ virtualized workstations.
Failover Testing and Verification That It Actually Works
This is the most critical step: practical verification that the HA cluster works as expected. Theoretical configurations are not enough.
Manual Failover Test
Simulate a VM failover to verify that it is correctly migrated and restarted on another node.
ha-manager crm-command migrate <vmid> <node>
Replace with the VM ID and with the destination node name. This is a controlled test. The real test is simulating a failure.
Simulating a Real Failure
Unplug the network from a node, forcibly shut down a node. Observe the VM recovery times. This is the only way to be sure that the HA configuration is robust. Measure the downtime and ensure it falls within the expected SLAs. In an enterprise environment, a failover must occur within minutes, not 45.
Cluster Monitoring with Prometheus + Grafana
An HA cluster without proactive monitoring is a ticking time bomb. Integrate Proxmox with Prometheus for metric collection and Grafana for visualization and alerts. Monitor quorum status, node resource utilization, storage latency, and HA VM status. Critical alerts for corosync_quorum_votes_total or proxmox_ha_resource_status can save you from disaster.
Common Installation Errors and How to Avoid Them
- Lack of dedicated network for Corosync: Causes latency and instability. Solution: configure a separate network interface and a bond.
- Unsynchronized NTP: Leads to split-brain. Solution: configure NTP servers on all nodes and verify regularly.
- Insufficient quorum in 2-node clusters: The cluster halts if one node fails. Solution: implement a QDevice.
- Non-HA shared storage: If storage fails, everything fails. Solution: use Ceph or SAN/NAS storage with redundancy.
- Lack of real failover tests: Theory is not enough. Solution: simulate real failures and measure recovery times.
- Incorrectly configured firewall: Blocks Corosync traffic. Solution: open UDP ports 5404 and 5405.
- Mismatched Proxmox versions: Compatibility issues. Solution: keep all nodes at the same version and patch level.
FAQ — Frequently Asked Questions
How can I add a Proxmox node to an existing cluster?
To add a node, first install Proxmox on the new hardware. Then, from an existing node, create a join token with pvecm add . From the new node, use this token to join it to the cluster. Ensure network and NTP settings are correct. The procedure is well-documented in the official Proxmox documentation.
What is the difference between an HA Group and an HA Policy?
An HA Group defines a logical grouping of resources (VMs/CTs) and nodes. It allows you to specify on which nodes a group of VMs can run. HA Policies, on the other hand, define the system’s behavior in case of a resource failure, such as how many times to attempt restarting a VM or whether to migrate it to another node. You can have multiple policies for different levels of VM criticality.
Is a QDevice mandatory for a Proxmox HA cluster?
Not strictly mandatory for clusters with more than two nodes, but it is highly recommended for two-node clusters. In a two-node cluster, without a QDevice, if one node fails, there is no majority of votes to maintain quorum, and the cluster completely halts. The QDevice provides an external third vote, allowing the surviving node to maintain the majority and continue operating.
How can I monitor the status of my Proxmox cluster?
For effective monitoring, integrate Prometheus and Grafana. Proxmox exposes metrics via an exporter that Prometheus can collect. Grafana allows you to create custom dashboards and configure alerts for critical events such as quorum loss, excessive resource utilization, or the failure state of HA VMs. This enables you to detect problems before they become service interruptions.
Conclusions with Operational Takeaways
Implementing a Proxmox HA cluster is a fundamental step to ensure operational continuity. However, installation is just the beginning. The real challenge lies in continuous verification and maintenance. Remember: a dedicated network for Corosync, synchronized NTP, and an adequate Quorum Device are non-negotiable pillars. Integrate robust monitoring and, above all, test failover under real-world conditions. Only then can you be sure your cluster will withstand pressure when production is at risk. The peace of mind of a stable IT environment results from meticulous planning and rigorous testing.
Read also: Prometheus Grafana Monitoring: Installation and Critical Alerts (2026)
Read also: SSH Hardening Linux: Complete Guide 2026 (10 Critical Settings)
Updated: July 2026