Proxmox Cluster Node Shows Offline – Fix Step by Step

First Test – 30 Seconds

Open a shell on any node that's still online. Run pvecm status. It shows the cluster state and member IPs. If your offline node isn't listed, or shows 'offline', the real question is: can the other nodes reach it?

pvecm status
# Output example:
# Membership information
#    Nodeid      Votes Name
# 1          1 node01 (192.168.1.10)
# 2          0 node02 (192.168.1.11)   # vote 0 means offline

Now ping the offline node's IP from an online node. Use its corosync IP, not the management IP (they're often the same, but not always).

ping -c 3 192.168.1.11

If no reply, skip to the network section. If it replies, the issue is likely corosync itself. Try restarting corosync on the offline node (if you can SSH in):

systemctl restart corosync

Then run pvecm status again. If the node comes back online, you're done. This happens sometimes after a node reboot without waiting for cluster sync. But if it stays offline after restart, move on.

Moderate Fix – 5 Minutes: Check Corosync Config and Firewall

Corosync uses UDP port 5404 and 5405 by default, plus multicast or unicast. Proxmox 6+ uses unicast (knet) by default. The most common reason a node shows offline is a firewall rule blocking the corosync traffic.

On each node, check if the firewall is enabled:

pve-firewall status
# or check via datacenter view in web UI

If it's on, create a rule to allow corosync. In the Proxmox web UI, go to Datacenter -> Firewall -> Rules, add 'Accept' for source IP of each cluster node, destination IP of each node, protocol UDP, port 5404-5405. Or do it from CLI:

# on each node, edit /etc/pve/firewall/cluster.fw
# Add lines like:
# [RULES]
# GROUP corosync

But the simpler test: temporarily disable the firewall on the offline node and see if it comes back. Only do this if you have physical or IPMI access, because remote SSH might also be blocked.

systemctl stop pve-firewall
systemctl stop pvefw-logger
pvecm status

If the node shows online now, the firewall was the problem. Re-enable it and configure the rules properly.

Next, check the corosync config on the offline node. It lives at /etc/pve/corosync.conf (this is synced across the cluster, but local node might have a stale copy). You can view it with:

cat /etc/pve/corosync.conf

Look for the 'interface' section. Make sure the bindnetaddr matches the IP on the network interface corosync uses. For example, if your node IP is 192.168.1.11 but bindnetaddr is 192.168.100.0, it won't work. That mismatch happens after you move a node to a new subnet.

nodelist {
  node {
    name: node02
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.1.11
  }
}

If the ring0_addr is wrong, you need to fix it. But you can't edit that file directly – Proxmox regenerates it from the cluster database. The proper way to change a node's corosync IP is pvecm updatecerts or re-adding the node. But for a quick fix, you can stop corosync, edit the file temporarily, restart. Warning: that might cause a split-brain if done wrong. Better to do the advanced fix below.

Advanced Fix – 15+ Minutes: Rebuild Corosync and Rejoin Node

If nothing above works, the corosync state on the offline node is probably corrupted. What's happening here is the node can't sync its token ring with the rest. The nuclear option is to remove the node from the cluster and re-add it. Backup your VMs and CTs first – especially if this node is a quorum voter.

Step 1: On the offline node, stop corosync and pve-cluster:

systemctl stop pve-cluster
systemctl stop corosync
pmxcfs -l   # start local cluster filesystem in local mode

Step 2: Delete the corosync config files:

rm -f /etc/corosync/corosync.conf
rm -rf /var/lib/corosync/*

Step 3: Stop the local cluster fs:

killall pmxcfs
systemctl start pve-cluster   # this will start fresh, no cluster membership

Now this node is isolated. On another online node, remove the offline node from the cluster:

pvecm delnode node02

If the command fails because the node is still in corosync's expected list, use the force flag:

pvecm delnode --force node02

Step 4: On the offline node, join the cluster again. You'll need the cluster join token from an online node:

# on online node
pvecm join add node02
# This prints a join token like: 1234567890abcdef:node01

Then on the offline node, actually join:

pvecm add 192.168.1.10 --force 1
# or use the token
pvecm add 192.168.1.10 --token 1234567890abcdef:node01

Enter the root password for the online node when prompted. After a few seconds, run pvecm status – you should see both nodes online.

One more thing: check quorum. If you have only 2 nodes, you need a third witness node (a tiny VM) or set expected_votes to 2 in corosync.conf. Without quorum, nodes will show offline after a few minutes. Fix quorum with:

pvecm expected 2
# for 2-node cluster without witness

Add that to /etc/pve/corosync.conf so it survives reboot:

totem {
  version: 2
  crypto_cipher: none
  crypto_hash: none
  interface {
    ...
  }
}
quorum {
  expected_votes: 2
}

That's it. If the node still shows offline after this, you have a deeper network issue – check VLANs, MTU, or physical link. But 90% of cases are fixed by one of these three steps.