🌐 Cluster Network Architecture, Dual-NIC Bonding & Wake-on-LAN

To eliminate network single points of failure, remove redundant bridges (vmbr1), and maximize throughput, cluster nodes are configured with Active-Backup Linux Bonding (bond0) attached to a single primary bridge (vmbr0), paired with hardware-level Wake-on-LAN (WoL).

🚀 Design & Failover Strategy

Bond Mode: active-backup (Mode 1).
Primary Slave: nic1 (2.5 Gbps Ethernet interface).
Secondary / Standby Slave: nic0 (1.0 Gbps Onboard Ethernet interface).
Link Monitoring: MII link monitoring every 100ms (bond-miimon 100).
Failover Logic: PVE utilizes nic1 (2.5G) at all times for maximum bandwidth. If nic1 link drops, the Linux bonding kernel driver instantaneously shifts traffic to nic0 (1G) with zero connection drop or packet loss. When nic1 recovers, traffic automatically shifts back to the 2.5G link (primary_reselect always).
Unified Bridge: vmbr0 bridges bond0, carrying management and all guest VNICs (veth*, tap*). The deprecated secondary bridge vmbr1 has been completely decommissioned and removed across the cluster.

🏷️ Interface Naming Standardization (`systemd .link`)

To guarantee that network interface names remain deterministic across kernel updates, PCI enumeration shifts, and reboots:

/etc/systemd/network/10-nic0.link binds MAC of the onboard 1G interface to nic0.
/etc/systemd/network/10-nic1.link binds MAC of the 2.5G interface to nic1.

Example /etc/systemd/network/10-nic0.link:

ini

[Match]
MACAddress=9c:7b:ef:2b:78:12

[Link]
Name=nic0
WakeOnLan=magic

⚙️ Node Network Configurations (`/etc/network/interfaces`)

PVE-1 (`pve`, `192.168.0.100`)

text

auto lo
iface lo inet loopback

iface nic0 inet manual
	post-up /usr/sbin/ethtool -s nic0 wol g

iface nic1 inet manual

auto bond0
iface bond0 inet manual
	bond-slaves nic0 nic1
	bond-miimon 100
	bond-mode active-backup
	bond-primary nic1

auto vmbr0
iface vmbr0 inet static
	address 192.168.0.100/24
	gateway 192.168.0.1
	bridge-ports bond0
	bridge-stp off
	bridge-fd 0

PVE-2 (`pve2`, `192.168.0.101`)

text

auto lo
iface lo inet loopback

iface nic0 inet manual
	post-up /usr/sbin/ethtool -s nic0 wol g

iface nic1 inet manual

auto bond0
iface bond0 inet manual
	bond-slaves nic0 nic1
	bond-miimon 100
	bond-mode active-backup
	bond-primary nic1

auto vmbr0
iface vmbr0 inet static
	address 192.168.0.101/24
	gateway 192.168.0.1
	bridge-ports bond0
	bridge-stp off
	bridge-fd 0

PVE-3 (`pve3`, `192.168.0.102`)

text

auto lo
iface lo inet loopback

iface nic0 inet manual
	post-up /usr/sbin/ethtool -s nic0 wol g

auto vmbr0
iface vmbr0 inet static
	address 192.168.0.102/24
	gateway 192.168.0.1
	bridge-ports nic0
	bridge-stp off
	bridge-fd 0

iface nic1 inet manual

source /etc/network/interfaces.d/*

NOTE

PVE-3 Hold-Off Status: PVE-3 currently operates with single interface nic0 (1GbE) attached directly to vmbr0 (192.168.0.102/24). Dual-NIC bonding on PVE-3 will be applied once an external USB 2.5GbE adapter is physically attached and provisioned as nic1. Do not apply bond0 referencing nic1 before the adapter is plugged in to avoid missing-device configuration warnings.

⚡ Wake-on-LAN (WoL) Configuration & Cluster Power Management

To allow powering on cluster nodes remotely via magic packets (from the Proxmox Web GUI, Home Assistant, automation scripts, or peer node CLI via pvenode wakeonlan), Wake-on-LAN must be explicitly enabled at the physical NIC and driver layer.

Critical Architecture Rules:

Physical Member NIC Target: In bonded (bond0) or bridged (vmbr0) topologies, magic packets are captured by the physical NIC hardware while the system is in ACPI S5 power-off (powered by motherboard 5VSB standby power). WoL cannot be enabled on virtual interfaces (bond0 or vmbr0)—it must be assigned to the physical onboard interface (nic0).
Dual-Layer Persistence:
- ifupdown2 Hook (/etc/network/interfaces): Includes post-up /usr/sbin/ethtool -s nic0 wol g under iface nic0 inet manual to re-assert the magic-packet flag whenever networking restarts.
- systemd-udevd Link Configuration (/etc/systemd/network/10-nic0.link): Add WakeOnLan=magic under the [Link] section so systemd enables WoL at hardware discovery, ensuring WoL remains enabled across soft reboots and kernel power state transitions:
  ini
```
[Match]
MACAddress=<NODE_NIC0_MAC>

[Link]
Name=nic0
WakeOnLan=magic
```
Prerequisite Package: Ensure ethtool is installed on all nodes:
bash
```
apt-get update && apt-get install -y ethtool
```

Node WoL Inventory

Node	Hostname / IP	Target WoL NIC	MAC Address	Interface Mapping History
PVE-1	`pve` (`192.168.0.100`)	`nic0` (Onboard 1G)	`9c:7b:ef:2b:78:12`	Formerly `eno1`
PVE-2	`pve2` (`192.168.0.101`)	`nic0` (Onboard 1G)	`7c:57:58:ff:58:20`	Deterministic via `10-nic0.link`
PVE-3	`pve3` (`192.168.0.102`)	`nic0` (Onboard 1G)	`7c:57:58:ff:50:55`	Formerly `enp2s0`

Proxmox Cluster WoL Integration

To allow waking nodes directly from the Proxmox Datacenter Web UI or from peer nodes:

Register the WoL MAC address in /etc/pve/nodes/<nodename>/config:
bash
```
# On PVE-3:
echo "wakeonlan: 7c:57:58:ff:50:55" >> /etc/pve/nodes/pve3/config
```
(Or in Proxmox Web GUI: <node> $\rightarrow$ System $\rightarrow$ Options $\rightarrow$ Wake-on-LAN MAC address).

To wake a sleeping or powered-off node from any cluster shell:

bash

# Using Proxmox cluster utility:
pvenode wakeonlan pve3

# Or standard broadcast magic packet:
wakeonlan 7c:57:58:ff:50:55

Verification

To confirm Wake-on-LAN is active on any node:

bash

ethtool nic0 | grep -i wake

Expected output:

text

Supports Wake-on: pumbg
Wake-on: g

(Flag g signifies MagicPacket™ mode).

🔍 Bonding Verification & Status Check

To inspect real-time bond status and failover health:

bash

cat /proc/net/bonding/bond0

Expected output confirms Bonding Mode: fault-tolerance (active-backup), Primary Slave: nic1, and current active slave.

To apply changes cleanly without reboot:

bash

ifreload -a

⚡ Realtek 2.5GbE (RTL8125) Driver & PHY Optimization

To prevent link flapping, 90%+ packet loss, or negotiation delays on nodes utilizing Realtek 2.5GbE NICs (nic1):

Driver: Use the standalone out-of-tree DKMS module (r8125-dkms) rather than the in-tree r8169 driver.
EEE Disabled: Energy Efficient Ethernet must be disabled via post-up /sbin/ethtool --set-eee nic1 eee off in /etc/network/interfaces.
PCIe Power Management: Disable PCIe ASPM via pcie_aspm=off in /etc/kernel/cmdline (proxmox-boot-tool refresh).

For the complete diagnostic and installation guide, see the Realtek 2.5GbE DKMS Runbook.

🌐 Cluster Network Architecture, Dual-NIC Bonding & Wake-on-LAN ​

🚀 Design & Failover Strategy ​

🏷️ Interface Naming Standardization (systemd .link) ​

⚙️ Node Network Configurations (/etc/network/interfaces) ​

PVE-1 (pve, 192.168.0.100) ​

PVE-2 (pve2, 192.168.0.101) ​

PVE-3 (pve3, 192.168.0.102) ​

⚡ Wake-on-LAN (WoL) Configuration & Cluster Power Management ​

Critical Architecture Rules: ​

Node WoL Inventory ​

Proxmox Cluster WoL Integration ​

Verification ​

🔍 Bonding Verification & Status Check ​

⚡ Realtek 2.5GbE (RTL8125) Driver & PHY Optimization ​

🔗 Related Documentation ​

🌐 Cluster Network Architecture, Dual-NIC Bonding & Wake-on-LAN

🚀 Design & Failover Strategy

🏷️ Interface Naming Standardization (`systemd .link`)

⚙️ Node Network Configurations (`/etc/network/interfaces`)

PVE-1 (`pve`, `192.168.0.100`)

PVE-2 (`pve2`, `192.168.0.101`)

PVE-3 (`pve3`, `192.168.0.102`)

⚡ Wake-on-LAN (WoL) Configuration & Cluster Power Management

Critical Architecture Rules:

Node WoL Inventory

Proxmox Cluster WoL Integration

Verification

🔍 Bonding Verification & Status Check

⚡ Realtek 2.5GbE (RTL8125) Driver & PHY Optimization

🔗 Related Documentation