🛠️ Runbook: Universal Media Storage Architecture & Cross-Node Migration (`tank/media_root`)

To allow containers (such as docker-media LXC 131) to migrate freely between PVE-1, PVE-2, and PVE-3 with zero configuration changes, the cluster enforces a Universal Cluster Mount Standard.

🛑 Why the Standard is Required: The Pre-Start Hook Trap

During cross-node migration testing of LXC 200 from PVE-1 to PVE-3:

PVE-1 originally mounted local /tank/media_root,acl=1.

When LXC 200 was migrated to PVE-3, container startup failed with:

text

run_buffer: 569 Script exited with status 20
lxc_init: 1037 Failed to run lxc.hook.pre-start for container "200"

Root Cause Analysis:
1. NFS Rejection of acl=1: On PVE-1, acl=1 is accepted by local ZFS. On PVE-2 and PVE-3, /mnt/pve/tank_media is an NFSv4 mount. Passing the acl mount option to an NFS mount causes the Linux VFS mount syscall to fail with EINVAL (EOPNOTSUPP), crashing Proxmox's /usr/share/lxc/pve-pre-start-hook with exit code 20.
2. Symlink Disallowance in LXC: Creating a symlink on the host (e.g. /tank/media_root -> /mnt/pve/tank_media) fails because Proxmox LXC security controls disallow host mountpoint paths that are symlinks.

🏛️ The Universal Architecture

To solve this permanently across all 3 nodes:

Standardize the host path on all nodes to /mnt/pve/tank_media.
Standardize all media container configs to use /mnt/pve/tank_media with shared=1 and without acl=1.

📋 Step-by-Step Implementation Guide

On PVE-1 (`192.168.0.100` - Storage Host):

Set the ZFS dataset mountpoint directly to /mnt/pve/tank_media. This natively mounts the dataset upon pool import without relying on /etc/fstab bind mounts or external systemd services, permanently avoiding boot race conditions:

bash

# 1. Set ZFS mountpoint directly on dataset (Native & Persistent across reboots)
zfs set mountpoint=/mnt/pve/tank_media tank/media_root

# 2. Maintain backwards compatibility symlink for host scripts
ln -sfn /mnt/pve/tank_media /tank/media_root

# 3. Configure NFS export for cluster client nodes (PVE-2 & PVE-3)
apt-get update && apt-get install -y nfs-kernel-server
if ! grep -q "/mnt/pve/tank_media" /etc/exports; then
  echo "/mnt/pve/tank_media 192.168.0.0/24(rw,sync,no_subtree_check,no_root_squash)" >> /etc/exports
fi
exportfs -ra
systemctl restart nfs-kernel-server

# 4. Ensure POSIX ACLs are configured for unprivileged LXCs (UID 100000) and nas_shares (GID 110000)
zfs set acltype=posixacl xattr=sa aclmode=passthrough tank/media_root
setfacl -R -m u:100000:rwx,d:u:100000:rwx,g:110000:rwx,d:g:110000:rwx /mnt/pve/tank_media

TIP

Setting zfs set mountpoint=/mnt/pve/tank_media ensures ZFS itself mounts the dataset directly during early pool import before Proxmox starts any LXC guests, eliminating any empty-directory mount issues on reboot.

On PVE-2 & PVE-3 (`192.168.0.101`, `192.168.0.102` - Client Nodes):

Ensure NFS client tools are installed and mount the share persistently with network-wait flags:

bash

apt-get update && apt-get install -y nfs-common
mkdir -p /mnt/pve/tank_media
if ! grep -q "/mnt/pve/tank_media" /etc/fstab; then
  echo "192.168.0.100:/mnt/pve/tank_media /mnt/pve/tank_media nfs4 defaults,_netdev,x-systemd.automount 0 0" >> /etc/fstab
fi
mount -a
ls -la /mnt/pve/tank_media

Step 2: Configure Universal Container Mount Point

For any container requiring media access (e.g. docker-media LXC 131), apply the identical config:

bash

# Apply universal mountpoint to container
pct set <VMID> -mp0 /mnt/pve/tank_media,mp=/mnt/media_root,shared=1

In /etc/pve/lxc/<VMID>.conf:

ini

mp0: /mnt/pve/tank_media,mp=/mnt/media_root,shared=1

NOTE

Why omit acl=1? The host ZFS dataset on PVE-1 already has acltype=posixacl and handles POSIX ACLs natively. Omitting acl=1 keeps the LXC mount compatible with NFS on PVE-2 and PVE-3 while preserving full ACL permissions.
Why shared=1? The shared=1 flag designates this mountpoint as existing across all cluster nodes, allowing Proxmox to live/offline migrate the container without blocking or modifying mount paths.

Step 3: Configure Dual-User Permissions inside the LXC Container

Inside the container (via SSH or pct enter <VMID>):

bash

# 1. Create or ensure nas_shares has GID 10000
sudo groupadd -g 10000 nas_shares 2>/dev/null || sudo groupmod -g 10000 nas_shares

# 2. Grant full access to BOTH rayman-ssh and admin-ssh
sudo usermod -aG nas_shares rayman-ssh
sudo usermod -aG nas_shares admin-ssh

# 3. Verify group memberships
id rayman-ssh
id admin-ssh

# 4. Verify read/write access
ls -la /mnt/media_root

Step 4: Configure Docker Compose Services

When running Docker services inside any media container that read or write to /mnt/media_root:

yaml

services:
  app:
    image: example/app:latest
    environment:
      - PUID=1000       # Matches container user
      - PGID=10000      # nas_shares GID for automatic ACL rwx permissions
      - UMASK=002       # Ensures group write bit is set on created files
    volumes:
      - /mnt/media_root:/data

Step 5: Seamless Cross-Node Migration Runbook

With the universal mount and unified local-zfs pool on all nodes, migrating an LXC takes a single command:

bash

# Example: Migrate LXC 200 from PVE-1 to PVE-3 with automatic restart
pct migrate 200 pve3 --restart

# Example: Migrate LXC 200 back to PVE-1
pct migrate 200 pve --restart

Proxmox replicates the rootfs via native ZFS send/receive (e.g. 47 GB transferred at ~500 MB/s), rebinds /mnt/pve/tank_media, and boots the container on the target node in seconds.

Step 6: Automated Mount Health Monitoring (Uptime Kuma Watchdog)

To ensure proactive notification if storage becomes unmounted or empty, a standalone one-liner cron job runs every minute under admin-ssh (or rayman-ssh) on docker-media (192.168.0.131):

Standalone Crontab Entry (crontab -e on docker-media):

text

* * * * * (df /mnt/media_root 2>/dev/null | grep -q "tank/media_root" && test -d /mnt/media_root/media) && curl -fsS -m 5 "http://192.168.0.183:3023/api/push/aE6xGaF4jD0ZlA1FWYVPpQdUbB37Zl2O?status=up&msg=OK&ping=" >/dev/null 2>&1 || curl -fsS -m 5 "http://192.168.0.183:3023/api/push/aE6xGaF4jD0ZlA1FWYVPpQdUbB37Zl2O?status=down&msg=StorageUnmounted" >/dev/null 2>&1

Uptime Kuma Configuration:
- Type: Push
- Name: Media Root ZFS Mount (docker-media)
- Heartbeat: 60s
- Alert: Triggers immediate push notification if a heartbeat is missed or status=down is reported.

🛠️ Runbook: Universal Media Storage Architecture & Cross-Node Migration (tank/media_root) ​

🛑 Why the Standard is Required: The Pre-Start Hook Trap ​

🏛️ The Universal Architecture ​

📋 Step-by-Step Implementation Guide ​

Step 1: Configure Host Mounts & NFS Sharing ​

On PVE-1 (192.168.0.100 - Storage Host): ​

On PVE-2 & PVE-3 (192.168.0.101, 192.168.0.102 - Client Nodes): ​

Step 2: Configure Universal Container Mount Point ​

Step 3: Configure Dual-User Permissions inside the LXC Container ​

Step 4: Configure Docker Compose Services ​

Step 5: Seamless Cross-Node Migration Runbook ​

Step 6: Automated Mount Health Monitoring (Uptime Kuma Watchdog) ​

🔗 Related Documentation ​

🛠️ Runbook: Universal Media Storage Architecture & Cross-Node Migration (`tank/media_root`)

🛑 Why the Standard is Required: The Pre-Start Hook Trap

🏛️ The Universal Architecture

📋 Step-by-Step Implementation Guide

Step 1: Configure Host Mounts & NFS Sharing

On PVE-1 (`192.168.0.100` - Storage Host):

On PVE-2 & PVE-3 (`192.168.0.101`, `192.168.0.102` - Client Nodes):

Step 2: Configure Universal Container Mount Point

Step 3: Configure Dual-User Permissions inside the LXC Container

Step 4: Configure Docker Compose Services

Step 5: Seamless Cross-Node Migration Runbook

Step 6: Automated Mount Health Monitoring (Uptime Kuma Watchdog)

🔗 Related Documentation