💾 Automated "Plug-and-Run" Offline Cold Backup Pipeline

This runbook documents the fully automated, zero-command offline cold backup architecture for PVE-1 (pve, 192.168.0.100), backing up the Immich photo library and homelab database dumps to a physically isolated 2TB Western Digital hard drive upon hardware insertion.

🏛️ Architecture & Operational Workflow

The cold backup pipeline implements the final tier of the homelab 3-2-1 backup strategy (air-gapped offline storage resistant to ransomware and physical disasters). The drive remains stored in a drawer until plugged in.

💽 Hardware & Storage Specification

Property	Value	Notes
Target Host	PVE-1 (`pve`, `192.168.0.100`)	Proxmox VE 9.2.11 / Debian 12
Drive Model	Western Digital 2TB (`WDC_WD20SDZW-11JJ8S0`)	5400 RPM 2.5" HDD
Drive Serial	`WD-WXL1EC84X4KR` (USB: `57584C314543383458344B52`)	Unique hardware identifier
By-ID Path	`/dev/disk/by-id/ata-WDC_WD20SDZW-11JJ8S0_WD-WXL1EC84X4KR`	Partition: `...-part1`
Filesystem	`ext4` (GPT)	Mount point: `/mnt/cold_backup`
Source Datasets	1. `/tank/media_root/backups/` 2. `/tank/media_root/media/photos/`	High-value air-gapped data
Workload Host	LXC 131 (`docker-media`)	Immich stack & database backups on mass storage
Telegram Bot	`@proxmox_pve_bot`	Token: `7245922764:AAGyR...`
Home Assistant Webhook	`http://192.168.0.199/api/webhook/cold_backup_status`	Automation: `automation.cold_backup_status_via_webhook`

⚙️ Key Pipeline Capabilities

1. Pre-Flight SMART Drive Health Checks

Before starting any rsync operations, the script queries disk hardware status via smartctl:

Overall Self-Assessment: Passed vs Failed.
Critical Attributes: Reallocated Sector Count (ID 5), Current Pending Sectors (ID 197), Offline Uncorrectable (ID 198).
Drive Telemetry: Temperature (°C) and Power-On Hours.
Any detected hardware anomalies are immediately flagged in both Telegram notifications and Home Assistant entities.

2. 🛡️ Bit Rot Protection & Checksum Integrity

Pre-Sync Verification: Validates existing SHA256 checksums in /mnt/cold_backup/.checksums/ against database dumps and a representative sampling of photos before syncing new data.
Post-Sync Generation: Generates updated SHA256 manifests (backups_manifest.sha256, photos_sample.sha256) for newly synced files.
Audit Logging: Appends integrity verification results to /mnt/cold_backup/.checksums/bitrot_audit.log.

3. 🏠 Home Assistant Integration & Dashboard Entities

The script dispatches real-time JSON payloads to http://192.168.0.199/api/webhook/cold_backup_status on start, finish, failure, and physical disconnect.

Home Assistant Entity	Helper Type	Purpose / Value
`input_boolean.cold_backup_running`	`input_boolean`	Active execution state (`on` / `off`)
`input_boolean.cold_backup_drive_connected`	`input_boolean`	Physical USB drive attached (`on` / `off`)
`input_boolean.cold_backup_warning`	`input_boolean`	Raised on SMART or Bit Rot warnings (`on` / `off`)
`input_text.cold_backup_status`	`input_text`	High-level status text (e.g. `Idle (Safe to Unplug)`, `Running`)
`input_text.cold_backup_drive_serial`	`input_text`	Connected drive serial (`WD-WXL1EC84X4KR`)
`input_text.cold_backup_smart_health`	`input_text`	SMART health & temperature (`PASSED (31°C)`)
`input_text.cold_backup_bit_rot_status`	`input_text`	Integrity verification summary (`✅ 18 files verified (0 errors)`)
`input_text.cold_backup_duration`	`input_text`	Last backup duration (`2m 4s`)
`input_text.cold_backup_net_change`	`input_text`	Storage change delta (`+84.9 GiB`)
`input_text.cold_backup_free_space`	`input_text`	Free space remaining on drive (`1.7T`)
`input_datetime.cold_backup_last_completed`	`input_datetime`	Timestamp of last successful backup
`automation.homelab_cold_backup_reminder`	`automation`	Weekly reminder (Sun 18:00) if backup > 30 days old

4. Drive State Persistence & History Comparison

Upon mounting /mnt/cold_backup:

Reads .backup_state.json on the drive and calculates elapsed time (e.g. 1d 4h ago).
Persists updated metadata JSON on completion and appends to .backup_history.log.

5. Graceful Container Quiescing (LXC 132)

Gracefully initiates shutdown (pct shutdown 132 --timeout 60).
Explicitly polls and verifies the container status reaches status: stopped before synchronization proceeds.
Restores Immich (pct start 132) immediately after sync completes.
Error traps guarantee container restart even if an unexpected error occurs.

6. 🛡️ Defense Safeguards (Data Loss & Loop Prevention)

Guard 1 (Source Sanity Check): Before running rsync, verifies that source directories exist, are readable, and contain at least 3 files. If a source dataset is empty (e.g. from an unmounted mount or accidental deletion), backup is immediately aborted to prevent wiping the destination cold drive.
Guard 2 (ZFS Storage Pool Validation): Validates that ZFS pool tank health is ONLINE and dataset tank/media_root is actively mounted (zfs get mounted tank/media_root == yes) before running any operations.
Guard 3 (3-Hour Cooldown Safeguard & Insertion Debounce):
- Insertion Debounce: 10-second debounce lock (/run/cold_backup_insert.lock) prevents USB bounce/flapping triggers when plugging in or wiggling cables.
- 3-Hour Cooldown: Upon mounting, reads .backup_state.json. If a successful backup was completed within the last 3 hours (< 10,800s), the script sends an informational Telegram & Home Assistant notification, unmounts, spins the drive down into 0 RPM standby (hdparm -y), and exits cleanly without stopping containers or writing data. Overridable with --force or -f.
Guard 4 (Max Deletion Cap): Runs a prospective dry-run rsync check to count file deletions. If deletions exceed MAX_DELETION_LIMIT (default 250 files), the backup halts immediately with a critical alert listing a sample of deleted files and providing the exact CLI inspection command (rsync -aAX -n --delete -i ... | grep '^\*deleting'). Overridable with --force-deletion or --max-deletions=N.
Guard 5 (Safe Recovery / Restore Mode): Dedicated non-destructive --restore mode that reverses the sync direction (/mnt/cold_backup/ -> /tank/media_root/) without --delete, verifies SHA256 checksums first, and quiesces LXC 132 during photo/backup restoration.

🚀 Deployment Instructions (On PVE-1 Host)

Run the following commands directly on PVE-1 (192.168.0.100) as root:

bash

# 1. Install prerequisites (if not present)
apt-get update && apt-get install -y rsync curl udisks2 hdparm smartmontools

# 2. Copy script to /usr/local/bin and set permissions
cp scripts/cold_backup.sh /usr/local/bin/cold_backup.sh
chmod 755 /usr/local/bin/cold_backup.sh

# 3. Copy systemd services and reload daemon
cp scripts/systemd/cold-backup.service /etc/systemd/system/cold-backup.service
cp scripts/systemd/cold-backup-unplug.service /etc/systemd/system/cold-backup-unplug.service
systemctl daemon-reload

# 4. Copy udev rules and reload udev
cp scripts/udev/99-cold-backup.rules /etc/udev/rules.d/99-cold-backup.rules
udevadm control --reload-rules && udevadm trigger

# 5. Create mount point directory
mkdir -p /mnt/cold_backup

🧪 CLI Testing & Operations Reference

bash

# Full Production Run (Default behavior upon plug-in)
/usr/local/bin/cold_backup.sh

# Fast Test Mode (Syncs small backup dumps only, skips LXC quiesce, keeps mounted)
/usr/local/bin/cold_backup.sh --test

# Dry Run Simulation (Simulates sync, zero disk writes)
/usr/local/bin/cold_backup.sh --dry-run

# Bypass 3-Hour Cooldown (Force immediate backup run)
/usr/local/bin/cold_backup.sh --force

# Override Deletion Cap (For intentional bulk deletions)
/usr/local/bin/cold_backup.sh --force-deletion

# Restore all targets from cold backup drive to ZFS (/tank/media_root)
/usr/local/bin/cold_backup.sh --restore

# Restore only Immich photos from cold backup drive
/usr/local/bin/cold_backup.sh --restore-photos

# Restore only system and database dumps
/usr/local/bin/cold_backup.sh --restore-backups

🔔 Telegram Notifications Reference

1. Drive Connected (First Message — Includes Last Backup Time)

🔌 [PVE-1] Cold Backup Drive Connected
📦 Drive: WD 2TB HDD (WD-WXL1EC84X4KR)
⚙️ Mode: PRODUCTION MODE
⏰ Connected At: 2026-10-03 14:00:00 BST
⏮️ Last Backup: 3 days ago (2026-09-30 18:30:24 BST)
📍 Device: /dev/sde
⏳ Running SMART health assessment & bit rot integrity audit...

2. Drive Connected (3-Hour Cooldown Active — Backup Skipped)

🔌 [PVE-1] Cold Backup Drive Connected — Cooldown Active
📦 Drive: WD 2TB HDD (WD-WXL1EC84X4KR)
⏰ Detected At: 2026-10-03 14:00:00 BST
⏮️ Last Backup: 45 min(s) ago (2026-10-03 13:15:00 BST)
⏳ Cooldown: 3-Hour Safeguard Active (2h 15m 0s remaining)
🛑 Backup skipped to prevent unnecessary duplicate runs. Drive will safely unmount and spin down.
💡 To override & run anyway: /usr/local/bin/cold_backup.sh --force

3. Pre-Flight Checks Passed — Backup Starting

🚀 [PVE-1] Pre-Flight Checks Passed — Backup Starting
📦 Drive: WD 2TB HDD (WD-WXL1EC84X4KR)
⚙️ Mode: Full Live Backup
⏰ Detected At: 2026-10-03 14:00:00 BST
⏮️ Last Backup: 3 days ago (2026-09-30 18:30:24 BST)
🩺 SMART Health: ✅ PASSED (23°C | Power-On: 38919h | Realloc: 0)
🛡️ Bit Rot Audit: ✅ 240 files verified (0 errors)
📂 Backup Targets:
• 🗄️ System & DB Backups: /tank/media_root/backups
• 📸 Immich Photos: /tank/media_root/media/photos
⏳ Quiescing containers & syncing data to cold storage...

4. Backup Completed

✅ [PVE-1] Cold Backup Complete!
🔌 Drive: WD 2TB HDD (WD-WXL1EC84X4KR)
⏱️ Total Duration: 2m 4s
🩺 SMART Health: ✅ PASSED (32°C | Realloc: 0)
🛡️ Bit Rot Integrity: ✅ 18 files verified (0 errors)
📊 Disk Space & Delta:
• Before: 1.8T free (520M used / 1%)
• After: 1.7T free (85.4G used / 5%)
• Net Change: +84.9 GiB
📋 Results:
• 🗄️ System & DB Backups: ✅ Synced (0m 5s)
• 📸 Immich Photos: ✅ Synced (1m 58s)
• 🔄 Immich (LXC 132): Quiesced & restored online
💾 Drive State: Metadata, history & checksum manifests saved to cold storage.
🔒 Safety: Filesystem flushed, cleanly unmounted, and drive spun down.
🟢 SAFE TO UNPLUG: Safe to disconnect the drive and return to cold storage.

5. Physical Disconnect

🔌 [PVE-1] Cold Backup Drive Unplugged
📦 Drive: WD 2TB HDD (WD-WXL1EC84X4KR)
🔒 Status: Physically disconnected from host USB port.
🟢 Cold Storage: Drive safely isolated offline.

6. Deletion Guard Alert (Data Protection Triggered)

🚨 [PVE-1] Cold Backup ABORTED — Deletion Guard Triggered!
⚠️ Target: 📸 Immich Photos (/mnt/cold_backup/photos)
🗑️ Prospective Deletions: 312 files (Safety Limit: 250)
🛡️ Action: Backup halted immediately to protect cold drive archives.
📄 Sample Files to be Deleted:
• photos/2026/08/IMG_0192.jpg
• photos/2026/08/IMG_0193.jpg
• photos/2026/08/IMG_0194.jpg
... and 309 more file(s)
🔍 Inspect All Deleted Files on PVE:
rsync -aAX -n --delete -i --exclude=thumbs/** --exclude=encoded-video/** /tank/media_root/media/photos/ /mnt/cold_backup/photos/ | grep '^\*deleting'
💡 To Override & Accept Deletions:
/usr/local/bin/cold_backup.sh --force-deletion
Or raise threshold: /usr/local/bin/cold_backup.sh --max-deletions=500

💾 Automated "Plug-and-Run" Offline Cold Backup Pipeline ​

🏛️ Architecture & Operational Workflow ​

💽 Hardware & Storage Specification ​

⚙️ Key Pipeline Capabilities ​

1. Pre-Flight SMART Drive Health Checks ​

2. 🛡️ Bit Rot Protection & Checksum Integrity ​

3. 🏠 Home Assistant Integration & Dashboard Entities ​

4. Drive State Persistence & History Comparison ​

5. Graceful Container Quiescing (LXC 132) ​

6. 🛡️ Defense Safeguards (Data Loss & Loop Prevention) ​

🚀 Deployment Instructions (On PVE-1 Host) ​

🧪 CLI Testing & Operations Reference ​

🔔 Telegram Notifications Reference ​

1. Drive Connected (First Message — Includes Last Backup Time) ​

2. Drive Connected (3-Hour Cooldown Active — Backup Skipped) ​

3. Pre-Flight Checks Passed — Backup Starting ​

4. Backup Completed ​

5. Physical Disconnect ​

6. Deletion Guard Alert (Data Protection Triggered) ​

🔗 Related Documentation ​