⚓ Dockhand Central Management & 2-Way GitOps Sync

This document details the centralized Docker management architecture using Dockhand and Hawser, and the automated 2-way Git synchronization between /opt/stacks/ on docker-edge (LXC 183) and the homelab-ops GitHub repository.

🏛️ Architecture Overview

The homelab uses a Centralized Control Plane model. All Compose stack definitions and environment files are stored centrally on docker-edge (where Dockhand runs), while container workloads execute on their respective nodes via lightweight Hawser agents.

📁 Directory Structure (`/opt/stacks/`)

All stack definitions are organized by target execution domain under /opt/stacks/:

text

/opt/stacks/
├── edge/                     # Stacks running locally on LXC 183 (docker-edge)
│   ├── traefik/              # compose.yaml & .env
│   ├── cloudflared/          # compose.yaml
│   ├── auth/                 # Pocket-ID & TinyAuth
│   ├── crowdsec/             # Security engine & log bouncer
│   ├── glance/               # Dashboard
│   ├── termix/               # Web terminal
│   └── dockhand/             # Dockhand controller compose
├── admin/                    # Stacks running remotely on LXC 129 (docker-admin)
│   ├── seq/                  # Seq & Syslog ingestion
│   ├── automation/           # n8n & Bambu Studio API
│   ├── diagnostics/          # Speedtest tracker & OpenSpeedTest
│   ├── mealie/               # Recipe manager
│   ├── obsidian/             # CouchDB LiveSync
│   └── monitoring/           # Dozzle / Uptime Kuma
└── media/                    # Stacks running remotely on LXC 131 (docker-media)
    ├── immich/               # Immich server, ML, Valkey, Postgres, Power Tools, Pet Tagger
    ├── arr-main/             # Radarr, Sonarr, Prowlarr, Bazarr, Seerr
    ├── arr-anime/            # Anime Arrs suite
    ├── downloads/            # Gluetun VPN, qBittorrent, Sabnzbd
    ├── streaming/            # Plex, Jellyfin, Watchstate
    ├── filebrowser/          # File management
    └── gaming/               # RomM & MariaDB

🛠️ Step-by-Step Setup: 2-Way Git Sync (`admin-ssh` Account)

All commands are executed as the admin-ssh user with sudo privileges.

Step 1: Generate SSH Deploy Key for GitHub

SSH into docker-edge as admin-ssh:
bash
```
ssh admin-ssh@192.168.0.183
```

Generate a dedicated Ed25519 SSH keypair:

bash

ssh-keygen -t ed25519 -C "docker-edge-stacks-sync" -f ~/.ssh/id_ed25519_stacks -N ""

Display the public key:
bash
```
cat ~/.ssh/id_ed25519_stacks.pub
```
Add the key to GitHub:
- Navigate to: https://github.com/malkinskir/homelab-ops $\rightarrow$ Settings $\rightarrow$ Deploy keys $\rightarrow$ Add deploy key.
- Title: docker-edge-stacks-sync
- Key: (Paste output of public key)
- Allow write access: ✅ Checked (Required for 2-way sync)
- Click Add key.

Configure SSH client configuration (~/.ssh/config):

bash

cat << 'EOF' >> ~/.ssh/config
Host github.com
    IdentityFile ~/.ssh/id_ed25519_stacks
    StrictHostKeyChecking accept-new
EOF
chmod 600 ~/.ssh/config

Test GitHub authentication:

bash

ssh -T git@github.com
# Expected output: Hi malkinskir/homelab-ops! You've successfully authenticated...

Step 2: Configure Directory Permissions & Git

Ensure /opt/stacks has proper ownership and initialize the repository:

bash

# Set ownership and group permissions (persists multi-user editing for admin-ssh, rayman-ssh, and Dockhand)
sudo chown -R admin-ssh:docker /opt/stacks
sudo chmod -R 775 /opt/stacks

# Configure Git user and safe directory
git config --global user.name "Rey (docker-edge)"
git config --global user.email "malkinskir@gmail.com"
git config --global init.defaultBranch main
git config --global --add safe.directory /opt/stacks

# Initialize repository in /opt/stacks with group sharing
cd /opt/stacks
git init
git config core.sharedRepository group
git remote add origin git@github.com:malkinskir/homelab-ops.git

Step 3: Create the 2-Way Sync Script

Create the sync script at /usr/local/bin/git-sync-stacks.sh:

bash

sudo tee /usr/local/bin/git-sync-stacks.sh > /dev/null << 'EOF'
#!/bin/bash
set -e

STACKS_DIR="/opt/stacks"
cd "$STACKS_DIR"

# 1. Fetch remote changes
git fetch origin main

# 2. Check for local modifications (made via Dockhand GUI or SSH)
if [ -n "$(git status --porcelain)" ]; then
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] Local changes detected. Committing..."
    git add -A
    git commit -m "chore(stacks): auto-sync from docker-edge [$(date '+%Y-%m-%d %H:%M')]"
fi

# 3. Pull incoming changes with rebase
git pull --rebase origin main

# 4. Push local commits to GitHub
git push origin main

echo "[$(date '+%Y-%m-%d %H:%M:%S')] 2-way sync completed successfully."
EOF

sudo chmod +x /usr/local/bin/git-sync-stacks.sh

Step 4: Automate via Systemd Service & Timer

Using a non-root systemd unit running as admin-ssh:

1. Create Systemd Service (`/etc/systemd/system/git-sync-stacks.service`):

bash

sudo tee /etc/systemd/system/git-sync-stacks.service > /dev/null << 'EOF'
[Unit]
Description=2-Way Git Sync for /opt/stacks to GitHub
After=network-online.target

[Service]
Type=oneshot
User=admin-ssh
Group=docker
WorkingDirectory=/opt/stacks
ExecStart=/usr/local/bin/git-sync-stacks.sh
StandardOutput=journal
StandardError=journal
EOF

2. Create Systemd Timer (`/etc/systemd/system/git-sync-stacks.timer`):

bash

sudo tee /etc/systemd/system/git-sync-stacks.timer > /dev/null << 'EOF'
[Unit]
Description=Run 2-Way Git Sync for /opt/stacks every 15 minutes

[Timer]
OnBootSec=5min
OnUnitActiveSec=15min
Persistent=true

[Install]
WantedBy=timers.target
EOF

3. Enable and Start:

bash

sudo systemctl daemon-reload
sudo systemctl enable --now git-sync-stacks.timer

🔍 Verification & Monitoring Commands

Task	Command
Run Manual Sync	`/usr/local/bin/git-sync-stacks.sh`
View Live Sync Logs	`journalctl -u git-sync-stacks.service -f`
Check Next Scheduled Run	`systemctl list-timers git-sync-stacks.timer`
Check Service Status	`systemctl status git-sync-stacks.timer`

🤖 Automated Ansible Monitoring Integration

The sync timer, service status, and script executable state on docker-edge are actively monitored across the homelab via the Ansible monitoring role (roles/monitoring/tasks/main.yml):

Checks Performed on docker-edge:
1. /usr/local/bin/git-sync-stacks.sh is present and executable (+x).
2. git-sync-stacks.timer is active and scheduled.
3. git-sync-stacks.service has not failed on its last run.
Alert Channel: If any component fails or deactivates, a Telegram alert is sent immediately: ❌ dockhand sync: /opt/stacks 2-way git sync issue on docker-edge

🏷️ Dockhand Container Labels Standard

Dockhand uses an opt-out model via Docker container labels. All compose files across edge/, admin/, and media/ implement standardized labels:

Label	Scope / Values	Operational Purpose
`dockhand.update=false`	Infrastructure (`dockhand`, `hawser`, `traefik`, `crowdsec`, `cloudflared`, `gluetun`, `db`, `redis`)	Excludes stateful DBs and bootstrap daemons from automated/bulk updates in Dockhand.
`dockhand.notify=false`	Infrastructure & Agents	Suppresses noisy notifications for internal restart/lifecycle events on foundational services.
`dockhand.url=http://<host-ip>:<port>`	Web Services	Provides direct HTTP IP-based clickable badges in the Dockhand container list.
`dockhand.changelog.url=https://github.com/<org>/<repo>/releases`	Upstream Applications	Displays direct release notes links next to container update badges.

🔄 Hawser Remote Updates (`hawser-updater` Companion)

Updating Hawser directly via the Dockhand UI previously caused a deadlock because Hawser terminated its own process, severing the communication socket mid-operation.

To solve this, a companion container pattern is configured on remote nodes (docker-admin and docker-media):

yaml

services:
  hawser:
    image: fnsys/dockhand-hawser:latest
    container_name: hawser
    restart: unless-stopped
    ports:
      - "2376:2376"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /opt/stacks:/opt/stacks
    environment:
      - HAWSER_LISTEN=:2376
      - HAWSER_TOKEN=${HAWSER_TOKEN}
    labels:
      - "dockhand.update=false"
      - "dockhand.notify=false"

  hawser-updater:
    image: fnsys/dockhand:latest
    container_name: hawser-updater
    restart: "no"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: ["dockhand-update-hawser"]
    labels:
      - "dockhand.update=false"
      - "dockhand.notify=false"

Execution: Run docker start -a hawser-updater (or trigger via SSH/Ansible) to pull the newest image and recreate hawser asynchronously without losing host control.

🏷️ Dockhand Tag Taxonomy & Visual Catalog

Tags are stored centrally in /opt/docker_container_config/dockhand/data/db/dockhand.db across 8 functional categories:

Tag Name	Color	Description & Workloads
Ingress & Security	`indigo`	Reverse proxies, edge gateways, authentication, firewall (`traefik`, `crowdsec`, `pocket-id`, `tinyauth`, `cloudflared`, `whoami`)
Media & Streaming	`sky`	Media servers, streaming backends, companion playback tools (`plex`, `jellyfin`, `jellyseerr`, `tautulli`, `watchstate`, `fileflows`)
Arr & Downloads	`emerald`	Torrent/Usenet downloaders, indexing, Arr-suite automation (`radarr`, `sonarr`, `prowlarr`, `bazarr`, `qbittorrent`, `sabnzbd`, `gluetun`)
Photos & Files	`amber`	Photo storage, personal file managers, notes sync (`immich_server`, `immich_ml`, `filebrowser`, `obsidian`)
Monitoring & Diagnostics	`teal`	Dashboards, logs, health checks, speed tests (`uptime-kuma`, `dozzle`, `seq`, `scrutiny`, `speedtest-tracker`, `openspeedtest`, `glance`)
Automation & Tools	`rose`	Workflow managers, utility toolsets, container managers (`n8n`, `mealie`, `dockhand`, `hawser`, `termix`, `bambu-studio-api`)
Gaming	`purple`	ROM & game library managers (`romm`)
Infrastructure & DB	`slate`	Underlying relational & key-value databases (`postgres`, `redis`/`valkey`, `mariadb`, `couchdb`)

🛡️ Resource Limits & Healthcheck Guardrails

All Compose stacks in homelab-ops implement production guardrails:

Resource Limits (deploy.resources.limits):
- Every service defines a memory and cpus ceiling preventing any single runaway container or memory leak from degrading the host LXC.
Standardized Healthchecks (healthcheck.test):
- HTTP/TCP/Curl/Wget/Pgrep probes test service vitality, enabling Dockhand and Uptime Kuma to detect unhealthy states and initiate automated self-healing.

🔒 Operational Best Practices

Secrets & .env Files:
- In Dockhand, non-sensitive variables can be stored in .env files.
- Sensitive credentials (passwords, private API keys) should be created via Dockhand's Key icon (🔑) to ensure they are stored encrypted in Dockhand's SQLite database rather than written to plaintext files on disk.
Adopting Stacks in Dockhand:
- When creating or adopting a stack on any remote host, select Internal Stack and point to the corresponding path (e.g., /opt/stacks/admin/seq/compose.yaml).
- This gives you full in-browser YAML editing, linting, autocomplete, and seamless Hawser streaming.
Agent Management:
- Hawser is excluded from Dockhand autoupdates (dockhand.update=false). Use hawser-updater or Ansible to cycle agents safely.

⚓ Dockhand Central Management & 2-Way GitOps Sync ​

🏛️ Architecture Overview ​

📁 Directory Structure (/opt/stacks/) ​

🛠️ Step-by-Step Setup: 2-Way Git Sync (admin-ssh Account) ​

Step 1: Generate SSH Deploy Key for GitHub ​

Step 2: Configure Directory Permissions & Git ​

Step 3: Create the 2-Way Sync Script ​

Step 4: Automate via Systemd Service & Timer ​

1. Create Systemd Service (/etc/systemd/system/git-sync-stacks.service): ​

2. Create Systemd Timer (/etc/systemd/system/git-sync-stacks.timer): ​

3. Enable and Start: ​

🔍 Verification & Monitoring Commands ​

🤖 Automated Ansible Monitoring Integration ​

🏷️ Dockhand Container Labels Standard ​

🔄 Hawser Remote Updates (hawser-updater Companion) ​

🏷️ Dockhand Tag Taxonomy & Visual Catalog ​

🛡️ Resource Limits & Healthcheck Guardrails ​

🔒 Operational Best Practices ​

⚓ Dockhand Central Management & 2-Way GitOps Sync

🏛️ Architecture Overview

📁 Directory Structure (`/opt/stacks/`)

🛠️ Step-by-Step Setup: 2-Way Git Sync (`admin-ssh` Account)

Step 1: Generate SSH Deploy Key for GitHub

Step 2: Configure Directory Permissions & Git

Step 3: Create the 2-Way Sync Script

Step 4: Automate via Systemd Service & Timer

1. Create Systemd Service (`/etc/systemd/system/git-sync-stacks.service`):

2. Create Systemd Timer (`/etc/systemd/system/git-sync-stacks.timer`):

3. Enable and Start:

🔍 Verification & Monitoring Commands

🤖 Automated Ansible Monitoring Integration

🏷️ Dockhand Container Labels Standard

🔄 Hawser Remote Updates (`hawser-updater` Companion)

🏷️ Dockhand Tag Taxonomy & Visual Catalog

🛡️ Resource Limits & Healthcheck Guardrails

🔒 Operational Best Practices