All runbooks
runbookuai-explorerredacted view

Proxmox Storage Maintenance Runbook

Synced UAI Explorer runbook documentation from docs/runbooks/proxmox-storage-maintenance.md.

Updated 2026-06-21·Freshness: Reference·No secret-like patterns were observed in this view.

Proxmox Storage Maintenance Runbook

Created: 2026-06-20

Purpose

Provide a safe review path for HomeLab Proxmox storage pressure before snapshot-heavy UAI Explorer work, device imports, cluster changes, or backup cleanup.

This runbook is intentionally conservative. It records what to inspect and what requires operator approval; it is not approval to delete snapshots, remove backups, migrate workloads, or resize storage.

Scope

Visible HomeLab Proxmox nodes:

  • Director at 10.0.0.250
  • echo at 10.0.0.249
  • forge at 10.0.0.248

UAI Explorer production CT 121 and sandbox CT 126 are on Director.

No-Change Boundaries

  • Do not delete snapshots without explicit operator approval.
  • Do not prune backups, templates, or ISO files without explicit operator approval.
  • Do not resize disks, move volumes, migrate containers, or change cluster membership from UAI Explorer work.
  • Do not mutate production CT 121.
  • Do not create new UAI Explorer rollback snapshots while this maintenance review remains unresolved unless an operator explicitly accepts the storage risk.

Read-Only Preflight

Run from the Windows workstation.

ssh -J root@10.0.0.103 -o BatchMode=yes -o StrictHostKeyChecking=no `
  root@10.0.0.250 "pvesm status; vgs --units g; lvs -a --units g -o lv_name,vg_name,lv_size,data_percent,metadata_percent,origin,segtype,lv_attr; pct list; qm list"

ssh -o BatchMode=yes -o StrictHostKeyChecking=no `
  root@10.0.0.248 "pvesm status; vgs --units g; lvs -a --units g -o lv_name,vg_name,lv_size,data_percent,metadata_percent,origin,segtype,lv_attr; pct list; qm list"

ssh -o BatchMode=yes -o StrictHostKeyChecking=no `
  root@10.0.0.249 "pvesm status; vgs --units g; lvs -a --units g -o lv_name,vg_name,lv_size,data_percent,metadata_percent,origin,segtype,lv_attr; pct list; qm list"

Snapshot inventory:

ssh -J root@10.0.0.103 -o BatchMode=yes -o StrictHostKeyChecking=no `
  root@10.0.0.250 "pct listsnapshot 126; pct listsnapshot 115; pct listsnapshot 116; pct listsnapshot 117; pct listsnapshot 253"

Host filesystem pressure:

ssh -o BatchMode=yes -o StrictHostKeyChecking=no root@10.0.0.249 `
  "df -h / /var/lib/vz; du -sh /var/lib/vz/* 2>/dev/null | sort -h"

Current Review Targets

As of the 2026-06-20 read-only review:

  • Director: decide retention policy for CT 126 snapshots migration-ready-20260620-081634, mr022-20260620-190655, and mr022g-20260620-191542.
  • Director: review preservation snapshots for archived CTs 115, 116, and 117 before any deletion or reuse.
  • Director: review CT 253 snapshot pre-gpu-move.
  • forge: review CT 103 storage use; its root thin volume showed 98.46% data usage.
  • echo: review root filesystem and template storage; root filesystem showed 89% used with 11 GiB available.

Approval Checklist

Before any mutating maintenance:

  • Confirm the target node, CT/VM ID, snapshot name, and business purpose.
  • Confirm whether a newer backup exists and is restorable.
  • Confirm whether the snapshot is part of a rollback rehearsal, archive preservation, or migration marker.
  • Confirm no UAI Explorer production route, DNS, TLS, firewall, or reverse proxy change is bundled into the maintenance.
  • Record the before/after state in docs/evidence/.

UAI Explorer Rule

For UAI Explorer sandbox refreshes, prefer app-only refresh without creating new snapshots while storage policy is unresolved. Keep the current rollback snapshot mr022g-20260620-191542 documented until an operator approves a new snapshot strategy.