Skip to content

Recurring Patterns

Real throughlines that show up across more than one Infrastructure document, not a summary of each one individually.

Cloud-init snippet path confusion

Proxmox resolves storageID:snippets/<file> against that storage's path setting plus a snippets/ subfolder, not against /var/lib/vz/snippets/ by default. This gets re-discovered the hard way across several sessions, each time with a different storage path (/var/lib/vz/snips, local:snippets, CephFS-backed snips).

cicustom user-data silently falling back to defaults

The network snippet (network=storageID:snippets/docker-net.yml) applies reliably. The user-data snippet does not: Proxmox repeatedly embeds its own default #cloud-config instead of the custom YAML, even after the storage path, permissions, and YAML syntax are all confirmed correct. The eventual workaround was bypassing cicustom entirely with a manually built NoCloud cidata ISO.

VM 100 as the recurring trouble spot

The primary Docker host VM has been rebuilt from scratch more than once, and most of the Automation, Containers & Apps, Compute, and Storage incidents trace back to it. The recurring causes are duplicate /etc/fstab mount entries, Docker-data-disk mount ordering, and UUID-vs-LABEL drift between what cloud-init intended and what the live system actually had.

Host-level crashes that look like guest problems

Two separate node-level incidents initially presented as guest or Docker-level instability, and both turned out to be host-level root causes once the logs were read carefully: unstable RAM on mainframe, and an Intel e1000e NIC driver hang on a NUC node that took Ceph connectivity down with it. The shared lesson: separate guest failure, host failure, and storage/network dependency failure before troubleshooting deeper into any one of them.

mainframe's Ceph monitor keeps needing rebuilding

mon.mainframe has dropped out of quorum or crashed outright in more than one unrelated incident, each time recovered by rebuilding the local monitor store from the live monmap/keyring rather than trusting whatever local state was left behind.

Docker upgrades breaking Traefik's provider API

More than once, a Docker Engine upgrade has broken Traefik's ability to talk to the Docker provider because the Traefik release in use negotiated too old an API version. Each time, the fix was upgrading Traefik itself rather than pinning Docker back.

Offen restore path-stripping

Restoring from an Offen backup archive is never a direct extract: the archive is rooted at backup/my-app-backup/..., so every restore has needed the right --strip-components count (or equivalent rsync path handling) to land content directly under /opt/docker-apps instead of recreating the wrapper path.