Infrastructure Management
Quick Reference
Deploy with Colmena
# Single host
colmena apply --on <hostname> --impure
# Multiple hosts
colmena apply --on host1,host2,host3 --impure
# Build only (no deploy)
colmena build --on <hostname> --impureServer Inventory
Hetzner Servers (Colmena-managed, root user)
| Host | Type | Services |
|---|---|---|
| headscale | Hetzner VPS | Headscale VPN, Tailscale client |
| ironforge | Hetzner dedicated | nixarr (jellyfin, jellyseerr, sonarr, radarr, lidarr, prowlarr, sabnzbd, bazarr) |
| orgrimmar | Hetzner dedicated | gitea, woodpecker, paperless, calibre, resume |
WSL Hosts (Colmena-managed, nixos user with sudo)
| Host | Type | Purpose |
|---|---|---|
| anton | WSL NixOS on Windows laptop | Gaming and AI processing |
On-Prem Workstations (locally configured)
| Host | Type | Notes |
|---|---|---|
| fredpc | x86_64-linux | GUI, NVIDIA CUDA, glance dashboard |
| nixosaarch64vm | aarch64-linux | ARM64 build host |
Troubleshooting Workflows
Service Not Working
- Check service status:
ssh <hostname> "systemctl status <service>" - Check logs:
ssh <hostname> "journalctl -u <service> -n 100" - Restart service:
ssh <hostname> "systemctl restart <service>"
Note: For Hetzner servers, SSH as root. For anton, SSH as nixos and use sudo.
Podman/Container Issues
Check socket status:
ssh <hostname> "systemctl status podman.socket"List running containers:
ssh <hostname> "podman ps -a"SSH Connection Issues
If colmena fails with SSH errors:
- Verify the host is reachable:
ping <hostname> - Check if SSH is listening:
ssh <hostname> "ss -tlnp | grep 22" - For Hetzner servers, check via Hetzner console if needed
Common Colmena Patterns
Deploy All Hetzner Hosts
colmena apply --on headscale,ironforge,orgrimmar --impureDeploy All Hosts
colmena apply --on headscale,ironforge,orgrimmar,anton --impureUpdate Secrets Before Deploy
just update-secrets
colmena apply --on <hostname> --impureFile Locations
| Purpose | Path |
|---|---|
| Colmena host configs | colmena/hosts/<hostname>.nix |
| Hetzner common modules | colmena/hetzner-common/ |
| WSL common modules | colmena/wsl-common/ |
| NixOS host configs | modules/nixos/host/<hostname>/configuration.nix |
| Application configs | apps/<appname>.nix |
| Service modules (incl. secrets) | modules/services/<service>.nix |
| Container image SHAs | apps/fetcher/containers-sha.nix |
| Container definitions | apps/fetcher/containers.toml |
Related Skills
- provision-nixos-server: Create new Hetzner servers from scratch