# Homelab Infrastructure — Project Knowledge

## Host Machine
- **ThinkStation-P710** — Primary Proxmox host
  - IP: `192.168.88.25` (jg-hud)
  - Specs: Dual Xeon E5 v4, 110GB ECC RAM, 16 cores
  - OS: Proxmox VE
  - SSH: `root@192.168.88.25` (key: SHA256:i4whMVfdf+SxGttDKhL9PEltWv2ABPQs7BRUunCfcf8)

## Local WORKSTATION
- **ThinkStation-P710**
- Specs: Dual Xeon E5 v4
- OS Linux Mint

  - Also runs Frigate NVR directly at `192.168.88.41`

## Network
- **MikroTik router**: `192.168.88.1`
  - Main LAN: `192.168.88.0/24`
  - IoT VLAN: `192.168.2.0/24`
  - Static WAN IP: `184.170.161.177`
- **Synology RT6600ax**: WiFi AP only
- **DNS**: Split-DNS via MikroTik static entries + Pi-hole forwarding local domains
- **Cloudflare**:
  - Zone ID: `3511a9ac47fa469b24a5c0f411063da4`
  - DNS API token: `gEgJiSdJKSLhnCwQGXRiRWgz5WhTmkZQk0H89X8p`
  - DNS Edit token: `cfat_RhlfvleyZGst8k6rTVx9Ti7x3b8NuE3uOxExka3i9d128e8c`
  - All jgitta.com subdomains point to WAN IP

## Proxmox VMs & Containers

| ID | Name | IP | Specs | Role |
|---|---|---|---|---|
| VM102 | jellyfin | 192.168.88.10 | 2c/12GB | Jellyfin + *arr stack + qBittorrent/ProtonVPN |
| VM103 | next | 192.168.88.62 | 2c/8GB | Nextcloud + OnlyOffice |
| VM104 | Windows11-NVR | — | 8c/11GB | Windows VM |
| VM105 | openclaw-debian | — | 4c/4GB | — |
| VM106 | haos | 192.168.88.39 | 2c/8GB | Home Assistant OS |
| VM107 | pbs | 192.168.88.60 | 4c/8GB | Proxmox Backup Server (nightly full backups) |
| VM112 | siklos/docker-server | 192.168.88.27 | 4c/12GB | Main Docker host |
| VM113 | photos | 192.168.88.32 | 4c/16GB | PhotoPrism + Immich (dedicated photo VM) |
| VM113 | zorin-os | — | 2c/4GB | — |
| CT200 | gitea | 192.168.88.200 | 1c/512MB | Gitea git server (port 3000) |
| CT201 | openclaw | 192.168.88.29 | — | AI gateway (Gemini + Telegram bot, port 18789) |
| CT202 | caddy-proxy | 192.168.88.110 | 2c/4GB | Caddy reverse proxy |
| VM111 | debian-1 | — | — | Stopped/unused |

## Storage Pools (on jg-hud)
| Pool | Size | Type |
|---|---|---|
| nvme-thin | 4.3TB | LVM |
| SSD-1.7T | 1.7TB | — |
| big-11t | 10.8TB | — |
| HD4T-1 | 3.6TB | — |
| HD4T-2 | 3.6TB | — |
| nextcloud-fast | 1.8TB | ZFS |
| pbs | 39TB | PBS |

- **TrueNAS**: `192.168.88.24` — 25.5TB (TrueNAS-NFS pool)
- **Photos NFS**: `/mnt/big-11t/photos` on Proxmox host, NFS-exported to siklos and ThinkStation

## SSH Access Pattern
- Direct to Proxmox host: `root@192.168.88.25`
- To Siklos (VM112): `ssh -o StrictHostKeyChecking=no jgitta@192.168.88.27` (from jg-hud)
- To other VMs: use VM103 or VM107 as jump hosts
- Public key: `ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMRHmoQ63d1qi5yjYoFm8FgnBwUo5uNyRCPChW25DmjF root@jg-hud`
- **Note**: Reach Siklos via Proxmox host shell SSH — QEMU agent times out on long commands

## Caddy Reverse Proxy (CT202 — 192.168.88.110)
- Version: v2.11.2 with Cloudflare DNS plugin
- Config: `/etc/caddy/Caddyfile` imports `/etc/caddy/snippets.caddy` and `/etc/caddy/sites/*.caddy`
- TLS: DNS challenge via Cloudflare
- Snippets: `headers_base`, `internal_only`, `web_secure`, `web_media`, `stream_secure`, `proxy_timeouts`, `pve_proxy`

### sites/infrastructure.caddy
| Subdomain | Backend | Notes |
|---|---|---|
| proxmox.jgitta.com | 192.168.88.25:8006 | internal_only |
| pbs.jgitta.com | 192.168.88.60:8007 | internal_only |
| mesh.jgitta.com | 192.168.88.27:444 | MeshCentral |
| pihole.jgitta.com | 192.168.88.27:8080 | internal_only |
| homarr.jgitta.com | 192.168.88.27:7575 | — |
| apache.jgitta.com | 192.168.88.27:8383 | Guacamole |
| notes.jgitta.com | 192.168.88.27:3010 | Karakeep |
| links.jgitta.com | 192.168.88.27:3015 | Linkwarden |
| gitea.jgitta.com | 192.168.88.200:3000 | — |
| status.jgitta.com | 192.168.88.27:3001 | Uptime Kuma, internal_only |
| grafana.jgitta.com | 192.168.88.27:3020 | internal_only |
| glances.jgitta.com | 192.168.88.27:61208 | internal_only |
| dashboard.jgitta.com | 192.168.88.27:8096 | custom dashboard |
| blue.jgitta.com | 192.168.88.47:75 | — |
| claw.jgitta.com | 192.168.88.29:18789 | internal_only |
| office.jgitta.com | 192.168.88.27:8880 | OnlyOffice |
| jgitta.com / www | 192.168.88.27:8095 | WordPress |

### sites/media.caddy
| Subdomain | Backend | Notes |
|---|---|---|
| jellyfin.jgitta.com | 192.168.88.10:8096 | stream_secure |
| sonarr.jgitta.com | 192.168.88.10:8989 | internal_only |
| radarr.jgitta.com | 192.168.88.10:7878 | internal_only |
| prowlarr.jgitta.com | 192.168.88.10:9696 | internal_only |
| ha.jgitta.com | 192.168.88.39:8123 | Home Assistant |
| cameras.jgitta.com | 192.168.88.41:8971 | Frigate, stream_secure |
| next.jgitta.com | 192.168.88.62:80 | Nextcloud |
| collabora.jgitta.com | 192.168.88.62:9980 | Collabora (may be replaced by OnlyOffice) |
| hub.jgitta.com | 192.168.88.65 | Hubitat |
| pictures.jgitta.com | 192.168.88.32:2283 | Immich (VM113) |
| photos.jgitta.com | 192.168.88.32:2342 | PhotoPrism (VM113) |

### When adding new subdomains
Always handle DNS (Cloudflare) and Caddy changes together as a single operation.

## Jellyfin (VM102 — 192.168.88.10)
- URL: `https://jellyfin.jgitta.com`
- Port: 8096
- API Token: `b861664663924b68858a67b91f6d1188`
- Admin User ID: `12659173-DBBE-43E5-B116-DE70B63A470D`
- DB: `/var/lib/jellyfin/data/jellyfin.db` (SQLite)
- Virtual folders config: `/var/lib/jellyfin/root/default/<LibraryName>/options.xml`
- LiveTV config: `/etc/jellyfin/livetv.xml`
- HDHomeRun tuner: `http://192.168.88.22` (device ID: 10B00DE1)

### Media Libraries
| Library | CollectionFolder ID | Path |
|---|---|---|
| Shows | A656B907-EB3A-7353-2E40-E44B968D0225 | `/mnt/media/tv` |
| Movies | F137A2DD-21BB-C1B9-9AA5-C0F6BF02A805 | `/mnt/media/movies` |
| Recordings | 79A2726D-3C50-E769-A8AF-1E4184E4FCCF | `/mnt/media/recordings` |

Media mount: `/mnt/media` (5.8TB, ~55% used) — subdirs: `movies/`, `tv/`, `recordings/`, `torrents/`

### Critical: Library Path Management
**NEVER edit `options.xml` directly to change library paths.** Jellyfin ignores changes made directly to the file while running, and on restart it overwrites the file from its internal state.

Always use the API to add/remove paths:
```bash
# Add a path to a library
curl -X POST "http://localhost:8096/Library/VirtualFolders/Paths" \
  -H "Authorization: MediaBrowser Token=\"<token>\"" \
  -H "Content-Type: application/json" \
  -d '{"Name":"Shows","PathInfo":{"Path":"/mnt/media/tv"}}'

# Remove a path from a library
curl -X DELETE "http://localhost:8096/Library/VirtualFolders/Paths?name=Shows&path=%2Fmnt%2Fmedia%2Ftv&refreshLibrary=false" \
  -H "Authorization: MediaBrowser Token=\"<token>\""
```

### Root Cause History: TV Shows Not Appearing (April 2026)
The Recordings library was accidentally configured with path `/mnt/media` (the entire media mount) instead of `/mnt/media/recordings`. This caused:
- Recordings library to scan and "own" all TV shows and movies
- Shows library to have no path configured (empty PathInfos) — never scanned `/mnt/media/tv`
- All items getting `TopParentId` pointing to the `/mnt/media` Folder (not Shows CollectionFolder)
- Jellyfin auto-creating a `Recordings2` library whenever LiveTV checked `/mnt/media/recordings` didn't match Recordings' path `/mnt/media`

Fix: Use API to remove `/mnt/media` from Recordings, add `/mnt/media/recordings` to Recordings, add `/mnt/media/tv` to Shows, delete Recordings2, wipe BaseItems table (preserving UserData), restart + scan.

### DB Wipe Command (preserves UserData/watch history)
```sql
DELETE FROM AncestorIds; DELETE FROM BaseItemImageInfos;
DELETE FROM BaseItemMetadataFields; DELETE FROM BaseItemProviders;
DELETE FROM BaseItemTrailerTypes; DELETE FROM Chapters;
DELETE FROM MediaStreamInfos; DELETE FROM AttachmentStreamInfos;
DELETE FROM TrickplayInfos; DELETE FROM KeyframeData;
DELETE FROM PeopleBaseItemMap; DELETE FROM Peoples;
DELETE FROM MediaSegments; DELETE FROM ItemValuesMap;
DELETE FROM ItemValues; DELETE FROM BaseItems; VACUUM;
```
Run as: `sudo sqlite3 /var/lib/jellyfin/data/jellyfin.db "..."`
(Must stop Jellyfin first: `sudo systemctl stop jellyfin`)

### Recordings2 Auto-Creation
If a `Recordings2` library appears, it means LiveTV's recording path in `/etc/jellyfin/livetv.xml` doesn't match the Recordings library path. Fix via API (remove/add paths) rather than editing XML files.

## Dotfiles
- Gitea: `http://192.168.88.200:3000/jgitta/dotfiles.git`
- Token (read/write): `56ac53def371df34d8f9c4b5580b28d4f62ab1ab`
- Includes: micro config, bash aliases, `update-dotfiles` alias
- Preferred editor: `micro` (never nano)

## Backups
- PBS (VM107, 192.168.88.60): nightly full backups of all VMs
- TrueNAS (192.168.88.24): bulk storage backups

## Key Notes
- KSM enabled with ksmtuned on Proxmox host
- Watchtower runs on Siklos but Pi-hole is excluded from auto-updates
- When restoring VMs: restore all disks to HD4T-2 first, then `qm disk move` OS disk to nvme-thin
- `delete_vm` MCP tool removes all disks — always confirm before running

## DNS Resolution Flow (Split-DNS)
All `*.jgitta.com` subdomains resolve internally to Caddy (192.168.88.110) via MikroTik static DNS entries — **not** through Cloudflare/public DNS. If a subdomain doesn't resolve internally, check MikroTik static DNS first, then Caddy config.

- **Internal**: MikroTik static entries → 192.168.88.110 (Caddy)
- **External**: Cloudflare A records → 184.170.161.177 (WAN) → NAT → Caddy
- **Other names**: Pi-hole (192.168.88.27:53) handles upstream forwarding
- **`.homenet` names**: MikroTik static DNS only (e.g., `siklos.homenet`, `jellyfin.homenet`)
- **IoT DNS**: Force-redirected to Pi-hole via MikroTik DSTNAT rules

**Firewall gotcha**: `vlan20-IoT` is dynamically added to the WAN interface list. NAT rules using `in-interface-list=WAN` will match IoT traffic. WAN-facing rules must use `in-interface=ether1` instead.

## Service Architecture — Native vs Docker

**VM102 (jellyfin, 192.168.88.10)** — native systemd for core services:
- `jellyfin.service` — native systemd
- `qbittorrent-nox.service` — native systemd
- `me.proton.vpn.split_tunneling.service` — native systemd
- Radarr, Sonarr, Prowlarr, FlareSolverr — **Docker containers**

**VM103 (next, 192.168.88.62)** — Nextcloud native Apache:
- Nextcloud 33.0.0 — native Apache (`/etc/apache2/sites-enabled/nextcloud.conf`)
- Web root: `/var/www/nextcloud/`
- Config: `/var/www/nextcloud/config/config.php`
- Data dir: `/mnt/nextcloud-data` (local 2TB disk `/dev/sdb1`, ~65% used)
- DB: MySQL on localhost, db `nextcloud`, user `nextcloud`
- Tiered storage mounts:
  - Warm: `192.168.88.25:/nvme5-1tb/nextcloud-data` → `/mnt/warm-storage` (900GB NFS)
  - Cold: `192.168.88.24:/mnt/pool1/cold-storage` → `/mnt/cold-storage` (58TB NFS from TrueNAS)
- Media: `192.168.88.25:/mnt/big-11t/media` → `/mnt/media`

**VM112 / Siklos (192.168.88.27)** — all services Docker; compose files at `/srv/docker/<service>/` (RAM reduced to 12GB after photo services migrated out)

**VM113 / photos (192.168.88.32)** — dedicated photo services VM, Docker:
- PhotoPrism — port 2342, compose `/srv/docker/photoprism/docker-compose.yml`
  - Storage: `/srv/docker/photoprism/storage_data/` (14GB)
  - DB: MariaDB 11 at `/srv/docker/photoprism/db_data/`
- Immich — port 2283, compose `/srv/docker/immich/docker-compose.yml`
  - Library: `/srv/docker/immich/library/` (22GB)
  - DB: PostgreSQL at `/srv/docker/immich/postgres/`
- NFS: `/mnt/photos` from `192.168.88.25:/mnt/big-11t/photos`
- Node exporter: port 9100

**ThinkStation workstation (192.168.88.41)** — Linux Mint 22.3 (separate machine from Proxmox host at .25):
- Frigate NVR — Docker, image `ghcr.io/blakeblackshear/frigate:stable-tensorrt`
  - Compose: `/opt/frigate/docker-compose.yml`
  - Ports: 8971 (web UI), 5000, 8554–8555 (RTSP), 8555/udp
- Open-WebUI — Docker (no external ports)
- Disks: OS `/dev/sda2` (234GB, 73%), `/dev/sdb2` 14TB HD at `/mnt/14TB-HD` (72%), `/dev/nvme0n1p1` 469GB at `/mnt/INTEL-SSD`, `/dev/sdc1` 1.8TB at `/mnt/StorFly-SSD`
- NFS mount: `/mnt/photos` from `192.168.88.25:/mnt/big-11t/photos`

## VM Disk → Storage Pool Mapping

| VM | Disk | Pool | Size | Notes |
|---|---|---|---|---|
| VM102 jellyfin | scsi0 (OS) | big-11t | 32GB | |
| VM102 jellyfin | scsi1 (media) | big-11t | 6TB | `/mnt/media` inside VM |
| VM103 next | scsi0 (OS) | nvme-thin | 32GB | |
| VM103 next | scsi1 (data) | nvme-thin | 2TB | local nextcloud data disk |
| VM106 haos | scsi0 | HD4T-2 | 32GB | |
| VM107 pbs | scsi0 | SSD-1.7T | 32GB | |
| VM112 siklos | scsi0 | nvme-thin | 250GB | |

**Storage pool usage** (as of April 2026):
| Pool | Type | Total | Used% |
|---|---|---|---|
| nvme-thin | lvmthin | 4.4TB | 39% |
| big-11t | dir | 10.8TB | 59% |
| SSD-1.7T | dir | 1.7TB | 70% |
| HD4T-1 | dir | 3.6TB | 70% |
| HD4T-2 | dir | 3.6TB | 0.2% |
| pbs | PBS | 64TB | 23% |
| nextcloud-fast | zfspool | — | INACTIVE |
| nvme4-2tb | zfspool | 1.8TB | 0% |
| nvme5-1tb | zfspool | 900GB | 0% |

## NFS Export Map

**From jg-hud (192.168.88.25)**:
| Export | Clients | Mounted by |
|---|---|---|
| `/mnt/big-11t/media` | 192.168.88.0/24 | VM102 (`/mnt/media`), VM103 (`/mnt/media`) |
| `/mnt/big-11t/photos` | 192.168.88.0/24 | VM112 (`/mnt/photos`), ThinkStation (`/mnt/photos`) |
| `/nvme5-1tb/nextcloud-data` | 192.168.88.62 only | VM103 (`/mnt/warm-storage`) |

**From TrueNAS (192.168.88.24)**:
| Export | Mounted by |
|---|---|
| `/mnt/pool1/cold-storage` | VM103 (`/mnt/cold-storage`, 58TB) |
| PBS dataset | VM107 PBS (`/mnt/pbsdataset/pbs-store`) |

## Backup Schedule & Retention

**Job 1 — PBS (enabled)**: Mon/Wed/Fri at 21:00
- All VMs/CTs **except VM107** (PBS itself)
- Storage: `pbs` (PBS datastore `pbs-store` on TrueNAS NFS at `/mnt/pbsdataset/pbs-store`)
- Retention: keep-daily=7, keep-weekly=4, keep-monthly=3
- Email notification: `jgitta@jgitta.com`

**Job 2 — TrueNAS NFS (disabled)**: Sun/Tue/Thu/Sat at 00:00
- All VMs, storage: `Proxmox-TrueNAS-NFS`, keep-last=5

## TrueNAS (192.168.88.24)
- TrueNAS SCALE 24.10.2.4
- **pool1**: RAIDZ2, 8 disks, ~63.7TB used / ~104.4TB total, ONLINE/healthy
  - Hosts PBS backup store (NFS-exported to VM107)
  - Hosts cold-storage NFS export (58TB, mounted on VM103)
- API: use `curl -sk https://192.168.88.24/api/v2.0/<endpoint> -u "root:<password>"`
- MCP connection is broken — use curl API directly
- Disks with SMART warnings: sda/9JH31S2T (40 unreadable, 24 ATA errors), sdh/9JG47UTT (13 ATA errors)

## Log Locations

| Service | Location | Command |
|---|---|---|
| Jellyfin | `/var/log/jellyfin/` | `tail -f /var/log/jellyfin/jellyfin$(date +%Y%m%d).log` on VM102 |
| FFmpeg transcode | `/var/log/jellyfin/FFmpeg.Transcode-*.log` | VM102 |
| Caddy | systemd journal | `journalctl -u caddy -f` on CT202 |
| Pi-hole | FTL DB / journal | `/etc/pihole/pihole-FTL.db`, `journalctl -u pihole-FTL` on Siklos |
| Docker services (Siklos) | docker logs | `docker compose logs -f` in `/srv/docker/<service>/` |
| Proxmox | `/var/log/pve/` | on jg-hud (192.168.88.25) |
| qBittorrent | systemd journal | `journalctl -u qbittorrent-nox -f` on VM102 |
| ProtonVPN | systemd journal | `journalctl -u me.proton.vpn.split_tunneling -f` on VM102 |

## Uptime Kuma Monitors (Siklos, port 3001)
HTTP checks: Caddy Health, Gitea, Glances, Grafana, Homarr, Home Assistant, Immich, Jellyfin, Karakeep, Linkwarden, MeshCentral, Nextcloud, PBS, Pi-hole web, Prometheus
Port checks: MikroTik SSH (:22), Pi-hole DNS (:53), Proxmox SSH (:22), Siklos SSH (:22)
DNS check: Pi-hole DNS resolution (google.com via 192.168.88.27:53)
Metric check: Node Exporter on ThinkStation (192.168.88.41:9100)

**Not monitored**: VM102 SSH, qBittorrent, ProtonVPN status, Radarr/Sonarr/Prowlarr health, TrueNAS, Frigate