jgitta
/
claude-projects
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
claude-projects/Homelab Infrastructure/homelab-infrastructure.md

15 KiB
Raw Permalink Blame History

Homelab Infrastructure — Project Knowledge

Host Machine

  • ThinkStation-P710 — Primary Proxmox host
    • IP: 192.168.88.25 (jg-hud)
    • Specs: Dual Xeon E5 v4, 110GB ECC RAM, 16 cores
    • OS: Proxmox VE
    • SSH: root@192.168.88.25 (key: SHA256:i4whMVfdf+SxGttDKhL9PEltWv2ABPQs7BRUunCfcf8)

Local WORKSTATION

  • ThinkStation-P710

  • Specs: Dual Xeon E5 v4

  • OS Linux Mint

    • Also runs Frigate NVR directly at 192.168.88.41

Network

  • MikroTik router: 192.168.88.1
    • Main LAN: 192.168.88.0/24
    • IoT VLAN: 192.168.2.0/24
    • Static WAN IP: 184.170.161.177
  • Synology RT6600ax: WiFi AP only
  • DNS: Split-DNS via MikroTik static entries + Pi-hole forwarding local domains
  • Cloudflare:
    • Zone ID: 3511a9ac47fa469b24a5c0f411063da4
    • DNS API token: gEgJiSdJKSLhnCwQGXRiRWgz5WhTmkZQk0H89X8p
    • DNS Edit token: cfat_RhlfvleyZGst8k6rTVx9Ti7x3b8NuE3uOxExka3i9d128e8c
    • All jgitta.com subdomains point to WAN IP

Proxmox VMs & Containers

ID Name IP Specs Role
VM102 jellyfin 192.168.88.10 2c/12GB Jellyfin + *arr stack + qBittorrent/ProtonVPN
VM103 next 192.168.88.62 2c/8GB Nextcloud + OnlyOffice
VM104 Windows11-NVR — 8c/11GB Windows VM
VM105 openclaw-debian — 4c/4GB —
VM106 haos 192.168.88.39 2c/8GB Home Assistant OS
VM107 pbs 192.168.88.60 4c/8GB Proxmox Backup Server (nightly full backups)
VM112 siklos/docker-server 192.168.88.27 4c/12GB Main Docker host
VM113 photos 192.168.88.32 4c/16GB PhotoPrism + Immich (dedicated photo VM)
VM113 zorin-os — 2c/4GB —
CT200 gitea 192.168.88.200 1c/512MB Gitea git server (port 3000)
CT201 openclaw 192.168.88.29 — AI gateway (Gemini + Telegram bot, port 18789)
CT202 caddy-proxy 192.168.88.110 2c/4GB Caddy reverse proxy
VM111 debian-1 — — Stopped/unused

Storage Pools (on jg-hud)

Pool Size Type
nvme-thin 4.3TB LVM
SSD-1.7T 1.7TB —
big-11t 10.8TB —
HD4T-1 3.6TB —
HD4T-2 3.6TB —
nextcloud-fast 1.8TB ZFS
pbs 39TB PBS
  • TrueNAS: 192.168.88.24 — 25.5TB (TrueNAS-NFS pool)
  • Photos NFS: /mnt/big-11t/photos on Proxmox host, NFS-exported to siklos and ThinkStation

SSH Access Pattern

  • Direct to Proxmox host: root@192.168.88.25
  • To Siklos (VM112): ssh -o StrictHostKeyChecking=no jgitta@192.168.88.27 (from jg-hud)
  • To other VMs: use VM103 or VM107 as jump hosts
  • Public key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMRHmoQ63d1qi5yjYoFm8FgnBwUo5uNyRCPChW25DmjF root@jg-hud
  • Note: Reach Siklos via Proxmox host shell SSH — QEMU agent times out on long commands

Caddy Reverse Proxy (CT202 — 192.168.88.110)

  • Version: v2.11.2 with Cloudflare DNS plugin
  • Config: /etc/caddy/Caddyfile imports /etc/caddy/snippets.caddy and /etc/caddy/sites/*.caddy
  • TLS: DNS challenge via Cloudflare
  • Snippets: headers_base, internal_only, web_secure, web_media, stream_secure, proxy_timeouts, pve_proxy

sites/infrastructure.caddy

Subdomain Backend Notes
proxmox.jgitta.com 192.168.88.25:8006 internal_only
pbs.jgitta.com 192.168.88.60:8007 internal_only
mesh.jgitta.com 192.168.88.27:444 MeshCentral
pihole.jgitta.com 192.168.88.27:8080 internal_only
homarr.jgitta.com 192.168.88.27:7575 —
apache.jgitta.com 192.168.88.27:8383 Guacamole
notes.jgitta.com 192.168.88.27:3010 Karakeep
links.jgitta.com 192.168.88.27:3015 Linkwarden
gitea.jgitta.com 192.168.88.200:3000 —
status.jgitta.com 192.168.88.27:3001 Uptime Kuma, internal_only
grafana.jgitta.com 192.168.88.27:3020 internal_only
glances.jgitta.com 192.168.88.27:61208 internal_only
dashboard.jgitta.com 192.168.88.27:8096 custom dashboard
blue.jgitta.com 192.168.88.47:75 —
claw.jgitta.com 192.168.88.29:18789 internal_only
office.jgitta.com 192.168.88.27:8880 OnlyOffice
jgitta.com / www 192.168.88.27:8095 WordPress

sites/media.caddy

Subdomain Backend Notes
jellyfin.jgitta.com 192.168.88.10:8096 stream_secure
sonarr.jgitta.com 192.168.88.10:8989 internal_only
radarr.jgitta.com 192.168.88.10:7878 internal_only
prowlarr.jgitta.com 192.168.88.10:9696 internal_only
ha.jgitta.com 192.168.88.39:8123 Home Assistant
cameras.jgitta.com 192.168.88.41:8971 Frigate, stream_secure
next.jgitta.com 192.168.88.62:80 Nextcloud
collabora.jgitta.com 192.168.88.62:9980 Collabora (may be replaced by OnlyOffice)
hub.jgitta.com 192.168.88.65 Hubitat
pictures.jgitta.com 192.168.88.32:2283 Immich (VM113)
photos.jgitta.com 192.168.88.32:2342 PhotoPrism (VM113)

When adding new subdomains

Always handle DNS (Cloudflare) and Caddy changes together as a single operation.

Jellyfin (VM102 — 192.168.88.10)

  • URL: https://jellyfin.jgitta.com
  • Port: 8096
  • API Token: b861664663924b68858a67b91f6d1188
  • Admin User ID: 12659173-DBBE-43E5-B116-DE70B63A470D
  • DB: /var/lib/jellyfin/data/jellyfin.db (SQLite)
  • Virtual folders config: /var/lib/jellyfin/root/default/<LibraryName>/options.xml
  • LiveTV config: /etc/jellyfin/livetv.xml
  • HDHomeRun tuner: http://192.168.88.22 (device ID: 10B00DE1)

Media Libraries

Library CollectionFolder ID Path
Shows A656B907-EB3A-7353-2E40-E44B968D0225 /mnt/media/tv
Movies F137A2DD-21BB-C1B9-9AA5-C0F6BF02A805 /mnt/media/movies
Recordings 79A2726D-3C50-E769-A8AF-1E4184E4FCCF /mnt/media/recordings

Media mount: /mnt/media (5.8TB, ~55% used) — subdirs: movies/, tv/, recordings/, torrents/

Critical: Library Path Management

NEVER edit options.xml directly to change library paths. Jellyfin ignores changes made directly to the file while running, and on restart it overwrites the file from its internal state.

Always use the API to add/remove paths:

# Add a path to a library
curl -X POST "http://localhost:8096/Library/VirtualFolders/Paths" \
  -H "Authorization: MediaBrowser Token=\"<token>\"" \
  -H "Content-Type: application/json" \
  -d '{"Name":"Shows","PathInfo":{"Path":"/mnt/media/tv"}}'

# Remove a path from a library
curl -X DELETE "http://localhost:8096/Library/VirtualFolders/Paths?name=Shows&path=%2Fmnt%2Fmedia%2Ftv&refreshLibrary=false" \
  -H "Authorization: MediaBrowser Token=\"<token>\""

Root Cause History: TV Shows Not Appearing (April 2026)

The Recordings library was accidentally configured with path /mnt/media (the entire media mount) instead of /mnt/media/recordings. This caused:

  • Recordings library to scan and "own" all TV shows and movies
  • Shows library to have no path configured (empty PathInfos) — never scanned /mnt/media/tv
  • All items getting TopParentId pointing to the /mnt/media Folder (not Shows CollectionFolder)
  • Jellyfin auto-creating a Recordings2 library whenever LiveTV checked /mnt/media/recordings didn't match Recordings' path /mnt/media

Fix: Use API to remove /mnt/media from Recordings, add /mnt/media/recordings to Recordings, add /mnt/media/tv to Shows, delete Recordings2, wipe BaseItems table (preserving UserData), restart + scan.

DB Wipe Command (preserves UserData/watch history)

DELETE FROM AncestorIds; DELETE FROM BaseItemImageInfos;
DELETE FROM BaseItemMetadataFields; DELETE FROM BaseItemProviders;
DELETE FROM BaseItemTrailerTypes; DELETE FROM Chapters;
DELETE FROM MediaStreamInfos; DELETE FROM AttachmentStreamInfos;
DELETE FROM TrickplayInfos; DELETE FROM KeyframeData;
DELETE FROM PeopleBaseItemMap; DELETE FROM Peoples;
DELETE FROM MediaSegments; DELETE FROM ItemValuesMap;
DELETE FROM ItemValues; DELETE FROM BaseItems; VACUUM;

Run as: sudo sqlite3 /var/lib/jellyfin/data/jellyfin.db "..." (Must stop Jellyfin first: sudo systemctl stop jellyfin)

Recordings2 Auto-Creation

If a Recordings2 library appears, it means LiveTV's recording path in /etc/jellyfin/livetv.xml doesn't match the Recordings library path. Fix via API (remove/add paths) rather than editing XML files.

Dotfiles

  • Gitea: http://192.168.88.200:3000/jgitta/dotfiles.git
  • Token (read/write): 56ac53def371df34d8f9c4b5580b28d4f62ab1ab
  • Includes: micro config, bash aliases, update-dotfiles alias
  • Preferred editor: micro (never nano)

Backups

  • PBS (VM107, 192.168.88.60): nightly full backups of all VMs
  • TrueNAS (192.168.88.24): bulk storage backups

Key Notes

  • KSM enabled with ksmtuned on Proxmox host
  • Watchtower runs on Siklos but Pi-hole is excluded from auto-updates
  • When restoring VMs: restore all disks to HD4T-2 first, then qm disk move OS disk to nvme-thin
  • delete_vm MCP tool removes all disks — always confirm before running

DNS Resolution Flow (Split-DNS)

All *.jgitta.com subdomains resolve internally to Caddy (192.168.88.110) via MikroTik static DNS entries — not through Cloudflare/public DNS. If a subdomain doesn't resolve internally, check MikroTik static DNS first, then Caddy config.

  • Internal: MikroTik static entries → 192.168.88.110 (Caddy)
  • External: Cloudflare A records → 184.170.161.177 (WAN) → NAT → Caddy
  • Other names: Pi-hole (192.168.88.27:53) handles upstream forwarding
  • .homenet names: MikroTik static DNS only (e.g., siklos.homenet, jellyfin.homenet)
  • IoT DNS: Force-redirected to Pi-hole via MikroTik DSTNAT rules

Firewall gotcha: vlan20-IoT is dynamically added to the WAN interface list. NAT rules using in-interface-list=WAN will match IoT traffic. WAN-facing rules must use in-interface=ether1 instead.

Service Architecture — Native vs Docker

VM102 (jellyfin, 192.168.88.10) — native systemd for core services:

  • jellyfin.service — native systemd
  • qbittorrent-nox.service — native systemd
  • me.proton.vpn.split_tunneling.service — native systemd
  • Radarr, Sonarr, Prowlarr, FlareSolverr — Docker containers

VM103 (next, 192.168.88.62) — Nextcloud native Apache:

  • Nextcloud 33.0.0 — native Apache (/etc/apache2/sites-enabled/nextcloud.conf)
  • Web root: /var/www/nextcloud/
  • Config: /var/www/nextcloud/config/config.php
  • Data dir: /mnt/nextcloud-data (local 2TB disk /dev/sdb1, ~65% used)
  • DB: MySQL on localhost, db nextcloud, user nextcloud
  • Tiered storage mounts:
    • Warm: 192.168.88.25:/nvme5-1tb/nextcloud-data → /mnt/warm-storage (900GB NFS)
    • Cold: 192.168.88.24:/mnt/pool1/cold-storage → /mnt/cold-storage (58TB NFS from TrueNAS)
  • Media: 192.168.88.25:/mnt/big-11t/media → /mnt/media

VM112 / Siklos (192.168.88.27) — all services Docker; compose files at /srv/docker/<service>/ (RAM reduced to 12GB after photo services migrated out)

VM113 / photos (192.168.88.32) — dedicated photo services VM, Docker:

  • PhotoPrism — port 2342, compose /srv/docker/photoprism/docker-compose.yml
    • Storage: /srv/docker/photoprism/storage_data/ (14GB)
    • DB: MariaDB 11 at /srv/docker/photoprism/db_data/
  • Immich — port 2283, compose /srv/docker/immich/docker-compose.yml
    • Library: /srv/docker/immich/library/ (22GB)
    • DB: PostgreSQL at /srv/docker/immich/postgres/
  • NFS: /mnt/photos from 192.168.88.25:/mnt/big-11t/photos
  • Node exporter: port 9100

ThinkStation workstation (192.168.88.41) — Linux Mint 22.3 (separate machine from Proxmox host at .25):

  • Frigate NVR — Docker, image ghcr.io/blakeblackshear/frigate:stable-tensorrt
    • Compose: /opt/frigate/docker-compose.yml
    • Ports: 8971 (web UI), 5000, 8554–8555 (RTSP), 8555/udp
  • Open-WebUI — Docker (no external ports)
  • Disks: OS /dev/sda2 (234GB, 73%), /dev/sdb2 14TB HD at /mnt/14TB-HD (72%), /dev/nvme0n1p1 469GB at /mnt/INTEL-SSD, /dev/sdc1 1.8TB at /mnt/StorFly-SSD
  • NFS mount: /mnt/photos from 192.168.88.25:/mnt/big-11t/photos

VM Disk → Storage Pool Mapping

VM Disk Pool Size Notes
VM102 jellyfin scsi0 (OS) big-11t 32GB
VM102 jellyfin scsi1 (media) big-11t 6TB /mnt/media inside VM
VM103 next scsi0 (OS) nvme-thin 32GB
VM103 next scsi1 (data) nvme-thin 2TB local nextcloud data disk
VM106 haos scsi0 HD4T-2 32GB
VM107 pbs scsi0 SSD-1.7T 32GB
VM112 siklos scsi0 nvme-thin 250GB

Storage pool usage (as of April 2026):

Pool Type Total Used%
nvme-thin lvmthin 4.4TB 39%
big-11t dir 10.8TB 59%
SSD-1.7T dir 1.7TB 70%
HD4T-1 dir 3.6TB 70%
HD4T-2 dir 3.6TB 0.2%
pbs PBS 64TB 23%
nextcloud-fast zfspool — INACTIVE
nvme4-2tb zfspool 1.8TB 0%
nvme5-1tb zfspool 900GB 0%

NFS Export Map

From jg-hud (192.168.88.25):

Export Clients Mounted by
/mnt/big-11t/media 192.168.88.0/24 VM102 (/mnt/media), VM103 (/mnt/media)
/mnt/big-11t/photos 192.168.88.0/24 VM112 (/mnt/photos), ThinkStation (/mnt/photos)
/nvme5-1tb/nextcloud-data 192.168.88.62 only VM103 (/mnt/warm-storage)

From TrueNAS (192.168.88.24):

Export Mounted by
/mnt/pool1/cold-storage VM103 (/mnt/cold-storage, 58TB)
PBS dataset VM107 PBS (/mnt/pbsdataset/pbs-store)

Backup Schedule & Retention

Job 1 — PBS (enabled): Mon/Wed/Fri at 21:00

  • All VMs/CTs except VM107 (PBS itself)
  • Storage: pbs (PBS datastore pbs-store on TrueNAS NFS at /mnt/pbsdataset/pbs-store)
  • Retention: keep-daily=7, keep-weekly=4, keep-monthly=3
  • Email notification: jgitta@jgitta.com

Job 2 — TrueNAS NFS (disabled): Sun/Tue/Thu/Sat at 00:00

  • All VMs, storage: Proxmox-TrueNAS-NFS, keep-last=5

TrueNAS (192.168.88.24)

  • TrueNAS SCALE 24.10.2.4
  • pool1: RAIDZ2, 8 disks, ~63.7TB used / ~104.4TB total, ONLINE/healthy
    • Hosts PBS backup store (NFS-exported to VM107)
    • Hosts cold-storage NFS export (58TB, mounted on VM103)
  • API: use curl -sk https://192.168.88.24/api/v2.0/<endpoint> -u "root:<password>"
  • MCP connection is broken — use curl API directly
  • Disks with SMART warnings: sda/9JH31S2T (40 unreadable, 24 ATA errors), sdh/9JG47UTT (13 ATA errors)

Log Locations

Service Location Command
Jellyfin /var/log/jellyfin/ tail -f /var/log/jellyfin/jellyfin$(date +%Y%m%d).log on VM102
FFmpeg transcode /var/log/jellyfin/FFmpeg.Transcode-*.log VM102
Caddy systemd journal journalctl -u caddy -f on CT202
Pi-hole FTL DB / journal /etc/pihole/pihole-FTL.db, journalctl -u pihole-FTL on Siklos
Docker services (Siklos) docker logs docker compose logs -f in /srv/docker/<service>/
Proxmox /var/log/pve/ on jg-hud (192.168.88.25)
qBittorrent systemd journal journalctl -u qbittorrent-nox -f on VM102
ProtonVPN systemd journal journalctl -u me.proton.vpn.split_tunneling -f on VM102

Uptime Kuma Monitors (Siklos, port 3001)

HTTP checks: Caddy Health, Gitea, Glances, Grafana, Homarr, Home Assistant, Immich, Jellyfin, Karakeep, Linkwarden, MeshCentral, Nextcloud, PBS, Pi-hole web, Prometheus Port checks: MikroTik SSH (:22), Pi-hole DNS (:53), Proxmox SSH (:22), Siklos SSH (:22) DNS check: Pi-hole DNS resolution (google.com via 192.168.88.27:53) Metric check: Node Exporter on ThinkStation (192.168.88.41:9100)

Not monitored: VM102 SSH, qBittorrent, ProtonVPN status, Radarr/Sonarr/Prowlarr health, TrueNAS, Frigate