docs: update jellyfin-services.md with full Watchtower split-instance and auth details

jellyfin/jellyfin-services.md

@@ -28,48 +28,66 @@
 | prowlarr | lscr.io/linuxserver/prowlarr:latest | 9696 | /srv/docker/prowlarr/config |
 | flaresolverr | ghcr.io/flaresolverr/flaresolverr:latest | 8191 | — |
 | watchtower | containrrr/watchtower | — | see below |
+| watchtower-prowlarr | containrrr/watchtower | — | see below |
 
 Media volume: `/mnt/media` on jellyfin host → mounted as `/data` in radarr/sonarr.
 
 ---
 
-## Watchtower
+## Watchtower — Split-Instance Setup
 
-- Runs inside arr-stack on jellyfin (Portainer stack 35)
+Two Watchtower instances run in the arr-stack to stagger registry checks and avoid
-- Schedule: `0 0 4 * * *` — runs at 4:00 AM daily
+rate limiting lscr.io/ghcr.io with too many concurrent requests.
-- Notifications: email via Fastmail SMTP to jgitta@jgitta.com
-- Cleanup: enabled (removes old images after update)
-- Rolling restarts: enabled with 30s timeout, 60s wait before stopping
 
-### lscr.io Authentication (fix applied 2026-05-11)
+| Instance | Schedule | Scope | Manages |
+|---|---|---|---|
+| `watchtower` | 4:00 AM | (none) | radarr, sonarr, flaresolverr, portainer-agent |
+| `watchtower-prowlarr` | 4:05 AM | `prowlarr` | prowlarr only |
 
-**Problem**: Watchtower was hitting lscr.io's rate limiter on unauthenticated pulls. radarr,
+The scope works via Docker labels: prowlarr has
-sonarr, and prowlarr all use `lscr.io/linuxserver/` images; when Watchtower checks them
+`com.centurylinklabs.watchtower.scope=prowlarr` — this tells the main `watchtower`
-concurrently at 4 AM the burst briefly exceeded the anonymous quota. Error appeared in
+to skip it, and tells `watchtower-prowlarr` to manage it.
-email notifications as:
+
-```
+Both instances mount `/home/jgitta/.docker/config.json:/config.json:ro` for
-Unable to update container "/radarr": Error response from daemon: toomanyrequests:
+authenticated registry pulls.
-retry-after: 835.12µs, allowed: 44000/minute. Proceeding to next.
+
-```
+---
-Observed on: 2026-05-08 and 2026-05-11.
+
+## Registry Authentication (fixed 2026-05-20)
+
+**Problem**: Watchtower was hitting lscr.io/ghcr.io rate limits and timeouts with
+unauthenticated requests. lscr.io is a proxy that redirects to ghcr.io for the actual
+registry API — credentials for lscr.io alone were not enough; ghcr.io also needed auth.
+Two separate issues were compounding:
+1. Unauthenticated requests shared a small anonymous quota
+2. Two concurrent Watchtower instances were doubling all requests at 4 AM
 
 **Fix**:
-1. Logged in to lscr.io on the jellyfin host using the GitHub PAT (credentials.md):
-   ```
-   echo <github_pat> | docker login lscr.io -u jgitta --password-stdin
-   ```
-   Creates `/home/jgitta/.docker/config.json` with lscr.io auth token.
 
-2. Mounted that config into the Watchtower container (in docker-compose.yml):
+Three sets of credentials were configured — both as the `jgitta` user (for Watchtower's
-   ```yaml
+own registry digest checks) and as `root` (for the Docker daemon's actual image pulls):
-   volumes:
+
-     - /var/run/docker.sock:/var/run/docker.sock
+```bash
-     - /home/jgitta/.docker/config.json:/config.json:ro
+# As jgitta (Watchtower reads /home/jgitta/.docker/config.json via volume mount)
+echo <token> | docker login lscr.io -u jgitta --password-stdin
+echo <token> | docker login ghcr.io -u jgitta --password-stdin
+
+# As root (Docker daemon uses /root/.docker/config.json for docker pull)
+echo <token> | sudo docker login lscr.io -u jgitta --password-stdin
+echo <token> | sudo docker login ghcr.io -u jgitta --password-stdin
 ```
-   Watchtower reads `/config.json` inside the container for registry credentials.
 
-3. Updated Gitea (`jellyfin/arr-stack/docker-compose.yml`) and redeployed via Portainer.
+Token used: the no-expiry `read:packages` GitHub PAT in credentials.md.
 
-**If GitHub PAT expires**: Re-run the `docker login lscr.io` command on jellyfin with the
+**If token ever needs rotating**: run the four `docker login` commands above on jellyfin
-new token. No compose changes needed — the mounted file updates automatically.
+with the new token. No compose changes needed — both config files are read live.
-The PAT expiry date is tracked in credentials.md / github.com/settings/tokens.
+
+---
+
+## Notes
+- Do NOT run `docker restart watchtower` manually while `watchtower-prowlarr` is running —
+  Watchtower's multi-instance detection will cause one to kill the other. Use
+  `docker restart watchtower watchtower-prowlarr` to restart both together, or let
+  Portainer manage redeployment.
+- Portainer stack 35 is the authoritative runtime config. If it drifts from Gitea,
+  update via Portainer API with `prune: true` to remove orphaned services.