diff --git a/jellyfin/jellyfin-services.md b/jellyfin/jellyfin-services.md index ae062d2..308c3fe 100644 --- a/jellyfin/jellyfin-services.md +++ b/jellyfin/jellyfin-services.md @@ -28,48 +28,66 @@ | prowlarr | lscr.io/linuxserver/prowlarr:latest | 9696 | /srv/docker/prowlarr/config | | flaresolverr | ghcr.io/flaresolverr/flaresolverr:latest | 8191 | — | | watchtower | containrrr/watchtower | — | see below | +| watchtower-prowlarr | containrrr/watchtower | — | see below | Media volume: `/mnt/media` on jellyfin host → mounted as `/data` in radarr/sonarr. --- -## Watchtower +## Watchtower — Split-Instance Setup -- Runs inside arr-stack on jellyfin (Portainer stack 35) -- Schedule: `0 0 4 * * *` — runs at 4:00 AM daily -- Notifications: email via Fastmail SMTP to jgitta@jgitta.com -- Cleanup: enabled (removes old images after update) -- Rolling restarts: enabled with 30s timeout, 60s wait before stopping +Two Watchtower instances run in the arr-stack to stagger registry checks and avoid +rate limiting lscr.io/ghcr.io with too many concurrent requests. -### lscr.io Authentication (fix applied 2026-05-11) +| Instance | Schedule | Scope | Manages | +|---|---|---|---| +| `watchtower` | 4:00 AM | (none) | radarr, sonarr, flaresolverr, portainer-agent | +| `watchtower-prowlarr` | 4:05 AM | `prowlarr` | prowlarr only | -**Problem**: Watchtower was hitting lscr.io's rate limiter on unauthenticated pulls. radarr, -sonarr, and prowlarr all use `lscr.io/linuxserver/` images; when Watchtower checks them -concurrently at 4 AM the burst briefly exceeded the anonymous quota. Error appeared in -email notifications as: -``` -Unable to update container "/radarr": Error response from daemon: toomanyrequests: -retry-after: 835.12µs, allowed: 44000/minute. Proceeding to next. -``` -Observed on: 2026-05-08 and 2026-05-11. +The scope works via Docker labels: prowlarr has +`com.centurylinklabs.watchtower.scope=prowlarr` — this tells the main `watchtower` +to skip it, and tells `watchtower-prowlarr` to manage it. + +Both instances mount `/home/jgitta/.docker/config.json:/config.json:ro` for +authenticated registry pulls. + +--- + +## Registry Authentication (fixed 2026-05-20) + +**Problem**: Watchtower was hitting lscr.io/ghcr.io rate limits and timeouts with +unauthenticated requests. lscr.io is a proxy that redirects to ghcr.io for the actual +registry API — credentials for lscr.io alone were not enough; ghcr.io also needed auth. +Two separate issues were compounding: +1. Unauthenticated requests shared a small anonymous quota +2. Two concurrent Watchtower instances were doubling all requests at 4 AM **Fix**: -1. Logged in to lscr.io on the jellyfin host using the GitHub PAT (credentials.md): - ``` - echo | docker login lscr.io -u jgitta --password-stdin - ``` - Creates `/home/jgitta/.docker/config.json` with lscr.io auth token. -2. Mounted that config into the Watchtower container (in docker-compose.yml): - ```yaml - volumes: - - /var/run/docker.sock:/var/run/docker.sock - - /home/jgitta/.docker/config.json:/config.json:ro - ``` - Watchtower reads `/config.json` inside the container for registry credentials. +Three sets of credentials were configured — both as the `jgitta` user (for Watchtower's +own registry digest checks) and as `root` (for the Docker daemon's actual image pulls): -3. Updated Gitea (`jellyfin/arr-stack/docker-compose.yml`) and redeployed via Portainer. +```bash +# As jgitta (Watchtower reads /home/jgitta/.docker/config.json via volume mount) +echo | docker login lscr.io -u jgitta --password-stdin +echo | docker login ghcr.io -u jgitta --password-stdin -**If GitHub PAT expires**: Re-run the `docker login lscr.io` command on jellyfin with the -new token. No compose changes needed — the mounted file updates automatically. -The PAT expiry date is tracked in credentials.md / github.com/settings/tokens. +# As root (Docker daemon uses /root/.docker/config.json for docker pull) +echo | sudo docker login lscr.io -u jgitta --password-stdin +echo | sudo docker login ghcr.io -u jgitta --password-stdin +``` + +Token used: the no-expiry `read:packages` GitHub PAT in credentials.md. + +**If token ever needs rotating**: run the four `docker login` commands above on jellyfin +with the new token. No compose changes needed — both config files are read live. + +--- + +## Notes +- Do NOT run `docker restart watchtower` manually while `watchtower-prowlarr` is running — + Watchtower's multi-instance detection will cause one to kill the other. Use + `docker restart watchtower watchtower-prowlarr` to restart both together, or let + Portainer manage redeployment. +- Portainer stack 35 is the authoritative runtime config. If it drifts from Gitea, + update via Portainer API with `prune: true` to remove orphaned services.