diff --git a/README.md b/README.md index d991104..fbe0a4a 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,6 @@ All stacks are managed via **Portainer** at http://192.168.88.27:9000 (or https: | guacamole | 8080 | /srv/docker/guacamole | apache.jgitta.com | Apache Guacamole — remote desktop gateway | | homarr | 7575 | /srv/docker/homarr/.homarr/appdata | homarr.jgitta.com | Dashboard | | karakeep | 3010 | /srv/docker/karakeep | notes.jgitta.com | Bookmarks / read-later | -| kopia | 51515 | /srv/docker/kopia | kopia.jgitta.com | Backup tool (backs up to TrueNAS) | | linkwarden | 3015 | /srv/docker/linkwarden/pgdata | links.jgitta.com | Link manager | | litellm | 4000 | /srv/docker/litellm | — | LLM API gateway | | meshcentral | 444 | /srv/docker/meshcentral | mesh.jgitta.com | Remote management | @@ -53,6 +52,7 @@ All stacks are managed via **Portainer** at http://192.168.88.27:9000 (or https: | pihole | host network | /srv/docker/pihole | pihole.jgitta.com | DNS ad blocker + faster-whisper + open-webui | | portainer | 9000, 9443 | /srv/docker/portainer | portainer.jgitta.com | Container management UI | | uptime-kuma | 3001 | /srv/docker/uptime-kuma/data | status.jgitta.com | Uptime monitoring | +| ntfy | 2586 | /srv/docker/ntfy | ntfy.jgitta.com | Push notifications (replaces Telegram); topic: homelab-alerts; Caddy needs `transport http { versions 1.1 }` for WebSocket | | searxng | 8092 | /srv/docker/searxng | search.jgitta.com | Private search engine | | watchtower | — | — | — | Auto-updates containers | | wordpress | 8095 | /srv/docker/wordpress | jgitta.com | WordPress site | @@ -94,7 +94,7 @@ Config location: `/etc/caddy/` on the caddy VM. |---|---| | `Caddyfile` | Main file — sets global options, imports snippets and sites | | `snippets.caddy` | Reusable blocks: `web_secure`, `internal_only`, `proxy_timeouts`, etc. | -| `sites/infrastructure.caddy` | Management/tool services (portainer, kopia, beszel, pihole, etc.) | +| `sites/infrastructure.caddy` | Management/tool services (portainer, beszel, pihole, etc.) | | `sites/media.caddy` | Media and automation stack (jellyfin, sonarr, radarr, next, etc.) | | `sites/homenet.caddy` | Internal `.homenet` HTTP-only domains (no TLS) | | `sites/goldengamer.caddy` | goldengamer.org external domain | @@ -121,15 +121,6 @@ Old compose files were renamed to `.bak` after migration to Portainer. **2026-07-03 — external stacks converted to Portainer:** authentik (stack 58), beszel (57), searxng (56) were created in Portainer; watchtower (18), karakeep (21), frigate (31), immich (45) were redeployed so Portainer owns the running containers. Frigate and immich stack definitions were updated to match the newer host compose files (frigate: cpuset pinning + 8G memory limit; immich: /mnt/immich-photos path + vectorchord postgres image). Host compose files renamed to `.bak`. Do not run `docker compose up` from /srv/docker — deploy via Portainer only. -## Kopia Backup Notes (updated 2026-05-07) - -Kopia backs up Nextcloud data (VM 103) and other sources to TrueNAS NFS. -- Cache limit set to 4 GB content + 512 MB metadata (`kopia cache set`) -- `/tmp` and `/app/logs` are bind-mounted out of the container to prevent overlay2 growth -- After-snapshot hook pings Uptime Kuma heartbeat at `status.jgitta.com` after each backup -- If disk fills up again, check `/srv/docker/kopia/tmp/` for leftover verify/restore artifacts -- Ignore policy active for `/data/.claude` in the Nextcloud data source - ## Portainer Agent Notes Portainer agents run on jellyfin and jgpc, allowing the central Portainer instance on diff --git a/authentik/SETUP-GUIDE.md b/authentik/SETUP-GUIDE.md index 4469cf8..e3e273d 100644 --- a/authentik/SETUP-GUIDE.md +++ b/authentik/SETUP-GUIDE.md @@ -193,6 +193,27 @@ WEB_OIDC_CLIENT_ID= --- +#### Mealie (`recipes.jgitta.com`) + +**Redirect URIs**: `https://recipes.jgitta.com/login` and `https://recipes.jgitta.com/login?direct=1` + +Set in the Mealie stack (Portainer stack 59, `siklos/mealie/docker-compose.yml` in Gitea): +``` +OIDC_AUTH_ENABLED=true +OIDC_SIGNUP_ENABLED=true +OIDC_CONFIGURATION_URL=https://auth.jgitta.com/application/o/mealie/.well-known/openid-configuration +OIDC_CLIENT_ID= +OIDC_CLIENT_SECRET= +OIDC_PROVIDER_NAME=Authentik +OIDC_AUTO_REDIRECT=false +OIDC_REMEMBER_ME=true +``` +Client ID/secret and the Authentik API token used to create this provider are in API Codes.md. + +Mealie links accounts by email, so the existing Mealie admin account (`jgitta`) had its email changed to `admin@jgitta.com` to match the Authentik `akadmin` account — otherwise the first OIDC login would have created a brand-new, empty Mealie account instead of linking to the one with all the recipes. Password login (`ALLOW_PASSWORD_LOGIN`, default true) was left enabled as a fallback. + +--- + #### Home Assistant (`ha.jgitta.com`, VM106) **Redirect URI**: `https://ha.jgitta.com/auth/oidc/callback` diff --git a/homelab-overview.md b/homelab-overview.md index ed6e3d7..c13be3f 100644 --- a/homelab-overview.md +++ b/homelab-overview.md @@ -82,7 +82,7 @@ Proxmox runs all the VMs and LXC containers listed below. Backups are handled by | Name | IP | Role | |---|---|---| | jgpc | 192.168.88.41 | Workstation / AI / Ollama (bare metal, not a Proxmox guest) | -| TrueNAS | 192.168.88.24 | NAS — provides NFS storage for Kopia backups; separate physical machine | +| TrueNAS | 192.168.88.24 | NAS — provides NFS storage for PBS backups; separate physical machine | --- @@ -96,7 +96,7 @@ Config location on the Caddy VM: `/etc/caddy/` |---|---| | `Caddyfile` | Main entry point — global options, imports all site files | | `snippets.caddy` | Reusable blocks: `web_secure`, `internal_only`, `proxy_timeouts` | -| `sites/infrastructure.caddy` | Management tools (Portainer, Grafana, Pi-hole, Kopia, etc.) | +| `sites/infrastructure.caddy` | Management tools (Portainer, Grafana, Pi-hole, etc.) | | `sites/media.caddy` | Media stack (Jellyfin, Sonarr, Radarr, Nextcloud, etc.) | | `sites/homenet.caddy` | Internal `.homenet` HTTP-only domains | | `sites/goldengamer.caddy` | External `goldengamer.org` domain | @@ -137,13 +137,13 @@ All stack data volumes are stored at `/srv/docker//` on each host us | Karakeep | karakeep:release | 3010 | notes.jgitta.com | Bookmarks / read-later | | Linkwarden | linkwarden:latest | 3015 | links.jgitta.com | Link manager | | Gitea | gitea:latest | 3002, 2222 | gitea.jgitta.com | Self-hosted Git server | -| Kopia | kopia:latest | 51515 | kopia.jgitta.com | Backup tool → TrueNAS NFS | | Actual Budget | actual-budget | 5006 | budget.jgitta.com | Personal budget tracking | | WordPress | wordpress:php8.3-apache | 8095 | jgitta.com | Personal website | | WordPress DB | mariadb:10.11 | 3306 (internal) | — | WordPress database | | Grav | linuxserver/grav:latest | 8585 | — | Flat-file CMS | | Watchtower | containrrr/watchtower | — | — | Auto-updates containers (excludes Pi-hole) | | GoldenGames | goldengames (local build) | 3030 | goldengamer.org | Gaming site | +| Vaultwarden | vaultwarden/server:latest | 8222 (web), 3012 (websocket) | vault.jgitta.com | Self-hosted Bitwarden-compatible password manager | --- @@ -223,7 +223,6 @@ Prometheus scrapes **Node Exporter** (port 9100) from every host: siklos, proxmo | All VMs/CTs (except 107) | PBS job `pbs-backup` | Mon/Wed/Fri 21:00 | PBS datastore `pbs-store` (TrueNAS NFS) | | VMs 113 (photos) + 114 (ocis) | PBS job `B2 offsite` | Sun 03:00 | PBS datastore `b2-offsite` → Backblaze B2 | | PBS configuration | Cron on PBS VM | Nightly 02:30 | TrueNAS NFS (`pbs-config-backup/`) | -| Nextcloud data | Kopia (Docker on siklos) | — | TrueNAS NFS | | Docker configs | Gitea repo | — | `jgitta/homelab-configs` | ### PBS (VM 107, 192.168.88.60) — verified 2026-07-14 @@ -231,7 +230,7 @@ Prometheus scrapes **Node Exporter** (port 9100) from every host: siklos, proxmo PBS has two datastores: - **`pbs-store`** — local path `/mnt/pbs-dataset/pbs-store`, which is an NFS mount from TrueNAS (`192.168.88.24:/mnt/pool1/pbs-dataset`). Receives the Mon/Wed/Fri full-fleet backups. Retention: keep-daily=7, keep-weekly=4, keep-monthly=3. -- **`b2-offsite`** — S3 backend pointing at Backblaze bucket `jg-kopia` (endpoint `s3.us-east-005.backblazeb2.com`), local cache at `/pbs-data`. Receives the Sunday 3 AM backups of VMs 113 and 114 only. This replaced the old Kopia → B2 workflow. Retention: keep-daily=7, keep-weekly=2. +- **`b2-offsite`** — S3 backend pointing at Backblaze bucket `jg-kopia` (endpoint `s3.us-east-005.backblazeb2.com`), local cache at `/pbs-data`. Receives the Sunday 3 AM backups of VMs 113 and 114 only. Retention: keep-daily=7, keep-weekly=2. **Finding data in the Backblaze web UI:** browse `jg-kopia` bucket → `b2-offsite/` folder → `.chunks/`. PBS deduplicates, so weekly runs upload only new/changed chunks — sparse upload activity between Sundays is normal. Verify connectivity from the pbs host with: `proxmox-backup-manager s3 check backblaze-b2 jg-kopia` @@ -255,5 +254,5 @@ PBS has two datastores: | Immich | https://photos.jgitta.com | | AI (Open WebUI) | https://ai.jgitta.com | | Guacamole | https://apache.jgitta.com | -| Kopia | https://kopia.jgitta.com | +| Vaultwarden | https://vault.jgitta.com | diff --git a/owncloud/organize_photos.sh b/owncloud/organize_photos.sh index 81e094a..8bda07b 100755 --- a/owncloud/organize_photos.sh +++ b/owncloud/organize_photos.sh @@ -1,4 +1,35 @@ #!/usr/bin/env bash +# ============================================================================= +# organize_photos.sh +# ============================================================================= +# Organizes files in the ownCloud Instant Upload folders into a consistent +# date-based folder structure: SUBDIR/YYYY/MM/filename +# +# The ownCloud Android app pre-sorts videos into YYYY/MM/ subfolders +# automatically, but uses YYYY/YYYY-MM/ for images. This script: +# 1. Moves any new files dropped in the SUBDIR root into YYYY/MM/ +# 2. Migrates old YYYY/YYYY-MM/ folders → YYYY/MM/ (removes redundant year) +# +# Processed subdirectories (under BASE/): +# Camera – phone photos and videos (instant upload) +# BlueIris – BlueIris NVR clip exports +# Facebook – Facebook photo downloads +# scans – scanned documents +# EufyVideoDir – Eufy security camera recordings +# +# Files with no recognisable date go to BASE/_unsorted/ unchanged. +# +# Date extraction tries these patterns in order (first match wins): +# 1. YYYYMMDD embedded anywhere in the filename (e.g. PXL_20260802_...) +# 2. YYYY-MM-DD or YYYY_MM_DD separator format (e.g. 2026-08-02) +# 3. 13-digit Unix millisecond timestamp (e.g. 1722614400000) +# 4. MM_DD_YYYY_ prefix (e.g. 08_02_2026_clip) +# +# Scheduling: run via cron or systemd timer. Example cron (every 15 min): +# */15 * * * * /path/to/organize_photos.sh +# +# Log file: /var/log/organize_photos.log +# ============================================================================= BASE="/mnt/INTEL-SSD/ownCloud - Joe Gitta@cloud.jgitta.com/Personal/InstantUpload" UNSORTED="$BASE/_unsorted" @@ -7,31 +38,54 @@ moved=0; skipped=0; unsorted=0; errors=0 echo "=== $(date +%Y-%m-%d\ %H:%M:%S) Starting ===" >> "$LOG" +# ----------------------------------------------------------------------------- +# ts_to_ym TIMESTAMP_MS +# Converts a 13-digit Unix millisecond timestamp to "YYYY MM". +# ----------------------------------------------------------------------------- ts_to_ym() { local ts_ms="$1" local ts_s=$(( ts_ms / 1000 )) date -d "@$ts_s" "+%Y %m" 2>/dev/null } +# ----------------------------------------------------------------------------- +# get_ym FILEPATH +# Extracts "YYYY MM" from a filename using the patterns described in the header. +# Prints "YYYY MM" on success, empty string if no date found. +# ----------------------------------------------------------------------------- get_ym() { local f="$1" local b b=$(basename "$f") + + # Pattern 1: YYYYMMDD anywhere in filename (e.g. PXL_20260802_171610514.mp4) if [[ "$b" =~ (^|[^0-9])([0-9]{4})(0[1-9]|1[0-2])(0[1-9]|[12][0-9]|3[01]) ]]; then echo "${BASH_REMATCH[2]} ${BASH_REMATCH[3]}"; return fi + + # Pattern 2: YYYY-MM-DD or YYYY_MM_DD (e.g. 2026-08-02_clip.mp4) if [[ "$b" =~ ([0-9]{4})[-_](0[1-9]|1[0-2])[-_](0[1-9]|[12][0-9]|3[01]) ]]; then echo "${BASH_REMATCH[1]} ${BASH_REMATCH[2]}"; return fi + + # Pattern 3: 13-digit Unix ms timestamp (e.g. 1722614400000_eufy.mp4) if [[ "$b" =~ (^|[^0-9])([0-9]{13})([^0-9]|$) ]]; then ts_to_ym "${BASH_REMATCH[2]}"; return fi + + # Pattern 4: MM_DD_YYYY_ prefix (e.g. 08_02_2026_doorbell.mp4) if [[ "$b" =~ ^([0-9]{2})_([0-9]{2})_([0-9]{4})_ ]]; then echo "${BASH_REMATCH[3]} ${BASH_REMATCH[1]}"; return fi + echo "" } +# ----------------------------------------------------------------------------- +# move_file SRC ROOT +# Moves SRC into ROOT/YYYY/MM/filename using the date extracted from the +# filename. Sends undatable files to UNSORTED. Skips if destination exists. +# ----------------------------------------------------------------------------- move_file() { local src="$1" local root="$2" @@ -39,7 +93,9 @@ move_file() { fname=$(basename "$src") local ym ym=$(get_ym "$src") + if [[ -z "$ym" ]]; then + # No date found — move to _unsorted for manual review mkdir -p "$UNSORTED" if [[ ! -e "$UNSORTED/$fname" ]]; then mv "$src" "$UNSORTED/$fname" && ((unsorted++)) || ((errors++)) @@ -48,10 +104,11 @@ move_file() { fi return fi + local year month dest_dir dest year=$(echo "$ym" | cut -d' ' -f1) month=$(echo "$ym" | cut -d' ' -f2) - dest_dir="$root/$year/$year-$month" + dest_dir="$root/$year/$month" dest="$dest_dir/$fname" mkdir -p "$dest_dir" if [[ -e "$dest" ]]; then @@ -60,12 +117,43 @@ move_file() { mv "$src" "$dest" && ((moved++)) || { echo " ERROR: $fname" >> "$LOG"; ((errors++)); } } +# ============================================================================= +# Main loop — process each instant-upload subdirectory +# ============================================================================= for SUBDIR in Camera BlueIris Facebook scans EufyVideoDir; do SUBPATH="$BASE/$SUBDIR" [[ -d "$SUBPATH" ]] || continue + + # --- Pass 1: files dropped directly into the subdir root ---------------- + # These are new uploads not yet organised by the Android app. while IFS= read -r -d '' f; do move_file "$f" "$SUBPATH" done < <(find "$SUBPATH" -maxdepth 1 -type f -print0) + + # --- Pass 2: migrate old YYYY/YYYY-MM/ folders → YYYY/MM/ --------------- + # Older versions of this script (and some ownCloud client versions) created + # YYYY/YYYY-MM/ subfolders, which redundantly repeat the year. This pass + # flattens them to YYYY/MM/ and removes the now-empty YYYY-MM directory. + for year_dir in "$SUBPATH"/[0-9][0-9][0-9][0-9]; do + [[ -d "$year_dir" ]] || continue + year=$(basename "$year_dir") + for old_dir in "$year_dir"/"$year"-[0-9][0-9]; do + [[ -d "$old_dir" ]] || continue + month="${old_dir##*-}" # extract MM from YYYY-MM folder name + dest_dir="$SUBPATH/$year/$month" + mkdir -p "$dest_dir" + while IFS= read -r -d '' f; do + fname=$(basename "$f") + dest="$dest_dir/$fname" + if [[ ! -e "$dest" ]]; then + mv "$f" "$dest" && ((moved++)) || { echo " ERROR: $fname" >> "$LOG"; ((errors++)); } + else + ((skipped++)) + fi + done < <(find "$old_dir" -maxdepth 1 -type f -print0) + rmdir "$old_dir" 2>/dev/null # Remove if now empty + done + done done echo " Moved: $moved Unsorted: $unsorted Skipped: $skipped Errors: $errors" >> "$LOG"