Add kopia restore guide, update Caddy config, add .gitignore to exclude credentials

2026-05-08 22:21:08 -05:00
parent 748f3cafc6
commit 4a6cfa9ed0
5 changed files with 843 additions and 0 deletions
@@ -0,0 +1,4 @@
+credentials.md
+kopia-docker-compose-fixed.yml
+*.secret
+*.env
@@ -0,0 +1,97 @@
+# Caddy Configuration for Homelab
+# Location: /srv/docker/caddy/Caddyfile
+# Last Updated: April 29, 2026
+
+# ============================================================================
+# Kopia Web UI - HTTPS Reverse Proxy
+# ============================================================================
+kopia.jgitta.com {
+    # Reverse proxy to Kopia Web UI running on localhost:51515
+    reverse_proxy 127.0.0.1:51515 {
+        # Pass original headers for proper proxying
+        header_upstream X-Forwarded-Proto {scheme}
+        header_upstream X-Forwarded-Host {http.request.host}
+        header_upstream X-Forwarded-For {http.request.remote.host}
+
+        # WebSocket support (Kopia uses WebSockets for real-time updates)
+        header_upstream Connection {http.request.header.Connection}
+        header_upstream Upgrade {http.request.header.Upgrade}
+
+        # Timeout settings for backup operations (which can be slow)
+        timeout 300s
+    }
+
+    # Security headers
+    header {
+        # Prevent MIME type sniffing
+        X-Content-Type-Options "nosniff"
+
+        # Clickjacking protection
+        X-Frame-Options "SAMEORIGIN"
+
+        # XSS protection
+        X-XSS-Protection "1; mode=block"
+
+        # HTTPS enforcement (HSTS)
+        # max-age: 1 year, includeSubDomains: apply to subdomains
+        Strict-Transport-Security "max-age=31536000; includeSubDomains"
+
+        # Content Security Policy (basic)
+        Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'"
+
+        # Referrer policy
+        Referrer-Policy "strict-origin-when-cross-origin"
+
+        # Permissions policy
+        Permissions-Policy "geolocation=(), microphone=(), camera=()"
+    }
+
+    # Logging
+    log {
+        output stdout
+        format json
+    }
+
+    # Enable TLS (automatic, self-signed for internal use)
+    tls internal
+}
+
+# ============================================================================
+# HTTP to HTTPS Redirect
+# ============================================================================
+http://kopia.jgitta.com {
+    # Redirect all HTTP traffic to HTTPS
+    redir https://kopia.jgitta.com{uri} permanent
+}
+
+# ============================================================================
+# Portainer - Docker Management UI on siklos
+# ============================================================================
+portainer.jgitta.com {
+    # Reverse proxy to Portainer on siklos (HTTP port — Caddy handles TLS)
+    reverse_proxy 192.168.88.27:9000 {
+        header_upstream X-Forwarded-Proto {scheme}
+        header_upstream X-Forwarded-Host {http.request.host}
+        header_upstream X-Forwarded-For {http.request.remote.host}
+    }
+
+    # Security headers
+    header {
+        X-Content-Type-Options "nosniff"
+        X-Frame-Options "SAMEORIGIN"
+        X-XSS-Protection "1; mode=block"
+        Strict-Transport-Security "max-age=31536000; includeSubDomains"
+        Referrer-Policy "strict-origin-when-cross-origin"
+    }
+
+    log {
+        output stdout
+        format json
+    }
+
+    tls internal
+}
+
+http://portainer.jgitta.com {
+    redir https://portainer.jgitta.com{uri} permanent
+}
@@ -0,0 +1,153 @@
+# Kopia Restore Guide
+
+**Last Updated**: May 8, 2026  
+**Applies To**: Both Kopia repositories on siklos (192.168.88.27)
+
+---
+
+## ⚠️ Critical: What You Need to Restore
+
+You need **two things** to restore from Backblaze B2. Without either one, data is unrecoverable:
+
+1. **Repository password** — stored in `credentials.md`
+2. **Backblaze B2 account access** — credentials in `credentials.md`
+
+> **Do not store these only on siklos.** Keep them in a password manager or printed copy somewhere safe. If siklos dies and you haven't saved the password elsewhere, your backup is unreadable.
+
+---
+
+## 📦 Your Two Repositories
+
+| Container | B2 Bucket | What It Backs Up | Port |
+|-----------|-----------|-----------------|------|
+| `Kopia` | jg-kopia-nextcloud | Nextcloud files + database | 51515 |
+| `Kopia-Immich` | jg-kopia-immich | Immich photos + database | 51516 |
+
+---
+
+## 🔍 Step 1 — List Available Snapshots
+
+SSH to siklos first:
+```bash
+ssh jgitta@siklos
+```
+
+**Nextcloud snapshots:**
+```bash
+docker exec -e KOPIA_PASSWORD='<password>' Kopia kopia snapshot list
+```
+
+**Immich snapshots:**
+```bash
+docker exec -e KOPIA_PASSWORD='<password>' Kopia-Immich kopia snapshot list
+```
+
+Each snapshot has an ID like `k8adffae645082989d7c0f4f051439332` and a timestamp. You can restore from any snapshot in history — not just the latest.
+
+---
+
+## 📁 Step 2 — Browse a Snapshot (Find Specific Files)
+
+You can explore a snapshot like a folder before restoring anything:
+
+```bash
+# List the top-level contents of a snapshot
+docker exec -e KOPIA_PASSWORD='<password>' Kopia kopia ls <snapshot-id>
+
+# Drill into a subfolder
+docker exec -e KOPIA_PASSWORD='<password>' Kopia kopia ls <snapshot-id>/jgitta/files/Documents
+
+# Search for a specific filename across a snapshot
+docker exec -e KOPIA_PASSWORD='<password>' Kopia kopia find <snapshot-id> --name "*.jpg"
+```
+
+This lets you confirm exactly what you want to restore before doing it.
+
+---
+
+## ♻️ Step 3 — Restore Options
+
+### Restore a Single File
+```bash
+docker exec -e KOPIA_PASSWORD='<password>' Kopia kopia restore \
+  <snapshot-id>/path/to/filename.jpg \
+  /tmp/restore/filename.jpg
+```
+
+### Restore a Specific Folder
+```bash
+docker exec -e KOPIA_PASSWORD='<password>' Kopia kopia restore \
+  <snapshot-id>/jgitta/files/Photos \
+  /tmp/restore/Photos/
+```
+
+### Restore an Entire Snapshot
+```bash
+docker exec -e KOPIA_PASSWORD='<password>' Kopia kopia restore \
+  <snapshot-id> \
+  /tmp/restore/
+```
+
+> **Note:** Restored files land on siklos at `/tmp/restore/`. From there you can copy them to Nextcloud, Immich, or anywhere else via `scp` or `rsync`.
+
+---
+
+## 🚨 Emergency: Siklos Is Gone — Restore from Scratch
+
+If siklos itself is dead, you can restore to any Linux machine:
+
+### 1. Install Kopia
+```bash
+curl -s https://kopia.io/signing-key | sudo gpg --dearmor -o /usr/share/keyrings/kopia-keyring.gpg
+echo "deb [signed-by=/usr/share/keyrings/kopia-keyring.gpg] http://packages.kopia.io/apt/ stable main" | sudo tee /etc/apt/sources.list.d/kopia.list
+sudo apt update && sudo apt install kopia
+```
+
+### 2. Connect to Your B2 Repository
+
+**For Nextcloud backup:**
+```bash
+kopia repository connect b2 \
+  --bucket=jg-kopia-nextcloud \
+  --key-id=00522b2471e5f090000000003 \
+  --key=K005wv6tlgknxkM9arOJV/tMeOInOQY \
+  --password=<repository-password>
+```
+
+**For Immich backup:**
+```bash
+kopia repository connect b2 \
+  --bucket=jg-kopia-immich \
+  --key-id=00522b2471e5f090000000004 \
+  --key=K005gSepxpWk9hWx/MjWnCKtOiNuKnQ \
+  --password=<repository-password>
+```
+
+### 3. List and Restore
+```bash
+kopia snapshot list
+kopia restore <snapshot-id> /path/to/restore/
+```
+
+> The B2 key IDs and application keys above are in `credentials.md`. Replace `<repository-password>` with the password from `credentials.md`.
+
+---
+
+## 💡 Key Points
+
+- **Backblaze "Browse Files"** shows encrypted chunks — this is normal and expected. Files are not human-readable in B2 directly.
+- **Every snapshot is independent** — you can restore from last week's snapshot even if today's is corrupted.
+- **Partial restores are fully supported** — restore one file, one folder, or everything.
+- **Kopia is open source** — even if the project were abandoned, you can always download the binary from GitHub and reconnect to your B2 repository as long as you have your password.
+
+---
+
+## 📋 Restore Checklist
+
+- [ ] Have repository password (from `credentials.md`)  
+- [ ] Have B2 credentials (from `credentials.md`)  
+- [ ] SSH to siklos (or install Kopia on alternate machine)  
+- [ ] Run `snapshot list` to find correct snapshot by date  
+- [ ] Use `kopia ls` to browse and confirm the right files  
+- [ ] Run `kopia restore` with specific path for targeted restore  
+- [ ] Copy restored files to final destination  
@@ -0,0 +1,465 @@
+# Non-Disruptive Backup Strategy for Nextcloud
+
+**Last Updated**: April 28, 2026  
+**Problem**: Previous backup approach froze the system (load avg: 21.89 on 4 CPUs)  
+**Solution**: Implement intelligent backup strategy that doesn't impact live system
+
+---
+
+## What Went Wrong
+
+Your previous backup process:
+```
+tar --zstd -cf /mnt/warm-storage/.backup-staging/... /mnt/nextcloud-data/
+```
+
+**Problems:**
+1. Reading 1.4TB live data while the system is running
+2. Compressing with zstd (CPU-intensive) at same time
+3. Writing compressed data to disk (more I/O)
+4. All three happening simultaneously = I/O bottleneck
+5. Load average jumped to 21.89 (system completely frozen)
+6. Apache couldn't respond to requests (504 errors)
+7. SSH connections hung
+
+**Key insight**: Live backups of large, active datasets require careful planning to avoid saturating disk I/O.
+
+---
+
+## Backup Strategy Comparison
+
+### Strategy 1: LVM Snapshot + Backup ⭐ RECOMMENDED
+
+**How it works:**
+1. Create a frozen copy (snapshot) of `/mnt/nextcloud-data` 
+2. Backup the snapshot while original stays live
+3. Delete snapshot when done
+
+**Pros:**
+- Original data unaffected while backup reads snapshot
+- Read-only snapshot doesn't create new I/O load
+- Fast (COW - Copy-on-Write)
+- Works with any backup tool
+
+**Cons:**
+- Requires LVM (you have it: `/dev/sdb`)
+- Snapshot space depends on delta changes
+
+**Best for**: Your setup (1.4TB Nextcloud data)
+
+---
+
+### Strategy 2: Kopia Incremental Backups (Configured Properly)
+
+**How it works:**
+- Kopia only backs up changed blocks
+- First backup is full, then incrementals
+- Can throttle I/O impact
+
+**Pros:**
+- Incremental = smaller backups after first one
+- Built-in deduplication
+- Cloud-ready (Backblaze B2)
+
+**Cons:**
+- First full backup is still heavy
+- Needs I/O rate limiting configured
+- Incremental issues if not properly scheduled
+
+**Best for**: Ongoing maintenance backups
+
+---
+
+### Strategy 3: Database-Specific + File Sync
+
+**How it works:**
+- MariaDB dumps itself separately (consistent point-in-time)
+- Rsync/Kopia sync files incrementally
+- Lower I/O overall
+
+**Pros:**
+- Database integrity guaranteed
+- Can schedule separately
+- Flexible
+
+**Cons:**
+- More manual orchestration
+- Need to coordinate timing
+
+**Best for**: Hybrid approach with LVM snapshots
+
+---
+
+### Strategy 4: Separate Backup VM
+
+**How it works:**
+- NFS mount nextcloud-data on backup VM
+- Backup VM does all I/O work
+- Production VM untouched
+
+**Pros:**
+- Zero impact on production
+- Can backup without affecting users
+
+**Cons:**
+- Requires extra resources
+- Network I/O instead of local
+
+**Best for**: Very high-availability setups
+
+---
+
+## RECOMMENDED APPROACH: LVM Snapshot Strategy
+
+Your `/mnt/nextcloud-data` is on `/dev/sdc` (2TB disk). Create backup snapshots from there.
+
+### Implementation
+
+#### Step 1: Create Backup Script
+
+Create `/usr/local/bin/nextcloud-backup.sh`:
+
+```bash
+#!/bin/bash
+set -e
+
+# Configuration
+NEXTCLOUD_MOUNT="/mnt/nextcloud-data"
+BACKUP_DEST="/mnt/warm-storage/.backup-staging"
+SNAPSHOT_NAME="nextcloud-backup-$(date +%Y%m%d_%H%M%S)"
+SNAPSHOT_SIZE="200G"  # Max snapshot size (change if needed)
+BACKUP_LOG="/var/log/nextcloud-backup.log"
+
+# Logging function
+log() {
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$BACKUP_LOG"
+}
+
+# Error handling
+cleanup() {
+    if [ -n "$SNAPSHOT_MOUNT" ] && mountpoint -q "$SNAPSHOT_MOUNT"; then
+        log "Unmounting snapshot..."
+        umount "$SNAPSHOT_MOUNT"
+    fi
+    if [ -n "$SNAPSHOT_LV" ] && lvdisplay "$SNAPSHOT_LV" &>/dev/null; then
+        log "Removing snapshot $SNAPSHOT_LV..."
+        lvremove -f "$SNAPSHOT_LV"
+    fi
+}
+
+trap cleanup EXIT
+
+log "Starting Nextcloud backup to $BACKUP_DEST"
+
+# Find the LV backing the nextcloud mount
+SOURCE_LV=$(df "$NEXTCLOUD_MOUNT" | tail -1 | awk '{print $1}')
+log "Source LV: $SOURCE_LV"
+
+if [ -z "$SOURCE_LV" ]; then
+    log "ERROR: Could not determine LV for $NEXTCLOUD_MOUNT"
+    exit 1
+fi
+
+# Create snapshot
+SNAPSHOT_LV="/dev/vg-nextcloud/snapshot-$SNAPSHOT_NAME"
+log "Creating snapshot $SNAPSHOT_LV (size: $SNAPSHOT_SIZE)..."
+lvcreate -L"$SNAPSHOT_SIZE" -s -n "snapshot-$SNAPSHOT_NAME" "$SOURCE_LV"
+
+if [ $? -ne 0 ]; then
+    log "ERROR: Failed to create snapshot"
+    exit 1
+fi
+
+# Mount snapshot temporarily
+SNAPSHOT_MOUNT="/mnt/snapshot-backup-temp"
+mkdir -p "$SNAPSHOT_MOUNT"
+log "Mounting snapshot to $SNAPSHOT_MOUNT..."
+mount -r "$SNAPSHOT_LV" "$SNAPSHOT_MOUNT"
+
+if [ $? -ne 0 ]; then
+    log "ERROR: Failed to mount snapshot"
+    exit 1
+fi
+
+# Run backup (reads from snapshot, not live data)
+BACKUP_FILE="$BACKUP_DEST/nextcloud-backup-$SNAPSHOT_NAME.tar.zstd"
+mkdir -p "$BACKUP_DEST"
+
+log "Backing up snapshot to $BACKUP_FILE..."
+log "This may take 30-60 minutes (reading 1.4TB)..."
+
+# Nice and ionice reduce I/O impact on system
+nice -n 15 ionice -c3 tar --zstd -cf "$BACKUP_FILE" -C "$SNAPSHOT_MOUNT" .
+
+if [ $? -eq 0 ]; then
+    BACKUP_SIZE=$(du -h "$BACKUP_FILE" | cut -f1)
+    log "✓ Backup successful: $BACKUP_SIZE"
+else
+    log "✗ Backup failed"
+    exit 1
+fi
+
+# Cleanup snapshot (done automatically via trap)
+log "Backup complete. Cleaning up..."
+```
+
+#### Step 2: Make it Executable
+
+```bash
+chmod +x /usr/local/bin/nextcloud-backup.sh
+```
+
+#### Step 3: Create Systemd Service + Timer
+
+Create `/etc/systemd/system/nextcloud-backup.service`:
+
+```ini
+[Unit]
+Description=Nextcloud Backup Service
+After=local-fs.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/nextcloud-backup.sh
+StandardOutput=journal
+StandardError=journal
+User=root
+
+# Don't use too much I/O
+Nice=15
+IOSchedulingClass=idle
+```
+
+Create `/etc/systemd/system/nextcloud-backup.timer`:
+
+```ini
+[Unit]
+Description=Nextcloud Backup Timer
+Requires=nextcloud-backup.service
+
+[Timer]
+# Run at 2 AM daily when users are asleep
+OnCalendar=daily
+OnCalendar=*-*-* 02:00:00
+
+# Randomize within 5 minutes to avoid spikes
+RandomizedDelaySec=5min
+
+[Install]
+WantedBy=timers.target
+```
+
+#### Step 4: Enable and Test
+
+```bash
+# Reload systemd
+systemctl daemon-reload
+
+# Enable timer to start on boot
+systemctl enable nextcloud-backup.timer
+
+# Start the timer
+systemctl start nextcloud-backup.timer
+
+# Check status
+systemctl status nextcloud-backup.timer
+
+# View next scheduled run
+systemctl list-timers nextcloud-backup.timer
+
+# Manual test (runs immediately)
+systemctl start nextcloud-backup.service
+
+# Watch progress
+journalctl -u nextcloud-backup.service -f
+```
+
+---
+
+## Alternative: Kopia with Proper Rate Limiting
+
+If you prefer to keep Kopia:
+
+### Configure Kopia for Non-Disruptive Backups
+
+Edit Kopia config (usually `/srv/docker/kopia/config/kopia.json`):
+
+```json
+{
+  "uploads": {
+    "maxParallelFileWrites": 2,
+    "maxParallelSmallFileWrites": 2,
+    "ignoreFileErrors": false
+  },
+  "cache": {
+    "metadata": {
+      "maxCacheSize": 500000000
+    }
+  },
+  "logging": {
+    "level": "info"
+  }
+}
+```
+
+### Run Kopia with Nice/Ionice
+
+```bash
+# Instead of running kopia directly
+nice -n 15 ionice -c3 docker exec kopia kopia snapshot create
+```
+
+Or in crontab:
+```cron
+# 3 AM daily
+0 3 * * * nice -n 15 ionice -c3 docker exec kopia kopia snapshot create >> /var/log/kopia-backup.log 2>&1
+```
+
+---
+
+## Implementation Roadmap
+
+### Week 1: Preparation
+- [ ] Verify LVM setup on nextcloud VM
+- [ ] Create backup script
+- [ ] Test snapshot creation/deletion
+- [ ] Verify backup file integrity
+
+### Week 2: Automation
+- [ ] Install systemd service/timer
+- [ ] Run first automated backup
+- [ ] Monitor system load during backup
+- [ ] Verify load average stays below 10
+
+### Week 3: Refinement
+- [ ] Adjust backup schedule (time/frequency)
+- [ ] Add monitoring alerts
+- [ ] Document recovery procedures
+- [ ] Test restoration from backup
+
+### Week 4: Production
+- [ ] Run weekly backups
+- [ ] Monitor Backblaze B2 uploads
+- [ ] Schedule monthly test restores
+- [ ] Update documentation
+
+---
+
+## Key Differences: Before vs. After
+
+| Aspect | Before | After |
+|--------|--------|-------|
+| **Load during backup** | 21.89 (system frozen) | <8 (acceptable) |
+| **Web service** | 504 errors | Responsive |
+| **SSH** | Hangs | Responsive |
+| **Backup method** | Live tar + zstd | LVM snapshot |
+| **Schedule** | Manual (caused outage) | Automated (2 AM daily) |
+| **I/O strategy** | Full speed (killer) | Rate-limited (nice/ionice) |
+
+---
+
+## Monitoring
+
+### Check Backup Status
+
+```bash
+# View backup history
+ls -lh /mnt/warm-storage/.backup-staging/
+
+# Check backup size
+du -sh /mnt/warm-storage/.backup-staging/
+
+# View logs
+journalctl -u nextcloud-backup.service --since "1 day ago"
+```
+
+### Monitor Load During Backup
+
+```bash
+# On another terminal, watch system during backup
+watch -n 1 'uptime && echo "---" && iostat -x 1 1 | grep sdc'
+```
+
+**What to expect:**
+- Load: 4-8 (normal for 4-CPU system during backup)
+- Web users: Unaffected
+- SSH: Responsive
+- Apache/PHP: Normal response times
+
+---
+
+## Recovery: How to Restore from Backup
+
+When you need to restore:
+
+```bash
+# List available backups
+ls -lh /mnt/warm-storage/.backup-staging/
+
+# Extract to temp location (DON'T overwrite live data!)
+mkdir -p /mnt/restore-test
+tar --zstd -xf /mnt/warm-storage/.backup-staging/nextcloud-backup-20260428_020000.tar.zstd \
+    -C /mnt/restore-test
+
+# Verify files are there
+ls /mnt/restore-test/ | head -20
+
+# If good, proceed with restore to actual location
+# (This requires stopping Nextcloud first to avoid corruption)
+```
+
+---
+
+## Troubleshooting
+
+### "Failed to create snapshot: No space left on device"
+
+The snapshot storage filled up. Increase `$SNAPSHOT_SIZE` in the script:
+```bash
+SNAPSHOT_SIZE="500G"  # Instead of 200G
+```
+
+### "Backup file is too large"
+
+Disable compression to see raw size:
+```bash
+tar -cf - /mnt/nextcloud-data | wc -c
+# Divide by 1TB (1099511627776) to see size in TB
+```
+
+If uncompressed is >2TB, you need more storage on warm-storage.
+
+### Backups not running automatically
+
+```bash
+# Check timer is running
+systemctl status nextcloud-backup.timer
+
+# Enable it if disabled
+systemctl enable nextcloud-backup.timer
+
+# Check if service failed
+systemctl status nextcloud-backup.service
+journalctl -u nextcloud-backup.service -n 50
+```
+
+---
+
+## Summary
+
+**Use the LVM snapshot approach** because:
+1. ✅ Zero impact on live system during backup
+2. ✅ Consistent point-in-time copy
+3. ✅ Safe to interrupt without data loss
+4. ✅ Easy to automate with systemd
+5. ✅ Works with any backup tool (tar, Kopia, rsync)
+
+Next steps:
+1. SSH to nextcloud VM
+2. Copy backup script from above
+3. Create systemd service + timer
+4. Enable and test
+5. Monitor first backup run
+6. Adjust schedule based on timing
+
+You'll have reliable, non-disruptive backups running every night at 2 AM with zero system impact! 🎯
@@ -0,0 +1,124 @@
+# Siklos Docker Services — Project Knowledge
+
+## Host
+- **VM112 / siklos / docker-server**
+- IP: `192.168.88.27`
+- Specs: 4c/12GB RAM (reduced from 16GB after photo services migrated to VM113)
+- SSH: `jgitta@192.168.88.27` (via Proxmox host jump)
+- Docker compose files: `/srv/docker/<service>/docker-compose.yml`
+- Note: `vm.swappiness=10` set in `/etc/sysctl.conf` (April 2026)
+
+## Running Containers (as of April 2026)
+
+| Container | Image | Port(s) | Compose Path |
+|---|---|---|---|
+| portainer | portainer/portainer-ce:latest | 9000, 9443 | /srv/docker/portainer/ |
+| pihole | pihole/pihole:latest | 53, 8080 | /srv/docker/pihole/ |
+| open-webui | ghcr.io/open-webui/open-webui:main | 3000 | /srv/docker/open-webui/ |
+| onlyoffice | onlyoffice/documentserver | 8880 | /srv/docker/media/ |
+| homarr | homarr:latest | 7575 | /srv/docker/homarr/ |
+| uptime-kuma | uptime-kuma:2 | 3001 | /srv/docker/uptime-kuma/ |
+| grafana | grafana:latest | 3020 | /srv/docker/monitoring/ |
+| prometheus | prom/prometheus:latest | 9090 | /srv/docker/monitoring/ |
+| node-exporter | prom/node-exporter:latest | 9100 | /srv/docker/monitoring/ |
+| cadvisor | cadvisor:latest | 8090 | /srv/docker/monitoring/ |
+| graphite-exporter | prom/graphite-exporter:latest | 9108-9109 | /srv/docker/monitoring/ |
+| glances | nicolargo/glances:latest | 61208 | /srv/docker/glances/ |
+| meshcentral | typhonragewind/meshcentral:latest | 444 | /srv/docker/meshcentral/ |
+| guacamole | jwetzell/guacamole | 8383 | /srv/docker/guacamole/ |
+| karakeep-web-1 | karakeep:release | 3010 | /srv/docker/karakeep/ |
+| karakeep-meilisearch-1 | meilisearch:v1.13.3 | 7700 (internal) | /srv/docker/karakeep/ |
+| karakeep-chrome-1 | alpine-chrome:124 | — | /srv/docker/karakeep/ |
+| linkwarden-linkwarden-1 | linkwarden:latest | 3015 | /srv/docker/linkwarden/ |
+| linkwarden-postgres-1 | postgres:16-alpine | 5432 | /srv/docker/linkwarden/ |
+| wordpress | wordpress:php8.3-apache | 8095 | /srv/docker/wordpress/ |
+| wordpress-db | mariadb:10.11 | 3306 (internal) | /srv/docker/wordpress/ |
+| dashy | lissy93/dashy:latest | 8081 | /srv/docker/dashy/ |
+| dashboard | dashboard-dashboard | 8096 | /srv/docker/dashboard/ |
+| grav | linuxserver/grav:latest | 8585 | /srv/docker/grav/ |
+| watchtower | containrrr/watchtower | — | /srv/docker/watchtower/ |
+
+## Open WebUI
+- URL: `https://ai.jgitta.com`
+- Port: 3000
+- Compose: `/srv/docker/open-webui/docker-compose.yml`
+- Connects to Ollama backend running on ThinkStation (192.168.88.41, no external port)
+
+## Migrated Services
+- **PhotoPrism** and **Immich** were migrated to VM113 (photos, 192.168.88.32) in April 2026
+- See `/home/jgitta/Documents/Claude/Projects/Photos/photos.md` for current details
+
+## Pi-hole
+- Port: 53 (DNS), 8080 (web UI)
+- URL: `https://pihole.jgitta.com`
+- **Network mode: `host`** (no docker-proxy — eliminates the port-53 freeze bug)
+- Config: `listeningMode = "ALL"` in pihole.toml
+- **Upstream DNS: `192.168.88.1` only** (MikroTik) — MikroTik resolves local jgitta.com domains and forwards everything else to 1.1.1.1/8.8.8.8. Do NOT add 1.1.1.1 directly as a Pi-hole upstream — dnsmasq load-balances across all upstreams, which would cause local domains to sometimes resolve to the WAN IP instead of internal Caddy IP.
+- FTL DB: `/etc/pihole/pihole-FTL.db`
+- Rate limit: 300 concurrent queries
+- Excluded from Watchtower auto-updates
+- Pi-hole v6
+- **Watchdog**: native cron on Siklos at `*/5 * * * *` → `/usr/local/bin/pihole-watchdog.sh` (restarts if DNS fails); Claude scheduled task is a secondary layer
+- Uptime Kuma DNS monitor uses `192.168.88.27` (host IP, not Docker internal)
+
+## Monitoring Stack
+- Compose: `/srv/docker/monitoring/`
+- Grafana: port 3020 (`grafana.jgitta.com`), datasource UID: `cffiqslf48feod`
+- Prometheus: port 9090
+- Node Exporter on: siklos (.27), proxmox (.25), nextcloud (.62), jellyfin (.10), pbs (.60), caddy (.110), thinkstation (.41) — all port 9100
+- Grafana alert folder "Homelab Alerts":
+  - High RAM >90% for 5min
+  - Swap >50% for 5min
+  - CPU >90% for 10min
+  - Disk >85% for 5min
+  - Node Down 2min
+- Alert annotations: `{{ $labels.instance }}` and `{{ $values.B }}%`
+- Alerts use three-step reduce+threshold pipeline (not classic conditions)
+- Telegram: bot token `8758434542:AAEW6omM7twyInsb2INuy6mD1w2EWXHqmzE`, chat `8260387200`, repeat every 4h
+- Uptime Kuma: port 3001 (`status.jgitta.com`), joined to `pihole_default` network
+
+## OnlyOffice
+- URL: `https://office.jgitta.com`
+- Port: 8880 (all interfaces)
+- Compose: `/srv/docker/media/docker-compose.yml`
+- Replaces Collabora (`richdocuments` app is disabled in Nextcloud; `onlyoffice` app is enabled)
+- JWT secret (must match Nextcloud config): `4f2b0c719af2de99befacfec9ca5e8373cbdeb76`
+- Nextcloud `occ` settings (set on VM103/next):
+  - `DocumentServerUrl` = `https://office.jgitta.com/`
+  - `DocumentServerInternalUrl` = `http://192.168.88.27:8880/`
+  - `StorageUrl` = `https://next.jgitta.com/`
+  - `jwt_secret` = (matches container `local.json` above)
+  - `jwt_header` = `Authorization`
+- To reconfigure after container recreation: re-run `occ config:app:set onlyoffice jwt_secret --value="<secret from local.json>"`
+- Container JWT secret location: `/etc/onlyoffice/documentserver/local.json` → `.services.CoAuthoring.secret.inbox.string`
+
+## Key Notes
+- Watchtower excludes Pi-hole from auto-updates
+- NFS mount `/mnt/photos` was removed from Siklos `/etc/fstab` after PhotoPrism/Immich migration
+- OnlyOffice replaced Collabora (lighter RAM usage)
+- RAM reduced from 16GB → 12GB (April 2026, live via Proxmox balloon driver, no reboot)
+- `vm.swappiness=10` set to reduce swap pressure after photo services migrated out
+- OnlyOffice replaced Collabora (lighter RAM usage)
+
+## Portainer Setup (April 2026)
+
+Portainer CE manages Docker across three hosts from a single UI.
+
+**Server (web UI):**
+- URL: `http://192.168.88.27:9000` or `https://192.168.88.27:9443`
+- Future URL (once Caddy SSH fixed): `https://portainer.jgitta.com`
+- Compose: `/srv/docker/portainer/docker-compose.yml`
+
+**Agents (managed remote environments):**
+| Host | IP | Agent Port | Compose Path |
+|---|---|---|---|
+| jellyfin | 192.168.88.10 | 9001 | /srv/docker/portainer-agent/ |
+| thinkstation | 192.168.88.41 | 9001 | /srv/docker/portainer-agent/ |
+
+**Adding agents in Portainer UI (first-time setup):**
+1. Open Portainer at `http://192.168.88.27:9000` and create your admin account
+2. Go to **Environments → Add environment → Agent**
+3. For jellyfin: name `jellyfin`, URL `192.168.88.10:9001`
+4. For thinkstation: name `thinkstation`, URL `192.168.88.41:9001`
+
+**Caddy config:** Entry for `portainer.jgitta.com` is ready in workspace Caddyfile — needs deploying once Caddy VM SSH access is restored.