diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..2066fb9 --- /dev/null +++ b/.gitignore @@ -0,0 +1,4 @@ +credentials.md +kopia-docker-compose-fixed.yml +*.secret +*.env diff --git a/caddy/Caddyfile b/caddy/Caddyfile new file mode 100644 index 0000000..3c9fd08 --- /dev/null +++ b/caddy/Caddyfile @@ -0,0 +1,97 @@ +# Caddy Configuration for Homelab +# Location: /srv/docker/caddy/Caddyfile +# Last Updated: April 29, 2026 + +# ============================================================================ +# Kopia Web UI - HTTPS Reverse Proxy +# ============================================================================ +kopia.jgitta.com { + # Reverse proxy to Kopia Web UI running on localhost:51515 + reverse_proxy 127.0.0.1:51515 { + # Pass original headers for proper proxying + header_upstream X-Forwarded-Proto {scheme} + header_upstream X-Forwarded-Host {http.request.host} + header_upstream X-Forwarded-For {http.request.remote.host} + + # WebSocket support (Kopia uses WebSockets for real-time updates) + header_upstream Connection {http.request.header.Connection} + header_upstream Upgrade {http.request.header.Upgrade} + + # Timeout settings for backup operations (which can be slow) + timeout 300s + } + + # Security headers + header { + # Prevent MIME type sniffing + X-Content-Type-Options "nosniff" + + # Clickjacking protection + X-Frame-Options "SAMEORIGIN" + + # XSS protection + X-XSS-Protection "1; mode=block" + + # HTTPS enforcement (HSTS) + # max-age: 1 year, includeSubDomains: apply to subdomains + Strict-Transport-Security "max-age=31536000; includeSubDomains" + + # Content Security Policy (basic) + Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'" + + # Referrer policy + Referrer-Policy "strict-origin-when-cross-origin" + + # Permissions policy + Permissions-Policy "geolocation=(), microphone=(), camera=()" + } + + # Logging + log { + output stdout + format json + } + + # Enable TLS (automatic, self-signed for internal use) + tls internal +} + +# ============================================================================ +# HTTP to HTTPS Redirect +# ============================================================================ +http://kopia.jgitta.com { + # Redirect all HTTP traffic to HTTPS + redir https://kopia.jgitta.com{uri} permanent +} + +# ============================================================================ +# Portainer - Docker Management UI on siklos +# ============================================================================ +portainer.jgitta.com { + # Reverse proxy to Portainer on siklos (HTTP port — Caddy handles TLS) + reverse_proxy 192.168.88.27:9000 { + header_upstream X-Forwarded-Proto {scheme} + header_upstream X-Forwarded-Host {http.request.host} + header_upstream X-Forwarded-For {http.request.remote.host} + } + + # Security headers + header { + X-Content-Type-Options "nosniff" + X-Frame-Options "SAMEORIGIN" + X-XSS-Protection "1; mode=block" + Strict-Transport-Security "max-age=31536000; includeSubDomains" + Referrer-Policy "strict-origin-when-cross-origin" + } + + log { + output stdout + format json + } + + tls internal +} + +http://portainer.jgitta.com { + redir https://portainer.jgitta.com{uri} permanent +} diff --git a/kopia-restore-guide.md b/kopia-restore-guide.md new file mode 100644 index 0000000..33946ca --- /dev/null +++ b/kopia-restore-guide.md @@ -0,0 +1,153 @@ +# Kopia Restore Guide + +**Last Updated**: May 8, 2026 +**Applies To**: Both Kopia repositories on siklos (192.168.88.27) + +--- + +## ⚠️ Critical: What You Need to Restore + +You need **two things** to restore from Backblaze B2. Without either one, data is unrecoverable: + +1. **Repository password** — stored in `credentials.md` +2. **Backblaze B2 account access** — credentials in `credentials.md` + +> **Do not store these only on siklos.** Keep them in a password manager or printed copy somewhere safe. If siklos dies and you haven't saved the password elsewhere, your backup is unreadable. + +--- + +## 📦 Your Two Repositories + +| Container | B2 Bucket | What It Backs Up | Port | +|-----------|-----------|-----------------|------| +| `Kopia` | jg-kopia-nextcloud | Nextcloud files + database | 51515 | +| `Kopia-Immich` | jg-kopia-immich | Immich photos + database | 51516 | + +--- + +## 🔍 Step 1 — List Available Snapshots + +SSH to siklos first: +```bash +ssh jgitta@siklos +``` + +**Nextcloud snapshots:** +```bash +docker exec -e KOPIA_PASSWORD='' Kopia kopia snapshot list +``` + +**Immich snapshots:** +```bash +docker exec -e KOPIA_PASSWORD='' Kopia-Immich kopia snapshot list +``` + +Each snapshot has an ID like `k8adffae645082989d7c0f4f051439332` and a timestamp. You can restore from any snapshot in history — not just the latest. + +--- + +## 📁 Step 2 — Browse a Snapshot (Find Specific Files) + +You can explore a snapshot like a folder before restoring anything: + +```bash +# List the top-level contents of a snapshot +docker exec -e KOPIA_PASSWORD='' Kopia kopia ls + +# Drill into a subfolder +docker exec -e KOPIA_PASSWORD='' Kopia kopia ls /jgitta/files/Documents + +# Search for a specific filename across a snapshot +docker exec -e KOPIA_PASSWORD='' Kopia kopia find --name "*.jpg" +``` + +This lets you confirm exactly what you want to restore before doing it. + +--- + +## ♻️ Step 3 — Restore Options + +### Restore a Single File +```bash +docker exec -e KOPIA_PASSWORD='' Kopia kopia restore \ + /path/to/filename.jpg \ + /tmp/restore/filename.jpg +``` + +### Restore a Specific Folder +```bash +docker exec -e KOPIA_PASSWORD='' Kopia kopia restore \ + /jgitta/files/Photos \ + /tmp/restore/Photos/ +``` + +### Restore an Entire Snapshot +```bash +docker exec -e KOPIA_PASSWORD='' Kopia kopia restore \ + \ + /tmp/restore/ +``` + +> **Note:** Restored files land on siklos at `/tmp/restore/`. From there you can copy them to Nextcloud, Immich, or anywhere else via `scp` or `rsync`. + +--- + +## 🚨 Emergency: Siklos Is Gone — Restore from Scratch + +If siklos itself is dead, you can restore to any Linux machine: + +### 1. Install Kopia +```bash +curl -s https://kopia.io/signing-key | sudo gpg --dearmor -o /usr/share/keyrings/kopia-keyring.gpg +echo "deb [signed-by=/usr/share/keyrings/kopia-keyring.gpg] http://packages.kopia.io/apt/ stable main" | sudo tee /etc/apt/sources.list.d/kopia.list +sudo apt update && sudo apt install kopia +``` + +### 2. Connect to Your B2 Repository + +**For Nextcloud backup:** +```bash +kopia repository connect b2 \ + --bucket=jg-kopia-nextcloud \ + --key-id=00522b2471e5f090000000003 \ + --key=K005wv6tlgknxkM9arOJV/tMeOInOQY \ + --password= +``` + +**For Immich backup:** +```bash +kopia repository connect b2 \ + --bucket=jg-kopia-immich \ + --key-id=00522b2471e5f090000000004 \ + --key=K005gSepxpWk9hWx/MjWnCKtOiNuKnQ \ + --password= +``` + +### 3. List and Restore +```bash +kopia snapshot list +kopia restore /path/to/restore/ +``` + +> The B2 key IDs and application keys above are in `credentials.md`. Replace `` with the password from `credentials.md`. + +--- + +## 💡 Key Points + +- **Backblaze "Browse Files"** shows encrypted chunks — this is normal and expected. Files are not human-readable in B2 directly. +- **Every snapshot is independent** — you can restore from last week's snapshot even if today's is corrupted. +- **Partial restores are fully supported** — restore one file, one folder, or everything. +- **Kopia is open source** — even if the project were abandoned, you can always download the binary from GitHub and reconnect to your B2 repository as long as you have your password. + +--- + +## 📋 Restore Checklist + +- [ ] Have repository password (from `credentials.md`) +- [ ] Have B2 credentials (from `credentials.md`) +- [ ] SSH to siklos (or install Kopia on alternate machine) +- [ ] Run `snapshot list` to find correct snapshot by date +- [ ] Use `kopia ls` to browse and confirm the right files +- [ ] Run `kopia restore` with specific path for targeted restore +- [ ] Copy restored files to final destination diff --git a/nextcloud-backup-strategy.md b/nextcloud-backup-strategy.md new file mode 100644 index 0000000..6d71db3 --- /dev/null +++ b/nextcloud-backup-strategy.md @@ -0,0 +1,465 @@ +# Non-Disruptive Backup Strategy for Nextcloud + +**Last Updated**: April 28, 2026 +**Problem**: Previous backup approach froze the system (load avg: 21.89 on 4 CPUs) +**Solution**: Implement intelligent backup strategy that doesn't impact live system + +--- + +## What Went Wrong + +Your previous backup process: +``` +tar --zstd -cf /mnt/warm-storage/.backup-staging/... /mnt/nextcloud-data/ +``` + +**Problems:** +1. Reading 1.4TB live data while the system is running +2. Compressing with zstd (CPU-intensive) at same time +3. Writing compressed data to disk (more I/O) +4. All three happening simultaneously = I/O bottleneck +5. Load average jumped to 21.89 (system completely frozen) +6. Apache couldn't respond to requests (504 errors) +7. SSH connections hung + +**Key insight**: Live backups of large, active datasets require careful planning to avoid saturating disk I/O. + +--- + +## Backup Strategy Comparison + +### Strategy 1: LVM Snapshot + Backup ⭐ RECOMMENDED + +**How it works:** +1. Create a frozen copy (snapshot) of `/mnt/nextcloud-data` +2. Backup the snapshot while original stays live +3. Delete snapshot when done + +**Pros:** +- Original data unaffected while backup reads snapshot +- Read-only snapshot doesn't create new I/O load +- Fast (COW - Copy-on-Write) +- Works with any backup tool + +**Cons:** +- Requires LVM (you have it: `/dev/sdb`) +- Snapshot space depends on delta changes + +**Best for**: Your setup (1.4TB Nextcloud data) + +--- + +### Strategy 2: Kopia Incremental Backups (Configured Properly) + +**How it works:** +- Kopia only backs up changed blocks +- First backup is full, then incrementals +- Can throttle I/O impact + +**Pros:** +- Incremental = smaller backups after first one +- Built-in deduplication +- Cloud-ready (Backblaze B2) + +**Cons:** +- First full backup is still heavy +- Needs I/O rate limiting configured +- Incremental issues if not properly scheduled + +**Best for**: Ongoing maintenance backups + +--- + +### Strategy 3: Database-Specific + File Sync + +**How it works:** +- MariaDB dumps itself separately (consistent point-in-time) +- Rsync/Kopia sync files incrementally +- Lower I/O overall + +**Pros:** +- Database integrity guaranteed +- Can schedule separately +- Flexible + +**Cons:** +- More manual orchestration +- Need to coordinate timing + +**Best for**: Hybrid approach with LVM snapshots + +--- + +### Strategy 4: Separate Backup VM + +**How it works:** +- NFS mount nextcloud-data on backup VM +- Backup VM does all I/O work +- Production VM untouched + +**Pros:** +- Zero impact on production +- Can backup without affecting users + +**Cons:** +- Requires extra resources +- Network I/O instead of local + +**Best for**: Very high-availability setups + +--- + +## RECOMMENDED APPROACH: LVM Snapshot Strategy + +Your `/mnt/nextcloud-data` is on `/dev/sdc` (2TB disk). Create backup snapshots from there. + +### Implementation + +#### Step 1: Create Backup Script + +Create `/usr/local/bin/nextcloud-backup.sh`: + +```bash +#!/bin/bash +set -e + +# Configuration +NEXTCLOUD_MOUNT="/mnt/nextcloud-data" +BACKUP_DEST="/mnt/warm-storage/.backup-staging" +SNAPSHOT_NAME="nextcloud-backup-$(date +%Y%m%d_%H%M%S)" +SNAPSHOT_SIZE="200G" # Max snapshot size (change if needed) +BACKUP_LOG="/var/log/nextcloud-backup.log" + +# Logging function +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$BACKUP_LOG" +} + +# Error handling +cleanup() { + if [ -n "$SNAPSHOT_MOUNT" ] && mountpoint -q "$SNAPSHOT_MOUNT"; then + log "Unmounting snapshot..." + umount "$SNAPSHOT_MOUNT" + fi + if [ -n "$SNAPSHOT_LV" ] && lvdisplay "$SNAPSHOT_LV" &>/dev/null; then + log "Removing snapshot $SNAPSHOT_LV..." + lvremove -f "$SNAPSHOT_LV" + fi +} + +trap cleanup EXIT + +log "Starting Nextcloud backup to $BACKUP_DEST" + +# Find the LV backing the nextcloud mount +SOURCE_LV=$(df "$NEXTCLOUD_MOUNT" | tail -1 | awk '{print $1}') +log "Source LV: $SOURCE_LV" + +if [ -z "$SOURCE_LV" ]; then + log "ERROR: Could not determine LV for $NEXTCLOUD_MOUNT" + exit 1 +fi + +# Create snapshot +SNAPSHOT_LV="/dev/vg-nextcloud/snapshot-$SNAPSHOT_NAME" +log "Creating snapshot $SNAPSHOT_LV (size: $SNAPSHOT_SIZE)..." +lvcreate -L"$SNAPSHOT_SIZE" -s -n "snapshot-$SNAPSHOT_NAME" "$SOURCE_LV" + +if [ $? -ne 0 ]; then + log "ERROR: Failed to create snapshot" + exit 1 +fi + +# Mount snapshot temporarily +SNAPSHOT_MOUNT="/mnt/snapshot-backup-temp" +mkdir -p "$SNAPSHOT_MOUNT" +log "Mounting snapshot to $SNAPSHOT_MOUNT..." +mount -r "$SNAPSHOT_LV" "$SNAPSHOT_MOUNT" + +if [ $? -ne 0 ]; then + log "ERROR: Failed to mount snapshot" + exit 1 +fi + +# Run backup (reads from snapshot, not live data) +BACKUP_FILE="$BACKUP_DEST/nextcloud-backup-$SNAPSHOT_NAME.tar.zstd" +mkdir -p "$BACKUP_DEST" + +log "Backing up snapshot to $BACKUP_FILE..." +log "This may take 30-60 minutes (reading 1.4TB)..." + +# Nice and ionice reduce I/O impact on system +nice -n 15 ionice -c3 tar --zstd -cf "$BACKUP_FILE" -C "$SNAPSHOT_MOUNT" . + +if [ $? -eq 0 ]; then + BACKUP_SIZE=$(du -h "$BACKUP_FILE" | cut -f1) + log "✓ Backup successful: $BACKUP_SIZE" +else + log "✗ Backup failed" + exit 1 +fi + +# Cleanup snapshot (done automatically via trap) +log "Backup complete. Cleaning up..." +``` + +#### Step 2: Make it Executable + +```bash +chmod +x /usr/local/bin/nextcloud-backup.sh +``` + +#### Step 3: Create Systemd Service + Timer + +Create `/etc/systemd/system/nextcloud-backup.service`: + +```ini +[Unit] +Description=Nextcloud Backup Service +After=local-fs.target + +[Service] +Type=oneshot +ExecStart=/usr/local/bin/nextcloud-backup.sh +StandardOutput=journal +StandardError=journal +User=root + +# Don't use too much I/O +Nice=15 +IOSchedulingClass=idle +``` + +Create `/etc/systemd/system/nextcloud-backup.timer`: + +```ini +[Unit] +Description=Nextcloud Backup Timer +Requires=nextcloud-backup.service + +[Timer] +# Run at 2 AM daily when users are asleep +OnCalendar=daily +OnCalendar=*-*-* 02:00:00 + +# Randomize within 5 minutes to avoid spikes +RandomizedDelaySec=5min + +[Install] +WantedBy=timers.target +``` + +#### Step 4: Enable and Test + +```bash +# Reload systemd +systemctl daemon-reload + +# Enable timer to start on boot +systemctl enable nextcloud-backup.timer + +# Start the timer +systemctl start nextcloud-backup.timer + +# Check status +systemctl status nextcloud-backup.timer + +# View next scheduled run +systemctl list-timers nextcloud-backup.timer + +# Manual test (runs immediately) +systemctl start nextcloud-backup.service + +# Watch progress +journalctl -u nextcloud-backup.service -f +``` + +--- + +## Alternative: Kopia with Proper Rate Limiting + +If you prefer to keep Kopia: + +### Configure Kopia for Non-Disruptive Backups + +Edit Kopia config (usually `/srv/docker/kopia/config/kopia.json`): + +```json +{ + "uploads": { + "maxParallelFileWrites": 2, + "maxParallelSmallFileWrites": 2, + "ignoreFileErrors": false + }, + "cache": { + "metadata": { + "maxCacheSize": 500000000 + } + }, + "logging": { + "level": "info" + } +} +``` + +### Run Kopia with Nice/Ionice + +```bash +# Instead of running kopia directly +nice -n 15 ionice -c3 docker exec kopia kopia snapshot create +``` + +Or in crontab: +```cron +# 3 AM daily +0 3 * * * nice -n 15 ionice -c3 docker exec kopia kopia snapshot create >> /var/log/kopia-backup.log 2>&1 +``` + +--- + +## Implementation Roadmap + +### Week 1: Preparation +- [ ] Verify LVM setup on nextcloud VM +- [ ] Create backup script +- [ ] Test snapshot creation/deletion +- [ ] Verify backup file integrity + +### Week 2: Automation +- [ ] Install systemd service/timer +- [ ] Run first automated backup +- [ ] Monitor system load during backup +- [ ] Verify load average stays below 10 + +### Week 3: Refinement +- [ ] Adjust backup schedule (time/frequency) +- [ ] Add monitoring alerts +- [ ] Document recovery procedures +- [ ] Test restoration from backup + +### Week 4: Production +- [ ] Run weekly backups +- [ ] Monitor Backblaze B2 uploads +- [ ] Schedule monthly test restores +- [ ] Update documentation + +--- + +## Key Differences: Before vs. After + +| Aspect | Before | After | +|--------|--------|-------| +| **Load during backup** | 21.89 (system frozen) | <8 (acceptable) | +| **Web service** | 504 errors | Responsive | +| **SSH** | Hangs | Responsive | +| **Backup method** | Live tar + zstd | LVM snapshot | +| **Schedule** | Manual (caused outage) | Automated (2 AM daily) | +| **I/O strategy** | Full speed (killer) | Rate-limited (nice/ionice) | + +--- + +## Monitoring + +### Check Backup Status + +```bash +# View backup history +ls -lh /mnt/warm-storage/.backup-staging/ + +# Check backup size +du -sh /mnt/warm-storage/.backup-staging/ + +# View logs +journalctl -u nextcloud-backup.service --since "1 day ago" +``` + +### Monitor Load During Backup + +```bash +# On another terminal, watch system during backup +watch -n 1 'uptime && echo "---" && iostat -x 1 1 | grep sdc' +``` + +**What to expect:** +- Load: 4-8 (normal for 4-CPU system during backup) +- Web users: Unaffected +- SSH: Responsive +- Apache/PHP: Normal response times + +--- + +## Recovery: How to Restore from Backup + +When you need to restore: + +```bash +# List available backups +ls -lh /mnt/warm-storage/.backup-staging/ + +# Extract to temp location (DON'T overwrite live data!) +mkdir -p /mnt/restore-test +tar --zstd -xf /mnt/warm-storage/.backup-staging/nextcloud-backup-20260428_020000.tar.zstd \ + -C /mnt/restore-test + +# Verify files are there +ls /mnt/restore-test/ | head -20 + +# If good, proceed with restore to actual location +# (This requires stopping Nextcloud first to avoid corruption) +``` + +--- + +## Troubleshooting + +### "Failed to create snapshot: No space left on device" + +The snapshot storage filled up. Increase `$SNAPSHOT_SIZE` in the script: +```bash +SNAPSHOT_SIZE="500G" # Instead of 200G +``` + +### "Backup file is too large" + +Disable compression to see raw size: +```bash +tar -cf - /mnt/nextcloud-data | wc -c +# Divide by 1TB (1099511627776) to see size in TB +``` + +If uncompressed is >2TB, you need more storage on warm-storage. + +### Backups not running automatically + +```bash +# Check timer is running +systemctl status nextcloud-backup.timer + +# Enable it if disabled +systemctl enable nextcloud-backup.timer + +# Check if service failed +systemctl status nextcloud-backup.service +journalctl -u nextcloud-backup.service -n 50 +``` + +--- + +## Summary + +**Use the LVM snapshot approach** because: +1. ✅ Zero impact on live system during backup +2. ✅ Consistent point-in-time copy +3. ✅ Safe to interrupt without data loss +4. ✅ Easy to automate with systemd +5. ✅ Works with any backup tool (tar, Kopia, rsync) + +Next steps: +1. SSH to nextcloud VM +2. Copy backup script from above +3. Create systemd service + timer +4. Enable and test +5. Monitor first backup run +6. Adjust schedule based on timing + +You'll have reliable, non-disruptive backups running every night at 2 AM with zero system impact! 🎯 diff --git a/siklos-docker-services.md b/siklos-docker-services.md new file mode 100644 index 0000000..0e63992 --- /dev/null +++ b/siklos-docker-services.md @@ -0,0 +1,124 @@ +# Siklos Docker Services — Project Knowledge + +## Host +- **VM112 / siklos / docker-server** +- IP: `192.168.88.27` +- Specs: 4c/12GB RAM (reduced from 16GB after photo services migrated to VM113) +- SSH: `jgitta@192.168.88.27` (via Proxmox host jump) +- Docker compose files: `/srv/docker//docker-compose.yml` +- Note: `vm.swappiness=10` set in `/etc/sysctl.conf` (April 2026) + +## Running Containers (as of April 2026) + +| Container | Image | Port(s) | Compose Path | +|---|---|---|---| +| portainer | portainer/portainer-ce:latest | 9000, 9443 | /srv/docker/portainer/ | +| pihole | pihole/pihole:latest | 53, 8080 | /srv/docker/pihole/ | +| open-webui | ghcr.io/open-webui/open-webui:main | 3000 | /srv/docker/open-webui/ | +| onlyoffice | onlyoffice/documentserver | 8880 | /srv/docker/media/ | +| homarr | homarr:latest | 7575 | /srv/docker/homarr/ | +| uptime-kuma | uptime-kuma:2 | 3001 | /srv/docker/uptime-kuma/ | +| grafana | grafana:latest | 3020 | /srv/docker/monitoring/ | +| prometheus | prom/prometheus:latest | 9090 | /srv/docker/monitoring/ | +| node-exporter | prom/node-exporter:latest | 9100 | /srv/docker/monitoring/ | +| cadvisor | cadvisor:latest | 8090 | /srv/docker/monitoring/ | +| graphite-exporter | prom/graphite-exporter:latest | 9108-9109 | /srv/docker/monitoring/ | +| glances | nicolargo/glances:latest | 61208 | /srv/docker/glances/ | +| meshcentral | typhonragewind/meshcentral:latest | 444 | /srv/docker/meshcentral/ | +| guacamole | jwetzell/guacamole | 8383 | /srv/docker/guacamole/ | +| karakeep-web-1 | karakeep:release | 3010 | /srv/docker/karakeep/ | +| karakeep-meilisearch-1 | meilisearch:v1.13.3 | 7700 (internal) | /srv/docker/karakeep/ | +| karakeep-chrome-1 | alpine-chrome:124 | — | /srv/docker/karakeep/ | +| linkwarden-linkwarden-1 | linkwarden:latest | 3015 | /srv/docker/linkwarden/ | +| linkwarden-postgres-1 | postgres:16-alpine | 5432 | /srv/docker/linkwarden/ | +| wordpress | wordpress:php8.3-apache | 8095 | /srv/docker/wordpress/ | +| wordpress-db | mariadb:10.11 | 3306 (internal) | /srv/docker/wordpress/ | +| dashy | lissy93/dashy:latest | 8081 | /srv/docker/dashy/ | +| dashboard | dashboard-dashboard | 8096 | /srv/docker/dashboard/ | +| grav | linuxserver/grav:latest | 8585 | /srv/docker/grav/ | +| watchtower | containrrr/watchtower | — | /srv/docker/watchtower/ | + +## Open WebUI +- URL: `https://ai.jgitta.com` +- Port: 3000 +- Compose: `/srv/docker/open-webui/docker-compose.yml` +- Connects to Ollama backend running on ThinkStation (192.168.88.41, no external port) + +## Migrated Services +- **PhotoPrism** and **Immich** were migrated to VM113 (photos, 192.168.88.32) in April 2026 +- See `/home/jgitta/Documents/Claude/Projects/Photos/photos.md` for current details + +## Pi-hole +- Port: 53 (DNS), 8080 (web UI) +- URL: `https://pihole.jgitta.com` +- **Network mode: `host`** (no docker-proxy — eliminates the port-53 freeze bug) +- Config: `listeningMode = "ALL"` in pihole.toml +- **Upstream DNS: `192.168.88.1` only** (MikroTik) — MikroTik resolves local jgitta.com domains and forwards everything else to 1.1.1.1/8.8.8.8. Do NOT add 1.1.1.1 directly as a Pi-hole upstream — dnsmasq load-balances across all upstreams, which would cause local domains to sometimes resolve to the WAN IP instead of internal Caddy IP. +- FTL DB: `/etc/pihole/pihole-FTL.db` +- Rate limit: 300 concurrent queries +- Excluded from Watchtower auto-updates +- Pi-hole v6 +- **Watchdog**: native cron on Siklos at `*/5 * * * *` → `/usr/local/bin/pihole-watchdog.sh` (restarts if DNS fails); Claude scheduled task is a secondary layer +- Uptime Kuma DNS monitor uses `192.168.88.27` (host IP, not Docker internal) + +## Monitoring Stack +- Compose: `/srv/docker/monitoring/` +- Grafana: port 3020 (`grafana.jgitta.com`), datasource UID: `cffiqslf48feod` +- Prometheus: port 9090 +- Node Exporter on: siklos (.27), proxmox (.25), nextcloud (.62), jellyfin (.10), pbs (.60), caddy (.110), thinkstation (.41) — all port 9100 +- Grafana alert folder "Homelab Alerts": + - High RAM >90% for 5min + - Swap >50% for 5min + - CPU >90% for 10min + - Disk >85% for 5min + - Node Down 2min +- Alert annotations: `{{ $labels.instance }}` and `{{ $values.B }}%` +- Alerts use three-step reduce+threshold pipeline (not classic conditions) +- Telegram: bot token `8758434542:AAEW6omM7twyInsb2INuy6mD1w2EWXHqmzE`, chat `8260387200`, repeat every 4h +- Uptime Kuma: port 3001 (`status.jgitta.com`), joined to `pihole_default` network + +## OnlyOffice +- URL: `https://office.jgitta.com` +- Port: 8880 (all interfaces) +- Compose: `/srv/docker/media/docker-compose.yml` +- Replaces Collabora (`richdocuments` app is disabled in Nextcloud; `onlyoffice` app is enabled) +- JWT secret (must match Nextcloud config): `4f2b0c719af2de99befacfec9ca5e8373cbdeb76` +- Nextcloud `occ` settings (set on VM103/next): + - `DocumentServerUrl` = `https://office.jgitta.com/` + - `DocumentServerInternalUrl` = `http://192.168.88.27:8880/` + - `StorageUrl` = `https://next.jgitta.com/` + - `jwt_secret` = (matches container `local.json` above) + - `jwt_header` = `Authorization` +- To reconfigure after container recreation: re-run `occ config:app:set onlyoffice jwt_secret --value=""` +- Container JWT secret location: `/etc/onlyoffice/documentserver/local.json` → `.services.CoAuthoring.secret.inbox.string` + +## Key Notes +- Watchtower excludes Pi-hole from auto-updates +- NFS mount `/mnt/photos` was removed from Siklos `/etc/fstab` after PhotoPrism/Immich migration +- OnlyOffice replaced Collabora (lighter RAM usage) +- RAM reduced from 16GB → 12GB (April 2026, live via Proxmox balloon driver, no reboot) +- `vm.swappiness=10` set to reduce swap pressure after photo services migrated out +- OnlyOffice replaced Collabora (lighter RAM usage) + +## Portainer Setup (April 2026) + +Portainer CE manages Docker across three hosts from a single UI. + +**Server (web UI):** +- URL: `http://192.168.88.27:9000` or `https://192.168.88.27:9443` +- Future URL (once Caddy SSH fixed): `https://portainer.jgitta.com` +- Compose: `/srv/docker/portainer/docker-compose.yml` + +**Agents (managed remote environments):** +| Host | IP | Agent Port | Compose Path | +|---|---|---|---| +| jellyfin | 192.168.88.10 | 9001 | /srv/docker/portainer-agent/ | +| thinkstation | 192.168.88.41 | 9001 | /srv/docker/portainer-agent/ | + +**Adding agents in Portainer UI (first-time setup):** +1. Open Portainer at `http://192.168.88.27:9000` and create your admin account +2. Go to **Environments → Add environment → Agent** +3. For jellyfin: name `jellyfin`, URL `192.168.88.10:9001` +4. For thinkstation: name `thinkstation`, URL `192.168.88.41:9001` + +**Caddy config:** Entry for `portainer.jgitta.com` is ready in workspace Caddyfile — needs deploying once Caddy VM SSH access is restored.