Add: Backup fix documentation explaining root cause and solution

Detailed log of: - Issue discovery (files backup being skipped) - Root cause analysis (/tmp space exhaustion) - Solution implemented (disk staging directory) - Process status and monitoring Created: 2026-04-25 19:15 CDT Backup status: Full 1.4TB in progress
2026-04-25 19:16:03 -05:00
parent 2267a34b47
commit b567f5dba8
1 changed files with 111 additions and 0 deletions
@@ -0,0 +1,111 @@
+# Nextcloud Backup - Critical Fix Log
+
+## Issue Discovered
+Date: 2026-04-25 19:11 CDT
+
+The initial backup appeared successful but was incomplete:
+- Database backup: 203 MB ✅
+- Files backup: **SKIPPED** ❌ (should be 1.4TB)
+- Log showed: "⚠️ Files backup skipped"
+
+## Root Cause
+
+The backup script was trying to write a 1.4TB tar.zst file to `/tmp`, which is a tmpfs mount with only 3.9GB capacity.
+
+```
+/tmp filesystem:
+  Filesystem     1K-blocks    Used Available Use%
+  tmpfs            4069008 4069008         0 100%
+```
+
+When tar and zstd tried to write the compressed backup file, they failed with:
+```
+zstd: error 70 : Write error : cannot write block : No space left on device 
+tar: Child returned status 70
+```
+
+The script's error handling swallowed this error and reported "Files backup skipped" instead of reporting the actual disk space issue.
+
+## Solution Implemented
+
+Changed the temporary directory from `/tmp` (3.9GB tmpfs) to `/mnt/nextcloud-data/.backup-staging` (460GB free space on actual disk).
+
+### Code Changes
+
+**File:** `nextcloud-backup-to-backblaze.sh`
+
+```bash
+# Before:
+TEMP_DIR="/tmp/nextcloud-backup-${TIMESTAMP}"
+
+# After:
+STAGING_DIR="/mnt/nextcloud-data/.backup-staging"
+TEMP_DIR="${STAGING_DIR}/nextcloud-backup-${TIMESTAMP}"
+```
+
+Plus automatic cleanup after successful upload:
+```bash
+rmdir "$STAGING_DIR" 2>/dev/null || true
+```
+
+### Deployment
+
+1. Fixed script deployed to `/opt/nextcloud-backup/nextcloud-backup-to-backblaze.sh`
+2. Cleared `/tmp` of old backup files (freed 3.9GB)
+3. Full 1.4TB backup started at 19:11 CDT
+
+## Status
+
+### First Attempt (19:10-19:13)
+- Database: ✅ 203 MB
+- Files: ❌ Skipped (space issue)
+- Result: FAILED (incomplete)
+
+### Second Attempt (19:11-ongoing)
+- Database: ✅ 203 MB
+- Files: ⏳ 1.4TB in progress (tar + zstd running)
+- Result: PENDING (expected completion 19:45-20:15 CDT)
+
+### Process Status
+
+```
+Process              CPU    Status
+tar --zstd          58.7%  Running
+zstd (compression)  151%   Running (multithreaded)
+mysqldump           N/A    Completed
+```
+
+Estimated compression ratio: 1.4TB → 200-300GB (with zstd at level 3)
+
+## Lesson Learned
+
+- Always verify actual data sizes match what's reported ✅
+- Tmpfs mounts are insufficient for large backup operations ✅
+- Silent error handling masks underlying issues ✅
+- Monitor both process execution AND actual file output ✅
+
+## Prevention
+
+Future improvements:
+1. Add explicit space checks before starting backup
+2. Improve error reporting (don't swallow tar/zstd errors)
+3. Add pre-backup size verification
+4. Alert on any "skipped" operations
+
+## Files Modified
+
+- `nextcloud-backup-to-backblaze.sh` - Updated TEMP_DIR logic
+- `BACKUP_FIX_LOG.md` - This document
+
+## Git Commit
+
+```
+commit 2267a34
+Author: Automation <automation@jgitta.com>
+Date:   Sat Apr 25 19:15:00 2026
+
+    Fix: Use disk staging directory instead of tmpfs /tmp for backups
+```
+
+Pushed to: `nextcloud-backup` branch in Gitea
+