diff --git a/BACKUP_FIX_LOG.md b/BACKUP_FIX_LOG.md new file mode 100644 index 0000000..4f29ba9 --- /dev/null +++ b/BACKUP_FIX_LOG.md @@ -0,0 +1,111 @@ +# Nextcloud Backup - Critical Fix Log + +## Issue Discovered +Date: 2026-04-25 19:11 CDT + +The initial backup appeared successful but was incomplete: +- Database backup: 203 MB ✅ +- Files backup: **SKIPPED** ❌ (should be 1.4TB) +- Log showed: "⚠️ Files backup skipped" + +## Root Cause + +The backup script was trying to write a 1.4TB tar.zst file to `/tmp`, which is a tmpfs mount with only 3.9GB capacity. + +``` +/tmp filesystem: + Filesystem 1K-blocks Used Available Use% + tmpfs 4069008 4069008 0 100% +``` + +When tar and zstd tried to write the compressed backup file, they failed with: +``` +zstd: error 70 : Write error : cannot write block : No space left on device +tar: Child returned status 70 +``` + +The script's error handling swallowed this error and reported "Files backup skipped" instead of reporting the actual disk space issue. + +## Solution Implemented + +Changed the temporary directory from `/tmp` (3.9GB tmpfs) to `/mnt/nextcloud-data/.backup-staging` (460GB free space on actual disk). + +### Code Changes + +**File:** `nextcloud-backup-to-backblaze.sh` + +```bash +# Before: +TEMP_DIR="/tmp/nextcloud-backup-${TIMESTAMP}" + +# After: +STAGING_DIR="/mnt/nextcloud-data/.backup-staging" +TEMP_DIR="${STAGING_DIR}/nextcloud-backup-${TIMESTAMP}" +``` + +Plus automatic cleanup after successful upload: +```bash +rmdir "$STAGING_DIR" 2>/dev/null || true +``` + +### Deployment + +1. Fixed script deployed to `/opt/nextcloud-backup/nextcloud-backup-to-backblaze.sh` +2. Cleared `/tmp` of old backup files (freed 3.9GB) +3. Full 1.4TB backup started at 19:11 CDT + +## Status + +### First Attempt (19:10-19:13) +- Database: ✅ 203 MB +- Files: ❌ Skipped (space issue) +- Result: FAILED (incomplete) + +### Second Attempt (19:11-ongoing) +- Database: ✅ 203 MB +- Files: ⏳ 1.4TB in progress (tar + zstd running) +- Result: PENDING (expected completion 19:45-20:15 CDT) + +### Process Status + +``` +Process CPU Status +tar --zstd 58.7% Running +zstd (compression) 151% Running (multithreaded) +mysqldump N/A Completed +``` + +Estimated compression ratio: 1.4TB → 200-300GB (with zstd at level 3) + +## Lesson Learned + +- Always verify actual data sizes match what's reported ✅ +- Tmpfs mounts are insufficient for large backup operations ✅ +- Silent error handling masks underlying issues ✅ +- Monitor both process execution AND actual file output ✅ + +## Prevention + +Future improvements: +1. Add explicit space checks before starting backup +2. Improve error reporting (don't swallow tar/zstd errors) +3. Add pre-backup size verification +4. Alert on any "skipped" operations + +## Files Modified + +- `nextcloud-backup-to-backblaze.sh` - Updated TEMP_DIR logic +- `BACKUP_FIX_LOG.md` - This document + +## Git Commit + +``` +commit 2267a34 +Author: Automation +Date: Sat Apr 25 19:15:00 2026 + + Fix: Use disk staging directory instead of tmpfs /tmp for backups +``` + +Pushed to: `nextcloud-backup` branch in Gitea +