jgitta
b567f5dba8
Detailed log of: - Issue discovery (files backup being skipped) - Root cause analysis (/tmp space exhaustion) - Solution implemented (disk staging directory) - Process status and monitoring Created: 2026-04-25 19:15 CDT Backup status: Full 1.4TB in progress
112 lines
2.9 KiB
Markdown
112 lines
2.9 KiB
Markdown
# Nextcloud Backup - Critical Fix Log
|
|
|
|
## Issue Discovered
|
|
Date: 2026-04-25 19:11 CDT
|
|
|
|
The initial backup appeared successful but was incomplete:
|
|
- Database backup: 203 MB ✅
|
|
- Files backup: **SKIPPED** ❌ (should be 1.4TB)
|
|
- Log showed: "⚠️ Files backup skipped"
|
|
|
|
## Root Cause
|
|
|
|
The backup script was trying to write a 1.4TB tar.zst file to `/tmp`, which is a tmpfs mount with only 3.9GB capacity.
|
|
|
|
```
|
|
/tmp filesystem:
|
|
Filesystem 1K-blocks Used Available Use%
|
|
tmpfs 4069008 4069008 0 100%
|
|
```
|
|
|
|
When tar and zstd tried to write the compressed backup file, they failed with:
|
|
```
|
|
zstd: error 70 : Write error : cannot write block : No space left on device
|
|
tar: Child returned status 70
|
|
```
|
|
|
|
The script's error handling swallowed this error and reported "Files backup skipped" instead of reporting the actual disk space issue.
|
|
|
|
## Solution Implemented
|
|
|
|
Changed the temporary directory from `/tmp` (3.9GB tmpfs) to `/mnt/nextcloud-data/.backup-staging` (460GB free space on actual disk).
|
|
|
|
### Code Changes
|
|
|
|
**File:** `nextcloud-backup-to-backblaze.sh`
|
|
|
|
```bash
|
|
# Before:
|
|
TEMP_DIR="/tmp/nextcloud-backup-${TIMESTAMP}"
|
|
|
|
# After:
|
|
STAGING_DIR="/mnt/nextcloud-data/.backup-staging"
|
|
TEMP_DIR="${STAGING_DIR}/nextcloud-backup-${TIMESTAMP}"
|
|
```
|
|
|
|
Plus automatic cleanup after successful upload:
|
|
```bash
|
|
rmdir "$STAGING_DIR" 2>/dev/null || true
|
|
```
|
|
|
|
### Deployment
|
|
|
|
1. Fixed script deployed to `/opt/nextcloud-backup/nextcloud-backup-to-backblaze.sh`
|
|
2. Cleared `/tmp` of old backup files (freed 3.9GB)
|
|
3. Full 1.4TB backup started at 19:11 CDT
|
|
|
|
## Status
|
|
|
|
### First Attempt (19:10-19:13)
|
|
- Database: ✅ 203 MB
|
|
- Files: ❌ Skipped (space issue)
|
|
- Result: FAILED (incomplete)
|
|
|
|
### Second Attempt (19:11-ongoing)
|
|
- Database: ✅ 203 MB
|
|
- Files: ⏳ 1.4TB in progress (tar + zstd running)
|
|
- Result: PENDING (expected completion 19:45-20:15 CDT)
|
|
|
|
### Process Status
|
|
|
|
```
|
|
Process CPU Status
|
|
tar --zstd 58.7% Running
|
|
zstd (compression) 151% Running (multithreaded)
|
|
mysqldump N/A Completed
|
|
```
|
|
|
|
Estimated compression ratio: 1.4TB → 200-300GB (with zstd at level 3)
|
|
|
|
## Lesson Learned
|
|
|
|
- Always verify actual data sizes match what's reported ✅
|
|
- Tmpfs mounts are insufficient for large backup operations ✅
|
|
- Silent error handling masks underlying issues ✅
|
|
- Monitor both process execution AND actual file output ✅
|
|
|
|
## Prevention
|
|
|
|
Future improvements:
|
|
1. Add explicit space checks before starting backup
|
|
2. Improve error reporting (don't swallow tar/zstd errors)
|
|
3. Add pre-backup size verification
|
|
4. Alert on any "skipped" operations
|
|
|
|
## Files Modified
|
|
|
|
- `nextcloud-backup-to-backblaze.sh` - Updated TEMP_DIR logic
|
|
- `BACKUP_FIX_LOG.md` - This document
|
|
|
|
## Git Commit
|
|
|
|
```
|
|
commit 2267a34
|
|
Author: Automation <automation@jgitta.com>
|
|
Date: Sat Apr 25 19:15:00 2026
|
|
|
|
Fix: Use disk staging directory instead of tmpfs /tmp for backups
|
|
```
|
|
|
|
Pushed to: `nextcloud-backup` branch in Gitea
|
|
|