homelab-configs/BACKUP_FIX_LOG.md

Nextcloud Backup - Critical Fix Log

Issue Discovered

Date: 2026-04-25 19:11 CDT

The initial backup appeared successful but was incomplete:

• Database backup: 203 MB ✅
• Files backup: SKIPPED ❌ (should be 1.4TB)
• Log showed: "⚠️ Files backup skipped"

Root Cause

The backup script was trying to write a 1.4TB tar.zst file to /tmp, which is a tmpfs mount with only 3.9GB capacity.

/tmp filesystem:
  Filesystem     1K-blocks    Used Available Use%
  tmpfs            4069008 4069008         0 100%

When tar and zstd tried to write the compressed backup file, they failed with:

zstd: error 70 : Write error : cannot write block : No space left on device 
tar: Child returned status 70

The script's error handling swallowed this error and reported "Files backup skipped" instead of reporting the actual disk space issue.

Solution Implemented

Changed the temporary directory from /tmp (3.9GB tmpfs) to /mnt/nextcloud-data/.backup-staging (460GB free space on actual disk).

Code Changes

File: nextcloud-backup-to-backblaze.sh

# Before:
TEMP_DIR="/tmp/nextcloud-backup-${TIMESTAMP}"

# After:
STAGING_DIR="/mnt/nextcloud-data/.backup-staging"
TEMP_DIR="${STAGING_DIR}/nextcloud-backup-${TIMESTAMP}"

Plus automatic cleanup after successful upload:

rmdir "$STAGING_DIR" 2>/dev/null || true

Deployment

1. Fixed script deployed to /opt/nextcloud-backup/nextcloud-backup-to-backblaze.sh
2. Cleared /tmp of old backup files (freed 3.9GB)
3. Full 1.4TB backup started at 19:11 CDT

Status

First Attempt (19:10-19:13)

• Database: ✅ 203 MB
• Files: ❌ Skipped (space issue)
• Result: FAILED (incomplete)

Second Attempt (19:11-ongoing)

• Database: ✅ 203 MB
• Files: ⏳ 1.4TB in progress (tar + zstd running)
• Result: PENDING (expected completion 19:45-20:15 CDT)

Process Status

Process              CPU    Status
tar --zstd          58.7%  Running
zstd (compression)  151%   Running (multithreaded)
mysqldump           N/A    Completed

Estimated compression ratio: 1.4TB → 200-300GB (with zstd at level 3)

Lesson Learned

• Always verify actual data sizes match what's reported ✅
• Tmpfs mounts are insufficient for large backup operations ✅
• Silent error handling masks underlying issues ✅
• Monitor both process execution AND actual file output ✅

Prevention

Future improvements:

1. Add explicit space checks before starting backup
2. Improve error reporting (don't swallow tar/zstd errors)
3. Add pre-backup size verification
4. Alert on any "skipped" operations

Files Modified

• nextcloud-backup-to-backblaze.sh - Updated TEMP_DIR logic
• BACKUP_FIX_LOG.md - This document

Git Commit

commit 2267a34
Author: Automation <automation@jgitta.com>
Date:   Sat Apr 25 19:15:00 2026

    Fix: Use disk staging directory instead of tmpfs /tmp for backups

Pushed to: nextcloud-backup branch in Gitea