# Nextcloud Backup - Critical Fix Log

## Issue Discovered
Date: 2026-04-25 19:11 CDT

The initial backup appeared successful but was incomplete:
- Database backup: 203 MB ✅
- Files backup: **SKIPPED** ❌ (should be 1.4TB)
- Log showed: "⚠️ Files backup skipped"

## Root Cause

The backup script was trying to write a 1.4TB tar.zst file to `/tmp`, which is a tmpfs mount with only 3.9GB capacity.

```
/tmp filesystem:
  Filesystem     1K-blocks    Used Available Use%
  tmpfs            4069008 4069008         0 100%
```

When tar and zstd tried to write the compressed backup file, they failed with:
```
zstd: error 70 : Write error : cannot write block : No space left on device 
tar: Child returned status 70
```

The script's error handling swallowed this error and reported "Files backup skipped" instead of reporting the actual disk space issue.

## Solution Implemented

Changed the temporary directory from `/tmp` (3.9GB tmpfs) to `/mnt/nextcloud-data/.backup-staging` (460GB free space on actual disk).

### Code Changes

**File:** `nextcloud-backup-to-backblaze.sh`

```bash
# Before:
TEMP_DIR="/tmp/nextcloud-backup-${TIMESTAMP}"

# After:
STAGING_DIR="/mnt/nextcloud-data/.backup-staging"
TEMP_DIR="${STAGING_DIR}/nextcloud-backup-${TIMESTAMP}"
```

Plus automatic cleanup after successful upload:
```bash
rmdir "$STAGING_DIR" 2>/dev/null || true
```

### Deployment

1. Fixed script deployed to `/opt/nextcloud-backup/nextcloud-backup-to-backblaze.sh`
2. Cleared `/tmp` of old backup files (freed 3.9GB)
3. Full 1.4TB backup started at 19:11 CDT

## Status

### First Attempt (19:10-19:13)
- Database: ✅ 203 MB
- Files: ❌ Skipped (space issue)
- Result: FAILED (incomplete)

### Second Attempt (19:11-ongoing)
- Database: ✅ 203 MB
- Files: ⏳ 1.4TB in progress (tar + zstd running)
- Result: PENDING (expected completion 19:45-20:15 CDT)

### Process Status

```
Process              CPU    Status
tar --zstd          58.7%  Running
zstd (compression)  151%   Running (multithreaded)
mysqldump           N/A    Completed
```

Estimated compression ratio: 1.4TB → 200-300GB (with zstd at level 3)

## Lesson Learned

- Always verify actual data sizes match what's reported ✅
- Tmpfs mounts are insufficient for large backup operations ✅
- Silent error handling masks underlying issues ✅
- Monitor both process execution AND actual file output ✅

## Prevention

Future improvements:
1. Add explicit space checks before starting backup
2. Improve error reporting (don't swallow tar/zstd errors)
3. Add pre-backup size verification
4. Alert on any "skipped" operations

## Files Modified

- `nextcloud-backup-to-backblaze.sh` - Updated TEMP_DIR logic
- `BACKUP_FIX_LOG.md` - This document

## Git Commit

```
commit 2267a34
Author: Automation <automation@jgitta.com>
Date:   Sat Apr 25 19:15:00 2026

    Fix: Use disk staging directory instead of tmpfs /tmp for backups
```

Pushed to: `nextcloud-backup` branch in Gitea