homelab-configs/nextcloud-backup-strategy.md

# Non-Disruptive Backup Strategy for Nextcloud

**Last Updated**: April 28, 2026  
**Problem**: Previous backup approach froze the system (load avg: 21.89 on 4 CPUs)  
**Solution**: Implement intelligent backup strategy that doesn't impact live system

---

## What Went Wrong

Your previous backup process:
```
tar --zstd -cf /mnt/warm-storage/.backup-staging/... /mnt/nextcloud-data/
```

**Problems:**
1. Reading 1.4TB live data while the system is running
2. Compressing with zstd (CPU-intensive) at same time
3. Writing compressed data to disk (more I/O)
4. All three happening simultaneously = I/O bottleneck
5. Load average jumped to 21.89 (system completely frozen)
6. Apache couldn't respond to requests (504 errors)
7. SSH connections hung

**Key insight**: Live backups of large, active datasets require careful planning to avoid saturating disk I/O.

---

## Backup Strategy Comparison

### Strategy 1: LVM Snapshot + Backup ⭐ RECOMMENDED

**How it works:**
1. Create a frozen copy (snapshot) of `/mnt/nextcloud-data` 
2. Backup the snapshot while original stays live
3. Delete snapshot when done

**Pros:**
- Original data unaffected while backup reads snapshot
- Read-only snapshot doesn't create new I/O load
- Fast (COW - Copy-on-Write)
- Works with any backup tool

**Cons:**
- Requires LVM (you have it: `/dev/sdb`)
- Snapshot space depends on delta changes

**Best for**: Your setup (1.4TB Nextcloud data)

---

### Strategy 2: Kopia Incremental Backups (Configured Properly)

**How it works:**
- Kopia only backs up changed blocks
- First backup is full, then incrementals
- Can throttle I/O impact

**Pros:**
- Incremental = smaller backups after first one
- Built-in deduplication
- Cloud-ready (Backblaze B2)

**Cons:**
- First full backup is still heavy
- Needs I/O rate limiting configured
- Incremental issues if not properly scheduled

**Best for**: Ongoing maintenance backups

---

### Strategy 3: Database-Specific + File Sync

**How it works:**
- MariaDB dumps itself separately (consistent point-in-time)
- Rsync/Kopia sync files incrementally
- Lower I/O overall

**Pros:**
- Database integrity guaranteed
- Can schedule separately
- Flexible

**Cons:**
- More manual orchestration
- Need to coordinate timing

**Best for**: Hybrid approach with LVM snapshots

---

### Strategy 4: Separate Backup VM

**How it works:**
- NFS mount nextcloud-data on backup VM
- Backup VM does all I/O work
- Production VM untouched

**Pros:**
- Zero impact on production
- Can backup without affecting users

**Cons:**
- Requires extra resources
- Network I/O instead of local

**Best for**: Very high-availability setups

---

## RECOMMENDED APPROACH: LVM Snapshot Strategy

Your `/mnt/nextcloud-data` is on `/dev/sdc` (2TB disk). Create backup snapshots from there.

### Implementation

#### Step 1: Create Backup Script

Create `/usr/local/bin/nextcloud-backup.sh`:

```bash
#!/bin/bash
set -e

# Configuration
NEXTCLOUD_MOUNT="/mnt/nextcloud-data"
BACKUP_DEST="/mnt/warm-storage/.backup-staging"
SNAPSHOT_NAME="nextcloud-backup-$(date +%Y%m%d_%H%M%S)"
SNAPSHOT_SIZE="200G"  # Max snapshot size (change if needed)
BACKUP_LOG="/var/log/nextcloud-backup.log"

# Logging function
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$BACKUP_LOG"
}

# Error handling
cleanup() {
    if [ -n "$SNAPSHOT_MOUNT" ] && mountpoint -q "$SNAPSHOT_MOUNT"; then
        log "Unmounting snapshot..."
        umount "$SNAPSHOT_MOUNT"
    fi
    if [ -n "$SNAPSHOT_LV" ] && lvdisplay "$SNAPSHOT_LV" &>/dev/null; then
        log "Removing snapshot $SNAPSHOT_LV..."
        lvremove -f "$SNAPSHOT_LV"
    fi
}

trap cleanup EXIT

log "Starting Nextcloud backup to $BACKUP_DEST"

# Find the LV backing the nextcloud mount
SOURCE_LV=$(df "$NEXTCLOUD_MOUNT" | tail -1 | awk '{print $1}')
log "Source LV: $SOURCE_LV"

if [ -z "$SOURCE_LV" ]; then
    log "ERROR: Could not determine LV for $NEXTCLOUD_MOUNT"
    exit 1
fi

# Create snapshot
SNAPSHOT_LV="/dev/vg-nextcloud/snapshot-$SNAPSHOT_NAME"
log "Creating snapshot $SNAPSHOT_LV (size: $SNAPSHOT_SIZE)..."
lvcreate -L"$SNAPSHOT_SIZE" -s -n "snapshot-$SNAPSHOT_NAME" "$SOURCE_LV"

if [ $? -ne 0 ]; then
    log "ERROR: Failed to create snapshot"
    exit 1
fi

# Mount snapshot temporarily
SNAPSHOT_MOUNT="/mnt/snapshot-backup-temp"
mkdir -p "$SNAPSHOT_MOUNT"
log "Mounting snapshot to $SNAPSHOT_MOUNT..."
mount -r "$SNAPSHOT_LV" "$SNAPSHOT_MOUNT"

if [ $? -ne 0 ]; then
    log "ERROR: Failed to mount snapshot"
    exit 1
fi

# Run backup (reads from snapshot, not live data)
BACKUP_FILE="$BACKUP_DEST/nextcloud-backup-$SNAPSHOT_NAME.tar.zstd"
mkdir -p "$BACKUP_DEST"

log "Backing up snapshot to $BACKUP_FILE..."
log "This may take 30-60 minutes (reading 1.4TB)..."

# Nice and ionice reduce I/O impact on system
nice -n 15 ionice -c3 tar --zstd -cf "$BACKUP_FILE" -C "$SNAPSHOT_MOUNT" .

if [ $? -eq 0 ]; then
    BACKUP_SIZE=$(du -h "$BACKUP_FILE" | cut -f1)
    log "✓ Backup successful: $BACKUP_SIZE"
else
    log "✗ Backup failed"
    exit 1
fi

# Cleanup snapshot (done automatically via trap)
log "Backup complete. Cleaning up..."
```

#### Step 2: Make it Executable

```bash
chmod +x /usr/local/bin/nextcloud-backup.sh
```

#### Step 3: Create Systemd Service + Timer

Create `/etc/systemd/system/nextcloud-backup.service`:

```ini
[Unit]
Description=Nextcloud Backup Service
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/nextcloud-backup.sh
StandardOutput=journal
StandardError=journal
User=root

# Don't use too much I/O
Nice=15
IOSchedulingClass=idle
```

Create `/etc/systemd/system/nextcloud-backup.timer`:

```ini
[Unit]
Description=Nextcloud Backup Timer
Requires=nextcloud-backup.service

[Timer]
# Run at 2 AM daily when users are asleep
OnCalendar=daily
OnCalendar=*-*-* 02:00:00

# Randomize within 5 minutes to avoid spikes
RandomizedDelaySec=5min

[Install]
WantedBy=timers.target
```

#### Step 4: Enable and Test

```bash
# Reload systemd
systemctl daemon-reload

# Enable timer to start on boot
systemctl enable nextcloud-backup.timer

# Start the timer
systemctl start nextcloud-backup.timer

# Check status
systemctl status nextcloud-backup.timer

# View next scheduled run
systemctl list-timers nextcloud-backup.timer

# Manual test (runs immediately)
systemctl start nextcloud-backup.service

# Watch progress
journalctl -u nextcloud-backup.service -f
```

---

## Alternative: Kopia with Proper Rate Limiting

If you prefer to keep Kopia:

### Configure Kopia for Non-Disruptive Backups

Edit Kopia config (usually `/srv/docker/kopia/config/kopia.json`):

```json
{
  "uploads": {
    "maxParallelFileWrites": 2,
    "maxParallelSmallFileWrites": 2,
    "ignoreFileErrors": false
  },
  "cache": {
    "metadata": {
      "maxCacheSize": 500000000
    }
  },
  "logging": {
    "level": "info"
  }
}
```

### Run Kopia with Nice/Ionice

```bash
# Instead of running kopia directly
nice -n 15 ionice -c3 docker exec kopia kopia snapshot create
```

Or in crontab:
```cron
# 3 AM daily
0 3 * * * nice -n 15 ionice -c3 docker exec kopia kopia snapshot create >> /var/log/kopia-backup.log 2>&1
```

---

## Implementation Roadmap

### Week 1: Preparation
- [ ] Verify LVM setup on nextcloud VM
- [ ] Create backup script
- [ ] Test snapshot creation/deletion
- [ ] Verify backup file integrity

### Week 2: Automation
- [ ] Install systemd service/timer
- [ ] Run first automated backup
- [ ] Monitor system load during backup
- [ ] Verify load average stays below 10

### Week 3: Refinement
- [ ] Adjust backup schedule (time/frequency)
- [ ] Add monitoring alerts
- [ ] Document recovery procedures
- [ ] Test restoration from backup

### Week 4: Production
- [ ] Run weekly backups
- [ ] Monitor Backblaze B2 uploads
- [ ] Schedule monthly test restores
- [ ] Update documentation

---

## Key Differences: Before vs. After

| Aspect | Before | After |
|--------|--------|-------|
| **Load during backup** | 21.89 (system frozen) | <8 (acceptable) |
| **Web service** | 504 errors | Responsive |
| **SSH** | Hangs | Responsive |
| **Backup method** | Live tar + zstd | LVM snapshot |
| **Schedule** | Manual (caused outage) | Automated (2 AM daily) |
| **I/O strategy** | Full speed (killer) | Rate-limited (nice/ionice) |

---

## Monitoring

### Check Backup Status

```bash
# View backup history
ls -lh /mnt/warm-storage/.backup-staging/

# Check backup size
du -sh /mnt/warm-storage/.backup-staging/

# View logs
journalctl -u nextcloud-backup.service --since "1 day ago"
```

### Monitor Load During Backup

```bash
# On another terminal, watch system during backup
watch -n 1 'uptime && echo "---" && iostat -x 1 1 | grep sdc'
```

**What to expect:**
- Load: 4-8 (normal for 4-CPU system during backup)
- Web users: Unaffected
- SSH: Responsive
- Apache/PHP: Normal response times

---

## Recovery: How to Restore from Backup

When you need to restore:

```bash
# List available backups
ls -lh /mnt/warm-storage/.backup-staging/

# Extract to temp location (DON'T overwrite live data!)
mkdir -p /mnt/restore-test
tar --zstd -xf /mnt/warm-storage/.backup-staging/nextcloud-backup-20260428_020000.tar.zstd \
    -C /mnt/restore-test

# Verify files are there
ls /mnt/restore-test/ | head -20

# If good, proceed with restore to actual location
# (This requires stopping Nextcloud first to avoid corruption)
```

---

## Troubleshooting

### "Failed to create snapshot: No space left on device"

The snapshot storage filled up. Increase `$SNAPSHOT_SIZE` in the script:
```bash
SNAPSHOT_SIZE="500G"  # Instead of 200G
```

### "Backup file is too large"

Disable compression to see raw size:
```bash
tar -cf - /mnt/nextcloud-data | wc -c
# Divide by 1TB (1099511627776) to see size in TB
```

If uncompressed is >2TB, you need more storage on warm-storage.

### Backups not running automatically

```bash
# Check timer is running
systemctl status nextcloud-backup.timer

# Enable it if disabled
systemctl enable nextcloud-backup.timer

# Check if service failed
systemctl status nextcloud-backup.service
journalctl -u nextcloud-backup.service -n 50
```

---

## Summary

**Use the LVM snapshot approach** because:
1. ✅ Zero impact on live system during backup
2. ✅ Consistent point-in-time copy
3. ✅ Safe to interrupt without data loss
4. ✅ Easy to automate with systemd
5. ✅ Works with any backup tool (tar, Kopia, rsync)

Next steps:
1. SSH to nextcloud VM
2. Copy backup script from above
3. Create systemd service + timer
4. Enable and test
5. Monitor first backup run
6. Adjust schedule based on timing

You'll have reliable, non-disruptive backups running every night at 2 AM with zero system impact! 🎯