homelab-configs/nextcloud-backup-strategy.md

Non-Disruptive Backup Strategy for Nextcloud

Last Updated: April 28, 2026
Problem: Previous backup approach froze the system (load avg: 21.89 on 4 CPUs)
Solution: Implement intelligent backup strategy that doesn't impact live system

---

What Went Wrong

Your previous backup process:

tar --zstd -cf /mnt/warm-storage/.backup-staging/... /mnt/nextcloud-data/

Problems:

1. Reading 1.4TB live data while the system is running
2. Compressing with zstd (CPU-intensive) at same time
3. Writing compressed data to disk (more I/O)
4. All three happening simultaneously = I/O bottleneck
5. Load average jumped to 21.89 (system completely frozen)
6. Apache couldn't respond to requests (504 errors)
7. SSH connections hung

Key insight: Live backups of large, active datasets require careful planning to avoid saturating disk I/O.

---

Backup Strategy Comparison

Strategy 1: LVM Snapshot + Backup ⭐ RECOMMENDED

How it works:

1. Create a frozen copy (snapshot) of /mnt/nextcloud-data
2. Backup the snapshot while original stays live
3. Delete snapshot when done

Pros:

• Original data unaffected while backup reads snapshot
• Read-only snapshot doesn't create new I/O load
• Fast (COW - Copy-on-Write)
• Works with any backup tool

Cons:

• Requires LVM (you have it: /dev/sdb)
• Snapshot space depends on delta changes

Best for: Your setup (1.4TB Nextcloud data)

---

Strategy 2: Kopia Incremental Backups (Configured Properly)

How it works:

• Kopia only backs up changed blocks
• First backup is full, then incrementals
• Can throttle I/O impact

Pros:

• Incremental = smaller backups after first one
• Built-in deduplication
• Cloud-ready (Backblaze B2)

Cons:

• First full backup is still heavy
• Needs I/O rate limiting configured
• Incremental issues if not properly scheduled

Best for: Ongoing maintenance backups

---

Strategy 3: Database-Specific + File Sync

How it works:

• MariaDB dumps itself separately (consistent point-in-time)
• Rsync/Kopia sync files incrementally
• Lower I/O overall

Pros:

• Database integrity guaranteed
• Can schedule separately
• Flexible

Cons:

• More manual orchestration
• Need to coordinate timing

Best for: Hybrid approach with LVM snapshots

---

Strategy 4: Separate Backup VM

How it works:

• NFS mount nextcloud-data on backup VM
• Backup VM does all I/O work
• Production VM untouched

Pros:

• Zero impact on production
• Can backup without affecting users

Cons:

• Requires extra resources
• Network I/O instead of local

Best for: Very high-availability setups

---

RECOMMENDED APPROACH: LVM Snapshot Strategy

Your /mnt/nextcloud-data is on /dev/sdc (2TB disk). Create backup snapshots from there.

Implementation

Step 1: Create Backup Script

Create /usr/local/bin/nextcloud-backup.sh:

#!/bin/bash
set -e

# Configuration
NEXTCLOUD_MOUNT="/mnt/nextcloud-data"
BACKUP_DEST="/mnt/warm-storage/.backup-staging"
SNAPSHOT_NAME="nextcloud-backup-$(date +%Y%m%d_%H%M%S)"
SNAPSHOT_SIZE="200G"  # Max snapshot size (change if needed)
BACKUP_LOG="/var/log/nextcloud-backup.log"

# Logging function
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$BACKUP_LOG"
}

# Error handling
cleanup() {
    if [ -n "$SNAPSHOT_MOUNT" ] && mountpoint -q "$SNAPSHOT_MOUNT"; then
        log "Unmounting snapshot..."
        umount "$SNAPSHOT_MOUNT"
    fi
    if [ -n "$SNAPSHOT_LV" ] && lvdisplay "$SNAPSHOT_LV" &>/dev/null; then
        log "Removing snapshot $SNAPSHOT_LV..."
        lvremove -f "$SNAPSHOT_LV"
    fi
}

trap cleanup EXIT

log "Starting Nextcloud backup to $BACKUP_DEST"

# Find the LV backing the nextcloud mount
SOURCE_LV=$(df "$NEXTCLOUD_MOUNT" | tail -1 | awk '{print $1}')
log "Source LV: $SOURCE_LV"

if [ -z "$SOURCE_LV" ]; then
    log "ERROR: Could not determine LV for $NEXTCLOUD_MOUNT"
    exit 1
fi

# Create snapshot
SNAPSHOT_LV="/dev/vg-nextcloud/snapshot-$SNAPSHOT_NAME"
log "Creating snapshot $SNAPSHOT_LV (size: $SNAPSHOT_SIZE)..."
lvcreate -L"$SNAPSHOT_SIZE" -s -n "snapshot-$SNAPSHOT_NAME" "$SOURCE_LV"

if [ $? -ne 0 ]; then
    log "ERROR: Failed to create snapshot"
    exit 1
fi

# Mount snapshot temporarily
SNAPSHOT_MOUNT="/mnt/snapshot-backup-temp"
mkdir -p "$SNAPSHOT_MOUNT"
log "Mounting snapshot to $SNAPSHOT_MOUNT..."
mount -r "$SNAPSHOT_LV" "$SNAPSHOT_MOUNT"

if [ $? -ne 0 ]; then
    log "ERROR: Failed to mount snapshot"
    exit 1
fi

# Run backup (reads from snapshot, not live data)
BACKUP_FILE="$BACKUP_DEST/nextcloud-backup-$SNAPSHOT_NAME.tar.zstd"
mkdir -p "$BACKUP_DEST"

log "Backing up snapshot to $BACKUP_FILE..."
log "This may take 30-60 minutes (reading 1.4TB)..."

# Nice and ionice reduce I/O impact on system
nice -n 15 ionice -c3 tar --zstd -cf "$BACKUP_FILE" -C "$SNAPSHOT_MOUNT" .

if [ $? -eq 0 ]; then
    BACKUP_SIZE=$(du -h "$BACKUP_FILE" | cut -f1)
    log "✓ Backup successful: $BACKUP_SIZE"
else
    log "✗ Backup failed"
    exit 1
fi

# Cleanup snapshot (done automatically via trap)
log "Backup complete. Cleaning up..."

Step 2: Make it Executable

chmod +x /usr/local/bin/nextcloud-backup.sh

Step 3: Create Systemd Service + Timer

Create /etc/systemd/system/nextcloud-backup.service:

[Unit]
Description=Nextcloud Backup Service
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/nextcloud-backup.sh
StandardOutput=journal
StandardError=journal
User=root

# Don't use too much I/O
Nice=15
IOSchedulingClass=idle

Create /etc/systemd/system/nextcloud-backup.timer:

[Unit]
Description=Nextcloud Backup Timer
Requires=nextcloud-backup.service

[Timer]
# Run at 2 AM daily when users are asleep
OnCalendar=daily
OnCalendar=*-*-* 02:00:00

# Randomize within 5 minutes to avoid spikes
RandomizedDelaySec=5min

[Install]
WantedBy=timers.target

Step 4: Enable and Test

# Reload systemd
systemctl daemon-reload

# Enable timer to start on boot
systemctl enable nextcloud-backup.timer

# Start the timer
systemctl start nextcloud-backup.timer

# Check status
systemctl status nextcloud-backup.timer

# View next scheduled run
systemctl list-timers nextcloud-backup.timer

# Manual test (runs immediately)
systemctl start nextcloud-backup.service

# Watch progress
journalctl -u nextcloud-backup.service -f

---

Alternative: Kopia with Proper Rate Limiting

If you prefer to keep Kopia:

Configure Kopia for Non-Disruptive Backups

Edit Kopia config (usually /srv/docker/kopia/config/kopia.json):

{
  "uploads": {
    "maxParallelFileWrites": 2,
    "maxParallelSmallFileWrites": 2,
    "ignoreFileErrors": false
  },
  "cache": {
    "metadata": {
      "maxCacheSize": 500000000
    }
  },
  "logging": {
    "level": "info"
  }
}

Run Kopia with Nice/Ionice

# Instead of running kopia directly
nice -n 15 ionice -c3 docker exec kopia kopia snapshot create

Or in crontab:

# 3 AM daily
0 3 * * * nice -n 15 ionice -c3 docker exec kopia kopia snapshot create >> /var/log/kopia-backup.log 2>&1

---

Implementation Roadmap

Week 1: Preparation

• Verify LVM setup on nextcloud VM
• Create backup script
• Test snapshot creation/deletion
• Verify backup file integrity

Week 2: Automation

• Install systemd service/timer
• Run first automated backup
• Monitor system load during backup
• Verify load average stays below 10

Week 3: Refinement

• Adjust backup schedule (time/frequency)
• Add monitoring alerts
• Document recovery procedures
• Test restoration from backup

Week 4: Production

• Run weekly backups
• Monitor Backblaze B2 uploads
• Schedule monthly test restores
• Update documentation

---

Key Differences: Before vs. After

Aspect	Before	After
Load during backup	21.89 (system frozen)	<8 (acceptable)
Web service	504 errors	Responsive
SSH	Hangs	Responsive
Backup method	Live tar + zstd	LVM snapshot
Schedule	Manual (caused outage)	Automated (2 AM daily)
I/O strategy	Full speed (killer)	Rate-limited (nice/ionice)

---

Monitoring

Check Backup Status

# View backup history
ls -lh /mnt/warm-storage/.backup-staging/

# Check backup size
du -sh /mnt/warm-storage/.backup-staging/

# View logs
journalctl -u nextcloud-backup.service --since "1 day ago"

Monitor Load During Backup

# On another terminal, watch system during backup
watch -n 1 'uptime && echo "---" && iostat -x 1 1 | grep sdc'

What to expect:

• Load: 4-8 (normal for 4-CPU system during backup)
• Web users: Unaffected
• SSH: Responsive
• Apache/PHP: Normal response times

---

Recovery: How to Restore from Backup

When you need to restore:

# List available backups
ls -lh /mnt/warm-storage/.backup-staging/

# Extract to temp location (DON'T overwrite live data!)
mkdir -p /mnt/restore-test
tar --zstd -xf /mnt/warm-storage/.backup-staging/nextcloud-backup-20260428_020000.tar.zstd \
    -C /mnt/restore-test

# Verify files are there
ls /mnt/restore-test/ | head -20

# If good, proceed with restore to actual location
# (This requires stopping Nextcloud first to avoid corruption)

---

Troubleshooting

"Failed to create snapshot: No space left on device"

The snapshot storage filled up. Increase $SNAPSHOT_SIZE in the script:

SNAPSHOT_SIZE="500G"  # Instead of 200G

"Backup file is too large"

Disable compression to see raw size:

tar -cf - /mnt/nextcloud-data | wc -c
# Divide by 1TB (1099511627776) to see size in TB

If uncompressed is >2TB, you need more storage on warm-storage.

Backups not running automatically

# Check timer is running
systemctl status nextcloud-backup.timer

# Enable it if disabled
systemctl enable nextcloud-backup.timer

# Check if service failed
systemctl status nextcloud-backup.service
journalctl -u nextcloud-backup.service -n 50

---

Summary

Use the LVM snapshot approach because:

1. ✅ Zero impact on live system during backup
2. ✅ Consistent point-in-time copy
3. ✅ Safe to interrupt without data loss
4. ✅ Easy to automate with systemd
5. ✅ Works with any backup tool (tar, Kopia, rsync)

Next steps:

1. SSH to nextcloud VM
2. Copy backup script from above
3. Create systemd service + timer
4. Enable and test
5. Monitor first backup run
6. Adjust schedule based on timing

You'll have reliable, non-disruptive backups running every night at 2 AM with zero system impact! 🎯