Add kopia restore guide, update Caddy config, add .gitignore to exclude credentials
This commit is contained in:
@@ -0,0 +1,465 @@
|
||||
# Non-Disruptive Backup Strategy for Nextcloud
|
||||
|
||||
**Last Updated**: April 28, 2026
|
||||
**Problem**: Previous backup approach froze the system (load avg: 21.89 on 4 CPUs)
|
||||
**Solution**: Implement intelligent backup strategy that doesn't impact live system
|
||||
|
||||
---
|
||||
|
||||
## What Went Wrong
|
||||
|
||||
Your previous backup process:
|
||||
```
|
||||
tar --zstd -cf /mnt/warm-storage/.backup-staging/... /mnt/nextcloud-data/
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
1. Reading 1.4TB live data while the system is running
|
||||
2. Compressing with zstd (CPU-intensive) at same time
|
||||
3. Writing compressed data to disk (more I/O)
|
||||
4. All three happening simultaneously = I/O bottleneck
|
||||
5. Load average jumped to 21.89 (system completely frozen)
|
||||
6. Apache couldn't respond to requests (504 errors)
|
||||
7. SSH connections hung
|
||||
|
||||
**Key insight**: Live backups of large, active datasets require careful planning to avoid saturating disk I/O.
|
||||
|
||||
---
|
||||
|
||||
## Backup Strategy Comparison
|
||||
|
||||
### Strategy 1: LVM Snapshot + Backup ⭐ RECOMMENDED
|
||||
|
||||
**How it works:**
|
||||
1. Create a frozen copy (snapshot) of `/mnt/nextcloud-data`
|
||||
2. Backup the snapshot while original stays live
|
||||
3. Delete snapshot when done
|
||||
|
||||
**Pros:**
|
||||
- Original data unaffected while backup reads snapshot
|
||||
- Read-only snapshot doesn't create new I/O load
|
||||
- Fast (COW - Copy-on-Write)
|
||||
- Works with any backup tool
|
||||
|
||||
**Cons:**
|
||||
- Requires LVM (you have it: `/dev/sdb`)
|
||||
- Snapshot space depends on delta changes
|
||||
|
||||
**Best for**: Your setup (1.4TB Nextcloud data)
|
||||
|
||||
---
|
||||
|
||||
### Strategy 2: Kopia Incremental Backups (Configured Properly)
|
||||
|
||||
**How it works:**
|
||||
- Kopia only backs up changed blocks
|
||||
- First backup is full, then incrementals
|
||||
- Can throttle I/O impact
|
||||
|
||||
**Pros:**
|
||||
- Incremental = smaller backups after first one
|
||||
- Built-in deduplication
|
||||
- Cloud-ready (Backblaze B2)
|
||||
|
||||
**Cons:**
|
||||
- First full backup is still heavy
|
||||
- Needs I/O rate limiting configured
|
||||
- Incremental issues if not properly scheduled
|
||||
|
||||
**Best for**: Ongoing maintenance backups
|
||||
|
||||
---
|
||||
|
||||
### Strategy 3: Database-Specific + File Sync
|
||||
|
||||
**How it works:**
|
||||
- MariaDB dumps itself separately (consistent point-in-time)
|
||||
- Rsync/Kopia sync files incrementally
|
||||
- Lower I/O overall
|
||||
|
||||
**Pros:**
|
||||
- Database integrity guaranteed
|
||||
- Can schedule separately
|
||||
- Flexible
|
||||
|
||||
**Cons:**
|
||||
- More manual orchestration
|
||||
- Need to coordinate timing
|
||||
|
||||
**Best for**: Hybrid approach with LVM snapshots
|
||||
|
||||
---
|
||||
|
||||
### Strategy 4: Separate Backup VM
|
||||
|
||||
**How it works:**
|
||||
- NFS mount nextcloud-data on backup VM
|
||||
- Backup VM does all I/O work
|
||||
- Production VM untouched
|
||||
|
||||
**Pros:**
|
||||
- Zero impact on production
|
||||
- Can backup without affecting users
|
||||
|
||||
**Cons:**
|
||||
- Requires extra resources
|
||||
- Network I/O instead of local
|
||||
|
||||
**Best for**: Very high-availability setups
|
||||
|
||||
---
|
||||
|
||||
## RECOMMENDED APPROACH: LVM Snapshot Strategy
|
||||
|
||||
Your `/mnt/nextcloud-data` is on `/dev/sdc` (2TB disk). Create backup snapshots from there.
|
||||
|
||||
### Implementation
|
||||
|
||||
#### Step 1: Create Backup Script
|
||||
|
||||
Create `/usr/local/bin/nextcloud-backup.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Configuration
|
||||
NEXTCLOUD_MOUNT="/mnt/nextcloud-data"
|
||||
BACKUP_DEST="/mnt/warm-storage/.backup-staging"
|
||||
SNAPSHOT_NAME="nextcloud-backup-$(date +%Y%m%d_%H%M%S)"
|
||||
SNAPSHOT_SIZE="200G" # Max snapshot size (change if needed)
|
||||
BACKUP_LOG="/var/log/nextcloud-backup.log"
|
||||
|
||||
# Logging function
|
||||
log() {
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$BACKUP_LOG"
|
||||
}
|
||||
|
||||
# Error handling
|
||||
cleanup() {
|
||||
if [ -n "$SNAPSHOT_MOUNT" ] && mountpoint -q "$SNAPSHOT_MOUNT"; then
|
||||
log "Unmounting snapshot..."
|
||||
umount "$SNAPSHOT_MOUNT"
|
||||
fi
|
||||
if [ -n "$SNAPSHOT_LV" ] && lvdisplay "$SNAPSHOT_LV" &>/dev/null; then
|
||||
log "Removing snapshot $SNAPSHOT_LV..."
|
||||
lvremove -f "$SNAPSHOT_LV"
|
||||
fi
|
||||
}
|
||||
|
||||
trap cleanup EXIT
|
||||
|
||||
log "Starting Nextcloud backup to $BACKUP_DEST"
|
||||
|
||||
# Find the LV backing the nextcloud mount
|
||||
SOURCE_LV=$(df "$NEXTCLOUD_MOUNT" | tail -1 | awk '{print $1}')
|
||||
log "Source LV: $SOURCE_LV"
|
||||
|
||||
if [ -z "$SOURCE_LV" ]; then
|
||||
log "ERROR: Could not determine LV for $NEXTCLOUD_MOUNT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Create snapshot
|
||||
SNAPSHOT_LV="/dev/vg-nextcloud/snapshot-$SNAPSHOT_NAME"
|
||||
log "Creating snapshot $SNAPSHOT_LV (size: $SNAPSHOT_SIZE)..."
|
||||
lvcreate -L"$SNAPSHOT_SIZE" -s -n "snapshot-$SNAPSHOT_NAME" "$SOURCE_LV"
|
||||
|
||||
if [ $? -ne 0 ]; then
|
||||
log "ERROR: Failed to create snapshot"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Mount snapshot temporarily
|
||||
SNAPSHOT_MOUNT="/mnt/snapshot-backup-temp"
|
||||
mkdir -p "$SNAPSHOT_MOUNT"
|
||||
log "Mounting snapshot to $SNAPSHOT_MOUNT..."
|
||||
mount -r "$SNAPSHOT_LV" "$SNAPSHOT_MOUNT"
|
||||
|
||||
if [ $? -ne 0 ]; then
|
||||
log "ERROR: Failed to mount snapshot"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Run backup (reads from snapshot, not live data)
|
||||
BACKUP_FILE="$BACKUP_DEST/nextcloud-backup-$SNAPSHOT_NAME.tar.zstd"
|
||||
mkdir -p "$BACKUP_DEST"
|
||||
|
||||
log "Backing up snapshot to $BACKUP_FILE..."
|
||||
log "This may take 30-60 minutes (reading 1.4TB)..."
|
||||
|
||||
# Nice and ionice reduce I/O impact on system
|
||||
nice -n 15 ionice -c3 tar --zstd -cf "$BACKUP_FILE" -C "$SNAPSHOT_MOUNT" .
|
||||
|
||||
if [ $? -eq 0 ]; then
|
||||
BACKUP_SIZE=$(du -h "$BACKUP_FILE" | cut -f1)
|
||||
log "✓ Backup successful: $BACKUP_SIZE"
|
||||
else
|
||||
log "✗ Backup failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Cleanup snapshot (done automatically via trap)
|
||||
log "Backup complete. Cleaning up..."
|
||||
```
|
||||
|
||||
#### Step 2: Make it Executable
|
||||
|
||||
```bash
|
||||
chmod +x /usr/local/bin/nextcloud-backup.sh
|
||||
```
|
||||
|
||||
#### Step 3: Create Systemd Service + Timer
|
||||
|
||||
Create `/etc/systemd/system/nextcloud-backup.service`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Nextcloud Backup Service
|
||||
After=local-fs.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/bin/nextcloud-backup.sh
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
User=root
|
||||
|
||||
# Don't use too much I/O
|
||||
Nice=15
|
||||
IOSchedulingClass=idle
|
||||
```
|
||||
|
||||
Create `/etc/systemd/system/nextcloud-backup.timer`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Nextcloud Backup Timer
|
||||
Requires=nextcloud-backup.service
|
||||
|
||||
[Timer]
|
||||
# Run at 2 AM daily when users are asleep
|
||||
OnCalendar=daily
|
||||
OnCalendar=*-*-* 02:00:00
|
||||
|
||||
# Randomize within 5 minutes to avoid spikes
|
||||
RandomizedDelaySec=5min
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
```
|
||||
|
||||
#### Step 4: Enable and Test
|
||||
|
||||
```bash
|
||||
# Reload systemd
|
||||
systemctl daemon-reload
|
||||
|
||||
# Enable timer to start on boot
|
||||
systemctl enable nextcloud-backup.timer
|
||||
|
||||
# Start the timer
|
||||
systemctl start nextcloud-backup.timer
|
||||
|
||||
# Check status
|
||||
systemctl status nextcloud-backup.timer
|
||||
|
||||
# View next scheduled run
|
||||
systemctl list-timers nextcloud-backup.timer
|
||||
|
||||
# Manual test (runs immediately)
|
||||
systemctl start nextcloud-backup.service
|
||||
|
||||
# Watch progress
|
||||
journalctl -u nextcloud-backup.service -f
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Alternative: Kopia with Proper Rate Limiting
|
||||
|
||||
If you prefer to keep Kopia:
|
||||
|
||||
### Configure Kopia for Non-Disruptive Backups
|
||||
|
||||
Edit Kopia config (usually `/srv/docker/kopia/config/kopia.json`):
|
||||
|
||||
```json
|
||||
{
|
||||
"uploads": {
|
||||
"maxParallelFileWrites": 2,
|
||||
"maxParallelSmallFileWrites": 2,
|
||||
"ignoreFileErrors": false
|
||||
},
|
||||
"cache": {
|
||||
"metadata": {
|
||||
"maxCacheSize": 500000000
|
||||
}
|
||||
},
|
||||
"logging": {
|
||||
"level": "info"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Run Kopia with Nice/Ionice
|
||||
|
||||
```bash
|
||||
# Instead of running kopia directly
|
||||
nice -n 15 ionice -c3 docker exec kopia kopia snapshot create
|
||||
```
|
||||
|
||||
Or in crontab:
|
||||
```cron
|
||||
# 3 AM daily
|
||||
0 3 * * * nice -n 15 ionice -c3 docker exec kopia kopia snapshot create >> /var/log/kopia-backup.log 2>&1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Roadmap
|
||||
|
||||
### Week 1: Preparation
|
||||
- [ ] Verify LVM setup on nextcloud VM
|
||||
- [ ] Create backup script
|
||||
- [ ] Test snapshot creation/deletion
|
||||
- [ ] Verify backup file integrity
|
||||
|
||||
### Week 2: Automation
|
||||
- [ ] Install systemd service/timer
|
||||
- [ ] Run first automated backup
|
||||
- [ ] Monitor system load during backup
|
||||
- [ ] Verify load average stays below 10
|
||||
|
||||
### Week 3: Refinement
|
||||
- [ ] Adjust backup schedule (time/frequency)
|
||||
- [ ] Add monitoring alerts
|
||||
- [ ] Document recovery procedures
|
||||
- [ ] Test restoration from backup
|
||||
|
||||
### Week 4: Production
|
||||
- [ ] Run weekly backups
|
||||
- [ ] Monitor Backblaze B2 uploads
|
||||
- [ ] Schedule monthly test restores
|
||||
- [ ] Update documentation
|
||||
|
||||
---
|
||||
|
||||
## Key Differences: Before vs. After
|
||||
|
||||
| Aspect | Before | After |
|
||||
|--------|--------|-------|
|
||||
| **Load during backup** | 21.89 (system frozen) | <8 (acceptable) |
|
||||
| **Web service** | 504 errors | Responsive |
|
||||
| **SSH** | Hangs | Responsive |
|
||||
| **Backup method** | Live tar + zstd | LVM snapshot |
|
||||
| **Schedule** | Manual (caused outage) | Automated (2 AM daily) |
|
||||
| **I/O strategy** | Full speed (killer) | Rate-limited (nice/ionice) |
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Check Backup Status
|
||||
|
||||
```bash
|
||||
# View backup history
|
||||
ls -lh /mnt/warm-storage/.backup-staging/
|
||||
|
||||
# Check backup size
|
||||
du -sh /mnt/warm-storage/.backup-staging/
|
||||
|
||||
# View logs
|
||||
journalctl -u nextcloud-backup.service --since "1 day ago"
|
||||
```
|
||||
|
||||
### Monitor Load During Backup
|
||||
|
||||
```bash
|
||||
# On another terminal, watch system during backup
|
||||
watch -n 1 'uptime && echo "---" && iostat -x 1 1 | grep sdc'
|
||||
```
|
||||
|
||||
**What to expect:**
|
||||
- Load: 4-8 (normal for 4-CPU system during backup)
|
||||
- Web users: Unaffected
|
||||
- SSH: Responsive
|
||||
- Apache/PHP: Normal response times
|
||||
|
||||
---
|
||||
|
||||
## Recovery: How to Restore from Backup
|
||||
|
||||
When you need to restore:
|
||||
|
||||
```bash
|
||||
# List available backups
|
||||
ls -lh /mnt/warm-storage/.backup-staging/
|
||||
|
||||
# Extract to temp location (DON'T overwrite live data!)
|
||||
mkdir -p /mnt/restore-test
|
||||
tar --zstd -xf /mnt/warm-storage/.backup-staging/nextcloud-backup-20260428_020000.tar.zstd \
|
||||
-C /mnt/restore-test
|
||||
|
||||
# Verify files are there
|
||||
ls /mnt/restore-test/ | head -20
|
||||
|
||||
# If good, proceed with restore to actual location
|
||||
# (This requires stopping Nextcloud first to avoid corruption)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Failed to create snapshot: No space left on device"
|
||||
|
||||
The snapshot storage filled up. Increase `$SNAPSHOT_SIZE` in the script:
|
||||
```bash
|
||||
SNAPSHOT_SIZE="500G" # Instead of 200G
|
||||
```
|
||||
|
||||
### "Backup file is too large"
|
||||
|
||||
Disable compression to see raw size:
|
||||
```bash
|
||||
tar -cf - /mnt/nextcloud-data | wc -c
|
||||
# Divide by 1TB (1099511627776) to see size in TB
|
||||
```
|
||||
|
||||
If uncompressed is >2TB, you need more storage on warm-storage.
|
||||
|
||||
### Backups not running automatically
|
||||
|
||||
```bash
|
||||
# Check timer is running
|
||||
systemctl status nextcloud-backup.timer
|
||||
|
||||
# Enable it if disabled
|
||||
systemctl enable nextcloud-backup.timer
|
||||
|
||||
# Check if service failed
|
||||
systemctl status nextcloud-backup.service
|
||||
journalctl -u nextcloud-backup.service -n 50
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
**Use the LVM snapshot approach** because:
|
||||
1. ✅ Zero impact on live system during backup
|
||||
2. ✅ Consistent point-in-time copy
|
||||
3. ✅ Safe to interrupt without data loss
|
||||
4. ✅ Easy to automate with systemd
|
||||
5. ✅ Works with any backup tool (tar, Kopia, rsync)
|
||||
|
||||
Next steps:
|
||||
1. SSH to nextcloud VM
|
||||
2. Copy backup script from above
|
||||
3. Create systemd service + timer
|
||||
4. Enable and test
|
||||
5. Monitor first backup run
|
||||
6. Adjust schedule based on timing
|
||||
|
||||
You'll have reliable, non-disruptive backups running every night at 2 AM with zero system impact! 🎯
|
||||
Reference in New Issue
Block a user