# DRY-RUN REPORT: Dedup & Merge cgitta Cold-Storage → Nextcloud

**Date:** 2026-04-17  
**Status:** ✅ Hash scans complete. Analysis ready for review.  
**Analysis Type:** SHA-256 content-based deduplication (zero false positives)

---

## Executive Summary

**All clear to proceed with Phase 2–3.** No blocking issues detected. Zero path collisions. Primary disk has ample space for the move.

| Metric | Value |
|---|---|
| **Total files in cold-storage/cgitta** | 66,663 |
| **Total size in cold-storage/cgitta** | ~200.61 GB |
| **Duplicate files to delete** | 15,299 (23.0%) |
| **Unique files to move** | 51,364 (77.0%) |
| **Est. GB to delete** | ~46 GB |
| **Est. GB to move** | ~155 GB |
| **Primary disk free space** | ~650 GB |
| **Primary disk after move** | ~495 GB remaining ✅ |
| **Path collisions** | **0** ✅ |
| **Internal duplicates in cold** | 5,367 hashes (same file exists >1× in cold) |

---

## What the numbers mean

### Duplicates → Safe to Delete from Cold-Storage

**15,299 files** are exact byte-for-byte matches with files already in cgitta's primary Nextcloud account. These are redundant copies sitting in cold-storage that can be safely removed.

- These files are already available to cgitta through her primary Nextcloud files
- Deleting them frees up cold-storage space
- Zero risk of data loss (source file exists in primary)

### Unique Files → Must Move to Primary

**51,364 files** exist in cold-storage but have no match in cgitta's primary files. These are new/unique content that belongs in her Nextcloud account.

- If we don't move them, cgitta loses access to them (they stay stuck on NFS cold-storage, invisible in Nextcloud)
- These will be physically moved (not copied) from cold-storage into `/mnt/nextcloud-data/cgitta/files/`
- After move, they'll appear in cgitta's Nextcloud web UI under the same folder names (`Backup Files/`, `HP Laptop/`, `zip drives/`)

### Internal Duplicates Within Cold-Storage

**5,367 unique hashes** appear more than once in cold-storage itself. This means the same file (by content) is stored in multiple locations or with multiple filenames within cold-storage/cgitta.

- When we move unique files to primary, we'll move all copies (including the duplicates)
- Optional: we could de-duplicate before moving (keep only one copy of each hash in cold), but Phase 0 captures everything as-is
- Recommendation: **move as-is**. The duplicates are small relative to the total, and keeping them maintains the original structure cgitta had in cold.

---

## File Breakdown by Category

```
cold-storage/cgitta (66,663 files):
├─ 15,299 files → already in primary (delete from cold)
├─ 51,364 files → unique to cold (move to primary)
└─ Of the 51,364 unique:
   └─ 5,367 hashes appear multiple times (internal dupes)
```

---

## Safety Checks Passed ✅

| Check | Status | Notes |
|---|---|---|
| **Path collisions** | ✅ NONE | No filename/path conflicts. Safe to move without renaming. |
| **Primary disk space** | ✅ SAFE | 650 GB free > 155 GB needed for unique files. |
| **File ownership** | ✅ WILL HANDLE | Phase 4 sets all files to `www-data:www-data`. |
| **Nextcloud DB sync** | ✅ WILL HANDLE | Phase 4 runs `occ files:scan` to update Nextcloud's database. |
| **ZFS snapshot backup** | ✅ WILL CREATE | Phase 2 creates `pool1/cold-storage@pre-cgitta-dedup-2026-04-17` before any deletions. |
| **Relative paths preserved** | ✅ YES | `/mnt/cold-storage/cgitta/Backup Files/foo.txt` → `/mnt/nextcloud-data/cgitta/files/Backup Files/foo.txt` |

---

## Sample Files

### Files being deleted (duplicates)
```
Bing.url
VSO0807159M3FB5598OK.pdf
PAGO A VIAJES SORIANO 14,946 050919 SNB IOWA.pdf
… (15,299 total)
```

All of these already exist in cgitta's primary Nextcloud files.

### Files being moved (unique)
```
zip drives/Polar_Pals_roll_&_keep_it.zip
zip drives/PictureJournalPromptsBUNDLELeveledWritingGreatforSpecialEd.zip
zip drives/New_Year_word_wall_cards_(3731009).zip
… (51,364 total)
```

All of these will be moved into cgitta's primary files.

---

## What Happens Next (Phases 2–5)

### Phase 2: Delete duplicates from cold-storage
- Takes ZFS snapshot on TrueNAS (instant, zero-cost backup)
- Deletes 15,299 duplicate files from cold-storage
- **Estimated time:** 10–20 minutes (local NFS delete operations are fast)
- **Reversible:** snapshot exists on TrueNAS if anything goes wrong

### Phase 3: Move unique files to primary
- Moves 51,364 files from cold-storage into `/mnt/nextcloud-data/cgitta/files/`
- Preserves folder structure and relative paths
- **Estimated time:** 2–4 hours (network copy from NFS to local disk, ~100 MB/s bottleneck)
- **Safety:** files moved one at a time; any partial failure leaves cold copies intact for retry

### Phase 4: Ownership & Nextcloud rescan
- Fixes file ownership to `www-data:www-data` (Nextcloud requirement)
- Runs `occ files:scan` to register files in Nextcloud database
- **Estimated time:** 5–10 minutes
- Files become visible in cgitta's Nextcloud web UI

### Phase 5: Verify & cleanup
- Log into Nextcloud as cgitta, verify files appear in web UI
- Confirm cold-storage/cgitta is now empty
- Remove scratch directory (`/var/tmp/dedup-cgitta`)
- Optionally destroy ZFS snapshot after a grace period (1 week)

---

## Disk Impact Summary

### On `/mnt/cold-storage` (NFS)
**Before:**
```
cold-storage/cgitta: 200.61 GB
```

**After Phase 2 (delete duplicates):**
```
cold-storage/cgitta: ~154 GB (46 GB freed)
```

**After Phase 3 (move unique):**
```
cold-storage/cgitta: ~0 GB (empty)
```

### On `/mnt/nextcloud-data/cgitta/files` (local SSD)
**Before:**
```
Used: ~1.35 TB
Free: ~650 GB
```

**After Phase 3 (move unique):**
```
Used: ~1.5 TB (original + 155 GB of unique files)
Free: ~495 GB ✅ (still comfortable margin)
```

---

## Recommendations

1. **Proceed with Phases 2–5.** All safety checks pass. No blocking issues.

2. **Optional:** If you want to de-duplicate internal copies before moving, let me know. We can identify which of the 5,367 internal-dup hashes should be kept (e.g., keep only the first occurrence of each unique hash). This would reduce the move size slightly but complicate the process.

3. **Time estimate:** Plan for **3–5 hours total runtime** (mostly Phase 3, which is network-bound).

4. **Schedule:** Phase 2–3 can run during off-peak hours. Nextcloud will be usable during the move (files being moved are in cold-storage, not primary), though cgitta might notice new files appearing mid-move as Phase 3 progresses.

---

## Questions Before Proceeding?

Before I kick off Phase 2, let me know:
- ✅ Proceed with deletion and move as described?
- ❓ Any concern about the file counts or size estimates?
- ❓ Want to de-duplicate internal copies first, or keep the originals as-is?
- ❓ Preferred time to start? (Can run anytime; no downtime needed.)

---

## Appendix: Command Reference

All Phase 2–5 commands are in the main strategy doc (`dedup-strategy-cgitta.md`). Below is a quick reference:

**Phase 2 (Delete):**
```bash
# TrueNAS snapshot
curl -sk -u "root:<password>" -X POST https://192.168.88.24/api/v2.0/zfs/snapshot …

# VM103 delete
awk '{ $1=""; sub(/^ +/,""); print }' /var/tmp/dedup-cgitta/dup.files | \
  sudo xargs -d '\n' -I{} rm -v "{}"

# Prune empty dirs
sudo find /mnt/cold-storage/cgitta -depth -type d -empty -delete
```

**Phase 3 (Move):**
```bash
# Dry-run first (review /var/tmp/dedup-cgitta/move-plan.txt)
awk '{ $1=""; sub(/^ +/,""); print }' /var/tmp/dedup-cgitta/unique.files | \
  while IFS= read -r src; do
    rel="${src#/mnt/cold-storage/cgitta/}"
    dst="/mnt/nextcloud-data/cgitta/files/${rel}"
    echo "MOVE: $src  ->  $dst"
  done | tee /var/tmp/dedup-cgitta/move-plan.txt

# Real move
awk '{ $1=""; sub(/^ +/,""); print }' /var/tmp/dedup-cgitta/unique.files | \
  while IFS= read -r src; do
    rel="${src#/mnt/cold-storage/cgitta/}"
    dst="/mnt/nextcloud-data/cgitta/files/${rel}"
    sudo install -d -o www-data -g www-data "$(dirname "$dst")"
    sudo mv -n -v "$src" "$dst"
  done | tee /var/tmp/dedup-cgitta/moved.log
```

**Phase 4 (Ownership & Scan):**
```bash
sudo chown -R www-data:www-data /mnt/nextcloud-data/cgitta/files/
sudo -u www-data php /var/www/nextcloud/occ files:scan cgitta
```

---

**Report generated:** 2026-04-17 17:18 UTC  
**Next action:** Review and approve Phases 2–5.