homelab-configs/dedup-strategy-jgitta.md

Strategy: Dedup & Merge cold-storage/jgitta into Nextcloud jgitta

Goal: Take everything in /mnt/cold-storage/jgitta, compare it to jgitta's primary Nextcloud files, delete duplicates from cold-storage, and move unique files into jgitta's account.

Scope: jgitta only (separate from cgitta project, which is now complete).

---

The Situation

Location	Size	Notes
/mnt/cold-storage/jgitta	UNKNOWN (very large)	NFS-mounted from TrueNAS. ~25 subfolders: phone backups, Google Takeout, OneDrive dumps, etc.
/mnt/nextcloud-data/jgitta/files	239 GB	Local 2 TB SSD on VM103. jgitta's primary Nextcloud files.
Free space on /mnt/nextcloud-data	488 GB	This is the constraint. If cold-storage/jgitta > 488 GB of unique files, we'll run out of space.

⚠️ Critical: jgitta's cold-storage is likely multi-TB. If unique files exceed 488 GB, we need to expand the primary disk or use warm-storage for overflow.

---

Phase 0: Hash Scan (Read-only inventory)

Step 1: Start hash scans (may take 2-4 hours given size)

cd /var/tmp/dedup-jgitta
nohup /var/tmp/dedup-jgitta/scan.sh /mnt/cold-storage/jgitta /var/tmp/dedup-jgitta/cold > /dev/null 2>&1 &
nohup /var/tmp/dedup-jgitta/scan.sh /mnt/nextcloud-data/jgitta/files /var/tmp/dedup-jgitta/primary > /dev/null 2>&1 &

Step 2: When scans finish, build report (same as cgitta):

cd /var/tmp/dedup-jgitta
awk '{print $1}' cold.hashes    | sort -u > cold.hashset
awk '{print $1}' primary.hashes | sort -u > primary.hashset
comm -12 cold.hashset primary.hashset > dup.hashset
comm -23 cold.hashset primary.hashset > cold-only.hashset
grep -Ff dup.hashset      cold.hashes > dup.files
grep -Ff cold-only.hashset cold.hashes > unique.files

Step 3: Review numbers for capacity planning.

---

⚠️ CAPACITY DECISION POINT

Before proceeding to Phase 2, check:

1. Size of unique files: awk '{print $1}' unique.files | wc -l and estimate total GB
2. Available space: Currently 488 GB. If unique > 488 GB:
   • Option A: Expand primary disk (add storage to /mnt/nextcloud-data)
   • Option B: Move jgitta's cold-storage overflow to warm-storage tier (if available)
   • Option C: Dedup in two passes (move largest categories first, verify space)

---

Phases 2–5 (same as cgitta, if space is available)

Once capacity is confirmed:

• Phase 2: Delete duplicates from cold-storage (with ZFS snapshot backup)
• Phase 3: Move unique files to primary
• Phase 4: Fix ownership + Nextcloud rescan
• Phase 5: Verify + cleanup

---

Next Step

1. Start Phase 0 hash scans
2. Wait for completion (may take several hours)
3. Report numbers
4. Decide on capacity strategy before Phase 2