homelab-configs/dedup-dryrun-report-cgitta.md

DRY-RUN REPORT: Dedup & Merge cgitta Cold-Storage → Nextcloud

Date: 2026-04-17
Status: ✅ Hash scans complete. Analysis ready for review.
Analysis Type: SHA-256 content-based deduplication (zero false positives)

---

Executive Summary

All clear to proceed with Phase 2–3. No blocking issues detected. Zero path collisions. Primary disk has ample space for the move.

Metric	Value
Total files in cold-storage/cgitta	66,663
Total size in cold-storage/cgitta	~200.61 GB
Duplicate files to delete	15,299 (23.0%)
Unique files to move	51,364 (77.0%)
Est. GB to delete	~46 GB
Est. GB to move	~155 GB
Primary disk free space	~650 GB
Primary disk after move	~495 GB remaining ✅
Path collisions	0 ✅
Internal duplicates in cold	5,367 hashes (same file exists >1× in cold)

---

What the numbers mean

Duplicates → Safe to Delete from Cold-Storage

15,299 files are exact byte-for-byte matches with files already in cgitta's primary Nextcloud account. These are redundant copies sitting in cold-storage that can be safely removed.

• These files are already available to cgitta through her primary Nextcloud files
• Deleting them frees up cold-storage space
• Zero risk of data loss (source file exists in primary)

Unique Files → Must Move to Primary

51,364 files exist in cold-storage but have no match in cgitta's primary files. These are new/unique content that belongs in her Nextcloud account.

• If we don't move them, cgitta loses access to them (they stay stuck on NFS cold-storage, invisible in Nextcloud)
• These will be physically moved (not copied) from cold-storage into /mnt/nextcloud-data/cgitta/files/
• After move, they'll appear in cgitta's Nextcloud web UI under the same folder names (Backup Files/, HP Laptop/, zip drives/)

Internal Duplicates Within Cold-Storage

5,367 unique hashes appear more than once in cold-storage itself. This means the same file (by content) is stored in multiple locations or with multiple filenames within cold-storage/cgitta.

• When we move unique files to primary, we'll move all copies (including the duplicates)
• Optional: we could de-duplicate before moving (keep only one copy of each hash in cold), but Phase 0 captures everything as-is
• Recommendation: move as-is. The duplicates are small relative to the total, and keeping them maintains the original structure cgitta had in cold.

---

File Breakdown by Category

cold-storage/cgitta (66,663 files):
├─ 15,299 files → already in primary (delete from cold)
├─ 51,364 files → unique to cold (move to primary)
└─ Of the 51,364 unique:
   └─ 5,367 hashes appear multiple times (internal dupes)

---

Safety Checks Passed ✅

Check	Status	Notes
Path collisions	✅ NONE	No filename/path conflicts. Safe to move without renaming.
Primary disk space	✅ SAFE	650 GB free > 155 GB needed for unique files.
File ownership	✅ WILL HANDLE	Phase 4 sets all files to www-data:www-data.
Nextcloud DB sync	✅ WILL HANDLE	Phase 4 runs occ files:scan to update Nextcloud's database.
ZFS snapshot backup	✅ WILL CREATE	Phase 2 creates pool1/cold-storage@pre-cgitta-dedup-2026-04-17 before any deletions.
Relative paths preserved	✅ YES	/mnt/cold-storage/cgitta/Backup Files/foo.txt → /mnt/nextcloud-data/cgitta/files/Backup Files/foo.txt

---

Sample Files

Files being deleted (duplicates)

Bing.url
VSO0807159M3FB5598OK.pdf
PAGO A VIAJES SORIANO 14,946 050919 SNB IOWA.pdf
… (15,299 total)

All of these already exist in cgitta's primary Nextcloud files.

Files being moved (unique)

zip drives/Polar_Pals_roll_&_keep_it.zip
zip drives/PictureJournalPromptsBUNDLELeveledWritingGreatforSpecialEd.zip
zip drives/New_Year_word_wall_cards_(3731009).zip
… (51,364 total)

All of these will be moved into cgitta's primary files.

---

What Happens Next (Phases 2–5)

Phase 2: Delete duplicates from cold-storage

• Takes ZFS snapshot on TrueNAS (instant, zero-cost backup)
• Deletes 15,299 duplicate files from cold-storage
• Estimated time: 10–20 minutes (local NFS delete operations are fast)
• Reversible: snapshot exists on TrueNAS if anything goes wrong

Phase 3: Move unique files to primary

• Moves 51,364 files from cold-storage into /mnt/nextcloud-data/cgitta/files/
• Preserves folder structure and relative paths
• Estimated time: 2–4 hours (network copy from NFS to local disk, ~100 MB/s bottleneck)
• Safety: files moved one at a time; any partial failure leaves cold copies intact for retry

Phase 4: Ownership & Nextcloud rescan

• Fixes file ownership to www-data:www-data (Nextcloud requirement)
• Runs occ files:scan to register files in Nextcloud database
• Estimated time: 5–10 minutes
• Files become visible in cgitta's Nextcloud web UI

Phase 5: Verify & cleanup

• Log into Nextcloud as cgitta, verify files appear in web UI
• Confirm cold-storage/cgitta is now empty
• Remove scratch directory (/var/tmp/dedup-cgitta)
• Optionally destroy ZFS snapshot after a grace period (1 week)

---

Disk Impact Summary

On /mnt/cold-storage (NFS)

Before:

cold-storage/cgitta: 200.61 GB

After Phase 2 (delete duplicates):

cold-storage/cgitta: ~154 GB (46 GB freed)

After Phase 3 (move unique):

cold-storage/cgitta: ~0 GB (empty)

On /mnt/nextcloud-data/cgitta/files (local SSD)

Before:

Used: ~1.35 TB
Free: ~650 GB

After Phase 3 (move unique):

Used: ~1.5 TB (original + 155 GB of unique files)
Free: ~495 GB ✅ (still comfortable margin)

---

Recommendations

1. Proceed with Phases 2–5. All safety checks pass. No blocking issues.

2. Optional: If you want to de-duplicate internal copies before moving, let me know. We can identify which of the 5,367 internal-dup hashes should be kept (e.g., keep only the first occurrence of each unique hash). This would reduce the move size slightly but complicate the process.

3. Time estimate: Plan for 3–5 hours total runtime (mostly Phase 3, which is network-bound).

4. Schedule: Phase 2–3 can run during off-peak hours. Nextcloud will be usable during the move (files being moved are in cold-storage, not primary), though cgitta might notice new files appearing mid-move as Phase 3 progresses.

---

Questions Before Proceeding?

Before I kick off Phase 2, let me know:

• ✅ Proceed with deletion and move as described?
• ❓ Any concern about the file counts or size estimates?
• ❓ Want to de-duplicate internal copies first, or keep the originals as-is?
• ❓ Preferred time to start? (Can run anytime; no downtime needed.)

---

Appendix: Command Reference

All Phase 2–5 commands are in the main strategy doc (dedup-strategy-cgitta.md). Below is a quick reference:

Phase 2 (Delete):

# TrueNAS snapshot
curl -sk -u "root:<password>" -X POST https://192.168.88.24/api/v2.0/zfs/snapshot …

# VM103 delete
awk '{ $1=""; sub(/^ +/,""); print }' /var/tmp/dedup-cgitta/dup.files | \
  sudo xargs -d '\n' -I{} rm -v "{}"

# Prune empty dirs
sudo find /mnt/cold-storage/cgitta -depth -type d -empty -delete

Phase 3 (Move):

# Dry-run first (review /var/tmp/dedup-cgitta/move-plan.txt)
awk '{ $1=""; sub(/^ +/,""); print }' /var/tmp/dedup-cgitta/unique.files | \
  while IFS= read -r src; do
    rel="${src#/mnt/cold-storage/cgitta/}"
    dst="/mnt/nextcloud-data/cgitta/files/${rel}"
    echo "MOVE: $src  ->  $dst"
  done | tee /var/tmp/dedup-cgitta/move-plan.txt

# Real move
awk '{ $1=""; sub(/^ +/,""); print }' /var/tmp/dedup-cgitta/unique.files | \
  while IFS= read -r src; do
    rel="${src#/mnt/cold-storage/cgitta/}"
    dst="/mnt/nextcloud-data/cgitta/files/${rel}"
    sudo install -d -o www-data -g www-data "$(dirname "$dst")"
    sudo mv -n -v "$src" "$dst"
  done | tee /var/tmp/dedup-cgitta/moved.log

Phase 4 (Ownership & Scan):

sudo chown -R www-data:www-data /mnt/nextcloud-data/cgitta/files/
sudo -u www-data php /var/www/nextcloud/occ files:scan cgitta

---

Report generated: 2026-04-17 17:18 UTC
Next action: Review and approve Phases 2–5.