homelab-configs/dedup-strategy-cgitta.md

Strategy: Dedup & Merge cold-storage/cgitta into Nextcloud cgitta

Goal: Take everything currently sitting in /mnt/cold-storage/cgitta on VM103 (the Nextcloud VM), compare it file-by-file to what cgitta already has in her Nextcloud primary files, delete anything from cold-storage that's already a perfect duplicate, and physically move the rest into cgitta's Nextcloud account so it shows up in her normal files view.

Scope of this document: cgitta only. jgitta will be a separate, larger operation — covered in a follow-up doc once cgitta is complete and the process is validated.

---

The lay of the land (what we're working with)

Location	Size	Notes
/mnt/cold-storage/cgitta	193 GB	NFS-mounted from TrueNAS. Subfolders: Backup Files, HP Laptop, zip drives. This is the source.
/mnt/nextcloud-data/cgitta/files	256 GB	Local 2 TB SSD on VM103. This is cgitta's primary Nextcloud files. This is the destination.
Free space on /mnt/nextcloud-data	~650 GB	Plenty of room for 193 GB, no capacity adjustment needed.
/cold-storage (Nextcloud external mount)	shared	Currently visible to all users as one shared cold-storage folder. We'll leave this alone — it still serves its purpose for Entertainment/Family/etc.

A few things to know up front:

• NFS means "network filesystem". The cold-storage is physically on TrueNAS (192.168.88.24) but mounted into VM103 as if it were local. Reads are limited by network speed (~100 MB/s on gigabit).
• Nextcloud uses a database to track which files exist. If we just move files into /mnt/nextcloud-data/cgitta/files/ without telling Nextcloud, cgitta won't see them in the web UI. We have to run occ files:scan after the move — it's a command that makes Nextcloud re-read the filesystem and update its database.
• File ownership matters. Nextcloud runs as the www-data user. Every file in its data directory must be owned by www-data:www-data or Nextcloud will refuse to read it.

---

How we're going to decide two files are "the same"

You chose SHA-256 content hash. That means:

1. For every file in cold-storage/cgitta and every file in primary files, we compute a 64-character fingerprint based on the full byte-by-byte contents.
2. If two files have the same fingerprint, they are guaranteed to be byte-identical — it doesn't matter what the filename is or where in the tree it sits. A file called IMG_1234.JPG and a file called vacation copy.jpg with the same hash are, in fact, the same photo.
3. If the fingerprints differ — even by one byte — they're different files and both need to be preserved.

This is slow because every byte of every file has to be read, but it's the only method that gives zero false positives (no chance of accidentally deleting a file we thought was a duplicate but actually wasn't).

Estimated time for hash scan of cgitta: 30–60 minutes (193 GB NFS + 256 GB local disk).

---

Phases at a glance

Phase 0  →  Inventory & dry-run report    (READ ONLY — no changes)
Phase 1  →  User review of dry-run        (you decide: go / stop / adjust)
Phase 2  →  Delete duplicates from cold-storage
Phase 3  →  Move unique files into cgitta's primary files
Phase 4  →  Fix ownership, run Nextcloud rescan
Phase 5  →  Verify in the web UI, then remove scratch files

Each phase is described in detail below with exact commands.

---

Phase 0 — Inventory & dry-run report (READ ONLY)

Purpose: Build a complete picture of what's where without touching any file. Produces a report you can review before anything is deleted or moved.

What gets built

1. Two inventory files, one per side. Each line is <sha256> <full path>:
   • /var/tmp/dedup-cgitta/cold.hashes — every file in /mnt/cold-storage/cgitta
   • /var/tmp/dedup-cgitta/primary.hashes — every file in /mnt/nextcloud-data/cgitta/files
2. Size index files (one per side) so the report can show how many bytes each category represents.
3. A dry-run report: /var/tmp/dedup-cgitta/report.md summarizing:
   • Exact duplicates — same hash exists on both sides. These are the deletion candidates in cold-storage.
   • Unique to cold-storage — hashes not present in primary. These are the move candidates.
   • Already-in-primary-only — informational, no action.
   • Internal duplicates within cold-storage — same file appears more than once in cold-storage itself (you probably want to keep just one copy when moving).
   • Destination collisions — file paths that would conflict during the move (e.g. both sides have Backup Files/notes.txt but with different hashes). Each conflict needs a rename rule before moving.

Status: running now

The scan has been kicked off as two background processes on VM103:

/var/tmp/dedup-cgitta/scan.sh  /mnt/cold-storage/cgitta              /var/tmp/dedup-cgitta/cold
/var/tmp/dedup-cgitta/scan.sh  /mnt/nextcloud-data/cgitta/files      /var/tmp/dedup-cgitta/primary

How to check progress yourself

SSH into next and run:

ssh next
ls -la /var/tmp/dedup-cgitta/
# .hashes.partial → still in progress
# .hashes         → finished
wc -l /var/tmp/dedup-cgitta/cold.hashes /var/tmp/dedup-cgitta/primary.hashes  2>/dev/null
tail -n 3 /var/tmp/dedup-cgitta/cold.log /var/tmp/dedup-cgitta/primary.log    2>/dev/null

After scans complete — build the report

cd /var/tmp/dedup-cgitta

# Extract just the hashes from each side (first 64 chars).
awk '{print $1}' cold.hashes    | sort -u > cold.hashset
awk '{print $1}' primary.hashes | sort -u > primary.hashset

# Hashes in both → duplicates (safe-to-delete from cold-storage)
comm -12 cold.hashset primary.hashset > dup.hashset

# Hashes only in cold → unique (must be moved into primary)
comm -23 cold.hashset primary.hashset > cold-only.hashset

# Map hashes back to paths for the report
grep -Ff dup.hashset      cold.hashes > dup.files
grep -Ff cold-only.hashset cold.hashes > unique.files

echo "Duplicate files (to delete from cold-storage): $(wc -l < dup.files)"
echo "Unique files (to move to primary):            $(wc -l < unique.files)"

I'll wrap those exact commands into a build-report.sh once the scans complete and have the data to format the final report.

---

Phase 1 — Review the dry-run

Before anything destructive happens, you get to see the full report and decide:

• Does the duplicate count look sane? (If the report says "we're about to delete all 193 GB as duplicates" we should be very suspicious — investigate why cgitta already has every backup file.)
• Are there collisions? If Backup Files/foo.txt exists in both sides with different contents, we need a rule: do we rename one to foo.cold.txt, or always prefer primary, or always prefer cold?
• Are there files we want to keep in cold-storage anyway (e.g. the synology-backups directory lives next to cgitta/ in cold-storage and isn't touched here, but double-check nothing unexpected is in scope)?

No commands run in this phase. It's a checkpoint.

---

Phase 2 — Delete duplicates from cold-storage

Safety rail: before we delete anything, we take a snapshot on TrueNAS. Cold-storage lives on pool1 on TrueNAS, and ZFS snapshots are near-instant and cost nothing until the data diverges. If we delete something by mistake we can roll back.

# On TrueNAS (192.168.88.24) — API call
curl -sk -u "root:<password>" -X POST \
  https://192.168.88.24/api/v2.0/zfs/snapshot \
  -H "Content-Type: application/json" \
  -d '{"dataset":"pool1/cold-storage","name":"pre-cgitta-dedup-2026-04-17"}'

Then, on VM103:

# Delete each confirmed-duplicate file from cold-storage.
# The awk strips the hash, leaving just the path (which starts with /mnt/cold-storage/cgitta/).
awk '{ $1=""; sub(/^ +/,""); print }' /var/tmp/dedup-cgitta/dup.files \
  | sudo xargs -d '\n' -I{} rm -v "{}" \
  | tee /var/tmp/dedup-cgitta/deleted.log

What the command does, piece by piece:

• awk '{ $1=""; sub(/^ +/,""); print }' — strips the first column (the hash) and leading spaces, leaving the full file path.
• xargs -d '\n' -I{} rm -v "{}" — feeds each path to rm -v (verbose, prints what it deletes). -d '\n' tells xargs to split on newlines only, so filenames with spaces still work.
• tee deleted.log — writes a log of everything that was deleted, so we have an audit trail.

After deletion, prune now-empty directories:

sudo find /mnt/cold-storage/cgitta -depth -type d -empty -delete

---

Phase 3 — Move unique files into cgitta's primary files

The candidates are in unique.files. We move each to the same relative location under /mnt/nextcloud-data/cgitta/files/ — so /mnt/cold-storage/cgitta/Backup Files/foo.txt lands at /mnt/nextcloud-data/cgitta/files/Backup Files/foo.txt.

Collisions (same relative path, different hash) will have been identified in Phase 0 and a rename rule applied — most likely: the cold version gets suffixed with .from-cold-storage.

# Dry-run first — show what would happen
awk '{ $1=""; sub(/^ +/,""); print }' /var/tmp/dedup-cgitta/unique.files \
  | while IFS= read -r src; do
      rel="${src#/mnt/cold-storage/cgitta/}"
      dst="/mnt/nextcloud-data/cgitta/files/${rel}"
      echo "MOVE: $src  ->  $dst"
  done | tee /var/tmp/dedup-cgitta/move-plan.txt

# Review move-plan.txt. Then do the real move:
awk '{ $1=""; sub(/^ +/,""); print }' /var/tmp/dedup-cgitta/unique.files \
  | while IFS= read -r src; do
      rel="${src#/mnt/cold-storage/cgitta/}"
      dst="/mnt/nextcloud-data/cgitta/files/${rel}"
      sudo install -d -o www-data -g www-data "$(dirname "$dst")"
      sudo mv -n -v "$src" "$dst"
  done | tee /var/tmp/dedup-cgitta/moved.log

What each command does:

• install -d -o www-data -g www-data — creates the destination directory if missing, and makes sure it's owned by www-data. This is the safe alternative to mkdir -p + chown in one step.
• mv -n — "no-clobber". If the destination file already exists, mv refuses rather than overwriting. That's our safety net against unplanned collisions.
• -v — verbose.

Why mv instead of cp + delete? mv across filesystems (cold-storage is NFS, primary is local) is implemented as copy-then-delete under the hood, but it's atomic per-file from our perspective and doesn't leave an orphan if the machine loses power mid-operation. Since source and destination live on different physical disks, the data really is being copied over the network and then the source is removed. Plan on this phase taking roughly as long as the initial hash scan.

---

Phase 4 — Fix ownership and run Nextcloud rescan

# Make sure every newly-moved file is owned by www-data
sudo chown -R www-data:www-data /mnt/nextcloud-data/cgitta/files/

# Tell Nextcloud to re-scan cgitta's files so the database picks up the new files
sudo -u www-data php /var/www/nextcloud/occ files:scan cgitta

The files:scan command will output one line per directory it visits and a summary at the end. Expect it to take a few minutes for 200+ GB of new content — it reads file metadata (not contents), computes Nextcloud's own per-file etag, and inserts rows into the oc_filecache table.

---

Phase 5 — Verify, then clean up

1. Log into Nextcloud as cgitta at https://next.jgitta.com. Browse the files. Look for the newly-moved Backup Files, HP Laptop, zip drives folders. Open a few files to make sure they read correctly.

2. Confirm the old cold-storage path is now empty (or near-empty if we deliberately left shared content):

   sudo du -sh /mnt/cold-storage/cgitta
   sudo find /mnt/cold-storage/cgitta -type f | head
   

3. Once you're satisfied, remove the scratch directory:

   sudo rm -rf /var/tmp/dedup-cgitta
   

4. After a grace period (a week?), the ZFS snapshot taken in Phase 2 can be destroyed on TrueNAS:

   curl -sk -u "root:<password>" -X POST \
     https://192.168.88.24/api/v2.0/zfs/snapshot/delete \
     -d '{"id":"pool1/cold-storage@pre-cgitta-dedup-2026-04-17"}'
   

---

Things that could go wrong (and how we mitigate)

Risk	Mitigation
Scan picks up .htaccess / .ncdata / system files and we accidentally move Nextcloud internals	Scope is limited to /mnt/cold-storage/cgitta/* — we never touch /mnt/nextcloud-data/ except to add files under cgitta/files/.
Hash collision (two different files produce the same SHA-256)	SHA-256 collisions have never been found in the wild for unrelated data. Not a real concern.
Move operation runs out of space on primary disk	Primary has 650 GB free, cold-storage/cgitta is 193 GB. Safe margin. We'll check available space as part of Phase 3 before kicking off the move.
Nextcloud database gets confused by a partial scan	occ files:scan --path="cgitta/files/<subpath>" can target a subtree. If anything looks off, we rescan the specific folder.
Permissions wrong after move — Nextcloud shows "access denied" for the moved files	chown -R www-data:www-data in Phase 4 fixes this.
Power loss during move	Files are moved one at a time. Any file not yet moved is still intact in cold-storage. Any file fully moved is in primary. No file is ever in both places or neither. Resume by re-running Phase 0–3 — duplicates will be detected and skipped.
Two files have the same relative path but different content (a real "collision")	Flagged in Phase 0 report before we ever move. Rename rule applied: cold copy gets .from-cold-storage suffix. You approve before Phase 3.

---

What about jgitta?

The cold-storage/jgitta directory has 25 subfolders (phone backups, Google Takeout, OneDrive dumps, etc.) and is likely multi-TB. Strategy differences vs cgitta:

• Physical move into primary almost certainly won't fit on the 2 TB local disk. We'll need to either (a) expand the VM's data disk, (b) use the warm-storage tier for overflow, or (c) per-user external storage mount for the big archival stuff.
• Recommend: finish cgitta end-to-end first, validate the process, then come back for jgitta with specific capacity numbers and a tailored plan.

---

Next step

Wait for the Phase 0 scan to finish (30–60 min from kickoff at 07:17 UTC). I'll produce the actual report with real counts, then bring it back to you for review before anything destructive runs.