homelab-configs/dedup-strategy-cgitta.md

# Strategy: Dedup & Merge `cold-storage/cgitta` into Nextcloud cgitta

**Goal:** Take everything currently sitting in `/mnt/cold-storage/cgitta` on VM103 (the Nextcloud VM), compare it file-by-file to what cgitta already has in her Nextcloud primary files, delete anything from cold-storage that's already a perfect duplicate, and physically move the rest into cgitta's Nextcloud account so it shows up in her normal files view.

**Scope of this document:** cgitta only. jgitta will be a separate, larger operation — covered in a follow-up doc once cgitta is complete and the process is validated.

---

## The lay of the land (what we're working with)

| Location | Size | Notes |
|---|---|---|
| `/mnt/cold-storage/cgitta` | **193 GB** | NFS-mounted from TrueNAS. Subfolders: `Backup Files`, `HP Laptop`, `zip drives`. This is the source. |
| `/mnt/nextcloud-data/cgitta/files` | **256 GB** | Local 2 TB SSD on VM103. This is cgitta's primary Nextcloud files. This is the destination. |
| Free space on `/mnt/nextcloud-data` | **~650 GB** | Plenty of room for 193 GB, no capacity adjustment needed. |
| `/cold-storage` (Nextcloud external mount) | shared | Currently visible to all users as one shared cold-storage folder. We'll leave this alone — it still serves its purpose for Entertainment/Family/etc. |

**A few things to know up front:**

- *NFS means "network filesystem"*. The cold-storage is physically on TrueNAS (192.168.88.24) but mounted into VM103 as if it were local. Reads are limited by network speed (~100 MB/s on gigabit).
- *Nextcloud uses a database* to track which files exist. If we just move files into `/mnt/nextcloud-data/cgitta/files/` without telling Nextcloud, cgitta won't see them in the web UI. We have to run `occ files:scan` after the move — it's a command that makes Nextcloud re-read the filesystem and update its database.
- *File ownership matters*. Nextcloud runs as the `www-data` user. Every file in its data directory must be owned by `www-data:www-data` or Nextcloud will refuse to read it.

---

## How we're going to decide two files are "the same"

You chose **SHA-256 content hash**. That means:

1. For every file in cold-storage/cgitta and every file in primary files, we compute a 64-character fingerprint based on the full byte-by-byte contents.
2. If two files have the same fingerprint, they are **guaranteed** to be byte-identical — it doesn't matter what the filename is or where in the tree it sits. A file called `IMG_1234.JPG` and a file called `vacation copy.jpg` with the same hash are, in fact, the same photo.
3. If the fingerprints differ — even by one byte — they're different files and both need to be preserved.

This is slow because every byte of every file has to be read, but it's the only method that gives zero false positives (no chance of accidentally deleting a file we thought was a duplicate but actually wasn't).

**Estimated time for hash scan of cgitta:** 30–60 minutes (193 GB NFS + 256 GB local disk).

---

## Phases at a glance

```
Phase 0  →  Inventory & dry-run report    (READ ONLY — no changes)
Phase 1  →  User review of dry-run        (you decide: go / stop / adjust)
Phase 2  →  Delete duplicates from cold-storage
Phase 3  →  Move unique files into cgitta's primary files
Phase 4  →  Fix ownership, run Nextcloud rescan
Phase 5  →  Verify in the web UI, then remove scratch files
```

Each phase is described in detail below with exact commands.

---

## Phase 0 — Inventory & dry-run report (READ ONLY)

**Purpose:** Build a complete picture of what's where without touching any file. Produces a report you can review before anything is deleted or moved.

### What gets built

1. Two inventory files, one per side. Each line is `<sha256>  <full path>`:
   - `/var/tmp/dedup-cgitta/cold.hashes` — every file in `/mnt/cold-storage/cgitta`
   - `/var/tmp/dedup-cgitta/primary.hashes` — every file in `/mnt/nextcloud-data/cgitta/files`
2. Size index files (one per side) so the report can show how many bytes each category represents.
3. A dry-run report: `/var/tmp/dedup-cgitta/report.md` summarizing:
   - **Exact duplicates** — same hash exists on both sides. These are the deletion candidates in cold-storage.
   - **Unique to cold-storage** — hashes not present in primary. These are the move candidates.
   - **Already-in-primary-only** — informational, no action.
   - **Internal duplicates within cold-storage** — same file appears more than once in cold-storage itself (you probably want to keep just one copy when moving).
   - **Destination collisions** — file paths that would conflict during the move (e.g. both sides have `Backup Files/notes.txt` but with different hashes). Each conflict needs a rename rule before moving.

### Status: running now

The scan has been kicked off as two background processes on VM103:

```
/var/tmp/dedup-cgitta/scan.sh  /mnt/cold-storage/cgitta              /var/tmp/dedup-cgitta/cold
/var/tmp/dedup-cgitta/scan.sh  /mnt/nextcloud-data/cgitta/files      /var/tmp/dedup-cgitta/primary
```

### How to check progress yourself

SSH into next and run:

```bash
ssh next
ls -la /var/tmp/dedup-cgitta/
# .hashes.partial → still in progress
# .hashes         → finished
wc -l /var/tmp/dedup-cgitta/cold.hashes /var/tmp/dedup-cgitta/primary.hashes  2>/dev/null
tail -n 3 /var/tmp/dedup-cgitta/cold.log /var/tmp/dedup-cgitta/primary.log    2>/dev/null
```

### After scans complete — build the report

```bash
cd /var/tmp/dedup-cgitta

# Extract just the hashes from each side (first 64 chars).
awk '{print $1}' cold.hashes    | sort -u > cold.hashset
awk '{print $1}' primary.hashes | sort -u > primary.hashset

# Hashes in both → duplicates (safe-to-delete from cold-storage)
comm -12 cold.hashset primary.hashset > dup.hashset

# Hashes only in cold → unique (must be moved into primary)
comm -23 cold.hashset primary.hashset > cold-only.hashset

# Map hashes back to paths for the report
grep -Ff dup.hashset      cold.hashes > dup.files
grep -Ff cold-only.hashset cold.hashes > unique.files

echo "Duplicate files (to delete from cold-storage): $(wc -l < dup.files)"
echo "Unique files (to move to primary):            $(wc -l < unique.files)"
```

I'll wrap those exact commands into a `build-report.sh` once the scans complete and have the data to format the final report.

---

## Phase 1 — Review the dry-run

Before anything destructive happens, you get to see the full report and decide:

- Does the duplicate count look sane? (If the report says "we're about to delete all 193 GB as duplicates" we should be *very* suspicious — investigate why cgitta already has every backup file.)
- Are there collisions? If `Backup Files/foo.txt` exists in both sides with different contents, we need a rule: do we rename one to `foo.cold.txt`, or always prefer primary, or always prefer cold?
- Are there files we *want* to keep in cold-storage anyway (e.g. the `synology-backups` directory lives next to `cgitta/` in cold-storage and isn't touched here, but double-check nothing unexpected is in scope)?

**No commands run in this phase. It's a checkpoint.**

---

## Phase 2 — Delete duplicates from cold-storage

**Safety rail:** before we delete anything, we take a snapshot on TrueNAS. Cold-storage lives on `pool1` on TrueNAS, and ZFS snapshots are near-instant and cost nothing until the data diverges. If we delete something by mistake we can roll back.

```bash
# On TrueNAS (192.168.88.24) — API call
curl -sk -u "root:<password>" -X POST \
  https://192.168.88.24/api/v2.0/zfs/snapshot \
  -H "Content-Type: application/json" \
  -d '{"dataset":"pool1/cold-storage","name":"pre-cgitta-dedup-2026-04-17"}'
```

Then, on VM103:

```bash
# Delete each confirmed-duplicate file from cold-storage.
# The awk strips the hash, leaving just the path (which starts with /mnt/cold-storage/cgitta/).
awk '{ $1=""; sub(/^ +/,""); print }' /var/tmp/dedup-cgitta/dup.files \
  | sudo xargs -d '\n' -I{} rm -v "{}" \
  | tee /var/tmp/dedup-cgitta/deleted.log
```

**What the command does, piece by piece:**
- `awk '{ $1=""; sub(/^ +/,""); print }'` — strips the first column (the hash) and leading spaces, leaving the full file path.
- `xargs -d '\n' -I{} rm -v "{}"` — feeds each path to `rm -v` (verbose, prints what it deletes). `-d '\n'` tells xargs to split on newlines only, so filenames with spaces still work.
- `tee deleted.log` — writes a log of everything that was deleted, so we have an audit trail.

After deletion, prune now-empty directories:

```bash
sudo find /mnt/cold-storage/cgitta -depth -type d -empty -delete
```

---

## Phase 3 — Move unique files into cgitta's primary files

The candidates are in `unique.files`. We move each to the same relative location under `/mnt/nextcloud-data/cgitta/files/` — so `/mnt/cold-storage/cgitta/Backup Files/foo.txt` lands at `/mnt/nextcloud-data/cgitta/files/Backup Files/foo.txt`.

Collisions (same relative path, different hash) will have been identified in Phase 0 and a rename rule applied — most likely: the cold version gets suffixed with `.from-cold-storage`.

```bash
# Dry-run first — show what would happen
awk '{ $1=""; sub(/^ +/,""); print }' /var/tmp/dedup-cgitta/unique.files \
  | while IFS= read -r src; do
      rel="${src#/mnt/cold-storage/cgitta/}"
      dst="/mnt/nextcloud-data/cgitta/files/${rel}"
      echo "MOVE: $src  ->  $dst"
  done | tee /var/tmp/dedup-cgitta/move-plan.txt

# Review move-plan.txt. Then do the real move:
awk '{ $1=""; sub(/^ +/,""); print }' /var/tmp/dedup-cgitta/unique.files \
  | while IFS= read -r src; do
      rel="${src#/mnt/cold-storage/cgitta/}"
      dst="/mnt/nextcloud-data/cgitta/files/${rel}"
      sudo install -d -o www-data -g www-data "$(dirname "$dst")"
      sudo mv -n -v "$src" "$dst"
  done | tee /var/tmp/dedup-cgitta/moved.log
```

**What each command does:**
- `install -d -o www-data -g www-data` — creates the destination directory if missing, and makes sure it's owned by www-data. This is the safe alternative to `mkdir -p` + `chown` in one step.
- `mv -n` — "no-clobber". If the destination file already exists, `mv` refuses rather than overwriting. That's our safety net against unplanned collisions.
- `-v` — verbose.

**Why `mv` instead of `cp` + delete?** `mv` across filesystems (cold-storage is NFS, primary is local) is implemented as copy-then-delete under the hood, but it's atomic per-file from our perspective and doesn't leave an orphan if the machine loses power mid-operation. Since source and destination live on different physical disks, the data really is being copied over the network and then the source is removed. Plan on this phase taking roughly as long as the initial hash scan.

---

## Phase 4 — Fix ownership and run Nextcloud rescan

```bash
# Make sure every newly-moved file is owned by www-data
sudo chown -R www-data:www-data /mnt/nextcloud-data/cgitta/files/

# Tell Nextcloud to re-scan cgitta's files so the database picks up the new files
sudo -u www-data php /var/www/nextcloud/occ files:scan cgitta
```

The `files:scan` command will output one line per directory it visits and a summary at the end. Expect it to take a few minutes for 200+ GB of new content — it reads file metadata (not contents), computes Nextcloud's own per-file etag, and inserts rows into the `oc_filecache` table.

---

## Phase 5 — Verify, then clean up

1. Log into Nextcloud as cgitta at https://next.jgitta.com. Browse the files. Look for the newly-moved `Backup Files`, `HP Laptop`, `zip drives` folders. Open a few files to make sure they read correctly.
2. Confirm the old cold-storage path is now empty (or near-empty if we deliberately left shared content):

   ```bash
   sudo du -sh /mnt/cold-storage/cgitta
   sudo find /mnt/cold-storage/cgitta -type f | head
   ```

3. Once you're satisfied, remove the scratch directory:

   ```bash
   sudo rm -rf /var/tmp/dedup-cgitta
   ```

4. After a grace period (a week?), the ZFS snapshot taken in Phase 2 can be destroyed on TrueNAS:

   ```bash
   curl -sk -u "root:<password>" -X POST \
     https://192.168.88.24/api/v2.0/zfs/snapshot/delete \
     -d '{"id":"pool1/cold-storage@pre-cgitta-dedup-2026-04-17"}'
   ```

---

## Things that could go wrong (and how we mitigate)

| Risk | Mitigation |
|---|---|
| Scan picks up `.htaccess` / `.ncdata` / system files and we accidentally move Nextcloud internals | Scope is limited to `/mnt/cold-storage/cgitta/*` — we never touch `/mnt/nextcloud-data/` except to add files under `cgitta/files/`. |
| Hash collision (two different files produce the same SHA-256) | SHA-256 collisions have never been found in the wild for unrelated data. Not a real concern. |
| Move operation runs out of space on primary disk | Primary has 650 GB free, cold-storage/cgitta is 193 GB. Safe margin. We'll check available space as part of Phase 3 before kicking off the move. |
| Nextcloud database gets confused by a partial scan | `occ files:scan --path="cgitta/files/<subpath>"` can target a subtree. If anything looks off, we rescan the specific folder. |
| Permissions wrong after move — Nextcloud shows "access denied" for the moved files | `chown -R www-data:www-data` in Phase 4 fixes this. |
| Power loss during move | Files are moved one at a time. Any file not yet moved is still intact in cold-storage. Any file fully moved is in primary. No file is ever in both places or neither. Resume by re-running Phase 0–3 — duplicates will be detected and skipped. |
| Two files have the same relative path but different content (a real "collision") | Flagged in Phase 0 report before we ever move. Rename rule applied: cold copy gets `.from-cold-storage` suffix. You approve before Phase 3. |

---

## What about jgitta?

The cold-storage/jgitta directory has 25 subfolders (phone backups, Google Takeout, OneDrive dumps, etc.) and is likely multi-TB. Strategy differences vs cgitta:

- Physical move into primary almost certainly won't fit on the 2 TB local disk. We'll need to either (a) expand the VM's data disk, (b) use the warm-storage tier for overflow, or (c) per-user external storage mount for the big archival stuff.
- Recommend: finish cgitta end-to-end first, validate the process, then come back for jgitta with specific capacity numbers and a tailored plan.

---

## Next step

Wait for the Phase 0 scan to finish (30–60 min from kickoff at 07:17 UTC). I'll produce the actual report with real counts, then bring it back to you for review before anything destructive runs.