Add complete homelab documentation and configurations
Documentation: - OpenCode + LiteLLM integration guide - LiteLLM complete setup with 16 models - Gitea centralized configuration guide - Testing procedures and verification - API keys setup instructions Configurations: - OpenCode config pointing to LiteLLM - Updated LiteLLM config with all models - Nextcloud docker-compose template Scripts: - Gitea setup automation - OpenCode testing script Infrastructure: - Gitea: 192.168.88.200:3000 - LiteLLM: 192.168.88.27:4000 - Nextcloud: 192.168.88.62
This commit is contained in:
@@ -0,0 +1,234 @@
|
||||
# DRY-RUN REPORT: Dedup & Merge cgitta Cold-Storage → Nextcloud
|
||||
|
||||
**Date:** 2026-04-17
|
||||
**Status:** ✅ Hash scans complete. Analysis ready for review.
|
||||
**Analysis Type:** SHA-256 content-based deduplication (zero false positives)
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**All clear to proceed with Phase 2–3.** No blocking issues detected. Zero path collisions. Primary disk has ample space for the move.
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| **Total files in cold-storage/cgitta** | 66,663 |
|
||||
| **Total size in cold-storage/cgitta** | ~200.61 GB |
|
||||
| **Duplicate files to delete** | 15,299 (23.0%) |
|
||||
| **Unique files to move** | 51,364 (77.0%) |
|
||||
| **Est. GB to delete** | ~46 GB |
|
||||
| **Est. GB to move** | ~155 GB |
|
||||
| **Primary disk free space** | ~650 GB |
|
||||
| **Primary disk after move** | ~495 GB remaining ✅ |
|
||||
| **Path collisions** | **0** ✅ |
|
||||
| **Internal duplicates in cold** | 5,367 hashes (same file exists >1× in cold) |
|
||||
|
||||
---
|
||||
|
||||
## What the numbers mean
|
||||
|
||||
### Duplicates → Safe to Delete from Cold-Storage
|
||||
|
||||
**15,299 files** are exact byte-for-byte matches with files already in cgitta's primary Nextcloud account. These are redundant copies sitting in cold-storage that can be safely removed.
|
||||
|
||||
- These files are already available to cgitta through her primary Nextcloud files
|
||||
- Deleting them frees up cold-storage space
|
||||
- Zero risk of data loss (source file exists in primary)
|
||||
|
||||
### Unique Files → Must Move to Primary
|
||||
|
||||
**51,364 files** exist in cold-storage but have no match in cgitta's primary files. These are new/unique content that belongs in her Nextcloud account.
|
||||
|
||||
- If we don't move them, cgitta loses access to them (they stay stuck on NFS cold-storage, invisible in Nextcloud)
|
||||
- These will be physically moved (not copied) from cold-storage into `/mnt/nextcloud-data/cgitta/files/`
|
||||
- After move, they'll appear in cgitta's Nextcloud web UI under the same folder names (`Backup Files/`, `HP Laptop/`, `zip drives/`)
|
||||
|
||||
### Internal Duplicates Within Cold-Storage
|
||||
|
||||
**5,367 unique hashes** appear more than once in cold-storage itself. This means the same file (by content) is stored in multiple locations or with multiple filenames within cold-storage/cgitta.
|
||||
|
||||
- When we move unique files to primary, we'll move all copies (including the duplicates)
|
||||
- Optional: we could de-duplicate before moving (keep only one copy of each hash in cold), but Phase 0 captures everything as-is
|
||||
- Recommendation: **move as-is**. The duplicates are small relative to the total, and keeping them maintains the original structure cgitta had in cold.
|
||||
|
||||
---
|
||||
|
||||
## File Breakdown by Category
|
||||
|
||||
```
|
||||
cold-storage/cgitta (66,663 files):
|
||||
├─ 15,299 files → already in primary (delete from cold)
|
||||
├─ 51,364 files → unique to cold (move to primary)
|
||||
└─ Of the 51,364 unique:
|
||||
└─ 5,367 hashes appear multiple times (internal dupes)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Safety Checks Passed ✅
|
||||
|
||||
| Check | Status | Notes |
|
||||
|---|---|---|
|
||||
| **Path collisions** | ✅ NONE | No filename/path conflicts. Safe to move without renaming. |
|
||||
| **Primary disk space** | ✅ SAFE | 650 GB free > 155 GB needed for unique files. |
|
||||
| **File ownership** | ✅ WILL HANDLE | Phase 4 sets all files to `www-data:www-data`. |
|
||||
| **Nextcloud DB sync** | ✅ WILL HANDLE | Phase 4 runs `occ files:scan` to update Nextcloud's database. |
|
||||
| **ZFS snapshot backup** | ✅ WILL CREATE | Phase 2 creates `pool1/cold-storage@pre-cgitta-dedup-2026-04-17` before any deletions. |
|
||||
| **Relative paths preserved** | ✅ YES | `/mnt/cold-storage/cgitta/Backup Files/foo.txt` → `/mnt/nextcloud-data/cgitta/files/Backup Files/foo.txt` |
|
||||
|
||||
---
|
||||
|
||||
## Sample Files
|
||||
|
||||
### Files being deleted (duplicates)
|
||||
```
|
||||
Bing.url
|
||||
VSO0807159M3FB5598OK.pdf
|
||||
PAGO A VIAJES SORIANO 14,946 050919 SNB IOWA.pdf
|
||||
… (15,299 total)
|
||||
```
|
||||
|
||||
All of these already exist in cgitta's primary Nextcloud files.
|
||||
|
||||
### Files being moved (unique)
|
||||
```
|
||||
zip drives/Polar_Pals_roll_&_keep_it.zip
|
||||
zip drives/PictureJournalPromptsBUNDLELeveledWritingGreatforSpecialEd.zip
|
||||
zip drives/New_Year_word_wall_cards_(3731009).zip
|
||||
… (51,364 total)
|
||||
```
|
||||
|
||||
All of these will be moved into cgitta's primary files.
|
||||
|
||||
---
|
||||
|
||||
## What Happens Next (Phases 2–5)
|
||||
|
||||
### Phase 2: Delete duplicates from cold-storage
|
||||
- Takes ZFS snapshot on TrueNAS (instant, zero-cost backup)
|
||||
- Deletes 15,299 duplicate files from cold-storage
|
||||
- **Estimated time:** 10–20 minutes (local NFS delete operations are fast)
|
||||
- **Reversible:** snapshot exists on TrueNAS if anything goes wrong
|
||||
|
||||
### Phase 3: Move unique files to primary
|
||||
- Moves 51,364 files from cold-storage into `/mnt/nextcloud-data/cgitta/files/`
|
||||
- Preserves folder structure and relative paths
|
||||
- **Estimated time:** 2–4 hours (network copy from NFS to local disk, ~100 MB/s bottleneck)
|
||||
- **Safety:** files moved one at a time; any partial failure leaves cold copies intact for retry
|
||||
|
||||
### Phase 4: Ownership & Nextcloud rescan
|
||||
- Fixes file ownership to `www-data:www-data` (Nextcloud requirement)
|
||||
- Runs `occ files:scan` to register files in Nextcloud database
|
||||
- **Estimated time:** 5–10 minutes
|
||||
- Files become visible in cgitta's Nextcloud web UI
|
||||
|
||||
### Phase 5: Verify & cleanup
|
||||
- Log into Nextcloud as cgitta, verify files appear in web UI
|
||||
- Confirm cold-storage/cgitta is now empty
|
||||
- Remove scratch directory (`/var/tmp/dedup-cgitta`)
|
||||
- Optionally destroy ZFS snapshot after a grace period (1 week)
|
||||
|
||||
---
|
||||
|
||||
## Disk Impact Summary
|
||||
|
||||
### On `/mnt/cold-storage` (NFS)
|
||||
**Before:**
|
||||
```
|
||||
cold-storage/cgitta: 200.61 GB
|
||||
```
|
||||
|
||||
**After Phase 2 (delete duplicates):**
|
||||
```
|
||||
cold-storage/cgitta: ~154 GB (46 GB freed)
|
||||
```
|
||||
|
||||
**After Phase 3 (move unique):**
|
||||
```
|
||||
cold-storage/cgitta: ~0 GB (empty)
|
||||
```
|
||||
|
||||
### On `/mnt/nextcloud-data/cgitta/files` (local SSD)
|
||||
**Before:**
|
||||
```
|
||||
Used: ~1.35 TB
|
||||
Free: ~650 GB
|
||||
```
|
||||
|
||||
**After Phase 3 (move unique):**
|
||||
```
|
||||
Used: ~1.5 TB (original + 155 GB of unique files)
|
||||
Free: ~495 GB ✅ (still comfortable margin)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Recommendations
|
||||
|
||||
1. **Proceed with Phases 2–5.** All safety checks pass. No blocking issues.
|
||||
|
||||
2. **Optional:** If you want to de-duplicate internal copies before moving, let me know. We can identify which of the 5,367 internal-dup hashes should be kept (e.g., keep only the first occurrence of each unique hash). This would reduce the move size slightly but complicate the process.
|
||||
|
||||
3. **Time estimate:** Plan for **3–5 hours total runtime** (mostly Phase 3, which is network-bound).
|
||||
|
||||
4. **Schedule:** Phase 2–3 can run during off-peak hours. Nextcloud will be usable during the move (files being moved are in cold-storage, not primary), though cgitta might notice new files appearing mid-move as Phase 3 progresses.
|
||||
|
||||
---
|
||||
|
||||
## Questions Before Proceeding?
|
||||
|
||||
Before I kick off Phase 2, let me know:
|
||||
- ✅ Proceed with deletion and move as described?
|
||||
- ❓ Any concern about the file counts or size estimates?
|
||||
- ❓ Want to de-duplicate internal copies first, or keep the originals as-is?
|
||||
- ❓ Preferred time to start? (Can run anytime; no downtime needed.)
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Command Reference
|
||||
|
||||
All Phase 2–5 commands are in the main strategy doc (`dedup-strategy-cgitta.md`). Below is a quick reference:
|
||||
|
||||
**Phase 2 (Delete):**
|
||||
```bash
|
||||
# TrueNAS snapshot
|
||||
curl -sk -u "root:<password>" -X POST https://192.168.88.24/api/v2.0/zfs/snapshot …
|
||||
|
||||
# VM103 delete
|
||||
awk '{ $1=""; sub(/^ +/,""); print }' /var/tmp/dedup-cgitta/dup.files | \
|
||||
sudo xargs -d '\n' -I{} rm -v "{}"
|
||||
|
||||
# Prune empty dirs
|
||||
sudo find /mnt/cold-storage/cgitta -depth -type d -empty -delete
|
||||
```
|
||||
|
||||
**Phase 3 (Move):**
|
||||
```bash
|
||||
# Dry-run first (review /var/tmp/dedup-cgitta/move-plan.txt)
|
||||
awk '{ $1=""; sub(/^ +/,""); print }' /var/tmp/dedup-cgitta/unique.files | \
|
||||
while IFS= read -r src; do
|
||||
rel="${src#/mnt/cold-storage/cgitta/}"
|
||||
dst="/mnt/nextcloud-data/cgitta/files/${rel}"
|
||||
echo "MOVE: $src -> $dst"
|
||||
done | tee /var/tmp/dedup-cgitta/move-plan.txt
|
||||
|
||||
# Real move
|
||||
awk '{ $1=""; sub(/^ +/,""); print }' /var/tmp/dedup-cgitta/unique.files | \
|
||||
while IFS= read -r src; do
|
||||
rel="${src#/mnt/cold-storage/cgitta/}"
|
||||
dst="/mnt/nextcloud-data/cgitta/files/${rel}"
|
||||
sudo install -d -o www-data -g www-data "$(dirname "$dst")"
|
||||
sudo mv -n -v "$src" "$dst"
|
||||
done | tee /var/tmp/dedup-cgitta/moved.log
|
||||
```
|
||||
|
||||
**Phase 4 (Ownership & Scan):**
|
||||
```bash
|
||||
sudo chown -R www-data:www-data /mnt/nextcloud-data/cgitta/files/
|
||||
sudo -u www-data php /var/www/nextcloud/occ files:scan cgitta
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Report generated:** 2026-04-17 17:18 UTC
|
||||
**Next action:** Review and approve Phases 2–5.
|
||||
Reference in New Issue
Block a user