# Homelab Infrastructure - Complete Documentation **Last Updated**: April 27, 2026 **Status**: ✅ Operational and Documented --- ## 📍 Network Overview ### DNS & Routing - **MikroTik Router**: 192.168.88.1 (service user) - **Primary Domain**: jgitta.com - **DNS Records Configured**: - kopia.jgitta.com → 192.168.88.27 (Kopia backup server) - gitea.jgitta.com → 192.168.88.27 (Gitea repository) - next.jgitta.com → 192.168.88.62 (Nextcloud) ### Core Infrastructure | Host | IP | Role | Services | |------|----|----|----------| | **MikroTik** | 192.168.88.1 | Router/DNS | DHCP, DNS, Firewall | | **jg-hud (Proxmox)** | 192.168.88.25 | Hypervisor | VM/Container management | | **siklos (Docker)** | 192.168.88.27 | Docker Host | Gitea, LiteLLM, Kopia | | **Nextcloud** | 192.168.88.62 | Web Server | Nextcloud, MariaDB, Apache | --- ## 🗄️ Storage Architecture ### Nextcloud VM (192.168.88.62) #### Storage Devices ``` /dev/sda (128GB) - Root Filesystem ├─ /dev/sda1: 117.3GB (/root system) ├─ /dev/sda2: 1KB (partition) └─ /dev/sda5: 1.9GB (SWAP) /dev/sdb (1TB) - Warm Storage (LVM) └─ warm-storage-vg └─ warm-storage (980GB logical volume) └─ /mnt/warm-storage (ext4, 964GB available) /dev/sdc (2TB) - Nextcloud Data └─ /dev/sdc1 (2TB ext4) └─ /mnt/nextcloud-data (1.4TB used, 486GB free) ``` #### Mount Points & Usage | Mount Point | Size | Used | Available | Purpose | |------------|------|------|-----------|---------| | / | 116GB | 22GB | 89GB | Root filesystem | | /mnt/nextcloud-data | 2TB | 1.4TB | 486GB | Nextcloud database & files | | /mnt/warm-storage | 964GB | 13GB | 915GB | Backblaze backup staging | | /mnt/media | 11TB | 6.4TB | 3.9TB | Media files (NFS from TrueNAS) | #### Warm Storage Configuration **What It Is**: Temporary staging area for Backblaze backups **Technical Details**: - **Type**: Local LVM (Logical Volume Manager) - **Components**: - PV1: /dev/sdb (1000GB physical disk) - PV2: /dev/loop4 (500GB sparse file on /mnt/nextcloud-data) - VG: warm-storage-vg (1.38TB total) - LV: warm-storage (1.37TB allocated, 1.4TB displayed via df) **Why 1.4TB is Correct**: - Nextcloud data: 1.4TB - Warm storage needed: 1.4TB (for full backup staging) - Available: 915GB (62% of data size = safe buffer) **Backup Workflow**: ``` Nextcloud Data (1.4TB) ↓ Warm Storage (1.4TB local SSD) ↓ Kopia Backup (incremental) ↓ Backblaze B2 (encrypted cloud) ``` --- ## 🔐 Backup Infrastructure ### Kopia Backup Server **Location**: siklos (192.168.88.27) #### Docker Configuration ``` Container: kopia/kopia:latest Port: 51515 (Web UI) Config: /srv/docker/kopia/config/ Cache: /srv/docker/kopia/cache/ Mounts: /mnt/nextcloud → /data (read-only) ``` #### Repository Settings ``` Provider: Backblaze B2 Bucket: jg-kopia-nextcloud Bucket ID: 42c2db92c477c1fe95df0019 Key ID: 00522b2471e5f090000000003 Encryption: AES256-GCM-HMAC-SHA256 Hashing: BLAKE2B-256-128 Key Derivation: scrypt-65536-8-1 Repository PW: kopia123 (change recommended) ``` #### Backup Sources - **Path**: /data (mounted from /mnt/nextcloud on siklos) - **Includes**: - Nextcloud user data - Database backups - Configuration files - Media and uploads #### Retention Policy (Default) - Annual: 3 snapshots - Monthly: 24 snapshots - Weekly: 4 snapshots - Daily: 7 snapshots - Hourly: 48 snapshots - Latest: 10 snapshots #### Maintenance - Quick maintenance: Hourly - Full maintenance: Daily - Automatic deduplication - Incremental backups ### Web Access **Kopia Web UI** (via Caddy Reverse Proxy): - **HTTPS URL**: https://kopia.jgitta.com (recommended) - **Direct URL**: http://192.168.88.27:51515 (internal access) - **Caddy Reverse Proxy**: Configured on siklos port 443 - **Credentials**: admin / YourSecurePassword (update in docker-compose.yml) - **Features**: - Create snapshots - Browse backup history - Restore files - View statistics - WebSocket support for real-time updates **Caddy Configuration**: - Serves HTTPS on kopia.jgitta.com - Automatic TLS (self-signed for internal use) - Security headers (HSTS, X-Frame-Options, etc.) - Proxy to Kopia Web UI on localhost:51515 - See `caddy-kopia-reverse-proxy.md` for full configuration ### Database Backup **Location**: /mnt/nextcloud-data/nextcloud-db-backup.sql **Configuration**: - Database Type: MariaDB - Server: 192.168.88.62 - Database: nextcloud - Backup Size: ~1.1GB **Backup Methods**: 1. **File-based** (current): SQL dump included in Kopia snapshots 2. **Pre-snapshot hook** (optional): Auto-dump before each snapshot --- ## 🔧 Services & APIs ### Caddy Reverse Proxy **Location**: CT 202 (Proxmox Container) **Configuration**: - Service: Running on CT 202 - Port: 443 (HTTPS), 80 (HTTP redirect) - Config: Caddyfile (location varies by CT 202 setup) **Routes**: - `kopia.jgitta.com` → http://192.168.88.27:51515 (Kopia Web UI on siklos) **Features**: - Automatic HTTPS/TLS (self-signed for internal, Let's Encrypt ready) - Security headers (HSTS, X-Frame-Options, CSP) - WebSocket support - Zero-downtime reloads - Comprehensive logging **See Also**: - `caddy-kopia-integration.md` for CT 202 integration instructions - `caddy-kopia-reverse-proxy.md` for detailed configuration reference ### Gitea Repository **Location**: siklos (192.168.88.27:3002) **URL**: http://192.168.88.27:3002 (or http://gitea.jgitta.com) **Repository**: homelab-configs **Contents**: - Infrastructure documentation - Kopia setup guides - OpenCode configuration - LiteLLM models - Backup scripts ### LiteLLM API Gateway **Location**: siklos (192.168.88.27:4000) **Features**: - Centralized API key management - Multi-model support (Claude, GPT, Gemini) - 16 models available - Authentication via API keys ### Nextcloud **Location**: 192.168.88.62 (or next.jgitta.com) **Services**: - Web UI: Apache2 - Database: MariaDB - Data Storage: /mnt/nextcloud-data (2TB) - Backup Staging: /mnt/warm-storage (1.4TB) --- ## 📚 Complete File Structure ``` homelab-infrastructure/ ├── INFRASTRUCTURE-COMPLETE.md ← THIS FILE ├── README.md ├── AGENTS.md │ ├── Nextcloud & Backups │ ├── nextcloud-warm-storage-final-1.4tb.md │ ├── nextcloud-warm-storage-expansion.md │ ├── nextcloud-warm-storage-expansion-completed.md │ └── kopia-nextcloud-backblaze-setup.md │ ├── Reverse Proxy & HTTPS │ ├── caddy-kopia-reverse-proxy.md │ └── caddy/ │ ├── Caddyfile (to be deployed on siklos) │ └── docker-compose.yml (to be deployed on siklos) │ ├── LiteLLM Setup │ ├── litellm-complete-setup.md │ ├── litellm-setup-guide.md │ ├── litellm-api-keys-setup.md │ ├── litellm-models-update.md │ ├── litellm-testing-verification.md │ ├── litellm_config_updated.yaml │ └── litellm/ │ ├── OpenCode Configuration │ ├── opencode-litellm-ready.md │ └── opencode/ │ └── config.json │ ├── Gitea Integration │ ├── gitea-centralized-implementation.md │ └── gitea-centralized-setup.sh │ ├── Testing & Validation │ ├── opencode-testing-guide.md │ ├── dedup-strategy-jgitta.md │ ├── dedup-dryrun-report-cgitta.md │ └── opencode/config.json │ └── Docker Services ├── siklos-docker-services.md └── nextcloud-docker-compose-update.yml ``` --- ## 🚀 Operational Procedures ### Daily Operations **Monitor Backup Status**: ```bash # Check Kopia container docker ps | grep kopia # View recent logs docker logs -n 50 kopia # List latest snapshots docker exec -e KOPIA_PASSWORD='kopia123' kopia \ kopia snapshot list /data ``` **Check Storage Usage**: ```bash # SSH to Nextcloud VM ssh jgitta@next # View disk usage df -h /mnt/warm-storage /mnt/nextcloud-data # Monitor warm storage du -sh /mnt/warm-storage/* ``` ### Weekly Tasks - Review backup success rate - Verify Backblaze B2 uploads progressing - Check storage trends - Monitor system logs for errors ### Monthly Tasks - Test file restoration - Review backup costs on B2 - Audit retention policies - Verify database backup integrity ### Quarterly Tasks - Full test restore to alternate location - Review and update encryption keys if needed - Audit access logs - Plan capacity expansion if needed --- ## 🔐 Security Configuration ### Encryption at Rest - **Kopia Backups**: AES256-GCM-HMAC-SHA256 - **Database**: MariaDB native (can be encrypted) - **Storage**: LVM (unencrypted, but in private network) ### Encryption in Transit - **B2 Connection**: HTTPS/TLS - **SFTP to Nextcloud**: SSH key-based - **Internal Network**: Private LAN (192.168.88.0/24) ### Access Control | Service | Access | Auth | Location | |---------|--------|------|----------| | Kopia Web UI | 51515 | Username/Password | siklos | | Gitea | 3002 | SSH key / Password | siklos | | LiteLLM API | 4000 | API Key | siklos | | Nextcloud | 80/443 | User accounts | next | | MariaDB | 3306 | Local/socket | next | --- ## 📊 Capacity Planning ### Current Usage | System | Capacity | Used | Available | % Used | |--------|----------|------|-----------|--------| | Root FS | 116GB | 22GB | 89GB | 19% | | Nextcloud Data | 2TB | 1.4TB | 486GB | 70% | | Warm Storage | 1.4TB | 13GB | 915GB | 1% | | Media (NFS) | 11TB | 6.4TB | 3.9TB | 58% | ### Backup Capacity (B2) - Initial upload: ~630GB - Monthly incremental: ~10-50GB (varies) - Estimated monthly cost: $2-4 USD - Retention: 24+ monthly snapshots ### Growth Projections (Annual) - Nextcloud data: +10-20% - Warm storage: Stable (staging only) - Backups: +5-10% annually - Total annual cost estimate: $30-50 --- ## 🔗 Integration Points ### External Systems - **Backblaze B2**: Cloud backup storage - **MikroTik Router**: Network and DNS - **TrueNAS**: Media storage (11TB via NFS) ### Internal Services - **Kopia** → Backblaze B2 (encrypted backups) - **Nextcloud** → Warm Storage (staging) - **Gitea** → Central config repository - **LiteLLM** → API gateway for models - **OpenCode** → IDE with LiteLLM integration --- ## 📞 Troubleshooting Quick Reference ### Warm Storage Issues ```bash # Check mount mount | grep warm # Check LVM status lvs warm-storage-vg # Remount if needed sudo umount /mnt/warm-storage sudo mount /dev/warm-storage-vg/warm-storage /mnt/warm-storage ``` ### Kopia Issues ```bash # Check container docker ps | grep kopia # View logs docker logs -f kopia # Restart docker-compose -f /srv/docker/kopia/docker-compose.yml restart ``` ### Database Issues ```bash # SSH to Nextcloud VM ssh jgitta@next # Check MariaDB sudo systemctl status mariadb # Manual backup mysqldump -u nextcloud -p nextcloud > backup.sql ``` ### B2 Connectivity ```bash # From Kopia container docker exec kopia kopia repository validate-provider # Check network curl https://api.backblazeb2.com ``` --- ## 📝 Important Notes ### Passwords & Credentials - Kopia repository password: `kopia123` (CHANGE THIS) - Backblaze credentials: Stored in Kopia config - Database backup: Included in snapshots - Update docker-compose.yml credentials regularly ### Known Limitations - Warm storage is non-redundant (single drive) - Database backups are file-based (manual pre-backup setup recommended) - LVM loopback device needs remounting after reboot (can be automated) - Kopia Web UI uses HTTP (use HTTPS proxy in production) ### Future Improvements - [ ] Automate database dumps before backups - [ ] Set up Nginx reverse proxy for HTTPS - [ ] Add RAID to warm storage for redundancy - [ ] Configure scheduled automated snapshots - [ ] Implement backup verification tests - [ ] Add monitoring/alerting for backup failures --- ## ✅ Deployment Checklist - ✅ Nextcloud VM with 2TB data storage - ✅ Warm storage expanded to 1.4TB - ✅ LVM configured for flexibility - ✅ Kopia installed and configured - ✅ Backblaze B2 backend active - ✅ DNS records created (kopia.jgitta.com, gitea.jgitta.com) - ✅ Initial backups in progress - ⏳ Automated snapshot schedule (TODO) - ⏳ HTTPS/reverse proxy (TODO) - ⏳ Monitoring and alerting (TODO) --- ## 📖 Related Documentation ### Detailed Guides - [Nextcloud Warm Storage Expansion](nextcloud-warm-storage-final-1.4tb.md) - [Kopia Backup Setup](kopia-nextcloud-backblaze-setup.md) - [Caddy Reverse Proxy Integration (CT 202)](caddy-kopia-integration.md) - [Caddy Configuration Reference](caddy-kopia-reverse-proxy.md) - [LiteLLM Configuration](litellm-complete-setup.md) - [Gitea Integration](gitea-centralized-implementation.md) ### Configuration Files - Kopia: `/srv/docker/kopia/docker-compose.yml` - Nextcloud: `/etc/apache2/sites-available/nextcloud.conf` - LiteLLM: `litellm_config_updated.yaml` - OpenCode: `opencode/config.json` ### Repository - **Location**: http://192.168.88.27:3002/jgitta/homelab-configs - **Branch**: main - **Push Command**: `git push origin main` --- **Infrastructure Status**: ✅ Fully Operational **Last Updated**: April 27, 2026 **Maintained By**: jgitta@jgitta.com **Backup Status**: Active (Kopia → Backblaze B2)