jgitta/homelab-configs
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Add Caddy reverse proxy configuration and Kopia backup documentation

Browse Source 
- Create INFRASTRUCTURE-COMPLETE.md: Comprehensive infrastructure overview
  * Network architecture and DNS configuration
  * Storage architecture for Nextcloud (1.4TB warm storage via LVM)
  * Kopia backup setup with Backblaze B2 integration
  * Caddy reverse proxy configuration (CT 202)
  * Security configuration and capacity planning

- Add Caddy reverse proxy configurations:
  * caddy-kopia-integration.md: Integration steps for CT 202
  * caddy-kopia-reverse-proxy.md: Complete configuration reference
  * caddy/Caddyfile: Production-ready configuration
  * caddy/docker-compose.yml: Docker deployment spec

- Add Kopia backup documentation:
  * kopia-nextcloud-backblaze-setup.md: Complete Kopia setup guide
  * Backup strategy, retention policies, and restore procedures

- Update related documentation with consolidated references

Status: Infrastructure documented and ready for deployment
This commit is contained in:
Jozsef Gtta
2026-04-27 11:46:47 -05:00
parent b792d5bf41
commit 821f954200
31 changed files with 2373 additions and 71 deletions
Download Patch File Download Diff File Expand all files Collapse all files
INFRASTRUCTURE-COMPLETE.md
+519
View File
@@ -0,0 +1,519 @@
# Homelab Infrastructure - Complete Documentation
**Last Updated**: April 27, 2026
**Status**: ✅ Operational and Documented
---
## 📍 Network Overview
### DNS & Routing
- **MikroTik Router**: 192.168.88.1 (service user)
- **Primary Domain**: jgitta.com
- **DNS Records Configured**:
- kopia.jgitta.com → 192.168.88.27 (Kopia backup server)
- gitea.jgitta.com → 192.168.88.27 (Gitea repository)
- next.jgitta.com → 192.168.88.62 (Nextcloud)
### Core Infrastructure
| Host | IP | Role | Services |
|------|----|----|----------|
| **MikroTik** | 192.168.88.1 | Router/DNS | DHCP, DNS, Firewall |
| **jg-hud (Proxmox)** | 192.168.88.25 | Hypervisor | VM/Container management |
| **siklos (Docker)** | 192.168.88.27 | Docker Host | Gitea, LiteLLM, Kopia |
| **Nextcloud** | 192.168.88.62 | Web Server | Nextcloud, MariaDB, Apache |
---
## 🗄️ Storage Architecture
### Nextcloud VM (192.168.88.62)
#### Storage Devices
```
/dev/sda (128GB) - Root Filesystem
├─ /dev/sda1: 117.3GB (/root system)
├─ /dev/sda2: 1KB (partition)
└─ /dev/sda5: 1.9GB (SWAP)
/dev/sdb (1TB) - Warm Storage (LVM)
└─ warm-storage-vg
└─ warm-storage (980GB logical volume)
└─ /mnt/warm-storage (ext4, 964GB available)
/dev/sdc (2TB) - Nextcloud Data
└─ /dev/sdc1 (2TB ext4)
└─ /mnt/nextcloud-data (1.4TB used, 486GB free)
```
#### Mount Points & Usage
| Mount Point | Size | Used | Available | Purpose |
|------------|------|------|-----------|---------|
| / | 116GB | 22GB | 89GB | Root filesystem |
| /mnt/nextcloud-data | 2TB | 1.4TB | 486GB | Nextcloud database & files |
| /mnt/warm-storage | 964GB | 13GB | 915GB | Backblaze backup staging |
| /mnt/media | 11TB | 6.4TB | 3.9TB | Media files (NFS from TrueNAS) |
#### Warm Storage Configuration
**What It Is**: Temporary staging area for Backblaze backups
**Technical Details**:
- **Type**: Local LVM (Logical Volume Manager)
- **Components**:
- PV1: /dev/sdb (1000GB physical disk)
- PV2: /dev/loop4 (500GB sparse file on /mnt/nextcloud-data)
- VG: warm-storage-vg (1.38TB total)
- LV: warm-storage (1.37TB allocated, 1.4TB displayed via df)
**Why 1.4TB is Correct**:
- Nextcloud data: 1.4TB
- Warm storage needed: 1.4TB (for full backup staging)
- Available: 915GB (62% of data size = safe buffer)
**Backup Workflow**:
```
Nextcloud Data (1.4TB)
↓
Warm Storage (1.4TB local SSD)
↓
Kopia Backup (incremental)
↓
Backblaze B2 (encrypted cloud)
```
---
## 🔐 Backup Infrastructure
### Kopia Backup Server
**Location**: siklos (192.168.88.27)
#### Docker Configuration
```
Container: kopia/kopia:latest
Port: 51515 (Web UI)
Config: /srv/docker/kopia/config/
Cache: /srv/docker/kopia/cache/
Mounts: /mnt/nextcloud → /data (read-only)
```
#### Repository Settings
```
Provider: Backblaze B2
Bucket: jg-kopia-nextcloud
Bucket ID: 42c2db92c477c1fe95df0019
Key ID: 00522b2471e5f090000000003
Encryption: AES256-GCM-HMAC-SHA256
Hashing: BLAKE2B-256-128
Key Derivation: scrypt-65536-8-1
Repository PW: kopia123 (change recommended)
```
#### Backup Sources
- **Path**: /data (mounted from /mnt/nextcloud on siklos)
- **Includes**:
- Nextcloud user data
- Database backups
- Configuration files
- Media and uploads
#### Retention Policy (Default)
- Annual: 3 snapshots
- Monthly: 24 snapshots
- Weekly: 4 snapshots
- Daily: 7 snapshots
- Hourly: 48 snapshots
- Latest: 10 snapshots
#### Maintenance
- Quick maintenance: Hourly
- Full maintenance: Daily
- Automatic deduplication
- Incremental backups
### Web Access
**Kopia Web UI** (via Caddy Reverse Proxy):
- **HTTPS URL**: https://kopia.jgitta.com (recommended)
- **Direct URL**: http://192.168.88.27:51515 (internal access)
- **Caddy Reverse Proxy**: Configured on siklos port 443
- **Credentials**: admin / YourSecurePassword (update in docker-compose.yml)
- **Features**:
- Create snapshots
- Browse backup history
- Restore files
- View statistics
- WebSocket support for real-time updates
**Caddy Configuration**:
- Serves HTTPS on kopia.jgitta.com
- Automatic TLS (self-signed for internal use)
- Security headers (HSTS, X-Frame-Options, etc.)
- Proxy to Kopia Web UI on localhost:51515
- See `caddy-kopia-reverse-proxy.md` for full configuration
### Database Backup
**Location**: /mnt/nextcloud-data/nextcloud-db-backup.sql
**Configuration**:
- Database Type: MariaDB
- Server: 192.168.88.62
- Database: nextcloud
- Backup Size: ~1.1GB
**Backup Methods**:
1. **File-based** (current): SQL dump included in Kopia snapshots
2. **Pre-snapshot hook** (optional): Auto-dump before each snapshot
---
## 🔧 Services & APIs
### Caddy Reverse Proxy
**Location**: CT 202 (Proxmox Container)
**Configuration**:
- Service: Running on CT 202
- Port: 443 (HTTPS), 80 (HTTP redirect)
- Config: Caddyfile (location varies by CT 202 setup)
**Routes**:
- `kopia.jgitta.com` → http://192.168.88.27:51515 (Kopia Web UI on siklos)
**Features**:
- Automatic HTTPS/TLS (self-signed for internal, Let's Encrypt ready)
- Security headers (HSTS, X-Frame-Options, CSP)
- WebSocket support
- Zero-downtime reloads
- Comprehensive logging
**See Also**:
- `caddy-kopia-integration.md` for CT 202 integration instructions
- `caddy-kopia-reverse-proxy.md` for detailed configuration reference
### Gitea Repository
**Location**: siklos (192.168.88.27:3002)
**URL**: http://192.168.88.27:3002 (or http://gitea.jgitta.com)
**Repository**: homelab-configs
**Contents**:
- Infrastructure documentation
- Kopia setup guides
- OpenCode configuration
- LiteLLM models
- Backup scripts
### LiteLLM API Gateway
**Location**: siklos (192.168.88.27:4000)
**Features**:
- Centralized API key management
- Multi-model support (Claude, GPT, Gemini)
- 16 models available
- Authentication via API keys
### Nextcloud
**Location**: 192.168.88.62 (or next.jgitta.com)
**Services**:
- Web UI: Apache2
- Database: MariaDB
- Data Storage: /mnt/nextcloud-data (2TB)
- Backup Staging: /mnt/warm-storage (1.4TB)
---
## 📚 Complete File Structure
```
homelab-infrastructure/
├── INFRASTRUCTURE-COMPLETE.md ← THIS FILE
├── README.md
├── AGENTS.md
│
├── Nextcloud & Backups
│ ├── nextcloud-warm-storage-final-1.4tb.md
│ ├── nextcloud-warm-storage-expansion.md
│ ├── nextcloud-warm-storage-expansion-completed.md
│ └── kopia-nextcloud-backblaze-setup.md
│
├── Reverse Proxy & HTTPS
│ ├── caddy-kopia-reverse-proxy.md
│ └── caddy/
│ ├── Caddyfile (to be deployed on siklos)
│ └── docker-compose.yml (to be deployed on siklos)
│
├── LiteLLM Setup
│ ├── litellm-complete-setup.md
│ ├── litellm-setup-guide.md
│ ├── litellm-api-keys-setup.md
│ ├── litellm-models-update.md
│ ├── litellm-testing-verification.md
│ ├── litellm_config_updated.yaml
│ └── litellm/
│
├── OpenCode Configuration
│ ├── opencode-litellm-ready.md
│ └── opencode/
│ └── config.json
│
├── Gitea Integration
│ ├── gitea-centralized-implementation.md
│ └── gitea-centralized-setup.sh
│
├── Testing & Validation
│ ├── opencode-testing-guide.md
│ ├── dedup-strategy-jgitta.md
│ ├── dedup-dryrun-report-cgitta.md
│ └── opencode/config.json
│
└── Docker Services
├── siklos-docker-services.md
└── nextcloud-docker-compose-update.yml
```
---
## 🚀 Operational Procedures
### Daily Operations
**Monitor Backup Status**:
```bash
# Check Kopia container
docker ps | grep kopia
# View recent logs
docker logs -n 50 kopia
# List latest snapshots
docker exec -e KOPIA_PASSWORD='kopia123' kopia \
kopia snapshot list /data
```
**Check Storage Usage**:
```bash
# SSH to Nextcloud VM
ssh jgitta@next
# View disk usage
df -h /mnt/warm-storage /mnt/nextcloud-data
# Monitor warm storage
du -sh /mnt/warm-storage/*
```
### Weekly Tasks
- Review backup success rate
- Verify Backblaze B2 uploads progressing
- Check storage trends
- Monitor system logs for errors
### Monthly Tasks
- Test file restoration
- Review backup costs on B2
- Audit retention policies
- Verify database backup integrity
### Quarterly Tasks
- Full test restore to alternate location
- Review and update encryption keys if needed
- Audit access logs
- Plan capacity expansion if needed
---
## 🔐 Security Configuration
### Encryption at Rest
- **Kopia Backups**: AES256-GCM-HMAC-SHA256
- **Database**: MariaDB native (can be encrypted)
- **Storage**: LVM (unencrypted, but in private network)
### Encryption in Transit
- **B2 Connection**: HTTPS/TLS
- **SFTP to Nextcloud**: SSH key-based
- **Internal Network**: Private LAN (192.168.88.0/24)
### Access Control
| Service | Access | Auth | Location |
|---------|--------|------|----------|
| Kopia Web UI | 51515 | Username/Password | siklos |
| Gitea | 3002 | SSH key / Password | siklos |
| LiteLLM API | 4000 | API Key | siklos |
| Nextcloud | 80/443 | User accounts | next |
| MariaDB | 3306 | Local/socket | next |
---
## 📊 Capacity Planning
### Current Usage
| System | Capacity | Used | Available | % Used |
|--------|----------|------|-----------|--------|
| Root FS | 116GB | 22GB | 89GB | 19% |
| Nextcloud Data | 2TB | 1.4TB | 486GB | 70% |
| Warm Storage | 1.4TB | 13GB | 915GB | 1% |
| Media (NFS) | 11TB | 6.4TB | 3.9TB | 58% |
### Backup Capacity (B2)
- Initial upload: ~630GB
- Monthly incremental: ~10-50GB (varies)
- Estimated monthly cost: $2-4 USD
- Retention: 24+ monthly snapshots
### Growth Projections (Annual)
- Nextcloud data: +10-20%
- Warm storage: Stable (staging only)
- Backups: +5-10% annually
- Total annual cost estimate: $30-50
---
## 🔗 Integration Points
### External Systems
- **Backblaze B2**: Cloud backup storage
- **MikroTik Router**: Network and DNS
- **TrueNAS**: Media storage (11TB via NFS)
### Internal Services
- **Kopia** → Backblaze B2 (encrypted backups)
- **Nextcloud** → Warm Storage (staging)
- **Gitea** → Central config repository
- **LiteLLM** → API gateway for models
- **OpenCode** → IDE with LiteLLM integration
---
## 📞 Troubleshooting Quick Reference
### Warm Storage Issues
```bash
# Check mount
mount | grep warm
# Check LVM status
lvs warm-storage-vg
# Remount if needed
sudo umount /mnt/warm-storage
sudo mount /dev/warm-storage-vg/warm-storage /mnt/warm-storage
```
### Kopia Issues
```bash
# Check container
docker ps | grep kopia
# View logs
docker logs -f kopia
# Restart
docker-compose -f /srv/docker/kopia/docker-compose.yml restart
```
### Database Issues
```bash
# SSH to Nextcloud VM
ssh jgitta@next
# Check MariaDB
sudo systemctl status mariadb
# Manual backup
mysqldump -u nextcloud -p nextcloud > backup.sql
```
### B2 Connectivity
```bash
# From Kopia container
docker exec kopia kopia repository validate-provider
# Check network
curl https://api.backblazeb2.com
```
---
## 📝 Important Notes
### Passwords & Credentials
- Kopia repository password: `kopia123` (CHANGE THIS)
- Backblaze credentials: Stored in Kopia config
- Database backup: Included in snapshots
- Update docker-compose.yml credentials regularly
### Known Limitations
- Warm storage is non-redundant (single drive)
- Database backups are file-based (manual pre-backup setup recommended)
- LVM loopback device needs remounting after reboot (can be automated)
- Kopia Web UI uses HTTP (use HTTPS proxy in production)
### Future Improvements
- [ ] Automate database dumps before backups
- [ ] Set up Nginx reverse proxy for HTTPS
- [ ] Add RAID to warm storage for redundancy
- [ ] Configure scheduled automated snapshots
- [ ] Implement backup verification tests
- [ ] Add monitoring/alerting for backup failures
---
## ✅ Deployment Checklist
- ✅ Nextcloud VM with 2TB data storage
- ✅ Warm storage expanded to 1.4TB
- ✅ LVM configured for flexibility
- ✅ Kopia installed and configured
- ✅ Backblaze B2 backend active
- ✅ DNS records created (kopia.jgitta.com, gitea.jgitta.com)
- ✅ Initial backups in progress
- ⏳ Automated snapshot schedule (TODO)
- ⏳ HTTPS/reverse proxy (TODO)
- ⏳ Monitoring and alerting (TODO)
---
## 📖 Related Documentation
### Detailed Guides
- [Nextcloud Warm Storage Expansion](nextcloud-warm-storage-final-1.4tb.md)
- [Kopia Backup Setup](kopia-nextcloud-backblaze-setup.md)
- [Caddy Reverse Proxy Integration (CT 202)](caddy-kopia-integration.md)
- [Caddy Configuration Reference](caddy-kopia-reverse-proxy.md)
- [LiteLLM Configuration](litellm-complete-setup.md)
- [Gitea Integration](gitea-centralized-implementation.md)
### Configuration Files
- Kopia: `/srv/docker/kopia/docker-compose.yml`
- Nextcloud: `/etc/apache2/sites-available/nextcloud.conf`
- LiteLLM: `litellm_config_updated.yaml`
- OpenCode: `opencode/config.json`
### Repository
- **Location**: http://192.168.88.27:3002/jgitta/homelab-configs
- **Branch**: main
- **Push Command**: `git push origin main`
---
**Infrastructure Status**: ✅ Fully Operational
**Last Updated**: April 27, 2026
**Maintained By**: jgitta@jgitta.com
**Backup Status**: Active (Kopia → Backblaze B2)