homelab-configs/litellm-complete-setup.md

# LiteLLM Centralized API Gateway - Complete Setup Guide

## ✅ Infrastructure Status

**All components installed and running:**

- ✅ **LiteLLM Container**: Running on docker-server (siklos, 192.168.88.27:4000)
- ✅ **Docker Compose**: Configured and ready at `/opt/litellm/`
- ✅ **Configuration**: Git initialized with config files in `/opt/litellm/config/`
- ✅ **Gitea**: Ready at http://192.168.88.27:3002 for repository management
- ✅ **All VMs**: Network access confirmed to docker-server

---

## 📁 Current Setup Directory Structure

```
docker-server (siklos @ 192.168.88.27)
└── /opt/litellm/
    ├── docker-compose.yml          # Docker service configuration
    ├── .env                         # Environment variables (API keys)
    ├── .git/                        # Git version control
    ├── .git-setup.sh                # Helper script for Gitea push
    ├── logs/                        # Service logs
    └── config/
        ├── README.md                # Configuration guide
        └── litellm_config.yaml      # Model definitions
```

---

## 🔧 Next Steps: Configure API Keys

### Step 1: Add Your Real API Keys

Edit the `.env` file with your actual API credentials:

```bash
ssh jgitta@192.168.88.27 "nano /opt/litellm/.env"
```

Update these lines with your real keys:
```bash
OPENAI_API_KEY=sk-your-real-openai-key-here
ANTHROPIC_API_KEY=sk-ant-your-real-anthropic-key-here
```

**⚠️ Security Note:** Never commit API keys to git. The `.env` file should never be pushed to Gitea.

### Step 2: Restart LiteLLM with Real Keys

```bash
ssh jgitta@192.168.88.27 << 'EOF'
cd /opt/litellm
docker-compose down
docker-compose up -d
sleep 10
docker logs litellm | tail -20
EOF
```

### Step 3: Test the Gateway

Once running with real keys, test from any VM:

```bash
curl -X POST http://192.168.88.27:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer litellm-local-key-change-in-production" \
  -d '{
    "model": "gpt-4",
    "messages": [
      {"role": "user", "content": "Hello! What is 2+2?"}
    ]
  }'
```

---

## 🎯 How It Works: Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                        Your VMs                                  │
│  (jellyfin, next, photos, haos, etc.)                           │
│                        ↓                                          │
│              Single API Endpoint:                               │
│          http://192.168.88.27:4000                             │
└─────────────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────────────┐
│          LiteLLM API Gateway (Docker Container)                 │
│              Running on docker-server (siklos)                  │
│                                                                  │
│  Routes requests to appropriate backend:                       │
│  - gpt-4 → OpenAI API                                          │
│  - claude-3 → Anthropic API                                    │
│  - local-* → Local models (if configured)                      │
│                                                                  │
│  Advantages:                                                     │
│  ✓ Single authentication point                                  │
│  ✓ Centralized API key management                              │
│  ✓ Easy to add/remove models                                   │
│  ✓ Request logging & monitoring                                │
│  ✓ Load balancing & fallbacks                                  │
└─────────────────────────────────────────────────────────────────┘
                           ↓
                    ┌──────┴──────┐
                    ↓             ↓
              OpenAI API    Anthropic API
```

---

## 📝 Configuration Files

### Docker Compose (`docker-compose.yml`)

The service is configured to:
- **Listen on**: `0.0.0.0:4000` (accessible from all VMs)
- **Load env from**: `.env` file
- **Mount volumes**:
  - `./config/` → `/app/config/` (read-only)
  - `./logs/` → `/app/logs/` (read-write)
- **Auto-restart**: Unless stopped manually

### Environment File (`.env`)

Contains your API keys. Format:
```
OPENAI_API_KEY=sk-xxx...
ANTHROPIC_API_KEY=sk-ant-xxx...
LITELLM_MASTER_KEY=your-master-auth-key
```

### Configuration YAML (`config/litellm_config.yaml`)

Defines which models are available and how to route them. Current models:

```yaml
Models Configured:
  ✓ gpt-4 (OpenAI)
  ✓ gpt-3.5-turbo (OpenAI)
  ✓ claude-3-sonnet (Anthropic)
```

---

## 🔑 API Key Management Strategy

### Local Storage (Current Setup)
- API keys stored in `.env` file on docker-server
- Not committed to git (security best practice)
- Restart required after changes

### Optional: Gitea-Based Config (Future)
Once tested, you can:
1. Create a secure Gitea repository for non-sensitive config
2. Store API keys as Docker secrets or environment file separately
3. Pull config updates automatically

```bash
# Create litellm-config repo in Gitea, then:
cd /opt/litellm
git remote add gitea http://192.168.88.27:3002/jgitta/litellm-config.git
git pull gitea master
```

---

## 🧪 Testing & Monitoring

### Check Service Status

```bash
ssh jgitta@192.168.88.27 << 'EOF'
cd /opt/litellm
docker-compose ps
docker logs litellm -f  # Follow logs in real-time
EOF
```

### View Active Models

```bash
curl http://192.168.88.27:4000/models \
  -H "Authorization: Bearer litellm-local-key-change-in-production"
```

### Monitor Logs Real-Time

```bash
ssh jgitta@192.168.88.27 "docker logs litellm -f"
```

---

## 💻 Usage Examples from Your VMs

### Python Example

```python
import requests
import json

LITELLM_URL = "http://192.168.88.27:4000"
MASTER_KEY = "litellm-local-key-change-in-production"

response = requests.post(
    f"{LITELLM_URL}/chat/completions",
    headers={
        "Authorization": f"Bearer {MASTER_KEY}",
        "Content-Type": "application/json"
    },
    json={
        "model": "gpt-4",
        "messages": [
            {"role": "user", "content": "Explain this to me like I'm 5"}
        ]
    }
)

print(response.json())
```

### Bash Example

```bash
#!/bin/bash

LITELLM_URL="http://192.168.88.27:4000"
MASTER_KEY="litellm-local-key-change-in-production"
MODEL="gpt-4"

curl -s -X POST "$LITELLM_URL/chat/completions" \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "'$MODEL'",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ]
  }' | jq '.choices[0].message.content'
```

### Node.js Example

```javascript
const fetch = require('node-fetch');

const LITELLM_URL = 'http://192.168.88.27:4000';
const MASTER_KEY = 'litellm-local-key-change-in-production';

async function askLiteLLM(question, model = 'gpt-4') {
  const response = await fetch(`${LITELLM_URL}/chat/completions`, {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${MASTER_KEY}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      model: model,
      messages: [
        { role: 'user', content: question }
      ]
    })
  });

  const data = await response.json();
  return data.choices[0].message.content;
}

// Usage:
askLiteLLM('Hello, who are you?').then(response => console.log(response));
```

---

## 🔒 Security Considerations

### Current (Development) Setup
- Master key is a placeholder: `litellm-local-key-change-in-production`
- API keys stored in plain `.env` file
- Service accessible from all VMs on network

### Production Recommendations
1. **Change Master Key**
   ```bash
   # Edit .env and set a strong key
   LITELLM_MASTER_KEY=your-random-secure-32-char-key
   docker-compose restart
   ```

2. **Use Docker Secrets** (for Swarm/Kubernetes)
   ```bash
   echo "your-real-api-key" | docker secret create openai_key -
   ```

3. **Restrict Network Access**
   - Use firewall rules to limit which VMs can access port 4000
   - Only allow specific source IPs if possible

4. **SSL/TLS Encryption**
   - Run LiteLLM behind Caddy (your existing reverse proxy)
   - Use HTTPS instead of HTTP

5. **API Key Rotation**
   - Periodically update API keys in `.env`
   - Restart container after changes

---

## 🚀 Advanced Customization

### Add More Models

Edit `config/litellm_config.yaml`:

```yaml
model_list:
  - model_name: my-custom-model
    litellm_params:
      model: openai/gpt-3.5-turbo
      api_key: ${OPENAI_API_KEY}
```

Then restart:
```bash
cd /opt/litellm
docker-compose restart
```

### Enable Request Logging

Edit `docker-compose.yml` and add:
```yaml
environment:
  LOG_LEVEL: DEBUG
  LITELLM_LOG_REQUESTS: "true"
```

### Set Up Load Balancing

Multiple backend API keys for the same model:
```yaml
model_list:
  - model_name: gpt-4
    litellm_params:
      model: gpt-4-backup  # Primary
  - model_name: gpt-4
    litellm_params:
      model: gpt-4-fallback  # Fallback
```

---

## 📞 Troubleshooting

### Container Won't Start

```bash
# Check logs
docker logs litellm 2>&1 | tail -50

# Most common: Missing API keys
# Solution: Update .env with real keys

# Restart with fresh pull
docker-compose down
docker image pull ghcr.io/berriai/litellm:main
docker-compose up -d
```

### "Connection Refused" from VM

```bash
# Check container is running
docker ps | grep litellm

# Check port is listening
docker exec litellm netstat -tlnp | grep 4000

# Try from docker-server first
docker exec litellm curl http://localhost:4000/health
```

### "Unauthorized" Error

```bash
# Verify master key in request matches .env
# Default: litellm-local-key-change-in-production

# Test with correct key:
curl -H "Authorization: Bearer litellm-local-key-change-in-production" \
  http://192.168.88.27:4000/models
```

### Logs Show "Missing Credentials"

```bash
# Edit .env with real API keys
nano /opt/litellm/.env

# Restart the service
docker-compose down && docker-compose up -d
sleep 10
docker logs litellm | tail -20
```

---

## 📚 References & Documentation

- **LiteLLM Docs**: https://docs.litellm.ai/
- **LiteLLM Proxy Server**: https://docs.litellm.ai/docs/proxy_server
- **OpenAI API**: https://platform.openai.com/docs/
- **Anthropic Claude**: https://docs.anthropic.com/

---

## 🎓 Quick Commands Reference

```bash
# SSH to docker-server
ssh jgitta@192.168.88.27

# Navigate to litellm directory
cd /opt/litellm

# View status
docker-compose ps
docker logs litellm -f

# Restart service
docker-compose restart

# Edit API keys
nano .env
docker-compose down && docker-compose up -d

# View config
cat config/litellm_config.yaml

# Test from VM
curl -X POST http://192.168.88.27:4000/chat/completions \
  -H "Authorization: Bearer litellm-local-key-change-in-production" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}]}'
```

---

## ✨ What's Next

1. **Add Real API Keys** to `.env`
2. **Test the Gateway** from a VM
3. **Monitor Logs** to ensure everything works
4. **Document Your Models** in `config/README.md`
5. **(Optional) Set up Gitea** for config version control
6. **(Optional) Add Caddy** in front for HTTPS/SSL

---

**Your centralized API gateway is ready to go! All VMs now have a single endpoint to access multiple AI models. 🚀**