homelab-configs/litellm/litellm-complete-setup.md

LiteLLM Centralized API Gateway - Complete Setup Guide

✅ Infrastructure Status

All components installed and running:

• ✅ LiteLLM Container: Running on docker-server (siklos, 192.168.88.27:4000)
• ✅ Docker Compose: Configured and ready at /opt/litellm/
• ✅ Configuration: Git initialized with config files in /opt/litellm/config/
• ✅ Gitea: Ready at http://192.168.88.27:3002 for repository management
• ✅ All VMs: Network access confirmed to docker-server

---

📁 Current Setup Directory Structure

docker-server (siklos @ 192.168.88.27)
└── /opt/litellm/
    ├── docker-compose.yml          # Docker service configuration
    ├── .env                         # Environment variables (API keys)
    ├── .git/                        # Git version control
    ├── .git-setup.sh                # Helper script for Gitea push
    ├── logs/                        # Service logs
    └── config/
        ├── README.md                # Configuration guide
        └── litellm_config.yaml      # Model definitions

---

🔧 Next Steps: Configure API Keys

Step 1: Add Your Real API Keys

Edit the .env file with your actual API credentials:

ssh jgitta@192.168.88.27 "nano /opt/litellm/.env"

Update these lines with your real keys:

OPENAI_API_KEY=sk-your-real-openai-key-here
ANTHROPIC_API_KEY=sk-ant-your-real-anthropic-key-here

⚠️ Security Note: Never commit API keys to git. The .env file should never be pushed to Gitea.

Step 2: Restart LiteLLM with Real Keys

ssh jgitta@192.168.88.27 << 'EOF'
cd /opt/litellm
docker-compose down
docker-compose up -d
sleep 10
docker logs litellm | tail -20
EOF

Step 3: Test the Gateway

Once running with real keys, test from any VM:

curl -X POST http://192.168.88.27:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer litellm-local-key-change-in-production" \
  -d '{
    "model": "gpt-4",
    "messages": [
      {"role": "user", "content": "Hello! What is 2+2?"}
    ]
  }'

---

🎯 How It Works: Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Your VMs                                  │
│  (jellyfin, next, photos, haos, etc.)                           │
│                        ↓                                          │
│              Single API Endpoint:                               │
│          http://192.168.88.27:4000                             │
└─────────────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────────────┐
│          LiteLLM API Gateway (Docker Container)                 │
│              Running on docker-server (siklos)                  │
│                                                                  │
│  Routes requests to appropriate backend:                       │
│  - gpt-4 → OpenAI API                                          │
│  - claude-3 → Anthropic API                                    │
│  - local-* → Local models (if configured)                      │
│                                                                  │
│  Advantages:                                                     │
│  ✓ Single authentication point                                  │
│  ✓ Centralized API key management                              │
│  ✓ Easy to add/remove models                                   │
│  ✓ Request logging & monitoring                                │
│  ✓ Load balancing & fallbacks                                  │
└─────────────────────────────────────────────────────────────────┘
                           ↓
                    ┌──────┴──────┐
                    ↓             ↓
              OpenAI API    Anthropic API

---

📝 Configuration Files

Docker Compose (docker-compose.yml)

The service is configured to:

• Listen on: 0.0.0.0:4000 (accessible from all VMs)
• Load env from: .env file
• Mount volumes:
  • ./config/ → /app/config/ (read-only)
  • ./logs/ → /app/logs/ (read-write)
• Auto-restart: Unless stopped manually

Environment File (.env)

Contains your API keys. Format:

OPENAI_API_KEY=sk-xxx...
ANTHROPIC_API_KEY=sk-ant-xxx...
LITELLM_MASTER_KEY=your-master-auth-key

Configuration YAML (config/litellm_config.yaml)

Defines which models are available and how to route them. Current models:

Models Configured:
  ✓ gpt-4 (OpenAI)
  ✓ gpt-3.5-turbo (OpenAI)
  ✓ claude-3-sonnet (Anthropic)

---

🔑 API Key Management Strategy

Local Storage (Current Setup)

• API keys stored in .env file on docker-server
• Not committed to git (security best practice)
• Restart required after changes

Optional: Gitea-Based Config (Future)

Once tested, you can:

1. Create a secure Gitea repository for non-sensitive config
2. Store API keys as Docker secrets or environment file separately
3. Pull config updates automatically

# Create litellm-config repo in Gitea, then:
cd /opt/litellm
git remote add gitea http://192.168.88.27:3002/jgitta/litellm-config.git
git pull gitea master

---

🧪 Testing & Monitoring

Check Service Status

ssh jgitta@192.168.88.27 << 'EOF'
cd /opt/litellm
docker-compose ps
docker logs litellm -f  # Follow logs in real-time
EOF

View Active Models

curl http://192.168.88.27:4000/models \
  -H "Authorization: Bearer litellm-local-key-change-in-production"

Monitor Logs Real-Time

ssh jgitta@192.168.88.27 "docker logs litellm -f"

---

💻 Usage Examples from Your VMs

Python Example

import requests
import json

LITELLM_URL = "http://192.168.88.27:4000"
MASTER_KEY = "litellm-local-key-change-in-production"

response = requests.post(
    f"{LITELLM_URL}/chat/completions",
    headers={
        "Authorization": f"Bearer {MASTER_KEY}",
        "Content-Type": "application/json"
    },
    json={
        "model": "gpt-4",
        "messages": [
            {"role": "user", "content": "Explain this to me like I'm 5"}
        ]
    }
)

print(response.json())

Bash Example

#!/bin/bash

LITELLM_URL="http://192.168.88.27:4000"
MASTER_KEY="litellm-local-key-change-in-production"
MODEL="gpt-4"

curl -s -X POST "$LITELLM_URL/chat/completions" \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "'$MODEL'",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ]
  }' | jq '.choices[0].message.content'

Node.js Example

const fetch = require('node-fetch');

const LITELLM_URL = 'http://192.168.88.27:4000';
const MASTER_KEY = 'litellm-local-key-change-in-production';

async function askLiteLLM(question, model = 'gpt-4') {
  const response = await fetch(`${LITELLM_URL}/chat/completions`, {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${MASTER_KEY}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      model: model,
      messages: [
        { role: 'user', content: question }
      ]
    })
  });

  const data = await response.json();
  return data.choices[0].message.content;
}

// Usage:
askLiteLLM('Hello, who are you?').then(response => console.log(response));

---

🔒 Security Considerations

Current (Development) Setup

• Master key is a placeholder: litellm-local-key-change-in-production
• API keys stored in plain .env file
• Service accessible from all VMs on network

Production Recommendations

1. Change Master Key

   # Edit .env and set a strong key
   LITELLM_MASTER_KEY=your-random-secure-32-char-key
   docker-compose restart
   

2. Use Docker Secrets (for Swarm/Kubernetes)

   echo "your-real-api-key" | docker secret create openai_key -
   

3. Restrict Network Access

   • Use firewall rules to limit which VMs can access port 4000
   • Only allow specific source IPs if possible

4. SSL/TLS Encryption

   • Run LiteLLM behind Caddy (your existing reverse proxy)
   • Use HTTPS instead of HTTP

5. API Key Rotation

   • Periodically update API keys in .env
   • Restart container after changes

---

🚀 Advanced Customization

Add More Models

Edit config/litellm_config.yaml:

model_list:
  - model_name: my-custom-model
    litellm_params:
      model: openai/gpt-3.5-turbo
      api_key: ${OPENAI_API_KEY}

Then restart:

cd /opt/litellm
docker-compose restart

Enable Request Logging

Edit docker-compose.yml and add:

environment:
  LOG_LEVEL: DEBUG
  LITELLM_LOG_REQUESTS: "true"

Set Up Load Balancing

Multiple backend API keys for the same model:

model_list:
  - model_name: gpt-4
    litellm_params:
      model: gpt-4-backup  # Primary
  - model_name: gpt-4
    litellm_params:
      model: gpt-4-fallback  # Fallback

---

📞 Troubleshooting

Container Won't Start

# Check logs
docker logs litellm 2>&1 | tail -50

# Most common: Missing API keys
# Solution: Update .env with real keys

# Restart with fresh pull
docker-compose down
docker image pull ghcr.io/berriai/litellm:main
docker-compose up -d

"Connection Refused" from VM

# Check container is running
docker ps | grep litellm

# Check port is listening
docker exec litellm netstat -tlnp | grep 4000

# Try from docker-server first
docker exec litellm curl http://localhost:4000/health

"Unauthorized" Error

# Verify master key in request matches .env
# Default: litellm-local-key-change-in-production

# Test with correct key:
curl -H "Authorization: Bearer litellm-local-key-change-in-production" \
  http://192.168.88.27:4000/models

Logs Show "Missing Credentials"

# Edit .env with real API keys
nano /opt/litellm/.env

# Restart the service
docker-compose down && docker-compose up -d
sleep 10
docker logs litellm | tail -20

---

📚 References & Documentation

• LiteLLM Docs: https://docs.litellm.ai/
• LiteLLM Proxy Server: https://docs.litellm.ai/docs/proxy_server
• OpenAI API: https://platform.openai.com/docs/
• Anthropic Claude: https://docs.anthropic.com/

---

🎓 Quick Commands Reference

# SSH to docker-server
ssh jgitta@192.168.88.27

# Navigate to litellm directory
cd /opt/litellm

# View status
docker-compose ps
docker logs litellm -f

# Restart service
docker-compose restart

# Edit API keys
nano .env
docker-compose down && docker-compose up -d

# View config
cat config/litellm_config.yaml

# Test from VM
curl -X POST http://192.168.88.27:4000/chat/completions \
  -H "Authorization: Bearer litellm-local-key-change-in-production" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}]}'

---

✨ What's Next

1. Add Real API Keys to .env
2. Test the Gateway from a VM
3. Monitor Logs to ensure everything works
4. Document Your Models in config/README.md
5. (Optional) Set up Gitea for config version control
6. (Optional) Add Caddy in front for HTTPS/SSL

---

Your centralized API gateway is ready to go! All VMs now have a single endpoint to access multiple AI models. 🚀