homelab-configs/docs/setup/litellm-setup-guide.md

LiteLLM Centralized API Gateway Setup

✅ What's Done

Your LiteLLM API gateway is now running and ready to use:

• LiteLLM Service: Running in Docker on siklos (192.168.88.27:4000)
• Configuration: Stored locally at /opt/litellm/config/
• Git Repository: Ready to push to Gitea (already initialized)

📋 Current Configuration

LiteLLM is configured to proxy these models:

Models Available:
  ✓ gpt-4 (OpenAI)
  ✓ gpt-3.5-turbo (OpenAI)
  ✓ claude-3-sonnet (Anthropic)
  
Config Location: /opt/litellm/config/litellm_config.yaml
API Endpoint: http://192.168.88.27:4000
Master Key: litellm-local-key-change-in-production (⚠️ CHANGE THIS in production)

---

🔑 Next Steps (Required)

Step 1: Create Gitea Repository (30 seconds)

Why? This centralizes your API key management in version control. Updates to one place affect all VMs.

1. Open: http://192.168.88.200:3000 (Your Gitea instance)
2. Login with your account (jgitta)
3. Click "+" icon (top right) → "New Repository"
4. Fill in:
   • Repository name: litellm-config
   • Description: "Centralized LiteLLM API configuration"
   • Leave other settings as defaults
5. Click "Create Repository"

Step 2: Push Config to Gitea

Once the repo is created, run:

ssh jgitta@192.168.88.27 "cd /opt/litellm && bash .git-setup.sh"

This will push your local config to Gitea and set up the git remote.

Step 3: Add Your API Keys

1. Edit the config file:

   ssh jgitta@192.168.88.27 "nano /opt/litellm/config/litellm_config.yaml"
   

2. Replace placeholder API keys:

   • Find: ${OPENAI_API_KEY} → Add your OpenAI key
   • Find: ${ANTHROPIC_API_KEY} → Add your Anthropic key

3. Save and push to Gitea:

   ssh jgitta@192.168.88.27 << 'EOF'
   cd /opt/litellm
   git add config/litellm_config.yaml
   git commit -m "Add API keys"
   git push origin master
   EOF
   

4. LiteLLM will automatically reload (check logs in ~10 seconds)

---

🔌 How VMs Use LiteLLM

Instead of calling OpenAI/Anthropic directly, VMs call LiteLLM:

From Any VM - Example Usage

# Test the gateway is working
curl -X POST http://192.168.88.27:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer litellm-local-key-change-in-production" \
  -d '{
    "model": "gpt-4",
    "messages": [
      {"role": "user", "content": "Hello, what is 2+2?"}
    ]
  }'

In Python (from any VM)

import requests

response = requests.post(
    "http://192.168.88.27:4000/chat/completions",
    headers={
        "Authorization": "Bearer litellm-local-key-change-in-production",
        "Content-Type": "application/json"
    },
    json={
        "model": "gpt-4",
        "messages": [{"role": "user", "content": "Hello!"}]
    }
)
print(response.json())

In Node.js (from any VM)

const response = await fetch('http://192.168.88.27:4000/chat/completions', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer litellm-local-key-change-in-production',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    model: 'gpt-4',
    messages: [{ role: 'user', content: 'Hello!' }]
  })
});
const data = await response.json();
console.log(data);

---

📂 File Locations

For Linux newbies - here's where everything is:

docker-server (siklos @ 192.168.88.27)
└── /opt/litellm/
    ├── docker-compose.yml          # Docker configuration
    ├── .git/                       # Git repository
    ├── .git-setup.sh               # Script to push to Gitea
    └── config/
        ├── litellm_config.yaml     # Model definitions & API keys
        └── README.md               # Config documentation

Gitea (gitea @ 192.168.88.200:3000)
└── jgitta/litellm-config/
    └── (same files as above)

---

🛠️ Common Tasks

Add a New Model

Edit /opt/litellm/config/litellm_config.yaml and add:

- model_name: my-new-model
  litellm_params:
    model: openai/gpt-3.5-turbo
    api_key: ${OPENAI_API_KEY}

Then commit and push to Gitea.

Change the Master Key (Recommended for Production)

1. Edit: /opt/litellm/config/litellm_config.yaml
2. Find: master_key: litellm-local-key-change-in-production
3. Change to a secure key
4. Restart container: docker restart litellm

View Logs

ssh jgitta@192.168.88.27 "docker logs litellm -f"

Stop/Start LiteLLM

ssh jgitta@192.168.88.27 "cd /opt/litellm && docker-compose down"
ssh jgitta@192.168.88.27 "cd /opt/litellm && docker-compose up -d"

---

🔒 Security Notes

• Master Key: Currently set to a placeholder. Change this before using in production.
• API Keys: Store in environment variables (not in config file directly) via .env file or Docker secrets.
• Network: LiteLLM listens on 192.168.88.27:4000 - accessible from all VMs on your network. Restrict with firewall if needed.

---

❓ Troubleshooting

"LiteLLM not responding"

• Check if container is running: docker ps | grep litellm
• View logs: docker logs litellm
• Wait 30-60 seconds after startup

"API Key not working"

• Verify key is in litellm_config.yaml
• Check logs for error messages
• Ensure you're using correct model name

"Config not updating"

• After git push, wait 10-15 seconds for LiteLLM to reload
• Check logs: docker logs litellm | tail -20

---

📌 Summary

You now have: ✅ Centralized API Gateway - Single endpoint for all models
✅ Git-Versioned Config - Changes tracked in Gitea
✅ Easy Updates - Update config in one place, affects all VMs
✅ Simple API - Drop-in replacement for direct OpenAI/Anthropic calls

Next: Create the Gitea repo and push your config!