sakhiy/litellm-config
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

3 akun per provider — load balanced + fallback

Browse Source
This commit is contained in:
sakhiy
2026-05-27 01:09:21 +00:00
parent 3d857c5b47
commit 6cbb39a72c
3 changed files with 40 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
+9 -3
View File
@@ -1,14 +1,20 @@
# AI PROXY CONFIG # AI PROXY CONFIG
# Copy file ini ke .env dan isi API key masing-masing # Copy file ini ke .env dan isi API key masing-masing
# OpenRouter (Tier 1) # OpenRouter (Tier 1 - 3 akun)
OPENROUTER_API_KEY_1= OPENROUTER_API_KEY_1=
OPENROUTER_API_KEY_2=
OPENROUTER_API_KEY_3=
# Groq (Tier 2) # Groq (Tier 2 - 3 akun)
GROQ_API_KEY_1= GROQ_API_KEY_1=
GROQ_API_KEY_2=
GROQ_API_KEY_3=
# GitHub Models (Tier 3) # GitHub Models (Tier 3 - 3 token)
GITHUB_TOKEN_1= GITHUB_TOKEN_1=
GITHUB_TOKEN_2=
GITHUB_TOKEN_3=
# LiteLLM Admin # LiteLLM Admin
LITELLM_MASTER_KEY= LITELLM_MASTER_KEY=
README.md
+3 -3
View File
@@ -60,9 +60,9 @@ curl http://localhost:4000/v1/chat/completions \
| Tier | Model Name | Provider | Akun | Fungsi | | Tier | Model Name | Provider | Akun | Fungsi |
|------|-----------|----------|------|--------| |------|-----------|----------|------|--------|
| 1 | `primary` | GPT-4o Mini (OpenRouter) | 1 | Main | | 1 | `primary` | GPT-4o Mini (OpenRouter) | 3 | Main — load balanced |
| 2 | `secondary` | Llama 3.3 70B (Groq) | 1 | Fallback pertama | | 2 | `secondary` | Llama 3.3 70B (Groq) | 3 | Fallback — load balanced |
| 3 | `tertiary` | GPT-4o Mini (GitHub) | 1 | Fallback terakhir | | 3 | `tertiary` | GPT-4o Mini (GitHub) | 3 | Fallback terakhir — load balanced |
## Port ## Port
config.yaml
+28 -4
View File
@@ -1,21 +1,45 @@
model_list: model_list:
# --- TIER 1: OPENROUTER (Free) --- # --- TIER 1: OPENROUTER (3 Akun Load Balancing) ---
- model_name: primary - model_name: primary
litellm_params: litellm_params:
model: openrouter/openai/gpt-4o-mini model: openrouter/openai/gpt-4o-mini
api_key: os.environ/OPENROUTER_API_KEY_1 api_key: os.environ/OPENROUTER_API_KEY_1
- model_name: primary
litellm_params:
model: openrouter/openai/gpt-4o-mini
api_key: os.environ/OPENROUTER_API_KEY_2
- model_name: primary
litellm_params:
model: openrouter/openai/gpt-4o-mini
api_key: os.environ/OPENROUTER_API_KEY_3
# --- TIER 2: GROQ (Free) --- # --- TIER 2: GROQ (3 Akun Load Balancing) ---
- model_name: secondary - model_name: secondary
litellm_params: litellm_params:
model: groq/llama-3.3-70b-versatile model: groq/llama-3.3-70b-versatile
api_key: os.environ/GROQ_API_KEY_1 api_key: os.environ/GROQ_API_KEY_1
- model_name: secondary
litellm_params:
model: groq/llama-3.3-70b-versatile
api_key: os.environ/GROQ_API_KEY_2
- model_name: secondary
litellm_params:
model: groq/llama-3.3-70b-versatile
api_key: os.environ/GROQ_API_KEY_3
# --- TIER 3: GITHUB MODELS (via token) --- # --- TIER 3: GITHUB MODELS (3 Akun) ---
- model_name: tertiary - model_name: tertiary
litellm_params: litellm_params:
model: github/gpt-4o-mini model: github/gpt-4o-mini
api_key: os.environ/GITHUB_TOKEN_1 api_key: os.environ/GITHUB_TOKEN_1
- model_name: tertiary
litellm_params:
model: github/gpt-4o-mini
api_key: os.environ/GITHUB_TOKEN_2
- model_name: tertiary
litellm_params:
model: github/gpt-4o-mini
api_key: os.environ/GITHUB_TOKEN_3
router_settings: router_settings:
routing_strategy: simple-shuffle routing_strategy: simple-shuffle
@@ -26,5 +50,5 @@ router_settings:
retry_after: 0 retry_after: 0
num_retries: 2 num_retries: 2
timeout: 60 timeout: 60
allowed_fails: 2 allowed_fails: 3 # per akun — jadi tiap akun boleh fail 3x sebelum cooldown
cooldown_time: 60 cooldown_time: 60