LiteLLM

Różne notatki. Prace w trakcie.

requirements.txt

aiohappyeyeballs==2.7.1
aiohttp==3.14.3
aiosignal==1.4.0
annotated-doc==0.0.5 
annotated-types==0.8.0
anyio==4.15.1  
APScheduler==3.11.3
attrs==26.1.0 
azure-core==1.41.0       
azure-identity==1.25.3
azure-storage-blob==12.30.1
backoff==2.2.1  
boto3==1.43.89
botocore==1.43.89
certifi==2026.7.22       
cffi==2.1.1          
charset-normalizer==3.5.1
click==8.5.0 
croniter==6.2.4
cryptography==50.0.1
distro==1.9.0               
dnspython==2.8.0    
email-validator==2.3.0  
Expression==5.7.0
fastapi==0.141.1
fastapi-sso==0.22.0
fastuuid==0.14.0
filelock==3.32.5
frozenlist==1.8.0    
fsspec==2026.7.0
granian==2.8.2   
gunicorn==23.0.0
h11==0.16.0       
hf-xet==1.6.0
hiredis==3.4.1
httpcore==1.0.9  
httpx==0.28.1        
httpx-sse==0.4.3
huggingface_hub==1.30.0
idna==3.19        
importlib_metadata==8.9.0
inquirerpy==0.3.4
isodate==0.7.2          
Jinja2==3.1.6            
jiter==0.16.0 
jmespath==1.1.0
jsonschema==4.26.0
jsonschema-specifications==2025.9.1
litellm==1.99.0
litellm-enterprise==0.1.59
litellm-proxy-extras==0.4.89
markdown-it-py==4.2.0
MarkupSafe==3.0.3                                                                          
mcp==1.29.1
mdurl==0.1.2
msal==1.38.0
msal-extensions==1.3.1
multidict==6.7.1
nodeenv==1.10.0
numpy==2.5.2
oauthlib==3.3.1
openai==2.54.0
orjson==3.12.0
packaging==26.3
pfzy==0.3.4
polars==1.44.1
polars-runtime-32==1.44.1
prisma==0.13.0
prompt_toolkit==3.0.53
propcache==0.5.2
pycparser==3.0
pydantic==2.13.5
pydantic-settings==2.15.0
pydantic_core==2.46.5
Pygments==2.21.0
PyJWT==2.13.0
PyNaCl==1.6.2
pyroscope-io==0.8.16
python-dateutil==2.9.0.post0
python-dotenv==1.2.3
python-multipart==0.0.32
PyYAML==6.0.3
redis==8.1.0
referencing==0.37.0
regex==2026.9.3
requests==2.34.2
RestrictedPython==8.5
rich==13.9.4
rpds-py==2026.6.3
rq==2.12.0
s3transfer==0.19.2
six==1.17.0
sniffio==1.3.1
soundfile==0.14.0
sse-starlette==3.4.11
starlette==1.6.0
tiktoken==0.14.0
tokenizers==0.23.2
tomlkit==0.15.1
tqdm==4.70.0
typing-inspection==0.4.4
typing_extensions==4.16.0
tzlocal==5.4.4
urllib3==2.7.0
uvicorn==0.52.4
uvloop==0.22.1
wcwidth==0.8.3
websockets==15.0.1
yarl==1.24.5
zipp==4.1.0

Przykładowy plik litellm_config.yaml

model_list:
  - model_name: granite
    litellm_params:
      model: ollama/granite4.2:8b
      api_base: http://si.local:11434
      allow_client_keepalive_override: true
      temperature: 0.6
       
  - model_name: gemma
    litellm_params:
      model: ollama/gemma4:12b
      api_base: http://si.local:11434
      allow_client_keepalive_override: true
      temperature: 0.8
       
  - model_name: qwen
    litellm_params:
      model: ollama/qwen3.5:9b
      api_base: http://si.local:11434
      allow_client_keepalive_override: true
      temperature: 0.8
       
  - model_name: bielik
    litellm_params:
      model: ollama/bielik-11b:Q5_K_M
      api_base: http://si.local:11434
      allow_client_keepalive_override: true
      temperature: 0.8
       
router_settings:
  routing_strategy: latency-based-routing
  num_retries: 2
  timeout: 601
  fallbacks:
    - granite: ["qwen"]
    - bielik: ["gemma"]

general_settings:
  master_key: admin
  ###disable_env_credential_login: true
  ###store_model_in_db: true
  store_prompts_in_spend_logs: true
  database_url: "postgresql://rola:haslo@host:5432/bazadanych"

Rola i uprawnienia w bazie Postgres

# \c litellm
You are now connected to database "litellm" as user "postgres".
# grant all ON database litellm to litellm;
# grant CONNECT on database litellm to litellm;
# GRANT ALL PRIVILEGES ON SCHEMA public TO litellm;
# ALTER ROLE "litellm" WITH LOGIN;

Utworzenie schematu bazy

Będąc w katalogu z venv.

prisma generate --schema lib/python3.12/site-packages/litellm/proxy/schema.prisma

Utworzenie aplikacji

litellm --config litellm_config.yaml --port 4040

Można się zalogować jako użytkownik admin z hasłem admin na stronie http://127.0.0.1:4040/ui/login/ . Potem na stronie http://127.0.0.1:4040/ui/api-keys/ można wyklikać klucz API, powiązać go z modelami, grupami i użytkownikami.

AiderAI

Uruchomienie AiderAI bez połączenia z Ollama, ale z połączeniem do LiteLLM.

AIDER_ANALYTICS_DISABLE=true AIDER_CHECK_UPDATE=false AIDER_YES_ALWAYS=true AIDER_DARK_MODE=true AIDER_SHOW_MODEL_WARNINGS=false AIDER_MAP_TOKENS=1024 LITELLM_PROXY_API_BASE=http://127.0.0.1:4040 LITELLM_PROXY_API_KEY=sk-[token_w_litellm] aider --model=litellm_proxy/granite

Shell GPT

Plik konfiguracyjny ~/.config/shell_gpt/.sgptrc komunikujący klienta z LiteLLM.

CHAT_CACHE_PATH=/tmp/chat_cache
CACHE_PATH=/tmp/cache
CHAT_CACHE_LENGTH=100
CACHE_LENGTH=100
REQUEST_TIMEOUT=60
DEFAULT_MODEL=gemma
DEFAULT_COLOR=magenta
ROLE_STORAGE_PATH=/home/[uzytkownik]/.config/shell_gpt/roles
DEFAULT_EXECUTE_SHELL_CMD=false
DISABLE_STREAMING=false
CODE_THEME=dracula
OPENAI_FUNCTIONS_PATH=/home/[uzytkownik]/.config/shell_gpt/functions
OPENAI_USE_FUNCTIONS=false
SHOW_FUNCTIONS_OUTPUT=false
API_BASE_URL=http://127.0.0.1:4040
PRETTIFY_MARKDOWN=true
USE_LITELLM=false
SHELL_INTERACTION=true
OS_NAME=auto
SHELL_NAME=auto
OPENAI_API_KEY=sk-[klucz w LiteLLM]

Działa to tak:

((sgpt) ) [[uzytkownk]@[host] sgpt]$ sgpt 'Which user am I?'                                                                                                                             
To determine which user you are currently logged in as on your CentOS Stream 10 system, you can use the following command in your bash shell:                                         
                                                                                                                                                                                      
                                                                                                                                                                                      
 whoami                                                                                                                                                                               
                                                                                                                                                                                      
                                                                                                                                                                                      
Alternatively, you can check the USER environment variable:                                                                                                                           
                                                                                                                                                                                      
                                                                                                                                                                                      
 echo $USER                                                                                                                                                                           
                                                                                                                                                                                      
                                                                                                                                                                                      
These commands will return the username of the current session. If you are running with elevated privileges (e.g., via sudo), whoami will show the effective user identity.