litellm-mirror/litellm/proxy
Ishaan Jaff 51ffe93e77
(docs) add docstrings for all /key, /user, /team, /customer endpoints (#6804)
* use helper to handle_exception_on_proxy

* add doc string for /key/regenerate

* use 1 helper for handle_exception_on_proxy

* add doc string for /key/block

* add doc string for /key/unblock

* remove deprecated function

* remove deprecated endpoints

* remove incorrect tag for endpoint

* fix linting

* fix /key/regenerate

* fix regen key

* fix use port 4000 for user endpoints

* fix clean up - use separate file for customer endpoints

* add docstring for user/update

* fix imports

* doc string /user/list

* doc string for /team/delete

* fix team block endpoint

* fix import block user

* add doc string for /team/unblock

* add doc string for /team/list

* add doc string for /team/info

* add doc string for key endpoints

* fix customer_endpoints

* add doc string for customer endpoints

* fix import new_end_user

* fix testing

* fix import new_end_user

* fix add check for allow_user_auth
2024-11-18 19:44:06 -08:00
..
_experimental new ui build 2024-11-16 20:09:29 -08:00
analytics_endpoints Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
auth (fix) Fix - don't allow viewer roles to create virtual keys (#6764) 2024-11-15 18:02:13 -08:00
common_utils fix(pattern_match_deployments.py): default to user input if unable to… (#6632) 2024-11-08 00:55:57 +05:30
config_management_endpoints feat(ui): for adding pass-through endpoints 2024-08-15 21:58:11 -07:00
db (feat) log error class, function_name on prometheus service failure hook + only log DB related failures on DB service hook (#6650) 2024-11-07 17:01:18 -08:00
example_config_yaml Litellm router max depth (#6501) 2024-10-29 22:05:41 -07:00
fine_tuning_endpoints Add pyright to ci/cd + Fix remaining type-checking errors (#6082) 2024-10-05 17:04:00 -04:00
guardrails (code quality) add ruff check PLR0915 for too-many-statements (#6309) 2024-10-18 15:36:49 +05:30
health_endpoints (fix) Langfuse key based logging (#6372) 2024-10-23 18:24:22 +05:30
hooks (feat) Use litellm/ prefix when storing virtual keys in AWS secret manager (#6765) 2024-11-15 18:07:43 -08:00
management_endpoints (docs) add docstrings for all /key, /user, /team, /customer endpoints (#6804) 2024-11-18 19:44:06 -08:00
management_helpers fix create_audit_log_for_update 2024-10-25 16:48:25 +04:00
openai_files_endpoints Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
pass_through_endpoints (docs) add docstrings for all /key, /user, /team, /customer endpoints (#6804) 2024-11-18 19:44:06 -08:00
proxy_load_test Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
rerank_endpoints LiteLLM Minor Fixes & Improvements (09/26/2024) (#5925) (#5937) 2024-09-27 17:54:13 -07:00
spend_tracking LiteLLM Minor Fixes & Improvements (11/13/2024) (#6729) 2024-11-15 11:18:31 +05:30
ui_crud_endpoints ui - add Create, get, delete endpoints for IP Addresses 2024-07-09 15:12:08 -07:00
vertex_ai_endpoints LiteLLM Minor Fixes & Improvements (11/13/2024) (#6729) 2024-11-15 11:18:31 +05:30
.gitignore
__init__.py
_logging.py fix(_logging.py): fix timestamp format for json logs 2024-06-20 15:20:21 -07:00
_new_secret_config.yaml LiteLLM Minor Fixes & Improvement (11/14/2024) (#6730) 2024-11-15 01:02:54 +05:30
_super_secret_config.yaml docs(enterprise.md): cleanup docs 2024-07-15 14:52:08 -07:00
_types.py (feat) Use litellm/ prefix when storing virtual keys in AWS secret manager (#6765) 2024-11-15 18:07:43 -08:00
cached_logo.jpg (feat) use hosted images for custom branding 2024-02-22 14:51:40 -08:00
caching_routes.py (refactor) caching use LLMCachingHandler for async_get_cache and set_cache (#6208) 2024-10-14 16:34:01 +05:30
custom_sso.py Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
enterprise feat(llama_guard.py): add llama guard support for content moderation + new async_moderation_hook endpoint 2024-02-17 19:13:04 -08:00
health_check.py LiteLLM Minor Fixes and Improvements (09/14/2024) (#5697) 2024-09-14 10:32:39 -07:00
lambda.py
litellm_pre_call_utils.py LiteLLM Minor Fixes & Improvement (11/14/2024) (#6730) 2024-11-15 01:02:54 +05:30
llamaguard_prompt.txt feat(llama_guard.py): allow user to define custom unsafe content categories 2024-02-17 17:42:47 -08:00
logo.jpg (feat) admin ui custom branding 2024-02-21 17:34:42 -08:00
openapi.json
post_call_rules.py
prisma_migration.py Litellm expose disable schema update flag (#6085) 2024-10-05 21:26:51 -04:00
proxy_cli.py (Feat) Add support for storing virtual keys in AWS SecretManager (#6728) 2024-11-14 09:25:07 -08:00
proxy_config.yaml (Feat) Add support for storing virtual keys in AWS SecretManager (#6728) 2024-11-14 09:25:07 -08:00
proxy_server.py (docs) add docstrings for all /key, /user, /team, /customer endpoints (#6804) 2024-11-18 19:44:06 -08:00
README.md [Feat-Proxy] Allow using custom sso handler (#5809) 2024-09-20 19:14:33 -07:00
route_llm_request.py LiteLLM Minor Fixes & Improvements (11/06/2024) (#6624) 2024-11-07 04:37:32 +05:30
schema.prisma track created, updated at virtual keys 2024-10-25 07:19:29 +04:00
start.sh
utils.py (docs) add docstrings for all /key, /user, /team, /customer endpoints (#6804) 2024-11-18 19:44:06 -08:00

litellm-proxy

A local, fast, and lightweight OpenAI-compatible server to call 100+ LLM APIs.

usage

$ pip install litellm
$ litellm --model ollama/codellama 

#INFO: Ollama running on http://0.0.0.0:8000

replace openai base

import openai # openai v1.0.0+
client = openai.OpenAI(api_key="anything",base_url="http://0.0.0.0:8000") # set proxy to base_url
# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(model="gpt-3.5-turbo", messages = [
    {
        "role": "user",
        "content": "this is a test request, write a short poem"
    }
])

print(response)

See how to call Huggingface,Bedrock,TogetherAI,Anthropic, etc.


Folder Structure

Routes

  • proxy_server.py - all openai-compatible routes - /v1/chat/completion, /v1/embedding + model info routes - /v1/models, /v1/model/info, /v1/model_group_info routes.
  • health_endpoints/ - /health, /health/liveliness, /health/readiness
  • management_endpoints/key_management_endpoints.py - all /key/* routes
  • management_endpoints/team_endpoints.py - all /team/* routes
  • management_endpoints/internal_user_endpoints.py - all /user/* routes
  • management_endpoints/ui_sso.py - all /sso/* routes