* fix(azure/chat/gpt_transformation.py): add 'prediction' as a support azure param
Closes https://github.com/BerriAI/litellm/issues/8500
* build(model_prices_and_context_window.json): add new 'gemini-2.0-pro-exp-02-05' model
* style: cleanup invalid json trailing commma
* feat(utils.py): support passing 'tokenizer_config' to register_prompt_template
enables passing complete tokenizer config of model to litellm
Allows calling deepseek on bedrock with the correct prompt template
* fix(utils.py): fix register_prompt_template for custom model names
* test(test_prompt_factory.py): fix test
* test(test_completion.py): add e2e test for bedrock invoke deepseek ft model
* feat(base_invoke_transformation.py): support hf_model_name param for bedrock invoke calls
enables proxy admin to set base model for ft bedrock deepseek model
* feat(bedrock/invoke): support deepseek_r1 route for bedrock
makes it easy to apply the right chat template to that call
* feat(constants.py): store deepseek r1 chat template - allow user to get correct response from deepseek r1 without extra work
* test(test_completion.py): add e2e mock test for bedrock deepseek
* docs(bedrock.md): document new deepseek_r1 route for bedrock
allows us to use the right config
* fix(exception_mapping_utils.py): catch read operation timeout
* fix(utils.py): fix vertex ai optional param handling
don't pass max retries to unsupported route
Fixes https://github.com/BerriAI/litellm/issues/8254
* fix(get_supported_openai_params.py): fix linting error
* fix(get_supported_openai_params.py): default to openai-like spec
* test: fix test
* fix: fix linting error
* Improved wildcard route handling on `/models` and `/model_group/info` (#8473)
* fix(model_checks.py): update returning known model from wildcard to filter based on given model prefix
ensures wildcard route - `vertex_ai/gemini-*` just returns known vertex_ai/gemini- models
* test(test_proxy_utils.py): add unit testing for new 'get_known_models_from_wildcard' helper
* test(test_models.py): add e2e testing for `/model_group/info` endpoint
* feat(prometheus.py): support tracking total requests by user_email on prometheus
adds initial support for tracking total requests by user_email
* test(test_prometheus.py): add testing to ensure user email is always tracked
* test: update testing for new prometheus metric
* test(test_prometheus_unit_tests.py): add user email to total proxy metric
* test: update tests
* test: fix spend tests
* test: fix test
* fix(pagerduty.py): fix linting error
* (Bug fix) - Using `include_usage` for /completions requests + unit testing (#8484)
* pass stream options (#8419)
* test_completion_streaming_usage_metrics
* test_text_completion_include_usage
---------
Co-authored-by: Kaushik Deka <55996465+Kaushikdkrikhanu@users.noreply.github.com>
* fix naming docker stable release
* build(model_prices_and_context_window.json): handle azure model update
* docs(token_auth.md): clarify scopes can be a list or comma separated string
* docs: fix docs
* add sonar pricings (#8476)
* add sonar pricings
* Update model_prices_and_context_window.json
* Update model_prices_and_context_window.json
* Update model_prices_and_context_window_backup.json
* update load testing script
* fix test_async_router_context_window_fallback
* pplx - fix supports tool choice openai param (#8496)
* fix prom check startup (#8492)
* test_async_router_context_window_fallback
* ci(config.yml): mark daily docker builds with `-nightly` (#8499)
Resolves https://github.com/BerriAI/litellm/discussions/8495
* (Redis Cluster) - Fixes for using redis cluster + pipeline (#8442)
* update RedisCluster creation
* update RedisClusterCache
* add redis ClusterCache
* update async_set_cache_pipeline
* cleanup redis cluster usage
* fix redis pipeline
* test_init_async_client_returns_same_instance
* fix redis cluster
* update mypy_path
* fix init_redis_cluster
* remove stub
* test redis commit
* ClusterPipeline
* fix import
* RedisCluster import
* fix redis cluster
* Potential fix for code scanning alert no. 2129: Clear-text logging of sensitive information
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix naming of redis cluster integration
* test_redis_caching_ttl_pipeline
* fix async_set_cache_pipeline
---------
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Litellm UI stable version 02 12 2025 (#8497)
* fix(key_management_endpoints.py): fix `/key/list` to include `return_full_object` as a top-level query param
Allows user to specify they want the keys as a list of objects
* refactor(key_list.tsx): initial refactor of key table in user dashboard
offloads key filtering logic to backend api
prevents common error of user not being able to see their keys
* fix(key_management_endpoints.py): allow internal user to query `/key/list` to see their keys
* fix(key_management_endpoints.py): add validation checks and filtering to `/key/list` endpoint
allow internal user to see their keys. not anybody else's
* fix(view_key_table.tsx): fix issue where internal user could not see default team keys
* fix: fix linting error
* fix: fix linting error
* fix: fix linting error
* fix: fix linting error
* fix: fix linting error
* fix: fix linting error
* fix: fix linting error
* test_supports_tool_choice
* test_async_router_context_window_fallback
* fix: fix test (#8501)
* Litellm dev 02 12 2025 p1 (#8494)
* Resolves https://github.com/BerriAI/litellm/issues/6625 (#8459)
- enables no auth for SMTP
Signed-off-by: Regli Daniel <daniel.regli1@sanitas.com>
* add sonar pricings (#8476)
* add sonar pricings
* Update model_prices_and_context_window.json
* Update model_prices_and_context_window.json
* Update model_prices_and_context_window_backup.json
* test: fix test
---------
Signed-off-by: Regli Daniel <daniel.regli1@sanitas.com>
Co-authored-by: Dani Regli <1daniregli@gmail.com>
Co-authored-by: Lucca Zenóbio <luccazen@gmail.com>
* test: fix test
* UI Fixes p2 (#8502)
* refactor(admin.tsx): cleanup add new admin flow
removes buggy flow. Ensures just 1 simple way to add users / update roles.
* fix(user_search_modal.tsx): ensure 'add member' button is always visible
* fix(edit_membership.tsx): ensure 'save changes' button always visible
* fix(internal_user_endpoints.py): ensure user in org can be deleted
Fixes issue where user couldn't be deleted if they were a member of an org
* fix: fix linting error
* add phoenix docs for observability integration (#8522)
* Add files via upload
* Update arize_integration.md
* Update arize_integration.md
* add Phoenix docs
* Added custom_attributes to additional_keys which can be sent to athina (#8518)
* (UI) fix log details page (#8524)
* rollback changes to view logs page
* ui new build
* add interface for prefetch
* fix spread operation
* fix max size for request view page
* clean up table
* ui fix column on request logs page
* ui new build
* Add UI Support for Admins to Call /cache/ping and View Cache Analytics (#8475) (#8519)
* [Bug] UI: Newly created key does not display on the View Key Page (#8039)
- Fixed issue where all keys appeared blank for admin users.
- Implemented filtering of data via team settings to ensure all keys are displayed correctly.
* Fix:
- Updated the validator to allow model editing when `keyTeam.team_alias === "Default Team"`.
- Ensured other teams still follow the original validation rules.
* - added some classes in global.css
- added text wrap in output of request,response and metadata in index.tsx
- fixed styles of table in table.tsx
* - added full payload when we open single log entry
- added Combined Info Card in index.tsx
* fix: keys not showing on refresh for internal user
* merge
* main merge
* cache page
* ca remove
* terms change
* fix:places caching inside exp
---------
Signed-off-by: Regli Daniel <daniel.regli1@sanitas.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Kaushik Deka <55996465+Kaushikdkrikhanu@users.noreply.github.com>
Co-authored-by: Lucca Zenóbio <luccazen@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Dani Regli <1daniregli@gmail.com>
Co-authored-by: exiao <exiao@users.noreply.github.com>
Co-authored-by: vivek-athina <153479827+vivek-athina@users.noreply.github.com>
Co-authored-by: Taha Ali <123803932+tahaali-dev@users.noreply.github.com>
* fix(router.py): add more deployment timeout debug information for timeout errors
help understand why some calls in high-traffic don't respect their model-specific timeouts
* test(test_convert_dict_to_response.py): unit test ensuring empty str is not converted to None
Addresses https://github.com/BerriAI/litellm/issues/8507
* fix(convert_dict_to_response.py): handle empty message str - don't return back as 'None'
Fixes https://github.com/BerriAI/litellm/issues/8507
* test(test_completion.py): add e2e test
* fix(model_checks.py): update returning known model from wildcard to filter based on given model prefix
ensures wildcard route - `vertex_ai/gemini-*` just returns known vertex_ai/gemini- models
* test(test_proxy_utils.py): add unit testing for new 'get_known_models_from_wildcard' helper
* test(test_models.py): add e2e testing for `/model_group/info` endpoint
* feat(prometheus.py): support tracking total requests by user_email on prometheus
adds initial support for tracking total requests by user_email
* test(test_prometheus.py): add testing to ensure user email is always tracked
* test: update testing for new prometheus metric
* test(test_prometheus_unit_tests.py): add user email to total proxy metric
* test: update tests
* test: fix spend tests
* test: fix test
* fix(pagerduty.py): fix linting error
* update team info endpoint
* clean up model alias
* fix model alias
* fix model alias card
* clean up naming on docs
* fix model alias card
* fix _model_in_team_aliases
* team alias - fix litellm.model_alias_map
* fix _update_model_if_team_alias_exists
* fix test_aview_spend_per_user
* Test model alias functionality with teams:
* complete e2e test
* test_update_model_if_team_alias_exists
* update team info endpoint
* clean up model alias
* fix model alias
* fix model alias card
* clean up naming on docs
* fix model alias card
* fix _model_in_team_aliases
* fix key_model_access_denied
* test_can_key_call_model_with_aliases
* fix test_aview_spend_per_user
* fix(litellm_logging.py): support saving applied guardrails in logging object
allows list of applied guardrails to be logged for proxy admin's knowledge
* feat(spend_tracking_utils.py): log applied guardrails to spend logs
makes it easy for admin to know what guardrails were applied on a request
* ci(config.yml): uninstall posthog from ci/cd
* test: fix tests
* test: update test
* Fixed issue #8246 (#8250)
* Fixed issue #8246
* Added unit tests for discard() and for remove_callback_from_list_by_object()
* fix(openai.py): support dynamic passing of organization param to openai
handles scenario where client-side org id is passed to openai
---------
Co-authored-by: Erez Hadad <erezh@il.ibm.com>
* fix(client_initialization_utils.py): handle custom llm provider set with valid value not from model name
* fix(handle_jwt.py): handle groups not existing in jwt token
if user not in group, this won't exist
* fix(handle_jwt.py): add new `enforce_team_based_model_access` flag to jwt auth
allows proxy admin to enforce user can only call model if team has access
* feat(navbar.tsx): expose new dropdown in navbar - allow org admin to create teams within org context
* fix(navbar.tsx): remove non-functional cogicon
* fix(proxy/utils.py): include user-org memberships in `/user/info` response
return orgs user is a member of and the user role within org
* feat(organization_endpoints.py): allow internal user to query `/organizations/list` and get all orgs they belong to
enables org admin to select org they belong to, to create teams
* fix(navbar.tsx): show change in ui when org switcher clicked
* feat(page.tsx): update user role based on org they're in
allows org admin to create teams in the org context
* feat(teams.tsx): working e2e flow for allowing org admin to add new teams
* style(navbar.tsx): clarify switching orgs on UI is in BETA
* fix(organization_endpoints.py): handle getting but not setting members
* test: fix test
* fix(client_initialization_utils.py): revert custom llm provider handling fix - causing unintended issues
* docs(token_auth.md): cleanup docs
* fix(parallel_request_limiter.py): add back parallel request information to max parallel request limiter
Resolves https://github.com/BerriAI/litellm/issues/8392
* test: mark flaky test to handle time based tracking issues
* feat(model_management_endpoints.py): expose new patch `/model/{model_id}/update` endpoint
Allows updating specific values of a model in db - makes it easy for admin to know this by calling it a PA
TCH
* feat(edit_model_modal.tsx): allow user to update llm provider + api key on the ui
* fix: fix linting error
* add back streaming for base o3 (#8361)
* test(base_llm_unit_tests.py): add base test for o-series models - ensure streaming always works
* fix(base_llm_unit_tests.py): fix test for o series models
* refactor: move test
---------
Co-authored-by: Matteo Boschini <12133566+mbosc@users.noreply.github.com>
* fix(caching_routes.py): mask redis password on `/cache/ping` route
* fix(caching_routes.py): fix linting erro
* fix(caching_routes.py): fix linting error on caching routes
* fix: fix test - ignore mask_dict - has a breakpoint
* fix(azure.py): add timeout param + elapsed time in azure timeout error
* fix(http_handler.py): add elapsed time to http timeout request
makes it easier to debug how long request took before failing
* fix(azure.py): ensure max_retries=0 is respected
Fixes https://github.com/BerriAI/litellm/issues/6129
* fix(test_openai.py): add unit test to ensure openai sdk calls always respect max_retries = 0
* test(test_azure_openai.py): add unit testing for azure_text/ route
* fix(azure.py): fix passing max retries on streaming
* fix(azure.py): fix azure max retries on async completion + streaming
* fix(completion/handler.py): fix azure text async completion + streaming
* test(test_azure_openai.py): ensure azure openai max retries always respected
* test(test_azure_o_series.py): add testing to ensure max retries always respected
* Added gemini providers for 2.0-flash and 2.0-flash lite (#8321)
* Update model_prices_and_context_window.json
added gemini providers for 2.0-flash and 2.0-flash light
* Update model_prices_and_context_window.json
fixed URL
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
* Convert tool use arguments to string before counting tokens (#6989)
In at least some cases the `messages["tool_calls"]["function"]["arguments"]` is a dict, not a string. In order to tokenize it properly it needs to be a string. In the case that it is already a string this is a noop, which is also fine.
* build(model_prices_and_context_window.json): add gemini 2.0 flash lite pricing
* build(model_prices_and_context_window.json): add gemini commercial rate limits
* fix(utils.py): fix linting error
* refactor(utils.py): refactor to maintain function size
---------
Co-authored-by: Bardia Khosravi <bardiakhosravi95@gmail.com>
Co-authored-by: Josh Morrow <josh@jcmorrow.com>
* feat(handle_jwt.py): initial commit to allow scope based model access
* feat(handle_jwt.py): allow model access based on token scopes
allow admin to control model access from IDP
* test(test_jwt.py): add unit testing for scope based model access
* docs(token_auth.md): add scope based model access to docs
* docs(token_auth.md): update docs
* docs(token_auth.md): update docs
* build: add gemini commercial rate limits
* fix: fix linting error
* add initial test for assembly ai
* start using PassthroughEndpointRouter
* migrate to lllm passthrough endpoints
* add assembly ai as a known provider
* fix PassthroughEndpointRouter
* fix set_pass_through_credentials
* working EU request to assembly ai pass through endpoint
* add e2e test assembly
* test_assemblyai_routes_with_bad_api_key
* clean up pass through endpoint router
* e2e testing for assembly ai pass through
* test assembly ai e2e testing
* delete assembly ai models
* fix code quality
* ui working assembly ai api base flow
* fix install assembly ai
* update model call details with kwargs for pass through logging
* fix tracking assembly ai model in response
* _handle_assemblyai_passthrough_logging
* fix test_initialize_deployment_for_pass_through_unsupported_provider
* TestPassthroughEndpointRouter
* _get_assembly_transcript
* fix assembly ai pt logging tests
* fix assemblyai_proxy_route
* fix _get_assembly_region_from_url
* fix(utils.py): handle key error in msg validation
* Support running Aim Guard during LLM call (#7918)
* support running Aim Guard during LLM call
* Rename header
* adjust docs and fix type annotations
* fix(timeout.md): doc fix for openai example on dynamic timeouts
---------
Co-authored-by: Tomer Bin <117278227+hxtomer@users.noreply.github.com>
* refactor _get_langfuse_input_output_content
* test_langfuse_logging_completion_with_malformed_llm_response
* fix _get_langfuse_input_output_content
* fixes for langfuse linting
* unit testing for get chat/text content for langfuse
* fix _should_raise_content_policy_error
* fix(convert_dict_to_response.py): only convert if response is the response_format tool call passed in
Fixes https://github.com/BerriAI/litellm/issues/8241
* fix(gpt_transformation.py): makes sure response format / tools conversion doesn't remove previous tool calls
* refactor(gpt_transformation.py): refactor out json schema converstion to base config
keeps logic consistent across providers
* fix(o_series_transformation.py): support o3 mini native streaming
Fixes https://github.com/BerriAI/litellm/issues/8274
* fix(gpt_transformation.py): remove unused variables
* test: update test
* initial transform for invoke
* invoke transform_response
* working - able to make request
* working get_complete_url
* working - invoke now runs on llm_http_handler
* fix unused imports
* track litellm overhead ms
* working stream request
* sign_request transform
* sign_request update
* use has_async_custom_stream_wrapper property
* use get_async_custom_stream_wrapper in base llm http handler
* fix make_call in invoke handler
* fix invoke with streaming get_async_custom_stream_wrapper
* working bedrock async streaming with invoke
* fix make call handler for bedrock
* test_all_model_configs
* fix test_bedrock_custom_prompt_template
* sync streaming for bedrock invoke
* fix _add_stream_param_to_request_body
* test_async_text_completion_bedrock
* fix transform_request
* fix get_supported_openai_params
* fix test supports tool choice
* fix test_supports_tool_choice
* add unit test coverage for bedrock invoke transform
* fix location of transformation files
* update import loc
* fix bedrock invoke unit tests
* fix import for max completion tokens
* remove code block upserting master key hash to db
* run test to check if key upserted into db
* run ci/cd again
* litellm_proxy_security_tests
* litellm_proxy_security_tests
* run prisma entrypoint
* ci/cd run again
* fix test master key not in db
* refactor(deepseek/): move deepseek to base llm http handler
Fixes https://github.com/BerriAI/litellm/issues/8128#issuecomment-2635430457
* fix(gpt_transformation.py): support stream parsing for gpt-like calls
* test(test_deepseek_completion.py): add async streaming test
* fix(gpt_transformation.py): fix import
* fix(gpt_transformation.py): return full api base and content type
* feat(proxy/_types.py): add new jwt field params
allows users + services to auth into proxy
* feat(handle_jwt.py): allow team role proxy access
allows proxy admin to set allowed team roles
* fix(proxy/_types.py): add 'routes' to role based permissions
allow proxy admin to restrict what routes a team can access easily
* feat(handle_jwt.py): support more flexible role based route access
v2 on role based 'allowed_routes'
* test(test_jwt.py): add unit test for rbac for proxy routes
* feat(handle_jwt.py): ensure cost tracking always works for any jwt request with `enforce_rbac=True`
* docs(token_auth.md): add documentation on controlling model access via OIDC Roles
* test: increase time delay before retrying
* test: handle model overloaded for test
