* fix(main.py): fix key leak error when unknown provider given
don't return passed in args if unknown route on embedding
* fix(main.py): remove instances of {args} being passed in exception
prevent potential key leaks
* test(code_coverage/prevent_key_leaks_in_codebase.py): ban usage of {args} in codebase
* fix: fix linting errors
* fix: remove unused variable
* refactor(organization.tsx): initial commit with orgs tab refactor
make it similar to 'Teams' tab - simplifies org management actions
* fix(page.tsx): pass user orgs to component
* fix(organization_view.tsx): fix to pull info from org info endpoint
* feat(organization_endpoints.py): return org members when calling /org/info
* fix(organization_view.tsx): show org members on info page
* feat(organization_view.tsx): allow adding user to org via user email
Resolves https://github.com/BerriAI/litellm/issues/8330
* fix(organization_endpoints.py): raise better error when duplicate user_email found in db
* fix(organization_view.tsx): cleanup user_email for now
not in initial org info - will need to prefetch
* fix(page.tsx): fix getting user models on page load
allows passing down the user models to org
* fix(organizations.tsx): fix creating org on ui
* fix(proxy/_types.py): include org created at and updated at
cleans up ui
* fix(navbar.tsx): cleanup
* fix(organizations.tsx): fix tpm/rpm limits on org
* fix(organizations.tsx): fix linting error
* fix(organizations.tsx): fix linting \
* (Feat) - Add `/bedrock/meta.llama3-3-70b-instruct-v1:0` tool calling support + cost tracking + base llm unit test for tool calling (#8545)
* Add support for bedrock meta.llama3-3-70b-instruct-v1:0 tool calling (#8512)
* fix(converse_transformation.py): fixing bedrock meta.llama3-3-70b tool calling
* test(test_bedrock_completion.py): adding llama3.3 tool compatibility check
* add TestBedrockTestSuite
* add bedrock llama 3.3 to base llm class
* us.meta.llama3-3-70b-instruct-v1:0
* test_basic_tool_calling
* TestAzureOpenAIO1
* test_basic_tool_calling
* test_basic_tool_calling
---------
Co-authored-by: miraclebakelaser <65143272+miraclebakelaser@users.noreply.github.com>
* fix(general_settings.tsx): filter out empty dictionaries post fallback delete (#8550)
Fixes https://github.com/BerriAI/litellm/issues/8331
* bump: version 1.61.3 → 1.61.4
* (perf) Fix memory leak on `/completions` route (#8551)
* initial mem util test
* fix _cached_get_model_info_helper
* test memory usage
* fix tests
* fix mem usage
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: miraclebakelaser <65143272+miraclebakelaser@users.noreply.github.com>
* fix(utils.py): fix vertex ai optional param handling
don't pass max retries to unsupported route
Fixes https://github.com/BerriAI/litellm/issues/8254
* fix(get_supported_openai_params.py): fix linting error
* fix(get_supported_openai_params.py): default to openai-like spec
* test: fix test
* fix: fix linting error
* Improved wildcard route handling on `/models` and `/model_group/info` (#8473)
* fix(model_checks.py): update returning known model from wildcard to filter based on given model prefix
ensures wildcard route - `vertex_ai/gemini-*` just returns known vertex_ai/gemini- models
* test(test_proxy_utils.py): add unit testing for new 'get_known_models_from_wildcard' helper
* test(test_models.py): add e2e testing for `/model_group/info` endpoint
* feat(prometheus.py): support tracking total requests by user_email on prometheus
adds initial support for tracking total requests by user_email
* test(test_prometheus.py): add testing to ensure user email is always tracked
* test: update testing for new prometheus metric
* test(test_prometheus_unit_tests.py): add user email to total proxy metric
* test: update tests
* test: fix spend tests
* test: fix test
* fix(pagerduty.py): fix linting error
* (Bug fix) - Using `include_usage` for /completions requests + unit testing (#8484)
* pass stream options (#8419)
* test_completion_streaming_usage_metrics
* test_text_completion_include_usage
---------
Co-authored-by: Kaushik Deka <55996465+Kaushikdkrikhanu@users.noreply.github.com>
* fix naming docker stable release
* build(model_prices_and_context_window.json): handle azure model update
* docs(token_auth.md): clarify scopes can be a list or comma separated string
* docs: fix docs
* add sonar pricings (#8476)
* add sonar pricings
* Update model_prices_and_context_window.json
* Update model_prices_and_context_window.json
* Update model_prices_and_context_window_backup.json
* update load testing script
* fix test_async_router_context_window_fallback
* pplx - fix supports tool choice openai param (#8496)
* fix prom check startup (#8492)
* test_async_router_context_window_fallback
* ci(config.yml): mark daily docker builds with `-nightly` (#8499)
Resolves https://github.com/BerriAI/litellm/discussions/8495
* (Redis Cluster) - Fixes for using redis cluster + pipeline (#8442)
* update RedisCluster creation
* update RedisClusterCache
* add redis ClusterCache
* update async_set_cache_pipeline
* cleanup redis cluster usage
* fix redis pipeline
* test_init_async_client_returns_same_instance
* fix redis cluster
* update mypy_path
* fix init_redis_cluster
* remove stub
* test redis commit
* ClusterPipeline
* fix import
* RedisCluster import
* fix redis cluster
* Potential fix for code scanning alert no. 2129: Clear-text logging of sensitive information
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix naming of redis cluster integration
* test_redis_caching_ttl_pipeline
* fix async_set_cache_pipeline
---------
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Litellm UI stable version 02 12 2025 (#8497)
* fix(key_management_endpoints.py): fix `/key/list` to include `return_full_object` as a top-level query param
Allows user to specify they want the keys as a list of objects
* refactor(key_list.tsx): initial refactor of key table in user dashboard
offloads key filtering logic to backend api
prevents common error of user not being able to see their keys
* fix(key_management_endpoints.py): allow internal user to query `/key/list` to see their keys
* fix(key_management_endpoints.py): add validation checks and filtering to `/key/list` endpoint
allow internal user to see their keys. not anybody else's
* fix(view_key_table.tsx): fix issue where internal user could not see default team keys
* fix: fix linting error
* fix: fix linting error
* fix: fix linting error
* fix: fix linting error
* fix: fix linting error
* fix: fix linting error
* fix: fix linting error
* test_supports_tool_choice
* test_async_router_context_window_fallback
* fix: fix test (#8501)
* Litellm dev 02 12 2025 p1 (#8494)
* Resolves https://github.com/BerriAI/litellm/issues/6625 (#8459)
- enables no auth for SMTP
Signed-off-by: Regli Daniel <daniel.regli1@sanitas.com>
* add sonar pricings (#8476)
* add sonar pricings
* Update model_prices_and_context_window.json
* Update model_prices_and_context_window.json
* Update model_prices_and_context_window_backup.json
* test: fix test
---------
Signed-off-by: Regli Daniel <daniel.regli1@sanitas.com>
Co-authored-by: Dani Regli <1daniregli@gmail.com>
Co-authored-by: Lucca Zenóbio <luccazen@gmail.com>
* test: fix test
* UI Fixes p2 (#8502)
* refactor(admin.tsx): cleanup add new admin flow
removes buggy flow. Ensures just 1 simple way to add users / update roles.
* fix(user_search_modal.tsx): ensure 'add member' button is always visible
* fix(edit_membership.tsx): ensure 'save changes' button always visible
* fix(internal_user_endpoints.py): ensure user in org can be deleted
Fixes issue where user couldn't be deleted if they were a member of an org
* fix: fix linting error
* add phoenix docs for observability integration (#8522)
* Add files via upload
* Update arize_integration.md
* Update arize_integration.md
* add Phoenix docs
* Added custom_attributes to additional_keys which can be sent to athina (#8518)
* (UI) fix log details page (#8524)
* rollback changes to view logs page
* ui new build
* add interface for prefetch
* fix spread operation
* fix max size for request view page
* clean up table
* ui fix column on request logs page
* ui new build
* Add UI Support for Admins to Call /cache/ping and View Cache Analytics (#8475) (#8519)
* [Bug] UI: Newly created key does not display on the View Key Page (#8039)
- Fixed issue where all keys appeared blank for admin users.
- Implemented filtering of data via team settings to ensure all keys are displayed correctly.
* Fix:
- Updated the validator to allow model editing when `keyTeam.team_alias === "Default Team"`.
- Ensured other teams still follow the original validation rules.
* - added some classes in global.css
- added text wrap in output of request,response and metadata in index.tsx
- fixed styles of table in table.tsx
* - added full payload when we open single log entry
- added Combined Info Card in index.tsx
* fix: keys not showing on refresh for internal user
* merge
* main merge
* cache page
* ca remove
* terms change
* fix:places caching inside exp
---------
Signed-off-by: Regli Daniel <daniel.regli1@sanitas.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Kaushik Deka <55996465+Kaushikdkrikhanu@users.noreply.github.com>
Co-authored-by: Lucca Zenóbio <luccazen@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Dani Regli <1daniregli@gmail.com>
Co-authored-by: exiao <exiao@users.noreply.github.com>
Co-authored-by: vivek-athina <153479827+vivek-athina@users.noreply.github.com>
Co-authored-by: Taha Ali <123803932+tahaali-dev@users.noreply.github.com>
* fix(router.py): add more deployment timeout debug information for timeout errors
help understand why some calls in high-traffic don't respect their model-specific timeouts
* test(test_convert_dict_to_response.py): unit test ensuring empty str is not converted to None
Addresses https://github.com/BerriAI/litellm/issues/8507
* fix(convert_dict_to_response.py): handle empty message str - don't return back as 'None'
Fixes https://github.com/BerriAI/litellm/issues/8507
* test(test_completion.py): add e2e test
* refactor(admin.tsx): cleanup add new admin flow
removes buggy flow. Ensures just 1 simple way to add users / update roles.
* fix(user_search_modal.tsx): ensure 'add member' button is always visible
* fix(edit_membership.tsx): ensure 'save changes' button always visible
* fix(internal_user_endpoints.py): ensure user in org can be deleted
Fixes issue where user couldn't be deleted if they were a member of an org
* fix: fix linting error
* fix(key_management_endpoints.py): fix `/key/list` to include `return_full_object` as a top-level query param
Allows user to specify they want the keys as a list of objects
* refactor(key_list.tsx): initial refactor of key table in user dashboard
offloads key filtering logic to backend api
prevents common error of user not being able to see their keys
* fix(key_management_endpoints.py): allow internal user to query `/key/list` to see their keys
* fix(key_management_endpoints.py): add validation checks and filtering to `/key/list` endpoint
allow internal user to see their keys. not anybody else's
* fix(view_key_table.tsx): fix issue where internal user could not see default team keys
* fix: fix linting error
* fix: fix linting error
* fix: fix linting error
* fix: fix linting error
* fix: fix linting error
* fix: fix linting error
* fix: fix linting error
* fix(model_checks.py): update returning known model from wildcard to filter based on given model prefix
ensures wildcard route - `vertex_ai/gemini-*` just returns known vertex_ai/gemini- models
* test(test_proxy_utils.py): add unit testing for new 'get_known_models_from_wildcard' helper
* test(test_models.py): add e2e testing for `/model_group/info` endpoint
* feat(prometheus.py): support tracking total requests by user_email on prometheus
adds initial support for tracking total requests by user_email
* test(test_prometheus.py): add testing to ensure user email is always tracked
* test: update testing for new prometheus metric
* test(test_prometheus_unit_tests.py): add user email to total proxy metric
* test: update tests
* test: fix spend tests
* test: fix test
* fix(pagerduty.py): fix linting error
* update team info endpoint
* clean up model alias
* fix model alias
* fix model alias card
* clean up naming on docs
* fix model alias card
* fix _model_in_team_aliases
* team alias - fix litellm.model_alias_map
* fix _update_model_if_team_alias_exists
* fix test_aview_spend_per_user
* Test model alias functionality with teams:
* complete e2e test
* test_update_model_if_team_alias_exists
* update team info endpoint
* clean up model alias
* fix model alias
* fix model alias card
* clean up naming on docs
* fix model alias card
* fix _model_in_team_aliases
* fix key_model_access_denied
* test_can_key_call_model_with_aliases
* fix test_aview_spend_per_user
* fix(litellm_logging.py): support saving applied guardrails in logging object
allows list of applied guardrails to be logged for proxy admin's knowledge
* feat(spend_tracking_utils.py): log applied guardrails to spend logs
makes it easy for admin to know what guardrails were applied on a request
* ci(config.yml): uninstall posthog from ci/cd
* test: fix tests
* test: update test
* feat(team_endpoints.py): support returning teams filtered by organization_id
allows user to just get teams they belong to, within the org
Enables org admin to see filtered list of teams on UI
* fix(teams.tsx): simple filter for team on ui - just filter team based on selected org id
* feat(ui/organizations): show 'default org' in switcher, filter teams based on selected org
* feat(user_dashboard.tsx): update team in switcher when org changes
* feat(schema.prisma): add new 'organization_id' value to key table
allow org admin to directly issue keys to a user within their org
* fix(view_key_table.tsx): fix regression where admin couldn't see keys
caused by bad console log statement
* fix(team_endpoints.py): handle default org value in /team/list
* fix(key_management_endpoints.py): allow proxy admin to create keys for team they're not in
* fix(team_endpoints.py): fix team endpoint to handle org id not being passed in
* build(config.yml): investigate what pkg is installing posthog in ci/cd
* ci(config.yml): uninstall posthog
prevent it from being added in ci/cd
* ci: auto-install ci
* Fixed issue #8246 (#8250)
* Fixed issue #8246
* Added unit tests for discard() and for remove_callback_from_list_by_object()
* fix(openai.py): support dynamic passing of organization param to openai
handles scenario where client-side org id is passed to openai
---------
Co-authored-by: Erez Hadad <erezh@il.ibm.com>
* fix(client_initialization_utils.py): handle custom llm provider set with valid value not from model name
* fix(handle_jwt.py): handle groups not existing in jwt token
if user not in group, this won't exist
* fix(handle_jwt.py): add new `enforce_team_based_model_access` flag to jwt auth
allows proxy admin to enforce user can only call model if team has access
* feat(navbar.tsx): expose new dropdown in navbar - allow org admin to create teams within org context
* fix(navbar.tsx): remove non-functional cogicon
* fix(proxy/utils.py): include user-org memberships in `/user/info` response
return orgs user is a member of and the user role within org
* feat(organization_endpoints.py): allow internal user to query `/organizations/list` and get all orgs they belong to
enables org admin to select org they belong to, to create teams
* fix(navbar.tsx): show change in ui when org switcher clicked
* feat(page.tsx): update user role based on org they're in
allows org admin to create teams in the org context
* feat(teams.tsx): working e2e flow for allowing org admin to add new teams
* style(navbar.tsx): clarify switching orgs on UI is in BETA
* fix(organization_endpoints.py): handle getting but not setting members
* test: fix test
* fix(client_initialization_utils.py): revert custom llm provider handling fix - causing unintended issues
* docs(token_auth.md): cleanup docs
* fix(parallel_request_limiter.py): add back parallel request information to max parallel request limiter
Resolves https://github.com/BerriAI/litellm/issues/8392
* test: mark flaky test to handle time based tracking issues
* feat(model_management_endpoints.py): expose new patch `/model/{model_id}/update` endpoint
Allows updating specific values of a model in db - makes it easy for admin to know this by calling it a PA
TCH
* feat(edit_model_modal.tsx): allow user to update llm provider + api key on the ui
* fix: fix linting error
* fix(caching_routes.py): mask redis password on `/cache/ping` route
* fix(caching_routes.py): fix linting erro
* fix(caching_routes.py): fix linting error on caching routes
* fix: fix test - ignore mask_dict - has a breakpoint
* fix(azure.py): add timeout param + elapsed time in azure timeout error
* fix(http_handler.py): add elapsed time to http timeout request
makes it easier to debug how long request took before failing
* feat(handle_jwt.py): initial commit to allow scope based model access
* feat(handle_jwt.py): allow model access based on token scopes
allow admin to control model access from IDP
* test(test_jwt.py): add unit testing for scope based model access
* docs(token_auth.md): add scope based model access to docs
* docs(token_auth.md): update docs
* docs(token_auth.md): update docs
* build: add gemini commercial rate limits
* fix: fix linting error
* fix(.globals.css): revert .md hard set
caused regression in invitation link display (and possibly other places)
* Fix keys not showing on refresh for internal users (#8312)
* [Bug] UI: Newly created key does not display on the View Key Page (#8039)
- Fixed issue where all keys appeared blank for admin users.
- Implemented filtering of data via team settings to ensure all keys are displayed correctly.
* Fix:
- Updated the validator to allow model editing when `keyTeam.team_alias === "Default Team"`.
- Ensured other teams still follow the original validation rules.
* - added some classes in global.css
- added text wrap in output of request,response and metadata in index.tsx
- fixed styles of table in table.tsx
* - added full payload when we open single log entry
- added Combined Info Card in index.tsx
* fix: keys not showing on refresh for internal user
* fixed user id passed as null when keyuser is you (#8271)
* fix(user_dashboard.tsx): ensure non admin can't view other keys
---------
Co-authored-by: Taha Ali <123803932+tahaali-dev@users.noreply.github.com>
Co-authored-by: Jaswanth Karani <karani.jaswanth@gmail.com>
* add initial test for assembly ai
* start using PassthroughEndpointRouter
* migrate to lllm passthrough endpoints
* add assembly ai as a known provider
* fix PassthroughEndpointRouter
* fix set_pass_through_credentials
* working EU request to assembly ai pass through endpoint
* add e2e test assembly
* test_assemblyai_routes_with_bad_api_key
* clean up pass through endpoint router
* e2e testing for assembly ai pass through
* test assembly ai e2e testing
* delete assembly ai models
* fix code quality
* ui working assembly ai api base flow
* fix install assembly ai
* update model call details with kwargs for pass through logging
* fix tracking assembly ai model in response
* _handle_assemblyai_passthrough_logging
* fix test_initialize_deployment_for_pass_through_unsupported_provider
* TestPassthroughEndpointRouter
* _get_assembly_transcript
* fix assembly ai pt logging tests
* fix assemblyai_proxy_route
* fix _get_assembly_region_from_url
* fix(utils.py): handle key error in msg validation
* Support running Aim Guard during LLM call (#7918)
* support running Aim Guard during LLM call
* Rename header
* adjust docs and fix type annotations
* fix(timeout.md): doc fix for openai example on dynamic timeouts
---------
Co-authored-by: Tomer Bin <117278227+hxtomer@users.noreply.github.com>
* remove code block upserting master key hash to db
* run test to check if key upserted into db
* run ci/cd again
* litellm_proxy_security_tests
* litellm_proxy_security_tests
* run prisma entrypoint
* ci/cd run again
* fix test master key not in db
* feat(proxy/_types.py): add new jwt field params
allows users + services to auth into proxy
* feat(handle_jwt.py): allow team role proxy access
allows proxy admin to set allowed team roles
* fix(proxy/_types.py): add 'routes' to role based permissions
allow proxy admin to restrict what routes a team can access easily
* feat(handle_jwt.py): support more flexible role based route access
v2 on role based 'allowed_routes'
* test(test_jwt.py): add unit test for rbac for proxy routes
* feat(handle_jwt.py): ensure cost tracking always works for any jwt request with `enforce_rbac=True`
* docs(token_auth.md): add documentation on controlling model access via OIDC Roles
* test: increase time delay before retrying
* test: handle model overloaded for test
* fix _decrypt_and_set_db_env_variables
* fix proxy config
* test callbacks in DB
* test langfuse callbacks in db
* test_e2e_langfuse_callbacks_in_db
* proxy_store_model_in_db_tests
* fix proxy_store_model_in_db_tests
* proxy_store_model_in_db_tests
* fix store_model_db_config.yaml
* fix check_langfuse_request
* fix test langfuse base url
* ci/cd run again
* track org id in spend logs
* read org id from team table
* show user_api_key_org_id in spend logs
* test_spend_logs_payload
* test_spend_logs_with_org_id
* test_spend_logs_with_org_id
