Commit graph

1697 commits

Author SHA1 Message Date
Krish Dholakia
2ab00bd12c
Litellm UI improvements 04 26 2025 p1 (#10346)
* fix(provider_info_helpers.tsx): fix together ai provider name

* fix(user_edit_view.tsx): all admin to edit user's personal models

* fix(model_dashboard.tsx): fix model filtering by team id check

* fix(proxy_server.py): support returning available models for a user on `/model_group/info`

Fixes issue where personal key with 'all team models' would see all proxy models (not just the ones user can call)

* test(test_models.py): add unit test for model group info check on personal key
2025-04-26 18:24:49 -07:00
Krish Dholakia
31313dd489
Fix - support azure dall e custom pricing (#10339)
* fix(cost_calculator.py): support custom pricing for image gen

Allow user to set custom pricing on azure image gen models

* test(test_cost_calculator.py): add unit test

* test(test_litellm_logging.py): add more unit testing

* fix(litellm_logging.py): fix ruff check
2025-04-26 17:16:03 -07:00
mdonaj
b99b0d5010
Fix: Prevent cache token overwrite by last chunk in streaming usage (#10284) 2025-04-26 12:06:12 -07:00
Krish Dholakia
93b6df96e0
Prisma Migrate - support setting custom migration dir (#10336)
* build(litellm-proxy-extras/utils.py): correctly generate baseline migration for non-empty db

* fix(litellm-proxy-extras/utils.py): Fix issue in migration, where if a migration fails during baselining, all are still marked as applied

* fix(prisma_client.py): don't pass separate schema.prisma to litellm-proxy-extras

use the one in litellm-proxy-extras

* fix(litellm-proxy-extras/utils.py): support passing custom dir for baselining db in read-only fs

Fixes https://github.com/BerriAI/litellm/issues/9885

* fix(utils.py): give helpful warning message when permission denied error raised in fs
2025-04-26 12:05:06 -07:00
Ishaan Jaff
331e784db4
[Feat] Responses API - Add session management support for non-openai models (#10321)
Some checks failed
Read Version from pyproject.toml / read-version (push) Successful in 49s
Helm unit test / unit-test (push) Successful in 55s
Publish Prisma Migrations / publish-migrations (push) Failing after 1m57s
* add session id in spendLogs

* fix log proxy server request as independant field

* use trace id for SpendLogs

* add _ENTERPRISE_ResponsesSessionHandler

* use _ENTERPRISE_ResponsesSessionHandler

* working session_ids

* working session management

* working session_ids

* test_async_gcs_pub_sub_v1

* test_spend_logs_payload_e2e

* working session_ids

* test_get_standard_logging_payload_trace_id

* test_get_standard_logging_payload_trace_id

* test_gcs_pub_sub.py

* fix all linting errors

* test_spend_logs_payload_with_prompts_enabled

* _ENTERPRISE_ResponsesSessionHandler

* _ENTERPRISE_ResponsesSessionHandler

* expose session id on ui

* get spend logs by session

* add sessionSpendLogsCall

* add session handling

* session logs

* ui session details

* fix on rowExpandDetails

* ui working sessions
2025-04-25 23:24:24 -07:00
Krish Dholakia
c66c821f96
UI (Keys Page) - Support cross filtering, filter by user id, filter by key hash (#10322)
* feat(filter.tsx): initial commit making filter component more generic - same style as user table filters

* refactor(all_keys_table.tsx): refactor to simplify update logic

* fix: partially revert changes - reduce scope of pr

* fix(filter_logic.tsx): fix filter update logic

* fix(all_keys_table.tsx): fix filtering + search logic

* refactor: cleanup unused params

* Revert "fix(all_keys_table.tsx): fix filtering + search logic"

This reverts commit 5fbc331970.

* feat(filter_logic.tsx): allow filter by user id

* fix(key_management_endpoints.py): support filtering `/key/list` by  key hash

Enables lookup by key hash on ui

* fix(key_list.tsx): fix update

* fix(key_management.py): fix linting error

* test: update testing

* fix(prometheus.py): fix key hash

* style(all_keys_table.tsx): style improvements

* test: fix test
2025-04-25 21:30:56 -07:00
Krish Dholakia
70f7c73def
Move UI to encrypted token usage (#10302)
All checks were successful
Read Version from pyproject.toml / read-version (push) Successful in 25s
Helm unit test / unit-test (push) Successful in 33s
* test(test_auth_checks.py): add unit tests for ExperimentalUIJWTToken

* test: add appropriate flag

* fix: fix ruff check

* test: add autouse fixture to test salt key

* fix(user_api_key_auth.py): fix auth flow logic

* test: skip flaky test - anthropic does not reliably return 'redacted_thinking'
2025-04-25 18:20:41 -07:00
Krish Dholakia
0f9ebc23a5
Fix SSO user login - invalid token error (#10298)
All checks were successful
Read Version from pyproject.toml / read-version (push) Successful in 18s
Helm unit test / unit-test (push) Successful in 24s
* fix(ui_sso.py): add info statements for litellm sso

* fix(ui_sso.py): use correct user id on existing user sso login

* refactor(ui_sso.py): break down function for easier testing

* test(test_ui_sso.py): add unit testing

* fix(ui_sso.py): fix passing user id from openid

* fix(ui_sso.py): fix returning user email

* fix(ui_sso.py): pass sso id on new sso user create

better tracking of when user is an sso user

* fix(ui_sso.py): don't auto create key for sso user

* docs(internal_user_endpoints.py): add 'sso_user_id' docstring
2025-04-25 09:48:54 -07:00
Ishaan Jaff
2e9fb2f6ba # expect an error when getting the response again since 2025-04-25 09:42:35 -07:00
Ishaan Jaff
8ed3557ce7 test_init_responses_api_endpoints
All checks were successful
Read Version from pyproject.toml / read-version (push) Successful in 17s
Helm unit test / unit-test (push) Successful in 24s
2025-04-24 21:43:59 -07:00
Ishaan Jaff
cb87dbbd51 fix responses test 2025-04-24 21:23:25 -07:00
Ishaan Jaff
5a8b35c3c5
Support max_completion_tokens on Sagemaker (#10243) (#10300)
Co-authored-by: Pranav Simha <pranav.simha@alteryx.com>
2025-04-24 21:15:54 -07:00
Ishaan Jaff
164017119d
[Bug Fix] Timestamp Granularities are not properly passed to whisper in Azure (#10299)
All checks were successful
Read Version from pyproject.toml / read-version (push) Successful in 23s
Helm unit test / unit-test (push) Successful in 29s
* test fix form data parsing

* test fix form data parsing

* fix types
2025-04-24 18:57:11 -07:00
Ishaan Jaff
5de101ab7b
[Feat] Add GET, DELETE Responses endpoints on LiteLLM Proxy (#10297)
* add GET responses endpoints on router

* add GET responses endpoints on router

* add GET responses endpoints on router

* add DELETE responses endpoints on proxy

* fixes for testing GET, DELETE endpoints

* test_basic_responses api e2e
2025-04-24 17:34:26 -07:00
Krrish Dholakia
2adb2fc6a5 test: handle service unavailable error 2025-04-23 22:10:46 -07:00
Krish Dholakia
be4152c8d5
UI - fix edit azure public model name + fix editing model name post create
* test(test_router.py): add unit test confirming fallbacks with tag based routing works as expected

* test: update testing

* test: update test to not use gemini-pro

google removed it

* fix(conditional_public_model_name.tsx): edit azure public model name

Fixes https://github.com/BerriAI/litellm/issues/10093

* fix(model_info_view.tsx): migrate to patch model updates

Enables changing model name easily
2025-04-23 21:56:56 -07:00
Krish Dholakia
acd2c1783c
fix(converse_transformation.py): support all bedrock - openai params for arn models (#10256)
Fixes https://github.com/BerriAI/litellm/issues/10207
2025-04-23 21:56:05 -07:00
Krrish Dholakia
2486a106f4 test: mark flaky tests 2025-04-23 21:50:25 -07:00
Ishaan Jaff
dc9b058dbd
[Feat] Add support for GET Responses Endpoint - OpenAI, Azure OpenAI (#10235)
* Added get responses API (#10234)

* test_basic_openai_responses_get_endpoint

* transform_get_response_api_request

* test_basic_openai_responses_get_endpoint

---------

Co-authored-by: Prathamesh Saraf <pratamesh1867@gmail.com>
2025-04-23 15:19:29 -07:00
Ishaan Jaff
2e58e47b43
[Bug Fix] Add Cost Tracking for gpt-image-1 when quality is unspecified (#10247)
* TestOpenAIGPTImage1

* fixes for cost calc

* fix ImageGenerationRequestQuality.MEDIUM
2025-04-23 15:16:40 -07:00
Krrish Dholakia
a649f10e63 test: update test to not use gemini-pro
google removed it
2025-04-23 11:31:09 -07:00
Krrish Dholakia
8184124217 test: update testing 2025-04-23 11:21:50 -07:00
Krrish Dholakia
174a1aa007 test: update test
All checks were successful
Read Version from pyproject.toml / read-version (push) Successful in 16s
Helm unit test / unit-test (push) Successful in 25s
2025-04-23 10:51:18 -07:00
Christian Owusu
47420d8d68
Require auth for all dashboard pages (#10229)
* Require authentication for all Dashboard pages

* Add test

* Add test
2025-04-23 07:08:25 -07:00
Krrish Dholakia
f5996b2f6b test: update test to skip 'gemini-pro' - model deprecated
All checks were successful
Read Version from pyproject.toml / read-version (push) Successful in 19s
Helm unit test / unit-test (push) Successful in 32s
2025-04-23 00:01:02 -07:00
Krish Dholakia
217681eb5e
Litellm dev 04 22 2025 p1 (#10206)
* fix(openai.py): initial commit adding generic event type for openai responses api streaming

Ensures handling for undocumented event types - e.g. "response.reasoning_summary_part.added"

* fix(transformation.py): handle unknown openai response type

* fix(datadog_llm_observability.py): handle dict[str, any] -> dict[str, str] conversion

Fixes https://github.com/BerriAI/litellm/issues/9494

* test: add more unit testing

* test: add unit test

* fix(common_utils.py): fix message with content list

* test: update testing
2025-04-22 23:58:43 -07:00
Krish Dholakia
f670ebeb2f
Users page - new user info pane (#10213)
* feat(user_info_view.tsx): be able to click in and see all teams user is part of

makes it easy to see which teams a user belongs to

* test(ui/): add unit testing for user info view

* fix(user_info_view.tsx): fix linting errors

* fix(login.ts): fix login

* fix: fix linting error
2025-04-22 21:55:47 -07:00
Ishaan Jaff
96e31d205c
feat: Added Missing Attributes For Arize & Phoenix Integration (#10043) (#10215)
* feat: Added Missing Attributes For Arize & Phoenix Integration

* chore: Added noqa for PLR0915 to suppress warning

* chore: Moved Contributor Test to Correct Location

* chore: Removed Redundant Fallback

Co-authored-by: Ali Saleh <saleh.a@turing.com>
2025-04-22 21:34:51 -07:00
Krish Dholakia
5f98d4d7de
UI - Users page - Enable global sorting (allows finding users with highest spend) (#10211)
* fix(view_users.tsx): add time tracking logic to debounce search - prevent new queries from being overwritten by previous ones

* fix(internal_user_endpoints.py): add sort functionality to user list endpoint

* feat(internal_user_endpoints.py): support sort by on `/user/list`

* fix(view_users.tsx): enable global sorting

allows finding user with highest spend

* feat(view_users.tsx): support filtering by sso user id

* test(search_users.spec.ts): add tests to ensure filtering works

* test: add more unit testing
2025-04-22 19:59:53 -07:00
Ishaan Jaff
0dba2886f0 fix test 2025-04-22 18:37:56 -07:00
Ishaan Jaff
868cdd0226
[Feat] Add Support for DELETE /v1/responses/{response_id} on OpenAI, Azure OpenAI (#10205)
* add transform_delete_response_api_request to base responses config

* add transform_delete_response_api_request

* add delete_response_api_handler

* fixes for deleting responses, response API

* add adelete_responses

* add async test_basic_openai_responses_delete_endpoint

* test_basic_openai_responses_delete_endpoint

* working delete for streaming on responses API

* fixes azure transformation

* TestAnthropicResponsesAPITest

* fix code check

* fix linting

* fixes for get_complete_url

* test_basic_openai_responses_streaming_delete_endpoint

* streaming fixes
2025-04-22 18:27:03 -07:00
Ishaan Jaff
44264ab6d6 fix failing agent ops test 2025-04-22 14:39:50 -07:00
Krish Dholakia
66680c421d
Add global filtering to Users tab (#10195)
* style(internal_user_endpoints.py): add response model to `/user/list` endpoint

make sure we maintain consistent response spec

* fix(key_management_endpoints.py): return 'created_at' and 'updated_at' on `/key/generate`

Show 'created_at' on UI when key created

* test(test_keys.py): add e2e test to ensure created at is always returned

* fix(view_users.tsx): support global search by user email

allows easier search

* test(search_users.spec.ts): add e2e test ensure user search works on admin ui

* fix(view_users.tsx): support filtering user by role and user id

More powerful filtering on internal users table

* fix(view_users.tsx): allow filtering users by team

* style(view_users.tsx): cleanup ui to show filters in consistent style

* refactor(view_users.tsx): cleanup to just use 1 variable for the data

* fix(view_users.tsx): cleanup use effect hooks

* fix(internal_user_endpoints.py): fix check to pass testing

* test: update tests

* test: update tests

* Revert "test: update tests"

This reverts commit 6553eeb232.

* fix(view_userts.tsx): add back in 'previous' and 'next' tabs for pagination
2025-04-22 13:59:43 -07:00
Dwij
b2955a2bdd
Add AgentOps Integration to LiteLLM (#9685)
* feat(sidebars): add new item for agentops integration in Logging & Observability category

* Update agentops_integration.md to enhance title formatting and remove redundant section

* Enhance AgentOps integration in documentation and codebase by removing LiteLLMCallbackHandler references, adding environment variable configurations, and updating logging initialization for AgentOps support.

* Update AgentOps integration documentation to include instructions for obtaining API keys and clarify environment variable setup.

* Add unit tests for AgentOps integration and improve error handling in token fetching

* Add unit tests for AgentOps configuration and token fetching functionality

* Corrected agentops test directory

* Linting fix

* chore: add OpenTelemetry dependencies to pyproject.toml

* chore: update OpenTelemetry dependencies and add new packages in pyproject.toml and poetry.lock
2025-04-22 10:29:01 -07:00
Krish Dholakia
a7db0df043
Gemini-2.5-flash improvements (#10198)
* fix(vertex_and_google_ai_studio_gemini.py): allow thinking budget = 0

Fixes https://github.com/BerriAI/litellm/issues/10121

* fix(vertex_and_google_ai_studio_gemini.py): handle nuance in counting exclusive vs. inclusive tokens

Addresses https://github.com/BerriAI/litellm/pull/10141#discussion_r2052272035
2025-04-21 22:48:00 -07:00
Ishaan Jaff
7cb95bcc96
[Bug Fix] caching does not account for thinking or reasoning_effort config (#10140)
* _get_litellm_supported_chat_completion_kwargs

* test caching with thinking
2025-04-21 22:39:40 -07:00
Ishaan Jaff
104e4cb1bc
[Feat] Add infinity embedding support (contributor pr) (#10196)
* Feature - infinity support for #8764 (#10009)

* Added support for infinity embeddings

* Added test cases

* Fixed tests and api base

* Updated docs and tests

* Removed unused import

* Updated signature

* Added support for infinity embeddings

* Added test cases

* Fixed tests and api base

* Updated docs and tests

* Removed unused import

* Updated signature

* Updated validate params

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>

* fix InfinityEmbeddingConfig

---------

Co-authored-by: Prathamesh Saraf <pratamesh1867@gmail.com>
2025-04-21 20:01:29 -07:00
Ishaan Jaff
0c2f705417
[Feat] Add Responses API - Routing Affinity logic for sessions (#10193)
* test for test_responses_api_routing_with_previous_response_id

* test_responses_api_routing_with_previous_response_id

* add ResponsesApiDeploymentCheck

* ResponsesApiDeploymentCheck

* ResponsesApiDeploymentCheck

* fix ResponsesApiDeploymentCheck

* test_responses_api_routing_with_previous_response_id

* ResponsesApiDeploymentCheck

* test_responses_api_deployment_check.py

* docs routing affinity

* simplify ResponsesApiDeploymentCheck

* test response id

* fix code quality check
2025-04-21 20:00:27 -07:00
Ishaan Jaff
4eac0f64f3
[Feat] Pass through endpoints - ensure PassthroughStandardLoggingPayload is logged and contains method, url, request/response body (#10194)
* ensure passthrough_logging_payload is filled in kwargs

* test_assistants_passthrough_logging

* test_assistants_passthrough_logging

* test_assistants_passthrough_logging

* test_threads_passthrough_logging

* test _init_kwargs_for_pass_through_endpoint

* _init_kwargs_for_pass_through_endpoint
2025-04-21 19:46:22 -07:00
Krish Dholakia
89131d8ed3
Remove user_id from url (#10192)
* fix(user_dashboard.tsx): initial commit using user id from jwt instead of url

* fix(proxy_server.py): remove user id from url

fixes security issue around sharing url's

* fix(user_dashboard.tsx): handle user id being null
2025-04-21 16:22:57 -07:00
Krish Dholakia
0c3b7bb37d
fix(router.py): handle edge case where user sets 'model_group' inside… (#10191)
* fix(router.py): handle edge case where user sets 'model_group' inside 'model_info'

* fix(key_management_endpoints.py): security fix - return hashed token in 'token' field

Ensures when creating a key on UI - only hashed token shown

* test(test_key_management_endpoints.py): add unit test

* test: update test
2025-04-21 16:17:45 -07:00
Nilanjan De
03245c732a
Fix: Potential SQLi in spend_management_endpoints.py (#9878)
* fix: Potential SQLi in spend_management_endpoints.py

* fix tests

* test: add tests for global spend keys endpoint

* chore: update error message

* chore: lint

* chore: rename test
2025-04-21 14:29:38 -07:00
Li Yang
10257426a2
fix(bedrock): wrong system prompt transformation (#10120)
All checks were successful
Read Version from pyproject.toml / read-version (push) Successful in 16s
Helm unit test / unit-test (push) Successful in 25s
* fix(bedrock): wrong system transformation

* chore: add one more test case

---------

Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
2025-04-21 08:48:14 -07:00
Krish Dholakia
e0a613f88a
fix(common_daily_activity.py): support empty entity id field (#10175)
* fix(common_daily_activity.py): support empty entity id field

allows returning empty response when user is not admin and does not belong to any team

* test(test_common_daily_activity.py): add unit testing
2025-04-19 22:20:28 -07:00
Krish Dholakia
55a17730fb
fix(transformation.py): pass back in gemini thinking content to api (#10173)
Ensures thinking content always returned
2025-04-19 18:03:05 -07:00
Ishaan Jaff
653570824a
Bug Fix - Responses API, Loosen restrictions on allowed environments for computer use tool (#10168)
* loosen allowed types on ComputerToolParam

* test_basic_computer_use_preview_tool_call
2025-04-19 14:40:32 -07:00
Ishaan Jaff
b0024bb229
[Bug Fix] Spend Tracking Bug Fix, don't modify in memory default litellm params (#10167)
* _update_kwargs_with_default_litellm_params

* test_update_kwargs_does_not_mutate_defaults_and_merges_metadata
2025-04-19 14:13:59 -07:00
Ishaan Jaff
0717369ae6
[Feat] Expose Responses API on LiteLLM UI Test Key Page (#10166)
* add /responses API on UI

* add makeOpenAIResponsesRequest

* add makeOpenAIResponsesRequest

* fix add responses API on UI

* fix endpoint selector

* responses API render chunks on litellm chat ui

* fixes to streaming iterator

* fix render responses completed events

* fixes for MockResponsesAPIStreamingIterator

* transform_responses_api_request_to_chat_completion_request

* fix for responses API

* test_basic_openai_responses_api_streaming

* fix base responses api tests
2025-04-19 13:18:54 -07:00
Krish Dholakia
03b5399f86
test(utils.py): handle scenario where text tokens + reasoning tokens … (#10165)
* test(utils.py): handle scenario where text tokens + reasoning tokens set, but reasoning tokens not charged separately

Addresses https://github.com/BerriAI/litellm/pull/10141#discussion_r2051555332

* fix(vertex_and_google_ai_studio.py): only set content if non-empty str
2025-04-19 12:32:38 -07:00
Krish Dholakia
5c929317cd
fix(triton/completion/transformation.py): remove bad_words / stop wor… (#10163)
* fix(triton/completion/transformation.py): remove bad_words / stop words from triton call

parameter 'bad_words' has invalid type. It should be either 'int', 'bool', or 'string'.

* fix(proxy_track_cost_callback.py): add debug logging for track cost callback error
2025-04-19 11:23:37 -07:00