Commit graph

1962 commits

Author SHA1 Message Date
Krish Dholakia
0865e52db3
fix(proxy_server.py): get master key from environment, if not set in … (#9617)
* fix(proxy_server.py): get master key from environment, if not set in general settings or general settings not set at all

* test: mark flaky test

* test(test_proxy_server.py): mock prisma client

* ci: add new github workflow for testing just the mock tests

* fix: fix linting error

* ci(conftest.py): add conftest.py to isolate proxy tests

* build(pyproject.toml): add respx to dev dependencies

* build(pyproject.toml): add prisma to dev dependencies

* test: fix mock prompt management tests to use a mock anthropic key

* ci(test-litellm.yml): parallelize mock testing

make it run faster

* build(pyproject.toml): add hypercorn as dev dep

* build(pyproject.toml): separate proxy vs. core dev dependencies

make it easier for non-proxy contributors to run tests locally - e.g. no need to install hypercorn

* ci(test-litellm.yml): pin python version

* test(test_rerank.py): move test - cannot be mocked, requires aws credentials for e2e testing

* ci: add thank you message to ci

* test: add mock env var to test

* test: add autouse to tests

* test: test mock env vars for e2e tests
2025-03-28 12:32:04 -07:00
Krrish Dholakia
3226477464 fix: cleanup 2025-03-27 23:31:12 -07:00
Krrish Dholakia
b155f4f4a6 fix(proxy_server.py): Fix https://github.com/BerriAI/litellm/issues/9576 2025-03-27 23:25:46 -07:00
Krish Dholakia
fb83567a03
Litellm new UI build (#9601)
* build: new ui build

* build: new ui build

* fix(proxy_server.py): only show user models their key can access on `/models`

* fix(model_management_endpoints.py): ensure team admin can add models

* test: update unit testing to reflect changes

* fix(model_dashboard.tsx): fix sizing on models page

* build: fix ui
2025-03-27 17:15:25 -07:00
Krish Dholakia
63c9f59373
Allow team admins to add/update/delete models on UI + show api base and model id on request logs (#9572)
* feat(view_logs.tsx): show model id + api base in request logs

easier debugging

* fix(index.tsx): fix length of api base

easier viewing

* refactor(leftnav.tsx): show models tab to team admin

* feat(model_dashboard.tsx): add explainer for what the 'models' page is for team admin

helps them understand how they can use it

* feat(model_management_endpoints.py): restrict model add by team to just team admin

allow team admin to add models via non-team keys (e.g. ui token)

* test(test_add_update_models.py): update unit testing for new behaviour

* fix(model_dashboard.tsx): show user the models

* feat(proxy_server.py): add new query param 'user_models_only' to `/v2/model/info`

Allows user to retrieve just the models they've added

Used in UI to show internal users just the models they've added

* feat(model_dashboard.tsx): allow team admins to view their own models

* fix: allow ui user to fetch model cost map

* feat(add_model_tab.tsx): require team admins to specify team when onboarding models

* fix(_types.py): add `/v1/model/info` to info route

`/model/info` was already there

* fix(model_info_view.tsx): allow user to edit a model they created

* fix(model_management_endpoints.py): allow team admin to update team model

* feat(model_managament_endpoints.py): allow team admin to delete team models

* fix(model_management_endpoints.py): don't require team id to be set when adding a model

* fix(proxy_server.py): fix linting error

* fix: fix ui linting error

* fix(model_management_endpoints.py): ensure consistent auth checks on all model calls

* test: remove old test - function no longer exists in same form

* test: add updated mock testing
2025-03-27 12:06:31 -07:00
Ishaan Jaff
34e58be36d fix order of _setup_prisma_client 2025-03-26 20:42:01 -07:00
Ishaan Jaff
427580eff5 fix _setup_prisma_client 2025-03-26 20:35:27 -07:00
Ishaan Jaff
15c04da735 refactor tests 2025-03-26 20:00:10 -07:00
Ishaan Jaff
88ef97b9d1 allow proxy to startup on DB unavailable 2025-03-26 19:50:57 -07:00
Krish Dholakia
37265c8b1c
Add Daily User Spend Aggregate view - allows UI Usage tab to work > 1m rows (#9538)
* ci: update github action

* build(schema.prisma): enable daily user spend table

allows storing aggregate view of user's daily spend

* build(schema.prisma): add new daily user spend table

* feat: working daily user spend tracking

maintains an aggregate view for easier querying in high traffic

* setup_google_dns

* ci: update ci yaml

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
2025-03-26 16:36:36 -07:00
Ishaan Jaff
08a4ba1b7e Merge branch 'main' into litellm_exp_mcp_server 2025-03-24 19:03:56 -07:00
Tyler Hutcherson
7864cd1f76 update redisvl dependency 2025-03-24 08:42:11 -04:00
Ishaan Jaff
f21a0c2da7 Merge branch 'main' into litellm_exp_mcp_server 2025-03-22 18:51:25 -07:00
Krrish Dholakia
790e063044 fix: fix linting error 2025-03-22 17:56:39 -07:00
Krrish Dholakia
59e14fc45c fix(router.py): fix get_model_list to return all wildcard models
enables viewing all wildcard models on `/model/info`
2025-03-22 15:39:23 -07:00
Krrish Dholakia
b44b3bd36b feat(llm_passthrough_endpoints.py): base case passing for refactored vertex passthrough route 2025-03-22 11:06:52 -07:00
Ishaan Jaff
fd95499e15 fix naming 2025-03-20 21:57:28 -07:00
Ishaan Jaff
2afa73d584 fix mcp router 2025-03-20 18:05:14 -07:00
Ishaan Jaff
dc33cfa074 init global_mcp_tool_registry 2025-03-20 17:53:37 -07:00
Krrish Dholakia
c13f6514bf fix(proxy_server.py): move db credential check to only run behind feature flag 2025-03-19 11:35:14 -07:00
Ishaan Jaff
de97cda445 refactor create_litellm_proxy_request_started_spen 2025-03-18 16:12:16 -07:00
Ishaan Jaff
537bed127b expose flag to disable_spend_updates 2025-03-17 20:45:49 -07:00
Krish Dholakia
d4caaae1be
Merge pull request #9274 from BerriAI/litellm_contributor_rebase_branch
All checks were successful
Read Version from pyproject.toml / read-version (push) Successful in 43s
Helm unit test / unit-test (push) Successful in 50s
Litellm contributor rebase branch
2025-03-14 21:57:49 -07:00
Ishaan Jaff
12188f286c
Merge pull request #9258 from BerriAI/litellm_fix_models_view_edit
(UI) Fix model edit + delete - instantly show edit + deletes to models
2025-03-14 21:21:46 -07:00
Krrish Dholakia
26226d475f feat(proxy_server.py): support retrieving models for a team, if user is a member - via /models?team_id
Allows user to see team models on UI when creating a key
2025-03-14 19:34:06 -07:00
Ishaan Jaff
c36e5cae50 backend instant delete model 2025-03-14 10:12:50 -07:00
Krrish Dholakia
dc3b02920f feat(model_management_endpoints.py): support audit logs on /model/add and /model/update endpoints
complete CUD endpoint audit logging on models + users
2025-03-13 19:17:40 -07:00
Krrish Dholakia
37b30395c9 feat(model_management_endpoints.py): emit audit logs on model delete 2025-03-13 18:48:38 -07:00
Tomer Bin
4a31b32a88 Support post-call guards for stream and non-stream responses 2025-03-13 08:53:54 +02:00
Krish Dholakia
1f2bddbc21
Merge pull request #9186 from BerriAI/litellm_dev_03_12_2025_p2
feat: initial commit adding support for credentials on proxy ui
2025-03-12 22:12:49 -07:00
Ishaan Jaff
db1f48bbfb responses_api 2025-03-12 20:38:05 -07:00
Krrish Dholakia
d604f52884 feat: refactor add models tab on UI to enable setting credentials 2025-03-12 20:32:01 -07:00
Ishaan Jaff
98718f6f09 working responses_api 2025-03-12 17:26:12 -07:00
Ishaan Jaff
f2041762e6 add responses_api 2025-03-12 17:08:16 -07:00
Ishaan Jaff
b2d9935567 use ProxyBaseLLMRequestProcessing 2025-03-12 16:54:33 -07:00
Krish Dholakia
2d957a0ed9
Merge branch 'main' into litellm_dev_03_10_2025_p3 2025-03-12 14:56:01 -07:00
Krrish Dholakia
b8d590da0c fix(azure/audio_transcriptions.py): support azure cost tracking
extract content time and log correctly as duration
2025-03-11 22:25:13 -07:00
Krrish Dholakia
01aa536716 fix: fix test 2025-03-10 21:46:05 -07:00
Krrish Dholakia
f56c5ca380 feat: working e2e credential management - support reusing existing credentials 2025-03-10 19:29:24 -07:00
Krrish Dholakia
f1cdc26967 feat(endpoints.py): initial set of crud endpoints for reusable credentials on proxy 2025-03-10 17:48:02 -07:00
Krrish Dholakia
4bd4bb16fd feat(proxy_server.py): move credential list to being a top-level param 2025-03-10 17:04:05 -07:00
Krrish Dholakia
f688fc8138 feat(proxy_server.py): check code before defaulting to status code 2025-03-10 15:34:06 -07:00
Ishaan Jaff
73df319f4e
(Clean up) - Allow switching off storing Error Logs in DB (#9084)
* fix - cleanup, dont store ErrorLogs in 2 tables

* async_post_call_failure_hook

* docs disable error logs

* disable_error_logs
2025-03-08 16:12:03 -08:00
Krish Dholakia
0e3caf92b9
UI - new API Playground for testing LiteLLM translation (#9073)
* feat: initial commit - enable dev to see translated request

* feat(utils.py): expose new endpoint - `/utils/transform_request` to see the raw request sent by litellm

* feat(transform_request.tsx): allow user to see their transformed request

* refactor(litellm_logging.py): return raw request in 3 parts - api_base, headers, request body

easier to render each individually on UI vs. extracting from combined string

* feat: transform_request.tsx

working e2e raw request viewing

* fix(litellm_logging.py): fix transform viewing for bedrock models

* fix(litellm_logging.py): don't return sensitive headers in raw request headers

prevent accidental leak

* feat(transform_request.tsx): style improvements
2025-03-07 19:39:31 -08:00
Krish Dholakia
5591354309
Support master key rotations (#9041)
* feat(key_management_endpoints.py): adding support for rotating master key

* feat(key_management_endpoints.py): support decryption-re-encryption of models in db, when master key rotated

* fix(user_api_key_auth.py): raise valid token is None error earlier

enables easier debugging with api key hash in error message

* feat(key_management_endpoints.py): rotate any env vars

* fix(key_management_endpoints.py): uncomment check

* fix: fix linting error
2025-03-06 23:13:30 -08:00
Ishaan Jaff
f47987e673
(Refactor) /v1/messages to follow simpler logic for Anthropic API spec (#9013)
* anthropic_messages_handler v0

* fix /messages

* working messages with router methods

* test_anthropic_messages_handler_litellm_router_non_streaming

* test_anthropic_messages_litellm_router_non_streaming_with_logging

* AnthropicMessagesConfig

* _handle_anthropic_messages_response_logging

* working with /v1/messages endpoint

* working /v1/messages endpoint

* refactor to use router factory function

* use aanthropic_messages

* use BaseConfig for Anthropic /v1/messages

* track api key, team on /v1/messages endpoint

* fix get_logging_payload

* BaseAnthropicMessagesTest

* align test config

* test_anthropic_messages_with_thinking

* test_anthropic_streaming_with_thinking

* fix - display anthropic url for debugging

* test_bad_request_error_handling

* test_anthropic_messages_router_streaming_with_bad_request

* fix ProxyException

* test_bad_request_error_handling_streaming

* use provider_specific_header

* test_anthropic_messages_with_extra_headers

* test_anthropic_messages_to_wildcard_model

* fix gcs pub sub test

* standard_logging_payload

* fix unit testing for anthopic /v1/messages support

* fix pass through anthropic messages api

* delete dead code

* fix anthropic pass through response

* revert change to spend tracking utils

* fix get_litellm_metadata_from_kwargs

* fix spend logs payload json

* proxy_pass_through_endpoint_tests

* TestAnthropicPassthroughBasic

* fix pass through tests

* test_async_vertex_proxy_route_api_key_auth

* _handle_anthropic_messages_response_logging

* vertex_credentials

* test_set_default_vertex_config

* test_anthropic_messages_litellm_router_non_streaming_with_logging

* test_ageneric_api_call_with_fallbacks_basic

* test__aadapter_completion
2025-03-06 00:43:08 -08:00
Ishaan Jaff
8d6815ce98 Revert "(UI) - Security Improvement, move to JWT Auth for Admin UI Sessions (#8995)"
This reverts commit 01a44a4e47.
2025-03-05 08:49:20 -08:00
Krish Dholakia
5e386c28b2
Litellm dev 03 04 2025 p3 (#8997)
* fix(core_helpers.py): handle litellm_metadata instead of 'metadata'

* feat(batches/): ensure batches logs are written to db

makes batches response dict compatible

* fix(cost_calculator.py): handle batch response being a dictionary

* fix(batches/main.py): modify retrieve endpoints to use @client decorator

enables logging to work on retrieve call

* fix(batches/main.py): fix retrieve batch response type to be 'dict' compatible

* fix(spend_tracking_utils.py): send unique uuid for retrieve batch call type

create batch and retrieve batch share the same id

* fix(spend_tracking_utils.py): prevent duplicate retrieve batch calls from being double counted

* refactor(batches/): refactor cost tracking for batches - do it on retrieve, and within the established litellm_logging pipeline

ensures cost is always logged to db

* fix: fix linting errors

* fix: fix linting error
2025-03-04 21:58:03 -08:00
Ishaan Jaff
01a44a4e47
(UI) - Security Improvement, move to JWT Auth for Admin UI Sessions (#8995)
* (UI) - Improvements to session handling logic  (#8970)

* add cookieUtils

* use utils for clearing cookies

* on logout use clearTokenCookies

* ui use correct clearTokenCookies

* navbar show userEmail on UserID page

* add timestamp on token cookie

* update generate_authenticated_redirect_response

* use common getAuthToken

* fix clearTokenCookies

* fixes for get auth token

* fix invitation link sign in logic

* Revert "fix invitation link sign in logic"

This reverts commit 30e5308cb3.

* fix getAuthToken

* update setAuthToken

* fix ui session handling

* fix ui session handler

* bug fix stop generating LiteLLM Virtual keys for access

* working JWT insert into cookies

* use central place to build UI JWT token

* add _validate_ui_token

* fix ui session handler

* fix fetchWithCredentials

* check allowed routes for ui session tokens

* expose validate_session endpoint

* validate session endpoint

* call sso/session/validate

* getUISessionDetails

* ui move to getUISessionDetails

* /sso/session/validate

* fix cookie utils

* use getUISessionDetails

* use ui_session_id

* "/spend/logs/ui" in spend_tracking_routes

* working sign in JWT flow for proxy admin

* allow proxy admin to access ui routes

* use check_route_access

* update types

* update login method

* fixes to ui session handler

* working flow for admin and internal users

* fixes for invite links

* use JWTs for SSO sign in

* fix /invitation/new flow

* fix code quality checks

* fix _get_ui_session_token_from_cookies

* /organization/list

* ui sso sign in

* TestUISessionHandler

* TestUISessionHandler
2025-03-04 21:48:23 -08:00
Krish Dholakia
f1a44d1fdc
fix(common_utils.py): handle $id in response schema when calling vert… (#8991)
* fix(common_utils.py): handle $id in response schema when calling vertex ai

Fixes issue where `$id` present in response_schema was not accepted by vertex ai

* test(test_vertex.py): add unit test to ensure $id stripped out of vertex schema
2025-03-04 21:19:50 -08:00