Commit graph

4653 commits

Author SHA1 Message Date
Ishaan Jaff
decb6649ec test_queue_flush_limit 2025-04-04 16:29:06 -07:00
Ishaan Jaff
e77a178a37 test_queue_size_reduction_with_large_volume 2025-04-04 16:21:29 -07:00
Ishaan Jaff
eb48cbdec6 aggregate_queue_updates 2025-04-04 15:54:07 -07:00
Ishaan Jaff
cdd351a03b
Merge pull request #9745 from BerriAI/litellm_sso_fixes_dev
[Feat] Allow assigning SSO users to teams on MSFT SSO
2025-04-04 15:40:19 -07:00
Ishaan Jaff
93068cb142 flush_all_updates_from_in_memory_queue 2025-04-04 15:34:56 -07:00
Ishaan Jaff
065477abb4 add _get_aggregated_spend_update_queue_item 2025-04-04 15:32:27 -07:00
Ishaan Jaff
9abaefea62 add logic for max size in memory queue 2025-04-04 15:31:40 -07:00
Ishaan Jaff
363fb0c46f add MAX_SIZE_IN_MEMORY_QUEUE 2025-04-04 15:31:09 -07:00
Ishaan Jaff
cba1dacc7d ui new build 2025-04-04 14:39:55 -07:00
Ishaan Jaff
984114adf0 fix sso callback 2025-04-03 22:13:46 -07:00
Ishaan Jaff
f1bc99a137 MSFT make it easier for using group ids with MSFT 2025-04-03 20:43:22 -07:00
Ishaan Jaff
add24d5999 debug show SSO callback result 2025-04-03 20:06:21 -07:00
Krish Dholakia
6dda1ba6dd
LiteLLM Minor Fixes & Improvements (04/02/2025) (#9725)
* Add date picker to usage tab + Add reasoning_content token tracking across all providers on streaming (#9722)

* feat(new_usage.tsx): add date picker for new usage tab

allow user to look back on their usage data

* feat(anthropic/chat/transformation.py): report reasoning tokens in completion token details

allows usage tracking on how many reasoning tokens are actually being used

* feat(streaming_chunk_builder.py): return reasoning_tokens in anthropic/openai streaming response

allows tracking reasoning_token usage across providers

* Fix update team metadata + fix bulk adding models on Ui  (#9721)

* fix(handle_add_model_submit.tsx): fix bulk adding models

* fix(team_info.tsx): fix team metadata update

Fixes https://github.com/BerriAI/litellm/issues/9689

* (v0) Unified file id - allow calling multiple providers with same file id (#9718)

* feat(files_endpoints.py): initial commit adding 'target_model_names' support

allow developer to specify all the models they want to call with the file

* feat(files_endpoints.py): return unified files endpoint

* test(test_files_endpoints.py): add validation test - if invalid purpose submitted

* feat: more updates

* feat: initial working commit of unified file id translation

* fix: additional fixes

* fix(router.py): remove model replace logic in jsonl on acreate_file

enables file upload to work for chat completion requests as well

* fix(files_endpoints.py): remove whitespace around model name

* fix(azure/handler.py): return acreate_file with correct response type

* fix: fix linting errors

* test: fix mock test to run on github actions

* fix: fix ruff errors

* fix: fix file too large error

* fix(utils.py): remove redundant var

* test: modify test to work on github actions

* test: update tests

* test: more debug logs to understand ci/cd issue

* test: fix test for respx

* test: skip mock respx test

fails on ci/cd - not clear why

* fix: fix ruff check

* fix: fix test

* fix(model_connection_test.tsx): fix linting error

* test: update unit tests
2025-04-03 11:48:52 -07:00
Ishaan Jaff
dd2d1dc2f4 Merge branch 'main' into litellm_metrics_pod_lock_manager 2025-04-02 21:35:55 -07:00
Krish Dholakia
8ee32291e0
Squashed commit of the following: (#9709)
commit b12a9892b7
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Wed Apr 2 08:09:56 2025 -0700

    fix(utils.py): don't modify openai_token_counter

commit 294de31803
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Mon Mar 24 21:22:40 2025 -0700

    fix: fix linting error

commit cb6e9fbe40
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Mon Mar 24 19:52:45 2025 -0700

    refactor: complete migration

commit bfc159172d
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Mon Mar 24 19:09:59 2025 -0700

    refactor: refactor more constants

commit 43ffb6a558
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Mon Mar 24 18:45:24 2025 -0700

    fix: test

commit 04dbe4310c
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Mon Mar 24 18:28:58 2025 -0700

    refactor: refactor: move more constants into constants.py

commit 3c26284aff
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Mon Mar 24 18:14:46 2025 -0700

    refactor: migrate hardcoded constants out of __init__.py

commit c11e0de69d
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Mon Mar 24 18:11:21 2025 -0700

    build: migrate all constants into constants.py

commit 7882bdc787
Author: Krrish Dholakia <krrishdholakia@gmail.com>
Date:   Mon Mar 24 18:07:37 2025 -0700

    build: initial test banning hardcoded numbers in repo
2025-04-02 21:24:54 -07:00
Ishaan Jaff
5222cce510 Merge branch 'main' into litellm_metrics_pod_lock_manager 2025-04-02 21:04:44 -07:00
Ishaan Jaff
80fb4ece97 prom emit size of DB TX queues for observability 2025-04-02 18:39:29 -07:00
Ishaan Jaff
05b30e28db clean up service metrics 2025-04-02 17:50:41 -07:00
Ishaan Jaff
73bbd0a446 emit lock acquired and released events 2025-04-02 17:40:25 -07:00
Ishaan Jaff
8b12a2e5dc fix pod lock manager 2025-04-02 14:52:55 -07:00
Ishaan Jaff
a64631edfb test pod lock manager 2025-04-02 14:39:40 -07:00
Ishaan Jaff
b48b8366c2 docs new deadlock fixing architecture 2025-04-02 13:24:53 -07:00
Krish Dholakia
053b0e741f
Add Google AI Studio /v1/files upload API support (#9645)
All checks were successful
Read Version from pyproject.toml / read-version (push) Successful in 16s
Helm unit test / unit-test (push) Successful in 23s
* test: fix import for test

* fix: fix bad error string

* docs: cleanup files docs

* fix(files/main.py): cleanup error string

* style: initial commit with a provider/config pattern for files api

google ai studio files api onboarding

* fix: test

* feat(gemini/files/transformation.py): support gemini files api response transformation

* fix(gemini/files/transformation.py): return file id as gemini uri

allows id to be passed in to chat completion request, just like openai

* feat(llm_http_handler.py): support async route for files api on llm_http_handler

* fix: fix linting errors

* fix: fix model info check

* fix: fix ruff errors

* fix: fix linting errors

* Revert "fix: fix linting errors"

This reverts commit 926a5a527f.

* fix: fix linting errors

* test: fix test

* test: fix tests
2025-04-02 08:56:58 -07:00
Krish Dholakia
453003c378
fix(gemini/): add gemini/ route optional param mapping support (#9677)
Fixes https://github.com/BerriAI/litellm/issues/9654
2025-04-02 08:56:32 -07:00
Krish Dholakia
6c69ad4c89
fix(model_management_endpoints.py): fix allowing team admins to update team models (#9697)
All checks were successful
Read Version from pyproject.toml / read-version (push) Successful in 17s
Helm unit test / unit-test (push) Successful in 22s
* fix(model_management_endpoints.py): fix allowing team admins to update their models

* test(test_models.py): add e2e test to for team model flow

ensure team admin can always add / edit / delete team models
2025-04-01 22:28:15 -07:00
Krish Dholakia
3d0313b15b
Litellm user daily activity allow non admin usage (#9695)
* feat(internal_user_endpoints.py): allow non-admin to view their own usage via `/user/daily/activity` route

* fix(leftnav.tsx): allow users to view their own usage via new_usage.tsx

allows internal users to see their usage via new api

Handles 1m+ spend logs scenario

* fix(leftnav.tsx): allow all users to see new usage tab
2025-04-01 22:27:26 -07:00
Tomer Bin
0690f7a3cb
Virtual key based policies in Aim Guardrails (#9499)
* report key alias to aim

* send litellm version to aim

* Update docs

* blacken

* add docs

* Add info part about virtual keys specific guards

* sort guardrails alphabetically

* fix ruff
2025-04-01 21:57:23 -07:00
Ishaan Jaff
ac3399238e
Merge pull request #9690 from BerriAI/litellm_fixes_for_deadlock_round_2
[Reliability Fixes] - Ensure no deadlocks occur when updating `DailyUserSpendTransaction`
2025-04-01 20:59:41 -07:00
Krrish Dholakia
0cfc3e822c build: new ui build 2025-04-01 19:44:23 -07:00
Krish Dholakia
60863bb134
UI - New Usage Tab fixes (#9696)
* fix(new_usage.tsx): enable smooth scrolling - remove double scroll bars

enables easier viewing of overflowing content

* fix(new_usage.tsx): fix ordering of daily spend

ensure always from earliest to latest date

* feat(internal_user_endpoints.py): return key alias on `/user/daily/activity`

Enables easier consumption on UI

* fix(new_usage.tsx): show key alias on usage tab

* feat(activity_metric.tsx): new activity panel - showing spend per model per day

allows debugging if models are not being tracked

* fix(top_key_view.tsx): use consistent param for selecting key
2025-04-01 19:36:00 -07:00
Ishaan Jaff
44bc8974c8 BaseUpdateQueue 2025-04-01 18:31:54 -07:00
Ishaan Jaff
8dc792139e refactor file structure 2025-04-01 18:30:48 -07:00
Ishaan Jaff
07fc5a72f1 add debug statement 2025-04-01 17:54:52 -07:00
Ishaan Jaff
827ad38e7f fix spend update queue 2025-04-01 17:49:28 -07:00
Ishaan Jaff
4c14550721 refactor daily spend updates to use new Queue DS 2025-04-01 17:45:19 -07:00
Ishaan Jaff
4ddca7a79c
Merge branch 'main' into litellm_fix_service_account_behavior 2025-04-01 12:04:28 -07:00
Ishaan Jaff
61b609f320
Merge pull request #9673 from BerriAI/litellm_qa_deadlock_fixes
[Reliability] - Ensure new Redis + DB architecture tracks spend accurately
2025-04-01 12:04:03 -07:00
Ishaan Jaff
01bd418e0b remove deprecated logic 2025-04-01 07:55:28 -07:00
Ishaan Jaff
f8ce30763e fix check_if_token_is_service_account 2025-04-01 07:36:35 -07:00
Krish Dholakia
62ad84fb64
UI (new_usage.tsx): Report 'total_tokens' + report success/failure calls (#9675)
* feat(internal_user_endpoints.py): return 'total_tokens' in `/user/daily/analytics`

* test(test_internal_user_endpoints.py): add unit test to assert spend metrics and dailyspend metadata always report the same fields

* build(schema.prisma): record success + failure calls to daily user table

allows understanding why model requests might exceed provider requests (e.g. user hit rate limit error)

* fix(internal_user_endpoints.py): report success / failure requests in API

* fix(proxy/utils.py): default to success

status can be missing or none at times for successful requests

* feat(new_usage.tsx): show success/failure calls on UI

* style(new_usage.tsx): ui cleanup

* fix: fix linting error

* fix: fix linting error

* feat(litellm-proxy-extras/): add new migration files
2025-03-31 22:48:43 -07:00
Krish Dholakia
f2a7edaddc
fix(proxy_server.py): Fix "Circular reference detected" error when max_parallel_requests = 0 (#9671)
* fix(proxy_server.py): remove non-functional parent backoff/retry on /chat/completion

Causes circular reference error

* fix(http_parsing_utils.py): safely return parsed body - don't allow mutation of cached request body by client functions

Root cause fix for circular reference error

* Revert "fix: Anthropic prompt caching on GCP Vertex AI (#9605)" (#9670)

This reverts commit a8673246dc.

* add type hints for AnthropicMessagesResponse

* define types for response form AnthropicMessagesResponse

* fix response typing

* allow using litellm.messages.acreate and litellm.messages.create

* fix anthropic_messages implementation

* add clear type hints to litellm.messages.create functions

* fix anthropic_messages

* working anthropic API tests

* fixes - anthropic messages interface

* use new anthropic interface

* fix code quality check

* docs anthropic messages endpoint

* add namespace_packages = True to mypy

* fix mypy lint errors

* docs anthropic messages interface

* test: fix unit test

* test(test_http_parsing_utils.py): update tests

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
2025-03-31 22:06:02 -07:00
Ishaan Jaff
d0a7e44a6e fix linting 2025-03-31 21:33:05 -07:00
Ishaan Jaff
115946d402 unit testing for SpendUpdateQueue 2025-03-31 21:25:24 -07:00
Krish Dholakia
5ad2fbcba6
Openrouter streaming fixes + Anthropic 'file' message support (#9667)
* fix(openrouter/transformation.py): Handle error in openrouter stream

Fixes https://github.com/Aider-AI/aider/issues/3550

* test(test_openrouter_chat_transformation.py): add unit tests

* feat(anthropic/chat/transformation.py): add openai 'file' message content type support

Closes https://github.com/BerriAI/litellm/issues/9463

* fix(factory.py): add bedrock converse support for openai 'file' message content type

Closes https://github.com/BerriAI/litellm/issues/9463
2025-03-31 21:22:59 -07:00
Ishaan Jaff
f7ddc583f0 linting fix get_aggregated_db_spend_update_transactions 2025-03-31 21:15:21 -07:00
Ishaan Jaff
5fa5c1154e add SpendUpdateQueue 2025-03-31 19:42:20 -07:00
Ishaan Jaff
6d4a6a8402 add spend tracking config.yaml 2025-03-31 19:42:00 -07:00
Ishaan Jaff
aa8261af89 test fixes 2025-03-31 19:33:10 -07:00
Ishaan Jaff
811f488ca3 get_aggregated_db_spend_update_transactions 2025-03-31 19:31:53 -07:00
Ishaan Jaff
71e772dd4a use typed data structure for queue 2025-03-31 19:28:17 -07:00