Commit graph

4674 commits

Author SHA1 Message Date
Krish Dholakia
60863bb134
UI - New Usage Tab fixes (#9696)
* fix(new_usage.tsx): enable smooth scrolling - remove double scroll bars

enables easier viewing of overflowing content

* fix(new_usage.tsx): fix ordering of daily spend

ensure always from earliest to latest date

* feat(internal_user_endpoints.py): return key alias on `/user/daily/activity`

Enables easier consumption on UI

* fix(new_usage.tsx): show key alias on usage tab

* feat(activity_metric.tsx): new activity panel - showing spend per model per day

allows debugging if models are not being tracked

* fix(top_key_view.tsx): use consistent param for selecting key
2025-04-01 19:36:00 -07:00
Ishaan Jaff
44bc8974c8 BaseUpdateQueue 2025-04-01 18:31:54 -07:00
Ishaan Jaff
8dc792139e refactor file structure 2025-04-01 18:30:48 -07:00
Ishaan Jaff
07fc5a72f1 add debug statement 2025-04-01 17:54:52 -07:00
Ishaan Jaff
827ad38e7f fix spend update queue 2025-04-01 17:49:28 -07:00
Ishaan Jaff
4c14550721 refactor daily spend updates to use new Queue DS 2025-04-01 17:45:19 -07:00
Ishaan Jaff
4ddca7a79c
Merge branch 'main' into litellm_fix_service_account_behavior 2025-04-01 12:04:28 -07:00
Ishaan Jaff
61b609f320
Merge pull request #9673 from BerriAI/litellm_qa_deadlock_fixes
[Reliability] - Ensure new Redis + DB architecture tracks spend accurately
2025-04-01 12:04:03 -07:00
Ishaan Jaff
01bd418e0b remove deprecated logic 2025-04-01 07:55:28 -07:00
Ishaan Jaff
f8ce30763e fix check_if_token_is_service_account 2025-04-01 07:36:35 -07:00
Krish Dholakia
62ad84fb64
UI (new_usage.tsx): Report 'total_tokens' + report success/failure calls (#9675)
* feat(internal_user_endpoints.py): return 'total_tokens' in `/user/daily/analytics`

* test(test_internal_user_endpoints.py): add unit test to assert spend metrics and dailyspend metadata always report the same fields

* build(schema.prisma): record success + failure calls to daily user table

allows understanding why model requests might exceed provider requests (e.g. user hit rate limit error)

* fix(internal_user_endpoints.py): report success / failure requests in API

* fix(proxy/utils.py): default to success

status can be missing or none at times for successful requests

* feat(new_usage.tsx): show success/failure calls on UI

* style(new_usage.tsx): ui cleanup

* fix: fix linting error

* fix: fix linting error

* feat(litellm-proxy-extras/): add new migration files
2025-03-31 22:48:43 -07:00
Krish Dholakia
f2a7edaddc
fix(proxy_server.py): Fix "Circular reference detected" error when max_parallel_requests = 0 (#9671)
* fix(proxy_server.py): remove non-functional parent backoff/retry on /chat/completion

Causes circular reference error

* fix(http_parsing_utils.py): safely return parsed body - don't allow mutation of cached request body by client functions

Root cause fix for circular reference error

* Revert "fix: Anthropic prompt caching on GCP Vertex AI (#9605)" (#9670)

This reverts commit a8673246dc.

* add type hints for AnthropicMessagesResponse

* define types for response form AnthropicMessagesResponse

* fix response typing

* allow using litellm.messages.acreate and litellm.messages.create

* fix anthropic_messages implementation

* add clear type hints to litellm.messages.create functions

* fix anthropic_messages

* working anthropic API tests

* fixes - anthropic messages interface

* use new anthropic interface

* fix code quality check

* docs anthropic messages endpoint

* add namespace_packages = True to mypy

* fix mypy lint errors

* docs anthropic messages interface

* test: fix unit test

* test(test_http_parsing_utils.py): update tests

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
2025-03-31 22:06:02 -07:00
Ishaan Jaff
d0a7e44a6e fix linting 2025-03-31 21:33:05 -07:00
Ishaan Jaff
115946d402 unit testing for SpendUpdateQueue 2025-03-31 21:25:24 -07:00
Krish Dholakia
5ad2fbcba6
Openrouter streaming fixes + Anthropic 'file' message support (#9667)
* fix(openrouter/transformation.py): Handle error in openrouter stream

Fixes https://github.com/Aider-AI/aider/issues/3550

* test(test_openrouter_chat_transformation.py): add unit tests

* feat(anthropic/chat/transformation.py): add openai 'file' message content type support

Closes https://github.com/BerriAI/litellm/issues/9463

* fix(factory.py): add bedrock converse support for openai 'file' message content type

Closes https://github.com/BerriAI/litellm/issues/9463
2025-03-31 21:22:59 -07:00
Ishaan Jaff
f7ddc583f0 linting fix get_aggregated_db_spend_update_transactions 2025-03-31 21:15:21 -07:00
Ishaan Jaff
5fa5c1154e add SpendUpdateQueue 2025-03-31 19:42:20 -07:00
Ishaan Jaff
6d4a6a8402 add spend tracking config.yaml 2025-03-31 19:42:00 -07:00
Ishaan Jaff
aa8261af89 test fixes 2025-03-31 19:33:10 -07:00
Ishaan Jaff
811f488ca3 get_aggregated_db_spend_update_transactions 2025-03-31 19:31:53 -07:00
Ishaan Jaff
71e772dd4a use typed data structure for queue 2025-03-31 19:28:17 -07:00
Ishaan Jaff
4959512170 fix update_end_user_spend 2025-03-31 19:05:58 -07:00
Ishaan Jaff
73d6af5572 fix docstring 2025-03-31 19:03:02 -07:00
Ishaan Jaff
95e674d1e9 test spend accuracy 2025-03-31 19:01:17 -07:00
Ishaan Jaff
3e16a51ca6 fix update_database helper on db_spend_update_writer 2025-03-31 19:01:00 -07:00
Ishaan Jaff
bcd49204f6 use spend_update_queue for RedisUpdateBuffer 2025-03-31 18:40:52 -07:00
Ishaan Jaff
efe6d375e9 add new SpendUpdateQueue 2025-03-31 18:40:03 -07:00
Ishaan Jaff
3d0d5e1473 run black formatting 2025-03-31 16:12:31 -07:00
Ishaan Jaff
cc80370e0c docs mcp litellm 2025-03-29 21:59:58 -07:00
Ishaan Jaff
db12adb3db docs mcp tools 2025-03-29 20:42:14 -07:00
Ishaan Jaff
5df985f964
Merge pull request #9642 from BerriAI/litellm_mcp_improvements_expose_sse_urls
[Feat] - MCP improvements, add support for using SSE MCP servers
2025-03-29 19:37:57 -07:00
Krish Dholakia
5c107c64dd
Add gemini audio input support + handle special tokens in sagemaker response (#9640)
* fix(internal_user_endpoints.py): cleanup unused variables on beta endpoint

no team/org split on daily user endpoint

* build(model_prices_and_context_window.json): gemini-2.0-flash supports audio input

* feat(gemini/transformation.py): support passing audio input to gemini

* test: fix test

* fix(gemini/transformation.py): support audio input as a url

enables passing google cloud bucket urls

* fix(gemini/transformation.py): support explicitly passing format of file

* fix(gemini/transformation.py): expand support for inferred file types from url

* fix(sagemaker/completion/transformation.py): fix special token error when counting sagemaker tokens

* test: fix import
2025-03-29 19:23:09 -07:00
Ishaan Jaff
10486dd17e fix listing mcp tools 2025-03-29 18:40:58 -07:00
Ishaan Jaff
385e8bff4c fix order of imports 2025-03-29 18:32:06 -07:00
Ishaan Jaff
c24470ee72 list_tool_rest_api 2025-03-29 17:55:37 -07:00
Ishaan Jaff
0e321eed1a fix import errors without mcp 2025-03-29 17:44:32 -07:00
Ishaan Jaff
7dd54112cb fix showing list of MCP tools 2025-03-29 16:23:58 -07:00
Ishaan Jaff
fe6c033427 render MCP tools on ui logs page 2025-03-29 16:01:34 -07:00
Ishaan Jaff
08a52f4389 log MCP tool call metadata in SLP 2025-03-29 15:50:13 -07:00
Krish Dholakia
1604f87663
install prisma migration files - connects litellm proxy to litellm's prisma migration files (#9637)
* build(README.md): initial commit adding a separate folder for additional proxy files. Meant to reduce size of core package

* build(litellm-proxy-extras/): new pip package for storing migration files

allows litellm proxy to use migration files, without adding them to core repo

* build(litellm-proxy-extras/): cleanup pyproject.toml

* build: move prisma migration files inside new proxy extras package

* build(run_migration.py): update script to write to correct folder

* build(proxy_cli.py): load in migration files from litellm-proxy-extras

Closes https://github.com/BerriAI/litellm/issues/9558

* build: add MIT license to litellm-proxy-extras

* test: update test

* fix: fix schema

* bump: version 0.1.0 → 0.1.1

* build(publish-proxy-extras.sh): add script for publishing new proxy-extras version

* build(liccheck.ini): add litellm-proxy-extras to authorized packages

* fix(litellm-proxy-extras/utils.py): move prisma migrate logic inside extra proxy pkg

easier since migrations folder already there

* build(pre-commit-config.yaml): add litellm_proxy_extras to ci tests

* docs(config_settings.md): document new env var

* build(pyproject.toml): bump relevant files when litellm-proxy-extras version changed

* build(pre-commit-config.yaml): run poetry check on litellm-proxy-extras as well
2025-03-29 15:27:09 -07:00
Ishaan Jaff
b7b9f9d9da working MCP tool call logging 2025-03-29 15:20:52 -07:00
Ishaan Jaff
a4a083041d working MCP call tool method 2025-03-29 14:42:19 -07:00
Ishaan Jaff
e0cff7595c endpoints to list and call tools 2025-03-29 14:31:35 -07:00
Ishaan Jaff
1cf5cbab50 REST API endpoint for MCP 2025-03-29 13:35:46 -07:00
Ishaan Jaff
a1ec0dd0e2 add testing mcp server 2025-03-29 12:52:46 -07:00
Ishaan Jaff
6aa660b3f5 mcp server manager 2025-03-29 12:51:16 -07:00
Ishaan Jaff
ec283f7289 use global_mcp_server_manager 2025-03-29 12:46:13 -07:00
Ishaan Jaff
fe3623d775 init mcp client manager 2025-03-29 11:26:18 -07:00
Krish Dholakia
9b7ebb6a7d
build(pyproject.toml): add new dev dependencies - for type checking (#9631)
* build(pyproject.toml): add new dev dependencies - for type checking

* build: reformat files to fit black

* ci: reformat to fit black

* ci(test-litellm.yml): make tests run clear

* build(pyproject.toml): add ruff

* fix: fix ruff checks

* build(mypy/): fix mypy linting errors

* fix(hashicorp_secret_manager.py): fix passing cert for tls auth

* build(mypy/): resolve all mypy errors

* test: update test

* fix: fix black formatting

* build(pre-commit-config.yaml): use poetry run black

* fix(proxy_server.py): fix linting error

* fix: fix ruff safe representation error
2025-03-29 11:02:13 -07:00
Ishaan Jaff
c95fb6c692 MAX_REDIS_BUFFER_DEQUEUE_COUNT 2025-03-28 20:59:28 -07:00