Commit graph

1687 commits

Author SHA1 Message Date
Ishaan Jaff
13aa7f75f6 test_enforced_params_check 2025-04-01 07:40:31 -07:00
Ishaan Jaff
55763ae276 test_end_user_transactions_reset 2025-04-01 07:13:25 -07:00
Ishaan Jaff
7a2442d6c0 test_batch_update_spend 2025-04-01 07:12:29 -07:00
Krish Dholakia
62ad84fb64
UI (new_usage.tsx): Report 'total_tokens' + report success/failure calls (#9675)
* feat(internal_user_endpoints.py): return 'total_tokens' in `/user/daily/analytics`

* test(test_internal_user_endpoints.py): add unit test to assert spend metrics and dailyspend metadata always report the same fields

* build(schema.prisma): record success + failure calls to daily user table

allows understanding why model requests might exceed provider requests (e.g. user hit rate limit error)

* fix(internal_user_endpoints.py): report success / failure requests in API

* fix(proxy/utils.py): default to success

status can be missing or none at times for successful requests

* feat(new_usage.tsx): show success/failure calls on UI

* style(new_usage.tsx): ui cleanup

* fix: fix linting error

* fix: fix linting error

* feat(litellm-proxy-extras/): add new migration files
2025-03-31 22:48:43 -07:00
Krish Dholakia
f2a7edaddc
fix(proxy_server.py): Fix "Circular reference detected" error when max_parallel_requests = 0 (#9671)
* fix(proxy_server.py): remove non-functional parent backoff/retry on /chat/completion

Causes circular reference error

* fix(http_parsing_utils.py): safely return parsed body - don't allow mutation of cached request body by client functions

Root cause fix for circular reference error

* Revert "fix: Anthropic prompt caching on GCP Vertex AI (#9605)" (#9670)

This reverts commit a8673246dc.

* add type hints for AnthropicMessagesResponse

* define types for response form AnthropicMessagesResponse

* fix response typing

* allow using litellm.messages.acreate and litellm.messages.create

* fix anthropic_messages implementation

* add clear type hints to litellm.messages.create functions

* fix anthropic_messages

* working anthropic API tests

* fixes - anthropic messages interface

* use new anthropic interface

* fix code quality check

* docs anthropic messages endpoint

* add namespace_packages = True to mypy

* fix mypy lint errors

* docs anthropic messages interface

* test: fix unit test

* test(test_http_parsing_utils.py): update tests

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
2025-03-31 22:06:02 -07:00
Krish Dholakia
722f3ff0e6
fix(cost_calculator.py): allows checking received + sent model name when checking for cost calculation (#9669)
Fixes issue introduced by dfb838eaff (r154667517)
2025-03-31 21:29:48 -07:00
Ishaan Jaff
115946d402 unit testing for SpendUpdateQueue 2025-03-31 21:25:24 -07:00
Krish Dholakia
5ad2fbcba6
Openrouter streaming fixes + Anthropic 'file' message support (#9667)
* fix(openrouter/transformation.py): Handle error in openrouter stream

Fixes https://github.com/Aider-AI/aider/issues/3550

* test(test_openrouter_chat_transformation.py): add unit tests

* feat(anthropic/chat/transformation.py): add openai 'file' message content type support

Closes https://github.com/BerriAI/litellm/issues/9463

* fix(factory.py): add bedrock converse support for openai 'file' message content type

Closes https://github.com/BerriAI/litellm/issues/9463
2025-03-31 21:22:59 -07:00
Ishaan Jaff
9951b356da test_long_term_spend_accuracy_with_bursts 2025-03-31 21:09:29 -07:00
Ishaan Jaff
923ac2303b test_end_user_transactions_reset 2025-03-31 20:55:13 -07:00
Ishaan Jaff
bc5cc51b9d
Merge pull request #9567 from BerriAI/litellm_anthropic_messages_improvements
[Refactor] - Expose litellm.messages.acreate() and  litellm.messages.create() to make LLM API calls in Anthropic API spec
2025-03-31 20:50:30 -07:00
Ishaan Jaff
271b8b95bc test spend accuracy 2025-03-31 19:35:07 -07:00
Ishaan Jaff
aa8261af89 test fixes 2025-03-31 19:33:10 -07:00
Ishaan Jaff
a753fc9d9f test_long_term_spend_accuracy_with_bursts 2025-03-31 19:17:13 -07:00
Ishaan Jaff
95e674d1e9 test spend accuracy 2025-03-31 19:01:17 -07:00
Krish Dholakia
46b3dbde8f
Revert "fix: Anthropic prompt caching on GCP Vertex AI (#9605)" (#9670)
This reverts commit a8673246dc.
2025-03-31 17:13:55 -07:00
Krrish Dholakia
b37cb642c1 test: fix b64 str 2025-03-31 17:10:06 -07:00
Ishaan Jaff
ca4ed9ff2e ref issue 2025-03-31 16:05:10 -07:00
Ishaan Jaff
bc66827537 test_aiter_bytes_valid_chunk_followed_by_unicode_error 2025-03-31 16:04:38 -07:00
Ishaan Jaff
bd39a395f1 use new anthropic interface 2025-03-31 14:31:09 -07:00
Ishaan Jaff
01d85d5fb7 Merge branch 'main' into litellm_anthropic_messages_improvements 2025-03-31 14:22:56 -07:00
Ishaan Jaff
ce5f55d04e test fix update spend 2025-03-31 14:20:47 -07:00
Sam
a8673246dc
fix: Anthropic prompt caching on GCP Vertex AI (#9605)
* fix: Anthropic prompt caching on GCP Vertex AI

* test(vertex): anthropic prompt caching
2025-03-29 23:40:34 -07:00
Ishaan Jaff
5df985f964
Merge pull request #9642 from BerriAI/litellm_mcp_improvements_expose_sse_urls
[Feat] - MCP improvements, add support for using SSE MCP servers
2025-03-29 19:37:57 -07:00
Krish Dholakia
5c107c64dd
Add gemini audio input support + handle special tokens in sagemaker response (#9640)
* fix(internal_user_endpoints.py): cleanup unused variables on beta endpoint

no team/org split on daily user endpoint

* build(model_prices_and_context_window.json): gemini-2.0-flash supports audio input

* feat(gemini/transformation.py): support passing audio input to gemini

* test: fix test

* fix(gemini/transformation.py): support audio input as a url

enables passing google cloud bucket urls

* fix(gemini/transformation.py): support explicitly passing format of file

* fix(gemini/transformation.py): expand support for inferred file types from url

* fix(sagemaker/completion/transformation.py): fix special token error when counting sagemaker tokens

* test: fix import
2025-03-29 19:23:09 -07:00
Ishaan Jaff
3919e24256 test fix 2025-03-29 18:36:13 -07:00
Ishaan Jaff
194327bb7c test fixes 2025-03-29 18:34:58 -07:00
Ishaan Jaff
a3df0269bb fix tests 2025-03-29 17:38:24 -07:00
Ishaan Jaff
4e106ce217 fix test 2025-03-29 17:11:46 -07:00
Ishaan Jaff
3e378f2bec async def test_spend_logs_payload_e2e(self): 2025-03-29 17:07:36 -07:00
Ishaan Jaff
047d767947 fix tests for gcs pub sub 2025-03-29 17:06:36 -07:00
Ishaan Jaff
79e8bbbfd4 fix types on tools.py 2025-03-29 16:48:15 -07:00
Ishaan Jaff
815263f7bc rename transform_openai_tool_call_request_to_mcp_tool_call_request 2025-03-29 16:28:23 -07:00
Krish Dholakia
1604f87663
install prisma migration files - connects litellm proxy to litellm's prisma migration files (#9637)
* build(README.md): initial commit adding a separate folder for additional proxy files. Meant to reduce size of core package

* build(litellm-proxy-extras/): new pip package for storing migration files

allows litellm proxy to use migration files, without adding them to core repo

* build(litellm-proxy-extras/): cleanup pyproject.toml

* build: move prisma migration files inside new proxy extras package

* build(run_migration.py): update script to write to correct folder

* build(proxy_cli.py): load in migration files from litellm-proxy-extras

Closes https://github.com/BerriAI/litellm/issues/9558

* build: add MIT license to litellm-proxy-extras

* test: update test

* fix: fix schema

* bump: version 0.1.0 → 0.1.1

* build(publish-proxy-extras.sh): add script for publishing new proxy-extras version

* build(liccheck.ini): add litellm-proxy-extras to authorized packages

* fix(litellm-proxy-extras/utils.py): move prisma migrate logic inside extra proxy pkg

easier since migrations folder already there

* build(pre-commit-config.yaml): add litellm_proxy_extras to ci tests

* docs(config_settings.md): document new env var

* build(pyproject.toml): bump relevant files when litellm-proxy-extras version changed

* build(pre-commit-config.yaml): run poetry check on litellm-proxy-extras as well
2025-03-29 15:27:09 -07:00
Ishaan Jaff
a1ec0dd0e2 add testing mcp server 2025-03-29 12:52:46 -07:00
Krish Dholakia
9b7ebb6a7d
build(pyproject.toml): add new dev dependencies - for type checking (#9631)
* build(pyproject.toml): add new dev dependencies - for type checking

* build: reformat files to fit black

* ci: reformat to fit black

* ci(test-litellm.yml): make tests run clear

* build(pyproject.toml): add ruff

* fix: fix ruff checks

* build(mypy/): fix mypy linting errors

* fix(hashicorp_secret_manager.py): fix passing cert for tls auth

* build(mypy/): resolve all mypy errors

* test: update test

* fix: fix black formatting

* build(pre-commit-config.yaml): use poetry run black

* fix(proxy_server.py): fix linting error

* fix: fix ruff safe representation error
2025-03-29 11:02:13 -07:00
Krrish Dholakia
217e8d7d44 test: make script to run clearer 2025-03-29 08:23:18 -07:00
Krish Dholakia
5ac61a7572
Add bedrock latency optimized inference support (#9623)
* fix(converse_transformation.py): add performanceConfig param support on bedrock

Closes https://github.com/BerriAI/litellm/issues/7606

* fix(converse_transformation.py): refactor to use more flexible single getter for params which are separate config blocks

* test(test_main.py): add e2e mock test for bedrock performance config

* build(model_prices_and_context_window.json): add versioned multimodal embedding

* refactor(multimodal_embeddings/): migrate to config pattern

* feat(vertex_ai/multimodalembeddings): calculate usage for multimodal embedding calls

Enables cost calculation for multimodal embeddings

* feat(vertex_ai/multimodalembeddings): get usage object for embedding calls

ensures accurate cost tracking for vertexai multimodal embedding calls

* fix(embedding_handler.py): remove unused imports

* fix: fix linting errors

* fix: handle response api usage calculation

* test(test_vertex_ai_multimodal_embedding_transformation.py): update tests

* test: mark flaky test

* feat(vertex_ai/multimodal_embeddings/transformation.py): support text+image+video input

* docs(vertex.md): document sending text + image to vertex multimodal embeddings

* test: remove incorrect file

* fix(multimodal_embeddings/transformation.py): fix linting error

* style: remove unused import
2025-03-29 00:23:09 -07:00
Ishaan Jaff
7e8a02099c Merge branch 'main' into litellm_use_redis_for_updates 2025-03-28 20:12:29 -07:00
Ishaan Jaff
ba550e2147 test local spend accuracy 2025-03-28 19:52:39 -07:00
NickGrab
220d4c07f4
Merge pull request #9625 from BerriAI/litellm_mar_28_vertex_fix
Add support to Vertex AI transformation for anyOf union type with null fields
2025-03-28 16:09:29 -07:00
Krish Dholakia
222898d727
Fix anthropic thinking + response_format (#9594)
* fix(anthropic/chat/transformation.py): Don't set tool choice on response_format conversion when thinking is enabled

Not allowed by Anthropic

Fixes https://github.com/BerriAI/litellm/issues/8901

* refactor: move test to base anthropic chat tests

ensures consistent behaviour across vertex/anthropic/bedrock

* fix(anthropic/chat/transformation.py): if thinking token is specified and max tokens is not - ensure max token to anthropic is higher than thinking tokens

* feat(converse_transformation.py): correctly handle thinking + response format on Bedrock Converse

Fixes https://github.com/BerriAI/litellm/issues/8901

* fix(converse_transformation.py): correctly handle adding max tokens

* test: handle service unavailable error
2025-03-28 15:57:40 -07:00
Nicholas Grabar
09daeac188 Rebasing 2 2025-03-28 15:18:09 -07:00
Nicholas Grabar
06a45706b2 Rebase 3 2025-03-28 15:18:05 -07:00
Krrish Dholakia
8c9ff23e19 test(test_caching_handler.py): move to in-memory cache - prevent redis flakiness from impacting ci/cd 2025-03-28 15:16:15 -07:00
Krish Dholakia
205db622bf fix(proxy_server.py): get master key from environment, if not set in … (#9617)
* fix(proxy_server.py): get master key from environment, if not set in general settings or general settings not set at all

* test: mark flaky test

* test(test_proxy_server.py): mock prisma client

* ci: add new github workflow for testing just the mock tests

* fix: fix linting error

* ci(conftest.py): add conftest.py to isolate proxy tests

* build(pyproject.toml): add respx to dev dependencies

* build(pyproject.toml): add prisma to dev dependencies

* test: fix mock prompt management tests to use a mock anthropic key

* ci(test-litellm.yml): parallelize mock testing

make it run faster

* build(pyproject.toml): add hypercorn as dev dep

* build(pyproject.toml): separate proxy vs. core dev dependencies

make it easier for non-proxy contributors to run tests locally - e.g. no need to install hypercorn

* ci(test-litellm.yml): pin python version

* test(test_rerank.py): move test - cannot be mocked, requires aws credentials for e2e testing

* ci: add thank you message to ci

* test: add mock env var to test

* test: add autouse to tests

* test: test mock env vars for e2e tests
2025-03-28 15:16:15 -07:00
Ishaan Jaff
193052ed70 test pod lock manager 2025-03-28 15:05:17 -07:00
Krrish Dholakia
28a9edb547 test(test_caching_handler.py): move to in-memory cache - prevent redis flakiness from impacting ci/cd
All checks were successful
Read Version from pyproject.toml / read-version (push) Successful in 15s
Helm unit test / unit-test (push) Successful in 19s
2025-03-28 13:32:04 -07:00
Ishaan Jaff
1eaf847f8a test pod lock manager 2025-03-28 13:31:45 -07:00
Nicholas Grabar
1f2bbda11d Add recursion depth to convert_anyof_null_to_nullable, constants.py. Fix recursive_detector.py raise error state 2025-03-28 13:11:19 -07:00