Krish Dholakia
3933fba41f
LiteLLM Minor Fixes & Improvements (09/19/2024) ( #5793 )
...
* fix(model_prices_and_context_window.json): add cost tracking for more vertex llama3.1 model
8b and 70b models
* fix(proxy/utils.py): handle data being none on pre-call hooks
* fix(proxy/): create views on initial proxy startup
fixes base case, where user starts proxy for first time
Fixes https://github.com/BerriAI/litellm/issues/5756
* build(config.yml): fix vertex version for test
* feat(ui/): support enabling/disabling slack alerting
Allows admin to turn on/off slack alerting through ui
* feat(rerank/main.py): support langfuse logging
* fix(proxy/utils.py): fix linting errors
* fix(langfuse.py): log clean metadata
* test(tests): replace deprecated openai model
2024-09-20 08:19:52 -07:00
Krish Dholakia
d46660ea0f
LiteLLM Minor Fixes & Improvements (09/18/2024) ( #5772 )
...
* fix(proxy_server.py): fix azure key vault logic to not require client id/secret
* feat(cost_calculator.py): support fireworks ai cost tracking
* build(docker-compose.yml): add lines for mounting config.yaml to docker compose
Closes https://github.com/BerriAI/litellm/issues/5739
* fix(input.md): update docs to clarify litellm supports content as a list of dictionaries
Fixes https://github.com/BerriAI/litellm/issues/5755
* fix(input.md): update input.md to include all message values
* fix(image_handling.py): follow image url redirects
Fixes https://github.com/BerriAI/litellm/issues/5763
* fix(router.py): Fix model key/base leak in error message
Fixes https://github.com/BerriAI/litellm/issues/5762
* fix(http_handler.py): fix linting error
* fix(azure.py): fix logging to show azure_ad_token being used
Fixes https://github.com/BerriAI/litellm/issues/5767
* fix(_redis.py): add redis sentinel support
Closes https://github.com/BerriAI/litellm/issues/4381
* feat(_redis.py): add redis sentinel support
Closes https://github.com/BerriAI/litellm/issues/4381
* test(test_completion_cost.py): fix test
* Databricks Integration: Integrate Databricks SDK as optional mechanism for fetching API base and token, if unspecified (#5746 )
* LiteLLM Minor Fixes & Improvements (09/16/2024) (#5723 )
* coverage (#5713 )
Signed-off-by: dbczumar <corey.zumar@databricks.com>
* Move (#5714 )
Signed-off-by: dbczumar <corey.zumar@databricks.com>
* fix(litellm_logging.py): fix logging client re-init (#5710 )
Fixes https://github.com/BerriAI/litellm/issues/5695
* fix(presidio.py): Fix logging_hook response and add support for additional presidio variables in guardrails config
Fixes https://github.com/BerriAI/litellm/issues/5682
* feat(o1_handler.py): fake streaming for openai o1 models
Fixes https://github.com/BerriAI/litellm/issues/5694
* docs: deprecated traceloop integration in favor of native otel (#5249 )
* fix: fix linting errors
* fix: fix linting errors
* fix(main.py): fix o1 import
---------
Signed-off-by: dbczumar <corey.zumar@databricks.com>
Co-authored-by: Corey Zumar <39497902+dbczumar@users.noreply.github.com>
Co-authored-by: Nir Gazit <nirga@users.noreply.github.com>
* feat(spend_management_endpoints.py): expose `/global/spend/refresh` endpoint for updating material view (#5730 )
* feat(spend_management_endpoints.py): expose `/global/spend/refresh` endpoint for updating material view
Supports having `MonthlyGlobalSpend` view be a material view, and exposes an endpoint to refresh it
* fix(custom_logger.py): reset calltype
* fix: fix linting errors
* fix: fix linting error
* fix
Signed-off-by: dbczumar <corey.zumar@databricks.com>
* fix: fix import
* Fix
Signed-off-by: dbczumar <corey.zumar@databricks.com>
* fix
Signed-off-by: dbczumar <corey.zumar@databricks.com>
* DB test
Signed-off-by: dbczumar <corey.zumar@databricks.com>
* Coverage
Signed-off-by: dbczumar <corey.zumar@databricks.com>
* progress
Signed-off-by: dbczumar <corey.zumar@databricks.com>
* fix
Signed-off-by: dbczumar <corey.zumar@databricks.com>
* fix
Signed-off-by: dbczumar <corey.zumar@databricks.com>
* fix
Signed-off-by: dbczumar <corey.zumar@databricks.com>
* fix test name
Signed-off-by: dbczumar <corey.zumar@databricks.com>
---------
Signed-off-by: dbczumar <corey.zumar@databricks.com>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Nir Gazit <nirga@users.noreply.github.com>
* test: fix test
* test(test_databricks.py): fix test
* fix(databricks/chat.py): handle custom endpoint (e.g. sagemaker)
* Apply code scanning fix for clear-text logging of sensitive information
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix(__init__.py): fix known fireworks ai models
---------
Signed-off-by: dbczumar <corey.zumar@databricks.com>
Co-authored-by: Corey Zumar <39497902+dbczumar@users.noreply.github.com>
Co-authored-by: Nir Gazit <nirga@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2024-09-19 13:25:29 -07:00
Krish Dholakia
0295a22561
LiteLLM Minor Fixes and Improvements (09/10/2024) ( #5618 )
...
* fix(cost_calculator.py): move to debug for noisy warning message on cost calculation error
Fixes https://github.com/BerriAI/litellm/issues/5610
* fix(databricks/cost_calculator.py): Handles model name issues for databricks models
* fix(main.py): fix stream chunk builder for multiple tool calls
Fixes https://github.com/BerriAI/litellm/issues/5591
* fix: correctly set user_alias when passed in
Fixes https://github.com/BerriAI/litellm/issues/5612
* fix(types/utils.py): allow passing role for message object
https://github.com/BerriAI/litellm/issues/5621
* fix(litellm_logging.py): Fix langfuse logging across multiple projects
Fixes issue where langfuse logger was re-using the old logging object
* feat(proxy/_types.py): support adding key-based tags for tag-based routing
Enable tag based routing at key-level
* fix(proxy/_types.py): fix inheritance
* test(test_key_generate_prisma.py): fix test
* test: fix test
* fix(litellm_logging.py): return used callback object
2024-09-11 11:30:29 -07:00
Krish Dholakia
2d2282101b
LiteLLM Minor Fixes and Improvements (09/09/2024) ( #5602 )
...
* fix(main.py): pass default azure api version as alternative in completion call
Fixes api error caused due to api version
Closes https://github.com/BerriAI/litellm/issues/5584
* Fixed gemini-1.5-flash pricing (#5590 )
* add /key/list endpoint
* bump: version 1.44.21 → 1.44.22
* docs architecture
* Fixed gemini-1.5-flash pricing
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
* fix(bedrock/chat.py): fix converse api stop sequence param mapping
Fixes https://github.com/BerriAI/litellm/issues/5592
* fix(databricks/cost_calculator.py): handle databricks model name changes
Fixes https://github.com/BerriAI/litellm/issues/5597
* fix(azure.py): support azure api version 2024-08-01-preview
Closes https://github.com/BerriAI/litellm/issues/5377
* fix(proxy/_types.py): allow dev keys to call cohere /rerank endpoint
Fixes issue where only admin could call rerank endpoint
* fix(azure.py): check if model is gpt-4o
* fix(proxy/_types.py): support /v1/rerank on non-admin routes as well
* fix(cost_calculator.py): fix split on `/` logic in cost calculator
---------
Co-authored-by: F1bos <44951186+F1bos@users.noreply.github.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
2024-09-09 21:56:12 -07:00
Krish Dholakia
be3c7b401e
LiteLLM Minor fixes + improvements (08/03/2024) ( #5488 )
...
* fix(internal_user_endpoints.py): set budget_reset_at for /user/update
* fix(vertex_and_google_ai_studio_gemini.py): handle accumulated json
Fixes https://github.com/BerriAI/litellm/issues/5479
* fix(vertex_ai_and_gemini.py): fix assistant message function call when content is not None
Fixes https://github.com/BerriAI/litellm/issues/5490
* fix(proxy_server.py): generic state uuid for okta sso
* fix(lago.py): improve debug logs
Debugging for https://github.com/BerriAI/litellm/issues/5477
* docs(bedrock.md): add bedrock cross-region inferencing to docs
* fix(azure.py): return azure response headers on aembedding call
* feat(azure.py): return azure response headers for `/audio/transcription`
* fix(types/utils.py): standardize deepseek / anthropic prompt caching usage information
Closes https://github.com/BerriAI/litellm/issues/5285
* docs(usage.md): add docs on litellm usage object
* test(test_completion.py): mark flaky test
2024-09-03 21:21:34 -07:00
Krish Dholakia
9c8f1d7815
anthropic prompt caching cost tracking ( #5453 )
...
* fix(utils.py): support 'drop_params' for embedding requests
Fixes https://github.com/BerriAI/litellm/issues/5444
* feat(anthropic/cost_calculation.py): Support calculating cost for prompt caching on anthropic
* feat(types/utils.py): allows us to migrate to openai's equivalent, once that comes out
* fix: fix linting errors
* test: mark flaky test
2024-08-31 14:09:35 -07:00
Krrish Dholakia
55217fa8d7
feat(cost_calculator.py): only override base model if custom pricing is set
2024-08-19 16:05:49 -07:00
Krrish Dholakia
b1bed459b4
bump: version 1.43.16 → 1.43.17
2024-08-16 21:34:35 -07:00
Krrish Dholakia
178139f18d
feat(litellm_logging.py): support logging model price information to s3 logs
2024-08-16 16:21:34 -07:00
Krrish Dholakia
22e2840daa
fix(cost_calculator.py): handle openai usage pydantic object
...
Fixes https://github.com/BerriAI/litellm/issues/5165
2024-08-12 15:45:21 -07:00
Krrish Dholakia
0a30ba9674
fix(types/utils.py): support passing prompt cache usage stats in usage object
...
Passes deepseek prompt caching values through to end user
2024-08-02 09:30:50 -07:00
Krrish Dholakia
46634af06f
fix(utils.py): fix model registeration to model cost map
...
Fixes https://github.com/BerriAI/litellm/issues/4972
2024-07-30 18:15:00 -07:00
Krrish Dholakia
6d5aedc48d
feat(databricks.py): support vertex mistral cost tracking
2024-07-27 20:22:35 -07:00
Krrish Dholakia
2f773d9cb6
fix(litellm_cost_calc/google.py): support meta llama vertex ai cost tracking
2024-07-25 22:12:07 -07:00
Krrish Dholakia
f64a3309d1
fix(utils.py): support raw response headers for streaming requests
2024-07-23 11:58:58 -07:00
Krrish Dholakia
dcb974dd1e
feat(utils.py): support passing openai response headers to client, if enabled
...
Allows openai/openai-compatible provider response headers to be sent to client, if 'return_response_headers' is enabled
2024-07-23 11:30:52 -07:00
Ishaan Jaff
3bb66ab939
ci/cd run again
2024-07-19 20:08:50 -07:00
Ishaan Jaff
d35aeb3f4f
fix test_vertex_ai_medlm_completion_cost
2024-07-19 19:58:44 -07:00
skucherlapati
2ad342e7bf
add medlm models to cost map
2024-07-17 21:17:58 -07:00
Ishaan Jaff
14f5cab09a
fix medllm test
2024-07-17 20:19:37 -07:00
skucherlapati
ce474ff008
fix failing tests on PR-4760
2024-07-17 19:32:17 -07:00
skucherlapati
db3d6925c6
add medlm cost calc
2024-07-17 14:54:54 -07:00
Krrish Dholakia
959c627dd3
fix(litellm_logging.py): log response_cost=0 for failed calls
...
Fixes https://github.com/BerriAI/litellm/issues/4604
2024-07-15 19:25:56 -07:00
Krrish Dholakia
b1be355d42
build(model_prices_and_context_window.json): add azure ai jamba instruct pricing + token details
...
Adds jamba instruct, mistral, llama3 pricing + token info for azure_ai
2024-07-13 16:34:31 -07:00
Krrish Dholakia
2163434ff3
fix(llm_cost_calc/google.py): fix google embedding cost calculation
...
Fixes https://github.com/BerriAI/litellm/issues/4630
2024-07-11 11:55:48 -07:00
Krrish Dholakia
407639cc7d
fix(cost_calculator.py): support openai+azure tts calls
2024-07-05 20:58:08 -07:00
Krish Dholakia
869275585a
Merge branch 'main' into litellm_response_cost_headers
2024-06-27 21:33:09 -07:00
Krrish Dholakia
94c069e869
fix(cost_calculator.py): infer provider name if not given
...
Fixes https://github.com/BerriAI/litellm/issues/4452
2024-06-27 18:41:04 -07:00
Krrish Dholakia
f533e1da09
fix(utils.py): return 'response_cost' in completion call
...
Closes https://github.com/BerriAI/litellm/issues/4335
2024-06-26 17:55:57 -07:00
spdustin@gmail.com
4acc2d50ad
fix: use per-token costs for claude via vertex_ai
2024-06-21 11:21:36 -05:00
Krrish Dholakia
16da21e839
feat(llm_cost_calc/google.py): do character based cost calculation for vertex ai
...
Calculate cost for vertex ai responses using characters in query/response
Closes https://github.com/BerriAI/litellm/issues/4165
2024-06-19 17:18:42 -07:00
Krrish Dholakia
f597aa432b
feat(cost_calculator.py): add cost calculating for dynamic context window (vertex ai / google ai studio)
2024-06-17 12:38:10 -07:00
Krrish Dholakia
3f9811c6a9
test(test_completion_cost.py): fix import
2024-06-15 12:08:49 -07:00
Krrish Dholakia
f3a845eff9
build(model_prices_and_context_window.json): update together ai model pricing - account for new categories
2024-06-08 19:56:35 -07:00
Krrish Dholakia
b26c3c7d22
fix(cost_calculator.py): fixes tgai unmapped model pricing
...
Fixes error where tgai helper function returned None. Enforces stronger type hints, refactors code, adds more unit testing.
2024-06-08 19:43:57 -07:00
Krrish Dholakia
15b226ca4f
test(test_completion_cost.py): cleanup test
2024-06-04 16:57:25 -07:00
Krrish Dholakia
52a2f5150c
fix(utils.py): fix cost calculation for openai-compatible streaming object
2024-06-04 10:36:25 -07:00
Krrish Dholakia
a2a5884df1
fix(utils.py): allow passing in custom pricing to completion_cost as params
2024-05-16 16:24:44 -07:00
Krrish Dholakia
b4a8665d11
fix(utils.py): fix custom pricing when litellm model != response obj model name
2024-05-13 15:25:35 -07:00
Ishaan Jaff
282b8d0ae4
test bedrock pricing
2024-05-08 15:26:53 -07:00
Krrish Dholakia
2cfb97141d
fix(utils.py): replicate now also has token based pricing for some models
2024-04-29 08:06:15 -07:00
Krrish Dholakia
7dd94c802e
fix(azure.py): support cost tracking for azure/dall-e-3
2024-03-12 10:55:54 -07:00
Krrish Dholakia
fa45c569fd
feat: add cost tracking + caching for transcription calls
2024-03-09 15:43:38 -08:00
Krrish Dholakia
478307d4cf
fix(bedrock.py): support anthropic messages api on bedrock (claude-3)
2024-03-04 17:15:47 -08:00
ishaan-jaff
85e9116758
(ci/cd) aws disabled our test account :(
2024-02-28 17:26:23 -08:00
Krrish Dholakia
4c951d20bc
test: removing aws tests - account suspended - pending their approval
2024-02-28 13:46:20 -08:00
Krrish Dholakia
49b2dc4180
test(test_completion_cost.py): fix test
2024-02-03 22:00:49 -08:00
Krrish Dholakia
a32639fa79
fix(utils.py): support max token adjustment for sagemaker
2024-01-31 19:09:54 -08:00
Krrish Dholakia
f5da95685a
feat(utils.py): support region based pricing for bedrock + use bedrock's token counts if given
2024-01-26 14:53:58 -08:00
Krrish Dholakia
ac131b3824
fix(utils.py): completion_cost support for image gen models
2024-01-25 18:08:18 -08:00
