Krish Dholakia
7338b24a74
refactor(redis_cache.py): use a default cache value when writing to r… ( #6358 )
...
* refactor(redis_cache.py): use a default cache value when writing to redis
prevent redis from blowing up in high traffic
* refactor(redis_cache.py): refactor all cache writes to use self.get_ttl
ensures default ttl always used when writing to redis
Prevents redis db from blowing up in prod
2024-10-21 16:42:12 -07:00
Krish Dholakia
7cc12bd5c6
LiteLLM Minor Fixes & Improvements (10/18/2024) ( #6320 )
...
* fix(converse_transformation.py): handle cross region model name when getting openai param support
Fixes https://github.com/BerriAI/litellm/issues/6291
* LiteLLM Minor Fixes & Improvements (10/17/2024) (#6293 )
* fix(ui_sso.py): fix faulty admin only check
Fixes https://github.com/BerriAI/litellm/issues/6286
* refactor(sso_helper_utils.py): refactor /sso/callback to use helper utils, covered by unit testing
Prevent future regressions
* feat(prompt_factory): support 'ensure_alternating_roles' param
Closes https://github.com/BerriAI/litellm/issues/6257
* fix(proxy/utils.py): add dailytagspend to expected views
* feat(auth_utils.py): support setting regex for clientside auth credentials
Fixes https://github.com/BerriAI/litellm/issues/6203
* build(cookbook): add tutorial for mlflow + langchain + litellm proxy tracing
* feat(argilla.py): add argilla logging integration
Closes https://github.com/BerriAI/litellm/issues/6201
* fix: fix linting errors
* fix: fix ruff error
* test: fix test
* fix: update vertex ai assumption - parts not always guaranteed (#6296 )
* docs(configs.md): add argila env var to docs
* docs(user_keys.md): add regex doc for clientside auth params
* docs(argilla.md): add doc on argilla logging
* docs(argilla.md): add sampling rate to argilla calls
* bump: version 1.49.6 → 1.49.7
* add gpt-4o-audio models to model cost map (#6306 )
* (code quality) add ruff check PLR0915 for `too-many-statements` (#6309 )
* ruff add PLR0915
* add noqa for PLR0915
* fix noqa
* add # noqa: PLR0915
* # noqa: PLR0915
* # noqa: PLR0915
* # noqa: PLR0915
* add # noqa: PLR0915
* # noqa: PLR0915
* # noqa: PLR0915
* # noqa: PLR0915
* # noqa: PLR0915
* doc fix Turn on / off caching per Key. (#6297 )
* (feat) Support `audio`, `modalities` params (#6304 )
* add audio, modalities param
* add test for gpt audio models
* add get_supported_openai_params for GPT audio models
* add supported params for audio
* test_audio_output_from_model
* bump openai to openai==1.52.0
* bump openai on pyproject
* fix audio test
* fix test mock_chat_response
* handle audio for Message
* fix handling audio for OAI compatible API endpoints
* fix linting
* fix mock dbrx test
* (feat) Support audio param in responses streaming (#6312 )
* add audio, modalities param
* add test for gpt audio models
* add get_supported_openai_params for GPT audio models
* add supported params for audio
* test_audio_output_from_model
* bump openai to openai==1.52.0
* bump openai on pyproject
* fix audio test
* fix test mock_chat_response
* handle audio for Message
* fix handling audio for OAI compatible API endpoints
* fix linting
* fix mock dbrx test
* add audio to Delta
* handle model_response.choices.delta.audio
* fix linting
* build(model_prices_and_context_window.json): add gpt-4o-audio audio token cost tracking
* refactor(model_prices_and_context_window.json): refactor 'supports_audio' to be 'supports_audio_input' and 'supports_audio_output'
Allows for flag to be used for openai + gemini models (both support audio input)
* feat(cost_calculation.py): support cost calc for audio model
Closes https://github.com/BerriAI/litellm/issues/6302
* feat(utils.py): expose new `supports_audio_input` and `supports_audio_output` functions
Closes https://github.com/BerriAI/litellm/issues/6303
* feat(handle_jwt.py): support single dict list
* fix(cost_calculator.py): fix linting errors
* fix: fix linting error
* fix(cost_calculator): move to using standard openai usage cached tokens value
* test: fix test
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
2024-10-19 22:23:27 -07:00
Ishaan Jaff
979e8ea526
(refactor) get_cache_key to be under 100 LOC function ( #6327 )
...
* refactor - use helpers for name space and hashing
* use openai to get the relevant supported params
* use helpers for getting cache key
* fix test caching
* use get/set helpers for preset cache keys
* make get_cache_key under 100 LOC
* fix _get_model_param_value
* fix _get_caching_group
* fix linting error
* add unit testing for get cache key
* test_generate_streaming_content
2024-10-19 15:21:11 +05:30
Ishaan Jaff
610974b4fc
(code quality) add ruff check PLR0915 for too-many-statements ( #6309 )
...
* ruff add PLR0915
* add noqa for PLR0915
* fix noqa
* add # noqa: PLR0915
* # noqa: PLR0915
* # noqa: PLR0915
* # noqa: PLR0915
* add # noqa: PLR0915
* # noqa: PLR0915
* # noqa: PLR0915
* # noqa: PLR0915
* # noqa: PLR0915
2024-10-18 15:36:49 +05:30
Ishaan Jaff
f724f3131d
(testing) add unit tests for LLMCachingHandler Class ( #6279 )
...
* add unit testing for test_async_set_cache
* test test_async_log_cache_hit_on_callbacks
* assert the correct response type is returned
* test_convert_cached_result_to_model_response
* unit testing for caching handler
2024-10-17 19:12:57 +05:30
Krish Dholakia
38a9a106d2
LiteLLM Minor Fixes & Improvements (10/16/2024) ( #6265 )
...
* fix(caching_handler.py): handle positional arguments in add cache logic
Fixes https://github.com/BerriAI/litellm/issues/6264
* feat(litellm_pre_call_utils.py): allow forwarding openai org id to backend client
https://github.com/BerriAI/litellm/issues/6237
* docs(configs.md): add 'forward_openai_org_id' to docs
* fix(proxy_server.py): return model info if user_model is set
Fixes https://github.com/BerriAI/litellm/issues/6233
* fix(hosted_vllm/chat/transformation.py): don't set tools unless non-none
* fix(openai.py): improve debug log for openai 'str' error
Addresses https://github.com/BerriAI/litellm/issues/6272
* fix(proxy_server.py): fix linting error
* fix(proxy_server.py): fix linting errors
* test: skip WIP test
* docs(openai.md): add docs on passing openai org id from client to openai
2024-10-16 22:16:23 -07:00
Ishaan Jaff
d9a71650e3
(refactor) - caching use separate files for each cache class ( #6251 )
...
* fix remove qdrant semantic caching to it's own folder
* refactor use 1 file for s3 caching
* fix use sep files for in mem and redis caching
* fix refactor caching
* add readme.md for caching folder
2024-10-16 13:17:21 +05:30
Ishaan Jaff
97ba4eea7d
(refactor) sync caching - use LLMCachingHandler class for get_cache ( #6249 )
...
* caching - use _sync_set_cache
* add sync _sync_add_streaming_response_to_cache
* use caching class for cache storage
* fix use _sync_get_cache
* fix circular import
* use _update_litellm_logging_obj_environment
* use one helper for _process_async_embedding_cached_response
* fix _is_call_type_supported_by_cache
* fix checking cache
* fix sync get cache
* fix use _combine_cached_embedding_response_with_api_result
* fix _update_litellm_logging_obj_environment
* adjust test_redis_cache_acompletion_stream_bedrock
2024-10-16 12:33:49 +05:30
Ishaan Jaff
da6a7c3a55
(refactor) caching - use _sync_set_cache ( #6224 )
...
* caching - use _sync_set_cache
* add sync _sync_add_streaming_response_to_cache
* use caching class for cache storage
2024-10-16 10:38:07 +05:30
Ishaan Jaff
a69c670baa
(refactor) use helper function _assemble_complete_response_from_streaming_chunks to assemble complete responses in caching and logging callbacks ( #6220 )
...
* (refactor) use _assemble_complete_response_from_streaming_chunks
* add unit test for test_assemble_complete_response_from_streaming_chunks_1
* fix assemble complete_streaming_response
* config add logging_testing
* add logging_coverage in codecov
* test test_assemble_complete_response_from_streaming_chunks_3
* add unit tests for _assemble_complete_response_from_streaming_chunks
* fix remove unused / junk function
* add test for streaming_chunks when error assembling
2024-10-15 12:45:12 +05:30
Ishaan Jaff
cda0a993e2
fix importing Cache from litellm ( #6219 )
2024-10-15 08:47:23 +05:30
Ishaan Jaff
4ebe6813c0
(refactor caching) use common _retrieve_from_cache helper ( #6212 )
...
* use folder for caching
* fix importing caching
* fix clickhouse pyright
* fix linting
* fix correctly pass kwargs and args
* fix test case for embedding
* fix linting
* fix embedding caching logic
* fix refactor handle utils.py
* refactor async set stream cache
* fix linting
* refactor - use _retrieve_from_cache
* refactor use _convert_cached_result_to_model_response
* fix linting errors
2024-10-14 19:12:41 +05:30
Ishaan Jaff
d1bef4ad81
(refactor caching) use LLMCachingHandler for caching streaming responses ( #6210 )
...
* use folder for caching
* fix importing caching
* fix clickhouse pyright
* fix linting
* fix correctly pass kwargs and args
* fix test case for embedding
* fix linting
* fix embedding caching logic
* fix refactor handle utils.py
* refactor async set stream cache
* fix linting
2024-10-14 17:46:45 +05:30
Ishaan Jaff
4d1b4beb3d
(refactor) caching use LLMCachingHandler for async_get_cache and set_cache ( #6208 )
...
* use folder for caching
* fix importing caching
* fix clickhouse pyright
* fix linting
* fix correctly pass kwargs and args
* fix test case for embedding
* fix linting
* fix embedding caching logic
* fix refactor handle utils.py
* fix test_embedding_caching_azure_individual_items_reordered
2024-10-14 16:34:01 +05:30
