phoenix/litellm-mirror
0
0
Fork 1
mirror of https://github.com/BerriAI/litellm.git synced 2025-04-25 10:44:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

[Perf Proxy] parallel request limiter - use one cache update call (#5932)

Browse source 
* fix parallel request limiter - use one cache update call

* ci/cd run again

* run ci/cd again

* use docker username password

* fix config.yml

* fix config

* fix config

* fix config.yml

* ci/cd run again

* use correct typing for batch set cache

* fix async_set_cache_pipeline

* fix only check user id tpm / rpm limits when limits set

* fix test_openai_azure_embedding_with_oidc_and_cf
This commit is contained in:
Ishaan Jaff 2024-09-27 10:26:15 -07:00
parent 71f68ac185
commit f4613a100d
7 changed files with 56 additions and 36 deletions
Download patch file Download diff file Expand all files Collapse all files

14
litellm/proxy/utils.py
View file

@ -242,6 +242,20 @@ class InternalUsageCache:
**kwargs,
)
async def async_batch_set_cache(
self,
cache_list: List,
litellm_parent_otel_span: Union[Span, None],
local_only: bool = False,
**kwargs,
) -> None:
return await self.dual_cache.async_batch_set_cache(
cache_list=cache_list,
local_only=local_only,
litellm_parent_otel_span=litellm_parent_otel_span,
**kwargs,
)
async def async_increment_cache(
self,
key,