phoenix/litellm-mirror
0
0
Fork 1
mirror of https://github.com/BerriAI/litellm.git synced 2025-04-26 19:24:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat(lowest_tpm_rpm_v2.py): move to using redis.incr and redis.mget for getting model usage from redis

Browse source 
makes routing work across multiple instances
This commit is contained in:
Krrish Dholakia 2024-04-10 14:56:23 -07:00
parent 06a0ca1e80
commit 31e2d4e6d1
5 changed files with 437 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

2
litellm/proxy/hooks/batch_redis_get.py
View file

@ -79,7 +79,7 @@ class _PROXY_BatchRedisRequests(CustomLogger):
self.print_verbose(f"redis keys: {keys}")
if len(keys) > 0:
key_value_dict = (
await litellm.cache.cache.async_get_cache_pipeline(
await litellm.cache.cache.async_batch_get_cache(
key_list=keys
)
)