phoenix/litellm-mirror
0
0
Fork 1
mirror of https://github.com/BerriAI/litellm.git synced 2025-04-26 03:04:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

(router_strategy/) ensure all async functions use async cache methods (#6489)

Browse source 
* fix router strat

* use async set / get cache in router_strategy

* add coverage for router strategy

* fix imports

* fix batch_get_cache

* use async methods for least busy

* fix least busy use async methods

* fix test_dual_cache_increment

* test async_get_available_deployment when routing_strategy="least-busy"
This commit is contained in:
Ishaan Jaff 2024-10-29 21:07:17 +05:30 • committed by GitHub
parent f9ba74ef87
commit 441adad3ae
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 202 additions and 20 deletions
Download patch file Download diff file Expand all files Collapse all files

4
litellm/router_strategy/lowest_latency.py
View file

@ -243,7 +243,7 @@ class LowestLatencyLoggingHandler(CustomLogger):
"latency"
][: self.routing_args.max_latency_list_size - 1] + [1000.0]
self.router_cache.set_cache(
await self.router_cache.async_set_cache(
key=latency_key,
value=request_count_dict,
ttl=self.routing_args.ttl,
@ -384,7 +384,7 @@ class LowestLatencyLoggingHandler(CustomLogger):
request_count_dict[id][precise_minute].get("rpm", 0) + 1
)
self.router_cache.set_cache(
await self.router_cache.async_set_cache(
key=latency_key, value=request_count_dict, ttl=self.routing_args.ttl
) # reset map within window