Krrish Dholakia
78474b1ce7
test(test_lowest_latency_routing.py): add time.sleep to reduce test flakiness
2024-06-07 11:28:33 -07:00
Krrish Dholakia
f007bf7e21
feat(lowest_latency.py): route by time to first token, for streaming requests (if available)
...
Closes https://github.com/BerriAI/litellm/issues/3574
2024-05-21 13:08:17 -07:00
Ishaan Jaff
2b788f1c0f
fix - test_lowest_latency_routing_first_pick
2024-05-15 14:24:13 -07:00
Krrish Dholakia
cb88ed4df8
fix(lowest_latency.py): fix the size of the latency list to 10 by default (can be modified)
2024-05-03 09:00:32 -07:00
Krrish Dholakia
7ae28bfcc9
fix(lowest_latency.py): allow setting a buffer for getting values within a certain latency threshold
...
if an endpoint is slow - it's completion time might not be updated till the call is completed. This prevents us from overloading those endpoints, in a simple way.
2024-04-30 12:00:26 -07:00
Ishaan Jaff
2c6ec254a0
test - lowest latency router
2024-04-29 15:51:01 -07:00
Ishaan Jaff
4934385af6
test - lowest latency logger
2024-04-24 16:35:43 -07:00
Krish Dholakia
b8d285d120
Merge pull request #2798 from CLARKBENHAM/main
...
add test for rate limits - Router isn't coroutine safe
2024-04-06 08:47:40 -07:00
Krrish Dholakia
48a5948081
fix(router.py): handle id being passed in as int
2024-04-04 14:23:10 -07:00
CLARKBENHAM
006d0237e4
formating
2024-04-02 19:56:07 -07:00
CLARKBENHAM
2dd0c32612
fix lowest latency tests
2024-04-02 19:10:40 -07:00
CLARKBENHAM
dada96ccf6
param both tests to include failure (also fix prev)
2024-04-02 18:53:42 -07:00
CLARKBENHAM
22ea7abc94
tests showing error
2024-04-02 18:45:05 -07:00
Krrish Dholakia
568f4ae461
test(test_lowest_latency_routing.py): use the correct cache key
2024-01-10 22:15:01 +05:30
Krrish Dholakia
744e8bccc7
refactor: cleanup duplicates
2024-01-10 21:42:20 +05:30
Krish Dholakia
e635ca2151
Merge branch 'main' into litellm_latency_routing_updates
2024-01-10 21:33:54 +05:30
Krrish Dholakia
7df19b2f7c
fix(router.py): allow user to control the latency routing time window
2024-01-10 20:56:52 +05:30
Krrish Dholakia
f288b12411
fix(lowest_latency.py): add back tpm/rpm checks, configurable time window
2024-01-10 20:52:01 +05:30
Krrish Dholakia
fb9ebfbedd
feat(lowest_latency.py): support expanded time window for latency based routing
...
uses a 1hr avg. of latency for deployments, to determine which to route to
https://github.com/BerriAI/litellm/issues/1361
2024-01-09 09:38:04 +05:30
Krrish Dholakia
d3dee9b20c
test(test_lowest_latency_routing.py): add more tests
2023-12-30 17:41:42 +05:30
Krrish Dholakia
25ee96271e
fix(router.py): fix latency based routing
2023-12-30 17:25:40 +05:30
