Krrish Dholakia
|
308a6e11f8
|
fix(lowest_tpm_rpm_v2.py): ensure backwards compatibility for python 3.8
|
2024-04-18 21:42:35 -07:00 |
|
Krrish Dholakia
|
376ee4e9d7
|
fix(test_lowest_tpm_rpm_routing_v2.py): unit testing for usage-based-routing-v2
|
2024-04-18 21:38:00 -07:00 |
|
Krrish Dholakia
|
72691e05f4
|
fix(tpm_rpm_routing_v2.py): fix tpm rpm routing
|
2024-04-18 20:01:22 -07:00 |
|
Krrish Dholakia
|
eb7f260efc
|
fix(lowest_tpm_rpm_v2.py): don't fail calls if redis fails to connect
|
2024-04-12 19:36:59 -07:00 |
|
Krrish Dholakia
|
c177407f7b
|
test(test_openai_endpoints.py): add concurrency testing for user defined rate limits on proxy
|
2024-04-12 18:56:13 -07:00 |
|
Krrish Dholakia
|
d9b8f63e86
|
fix(router.py): support pre_call_rpm_check for lowest_tpm_rpm_v2 routing
have routing strategies expose an ‘update rpm’ function; for checking + updating rpm pre call
|
2024-04-12 18:25:14 -07:00 |
|
Krrish Dholakia
|
8f06c2d8c4
|
fix(router.py): fix datetime object
|
2024-04-10 17:55:24 -07:00 |
|
Krrish Dholakia
|
384245e331
|
fix(router.py): make get_cooldown_deployment logic async
|
2024-04-10 16:57:01 -07:00 |
|
Krrish Dholakia
|
f5206d592a
|
fix(router.py): generate consistent model id's
having the same id for a deployment, lets redis usage caching work across multiple instances
|
2024-04-10 15:23:57 -07:00 |
|
Krrish Dholakia
|
31e2d4e6d1
|
feat(lowest_tpm_rpm_v2.py): move to using redis.incr and redis.mget for getting model usage from redis
makes routing work across multiple instances
|
2024-04-10 14:56:23 -07:00 |
|