165 commits

Author	SHA1	Message	Date
Krrish Dholakia	2cea8b0e83	fix(router.py): periodically re-initialize azure/openai clients to solve max conn issue	2023-12-30 15:48:34 +05:30
Krrish Dholakia	402d59d0ff	fix(lowest_tpm_rpm_routing.py): broaden scope of get deployment logic	2023-12-30 13:27:50 +05:30
Krrish Dholakia	e1925d0e29	fix(router.py): support retry and fallbacks for atext_completion	2023-12-30 11:19:32 +05:30
ishaan-jaff	b5e819300b	(feat) router, add ModelResponse type hints	2023-12-30 10:44:13 +05:30
Krrish Dholakia	a11940f4eb	fix(router.py): handle initial scenario for tpm/rpm routing	2023-12-30 07:28:45 +05:30
Krrish Dholakia	1933d44cbd	fix(router.py): fix int logic	2023-12-29 20:41:56 +05:30
Krrish Dholakia	a30f00276b	refactor(lowest_tpm_rpm.py): move tpm/rpm based routing to a separate file for better testing	2023-12-29 18:33:43 +05:30
Krrish Dholakia	ffe2350428	fix(least_busy.py): support consistent use of model id instead of deployment name	2023-12-29 17:05:26 +05:30
Krrish Dholakia	90ed04d992	fix(router.py): support wait_for for async completion calls	2023-12-29 15:27:20 +05:30
Krrish Dholakia	b9dd46de6c	feat(router.py): support 'retry_after' param, to set min timeout before retrying a failed request (default 0)	2023-12-29 15:18:28 +05:30
Krrish Dholakia	6bd6d68990	feat(proxy_server.py): support maxage cache control	2023-12-26 17:50:27 +05:30
Krrish Dholakia	8f3732eeeb	docs(user_keys.md): docs on passing user keys to litellm proxy	2023-12-26 13:55:28 +05:30
Krrish Dholakia	a4633c57c4	fix(router.py): accept dynamic api key	2023-12-26 13:16:22 +05:30
Krrish Dholakia	79978c44ba	refactor: add black formatting	2023-12-25 14:11:20 +05:30
Krrish Dholakia	d1dea7c87d	fix(utils.py): log user_id to langfuse	2023-12-23 12:14:09 +05:30
Krish Dholakia	a52bb3ff1c	Merge pull request #1182 from sumanth13131/usage-based-routing-fix ... usage_based_routing_fix	2023-12-23 11:50:34 +05:30
Krish Dholakia	f509ee4ee1	Merge pull request #1195 from AllentDan/fix-routing ... fix least_busy router by updating min_traffic	2023-12-23 11:45:35 +05:30
Krrish Dholakia	61ab8dd5c1	fix(proxy_server.py): handle misformatted json body in chat completion request	2023-12-22 12:30:36 +05:30
Krrish Dholakia	c084f04a35	fix(router.py): add support for async image generation endpoints	2023-12-21 14:38:44 +05:30
Krrish Dholakia	b873833340	fix(router.py): only do sync image gen fallbacks for now ... The customhttptransport we use for dall-e-2 only works for sync httpx calls, not async. Will need to spend some time writing the async version n	2023-12-20 19:10:59 +05:30
Krrish Dholakia	89b808d767	feat(router.py): support async image generation on router	2023-12-20 17:24:20 +05:30
AllentDan	4b84f0f049	fix least_busy router by updating min_traffic	2023-12-20 18:16:00 +08:00
sumanth	64dfd0d9a6	fix	2023-12-19 13:15:48 +05:30
Krrish Dholakia	00826e195e	fix(router.py): init deployment_latency_map even if model_list is empty	2023-12-18 17:50:26 -08:00
Krrish Dholakia	d9dfa3defc	fix(Dockerfile): support mac	2023-12-16 16:01:02 -08:00
Krrish Dholakia	68c53256ab	fix(proxy_server.py): setup depedencies on server startup	2023-12-16 11:56:11 -08:00
Krrish Dholakia	e76ed6be7d	feat(router.py): support caching groups	2023-12-15 21:45:51 -08:00
Krrish Dholakia	8e7491c67e	fix(router.py): init clients for azure/openai on set_model_list	2023-12-15 20:23:29 -08:00
Krrish Dholakia	6081602bf8	fix(router.py): support openai-compatible endpoints	2023-12-15 14:47:54 -08:00
Krrish Dholakia	07a54977a2	fix(router.py): adding mistral to list of openai-compatible endpoints	2023-12-15 14:24:43 -08:00
ishaan-jaff	5731b3bc23	(feat) router - remove confusing -ModelID-XXXX	2023-12-15 07:07:53 +05:30
Krrish Dholakia	bb5b883316	fix(main.py): support async streaming for text completions endpoint	2023-12-14 13:56:32 -08:00
ishaan-jaff	072fdac48c	(feat) caching - add supported call types	2023-12-14 22:27:14 +05:30
ishaan-jaff	18478382bf	(feat) add mistral to proxy + router	2023-12-14 18:20:08 +05:30
ishaan-jaff	0ea0e29bf1	(feat) proxy - use model_group_alias	2023-12-14 13:08:14 +05:30
Krrish Dholakia	0cf0c2d6dd	fix(router.py): deepcopy initial model list, don't mutate it	2023-12-12 09:54:06 -08:00
ishaan-jaff	5e9286ed41	(fix) use deepcopy for model list	2023-12-12 09:53:52 -08:00
Krrish Dholakia	bbf094dcf5	fix(router.py): reset caching correctly	2023-12-11 19:57:34 -08:00
Krrish Dholakia	4fd400015f	test(test_custom_callback_router.py): add async azure testing for router	2023-12-11 16:40:35 -08:00
ishaan-jaff	ae0cb38e47	(fix) router - only init cache when its none	2023-12-09 15:20:08 -08:00
Krrish Dholakia	a65c8919fc	fix(router.py): fix least-busy routing	2023-12-08 20:29:49 -08:00
ishaan-jaff	0740728be0	(feat) proxy: pass model_info as kwargs	2023-12-08 14:26:18 -08:00
Krrish Dholakia	30204497e0	fix: setting cache responses on proxy	2023-12-07 20:39:40 -08:00
ishaan-jaff	f744445db4	(fix) make print_verbose non blocking	2023-12-07 17:31:32 -08:00
Krrish Dholakia	69c34493ce	fix(router.py): fix default caching response value	2023-12-07 13:44:31 -08:00
ishaan-jaff	8920f12a8d	(linting) fix	2023-12-06 20:30:33 -08:00
ishaan-jaff	1177c54636	(feat) router - add model_group_alias_map	2023-12-06 20:13:33 -08:00
ishaan-jaff	2ecbd92f13	(fix) router - errors with reading timeout, stream timeout, max retries	2023-12-06 09:19:51 -08:00
ishaan-jaff	788da2ea68	(fix) router - set read os.environ/ values	2023-12-06 08:59:33 -08:00
ishaan-jaff	105da7bd58	(fix) router len(num_retries)	2023-12-05 22:05:47 -08:00