173 commits

Author	SHA1	Message	Date
Krrish Dholakia	2f1c5aa0c7	fix: setting cache responses on proxy	2023-12-07 20:39:40 -08:00
ishaan-jaff	762f28e4d7	(fix) make print_verbose non blocking	2023-12-07 17:31:32 -08:00
Krrish Dholakia	e5638e2c5d	fix(router.py): fix default caching response value	2023-12-07 13:44:31 -08:00
ishaan-jaff	7b389d10ca	(linting) fix	2023-12-06 20:30:33 -08:00
ishaan-jaff	ee70c4e822	(feat) router - add model_group_alias_map	2023-12-06 20:13:33 -08:00
ishaan-jaff	13f9e78799	(fix) router - errors with reading timeout, stream timeout, max retries	2023-12-06 09:19:51 -08:00
ishaan-jaff	aab6be654e	(fix) router - set read os.environ/ values	2023-12-06 08:59:33 -08:00
ishaan-jaff	ff028111cf	(fix) router len(num_retries)	2023-12-05 22:05:47 -08:00
ishaan-jaff	8e6c4c5310	(fix) router - allow users to call a specific_model explicit	2023-12-05 21:57:00 -08:00
Krrish Dholakia	acef6bd58d	refactor: linting fixes	2023-12-05 21:43:02 -08:00
Krrish Dholakia	7b83238cb5	fix(router.py): log when a call is retried or fallback happens	2023-12-05 21:29:58 -08:00
ishaan-jaff	56acded998	(router) better debugging using config.yaml	2023-12-05 18:07:27 -08:00
ishaan-jaff	e788a34da4	(chore) linting fix	2023-12-05 17:26:03 -08:00
ishaan-jaff	a532cf14ae	(feat) router - track original deployment names	2023-12-05 17:19:48 -08:00
ishaan-jaff	bb6a1968b3	(fix) router - allow user to call 1 deployment	2023-12-05 16:56:38 -08:00
ishaan-jaff	d2dab362df	(fix) proxy debugging display Init API key	2023-12-05 16:08:17 -08:00
Krrish Dholakia	d606a9cb4c	refactor(router.py): linting fixes	2023-12-05 13:33:44 -08:00
ishaan-jaff	63939c0a11	(fix) linting	2023-12-05 13:30:12 -08:00
Krrish Dholakia	58ab0a3f03	fix(router.py): fix cache init	2023-12-05 12:54:27 -08:00
ishaan-jaff	3f84ab04c4	(fix) router: Azure Client Init	2023-12-05 12:54:00 -08:00
ishaan-jaff	d9f083b5f8	(fix) router: remove misleading print statement	2023-12-05 12:54:00 -08:00
ishaan-jaff	951bcfc043	(fix) router init: raise error Azure API Base not set	2023-12-05 12:20:42 -08:00
Krrish Dholakia	88c95ca259	fix(_redis.py): support additional params for redis	2023-12-05 12:16:51 -08:00
ishaan-jaff	13261287ec	(fix) proxy: bug non OpenAI LLMs	2023-12-05 09:05:53 -08:00
ishaan-jaff	19646091fd	(feat) router: init stream, async stream, async, clients	2023-12-04 17:31:12 -08:00
Krrish Dholakia	e0ccb281d8	feat(utils.py): add async success callbacks for custom functions	2023-12-04 16:42:40 -08:00
ishaan-jaff	1247afb7a4	(feat) router: set max_retries + timeout	2023-12-04 16:09:31 -08:00
Krrish Dholakia	368fee224e	feat: support for azure key vault	2023-12-01 19:36:06 -08:00
Krrish Dholakia	ff4457e2d2	fix(router.py): back-off if no models available	2023-11-30 18:42:29 -08:00
Krrish Dholakia	032f71adb2	fix(router.py): support cloudflare ai gateway for azure models on router	2023-11-30 14:09:06 -08:00
ishaan-jaff	97ff0caf70	(feat) proxy: config - azure allow users to pass in base_url	2023-11-30 10:56:55 -08:00
Krrish Dholakia	c312ac4ca8	fix(main.py): don't pass stream to petals	2023-11-29 19:58:04 -08:00
ishaan-jaff	9780efca4b	(feat) router: async client Azure, OpenAI	2023-11-29 19:45:08 -08:00
Krrish Dholakia	1f5a1122fc	fix(replicate.py): fix custom prompt formatting	2023-11-29 19:44:09 -08:00
ishaan-jaff	3891462b29	(fix) router: azure/embedding support	2023-11-29 19:06:36 -08:00
ishaan-jaff	7bcc23e8e9	(fix) router: set default rpm/tpm when not set	2023-11-29 18:13:27 -08:00
ishaan-jaff	23af756531	(feat) router: random pick based on tpm/rpm	2023-11-29 17:54:06 -08:00
ishaan-jaff	088d2bc081	(fix) use weighted shuffle when rpm set	2023-11-29 17:13:11 -08:00
ishaan-jaff	3c6764efef	(feat) proxy+ router: support 1k request/second	2023-11-29 16:22:04 -08:00
ishaan-jaff	8a398a1777	(feat) proxy: add weighted shuffle + set cooldown to 1s	2023-11-29 16:09:31 -08:00
Krrish Dholakia	04a1c20bc5	fix(router.py): skip api key when generating model id for router deployments	2023-11-29 15:37:08 -08:00
Krrish Dholakia	383dd53e86	fix(main.py): passing client as a litellm-specific kwarg	2023-11-28 21:20:05 -08:00
ishaan-jaff	afd20098be	(feat) router: init client for OpenAI compatible providers	2023-11-28 17:49:53 -08:00
Krrish Dholakia	bb1267eb07	fix(router.py): fix exponential backoff to use retry-after if present in headers	2023-11-28 17:25:03 -08:00
ishaan-jaff	f5b558dde0	(fix) router red api_key, api_base, api_version	2023-11-28 17:10:20 -08:00
ishaan-jaff	282b9a37e5	(fix) router: passing client	2023-11-28 16:34:16 -08:00
ishaan-jaff	4d06c296e3	(router) re use client across requests	2023-11-28 16:21:16 -08:00
ishaan-jaff	94d35f1ec5	(feat) router: re-use the same client for high trafic	2023-11-28 15:44:56 -08:00
ishaan-jaff	2a69cab550	(feat) router track total, success, failed calls per model	2023-11-28 15:44:56 -08:00
Krrish Dholakia	094144de58	fix(router.py): removing model id before making call	2023-11-28 10:09:45 -08:00