685 commits

Author	SHA1	Message	Date
ishaan-jaff	8be91e2590	(feat) proxy+ router: support 1k request/second	2023-11-29 16:22:04 -08:00
ishaan-jaff	f32661661e	(feat) proxy: add weighted shuffle + set cooldown to 1s	2023-11-29 16:09:31 -08:00
Krrish Dholakia	e05063a399	fix(router.py): skip api key when generating model id for router deployments	2023-11-29 15:37:08 -08:00
Krrish Dholakia	ceadb1547d	fix(main.py): passing client as a litellm-specific kwarg	2023-11-28 21:20:05 -08:00
ishaan-jaff	8a22afd677	(feat) router: init client for OpenAI compatible providers	2023-11-28 17:49:53 -08:00
Krrish Dholakia	60d6b6bc37	fix(router.py): fix exponential backoff to use retry-after if present in headers	2023-11-28 17:25:03 -08:00
ishaan-jaff	d55df1c9c8	(fix) router red api_key, api_base, api_version	2023-11-28 17:10:20 -08:00
ishaan-jaff	9d69ea5b12	(fix) router: passing client	2023-11-28 16:34:16 -08:00
ishaan-jaff	493573f4d2	(router) re use client across requests	2023-11-28 16:21:16 -08:00
ishaan-jaff	57d774f3ad	(feat) router: re-use the same client for high trafic	2023-11-28 15:44:56 -08:00
ishaan-jaff	7ec948c9b9	(feat) router track total, success, failed calls per model	2023-11-28 15:44:56 -08:00
Krrish Dholakia	32a14e6cf0	fix(router.py): removing model id before making call	2023-11-28 10:09:45 -08:00
Krrish Dholakia	ae268a5c49	refactor(router.py): fix linting errors	2023-11-27 22:11:53 -08:00
Krrish Dholakia	861c66580f	refactor(router.py): fix linting errors	2023-11-27 22:08:48 -08:00
Krrish Dholakia	624ef83455	build: adding debug logs to gitignore	2023-11-27 22:05:07 -08:00
Krrish Dholakia	91e8ff439d	fix(main.py): fix linting errors	2023-11-27 19:11:38 -08:00
ishaan-jaff	741e923127	(feat) use api_base, api_key as model	2023-11-27 18:08:47 -08:00
Krrish Dholakia	e564f643f5	fix(router.py): speed improvements to the router	2023-11-27 17:35:26 -08:00
ishaan-jaff	5257fc1f9c	(fix) router: allow same model/name	2023-11-27 16:26:09 -08:00
Krrish Dholakia	2680f84cfc	fix(router.py): fix fallbacks	2023-11-25 19:34:20 -08:00
Krrish Dholakia	67fe8824b3	fix(router.py): check for fallbacks in completion params for router	2023-11-25 18:46:45 -08:00
Krrish Dholakia	8884ceb606	fix(proxy_server.py): expose a /health endpoint	2023-11-25 18:28:47 -08:00
Krrish Dholakia	2b9c5bf706	fix(router.py): check if fallbacks is none	2023-11-25 14:58:07 -08:00
Krrish Dholakia	b1b2d0c2b7	fix(utils.py): fix bedrock + cohere calls	2023-11-25 14:45:42 -08:00
Krrish Dholakia	e421642ba8	fix: fix linting issues	2023-11-24 15:46:25 -08:00
Krrish Dholakia	68168cc743	fix(router.py): fix retry logic	2023-11-24 13:27:44 -08:00
Krrish Dholakia	9618718080	test: refactor testing order	2023-11-24 12:47:28 -08:00
Krrish Dholakia	27fd144950	docs(simple_proxy.md): add tutorial for doing fallbacks + retries + timeouts on the proxy	2023-11-24 12:20:38 -08:00
Krrish Dholakia	9185d57f6d	fix(router.py): fixing embedding call	2023-11-23 21:07:02 -08:00
Krrish Dholakia	f3bef86848	fix(router.py): use an older version of async for compatibility	2023-11-23 21:00:53 -08:00
Krrish Dholakia	3a8d7ec835	fix(router.py): add modelgroup to call metadata	2023-11-23 20:55:49 -08:00
Krrish Dholakia	1b26a0931e	fix(utils.py): make failure logging sync	2023-11-23 20:19:27 -08:00
Krrish Dholakia	ae5d674c15	fix(router.py): fix linting errors	2023-11-23 16:50:19 -08:00
Krrish Dholakia	e4deb09eb6	fix(router.py): add support for context window fallbacks on router	2023-11-23 16:43:02 -08:00
ishaan-jaff	616b876f23	(fix) router	2023-11-23 16:28:19 -08:00
Krrish Dholakia	7f632e6e2f	fix(router.py): enable async completions with model fallbacks	2023-11-23 16:15:57 -08:00
Krrish Dholakia	59d084342d	fix(router.py): enable fallbacks for sync completions	2023-11-23 16:06:46 -08:00
Krrish Dholakia	7610b1f0af	feat(proxy_server.py): add in-memory caching for user api keys	2023-11-23 13:21:45 -08:00
Krrish Dholakia	0e3064ac8c	fix(router.py): fix caching for tracking cooldowns + usage	2023-11-23 11:13:32 -08:00
Krrish Dholakia	2df4791ae9	fix: fix linting errors	2023-11-22 19:59:25 -08:00
Krrish Dholakia	497419a766	fix(router.py): add support for cooldowns with redis	2023-11-22 19:54:22 -08:00
Krrish Dholakia	a2207d462e	feat(router.py): add server cooldown logic	2023-11-22 15:59:48 -08:00
Krrish Dholakia	73d70ef01c	feat(router.py): adding latency-based routing strategy	2023-11-21 21:19:27 -08:00
ishaan-jaff	2b1fc64f36	(fix) using callbacks with router	2023-11-20 19:08:53 -08:00
Krrish Dholakia	2ac804a42f	feat(proxy_server.py): enable model aliases	2023-11-20 16:51:04 -08:00
Krrish Dholakia	7472be1529	fix(routing.py): update token usage on streaming	2023-11-20 14:19:25 -08:00
Krrish Dholakia	5f4dca301b	refactor(router.py): adding user support message	2023-11-18 19:05:45 -08:00
Krrish Dholakia	43c26f3382	docs(routing.md): updating docs for managing multiple deployments	2023-11-18 19:02:50 -08:00
Krrish Dholakia	cf0a9f591c	fix(router.py): introducing usage-based-routing	2023-11-17 17:56:26 -08:00
Krrish Dholakia	452946b2f8	refactor(router.py): code cleanup	2023-11-17 17:05:46 -08:00