phoenix/litellm-mirror
0
0
Fork 1
mirror of https://github.com/BerriAI/litellm.git synced 2025-04-25 10:44:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
21443 commits 2480 branches 894 tags 534 MiB
Commit graph

153 commits

Author SHA1 Message Date
Lunik
9ba9b3891f
⚡️ perf: Remove test violation on each stream chunk 
Signed-off-by: Lunik <lunik@tiwabbit.fr>
 2024-05-03 20:51:40 +02:00
Lunik
e7405f105c
✅ ci: Add tests 
Signed-off-by: Lunik <lunik@tiwabbit.fr>
 2024-05-03 20:50:37 +02:00
Lunik
6cec252b07
✨ feat: Add Azure Content-Safety Proxy hooks 
Signed-off-by: Lunik <lunik@tiwabbit.fr>
 2024-05-02 23:21:08 +02:00
Krrish Dholakia
180cf9bd5c feat(lowest_tpm_rpm_v2.py): move to using redis.incr and redis.mget for getting model usage from redis 
makes routing work across multiple instances
 2024-04-10 14:56:23 -07:00
Krrish Dholakia
6467dd4e11 fix(tpm_rpm_limiter.py): fix cache init logic 2024-04-01 18:01:38 -07:00
Krrish Dholakia
383f12bbd3 test(test_max_tpm_rpm_limiter.py): unit tests for key + team based tpm rpm limits on proxy 2024-04-01 08:00:01 -07:00
Krrish Dholakia
f58fefd589 fix(tpm_rpm_limiter.py): enable redis caching for tpm/rpm checks on keys/user/teams 
allows tpm/rpm checks to work across instances

https://github.com/BerriAI/litellm/issues/2730
 2024-03-30 20:01:36 -07:00
Krrish Dholakia
5a117490ec fix(proxy_server.py): fix tpm/rpm limiting for jwt auth 
fixes tpm/rpm limiting for jwt auth and implements unit tests for jwt auth
 2024-03-28 21:19:34 -07:00
Krrish Dholakia
e10eb8f6fe feat(llm_guard.py): enable key-specific llm guard check 2024-03-26 17:21:51 -07:00
Ishaan Jaff
5d121a9f3c (fix) stop using f strings with logger 2024-03-25 10:47:18 -07:00
Krrish Dholakia
0521e8a1d9 fix(prompt_injection_detection.py): fix type check 2024-03-21 08:56:13 -07:00
Krrish Dholakia
8e8c4e214e fix: fix linting issue 2024-03-21 08:19:09 -07:00
Krrish Dholakia
d91f9a9f50 feat(proxy_server.py): enable llm api based prompt injection checks 
run user calls through an llm api to check for prompt injection attacks. This happens in parallel to th
e actual llm call using `async_moderation_hook`
 2024-03-20 22:43:42 -07:00
Krrish Dholakia
f24d3ffdb6 fix(proxy_server.py): fix import 2024-03-20 19:15:06 -07:00
Krrish Dholakia
3bb0e24cb7 fix(prompt_injection_detection.py): ensure combinations are actual phrases, not just 1-2 words 
reduces misflagging

https://github.com/BerriAI/litellm/issues/2601
 2024-03-20 19:09:38 -07:00
Krrish Dholakia
8a20ea795b feat(batch_redis_get.py): batch redis GET requests for a given key + call type 
reduces number of redis requests. 85ms latency improvement over 3 minutes of load (19k requests).
 2024-03-15 14:54:16 -07:00
Krrish Dholakia
226953e1d8 feat(batch_redis_get.py): batch redis GET requests for a given key + call type 
reduces the number of GET requests we're making in high-throughput scenarios
 2024-03-15 14:40:11 -07:00
Krrish Dholakia
7876aa2d75 fix(parallel_request_limiter.py): handle metadata being none 2024-03-14 10:02:41 -07:00
Krrish Dholakia
ad55f4dbb5 feat(proxy_server.py): retry if virtual key is rate limited 
currently for chat completions
 2024-03-05 19:00:03 -08:00
Krrish Dholakia
b3574f2b37 fix(parallel_request_limiter.py): handle none scenario 2024-02-26 20:09:06 -08:00
Krrish Dholakia
f86ab19067 fix(parallel_request_limiter.py): fix team rate limit enforcement 2024-02-26 18:06:13 -08:00
Krrish Dholakia
f84ac35000 feat(parallel_request_limiter.py): enforce team based tpm / rpm limits 2024-02-26 16:20:41 -08:00
ishaan-jaff
a13243652f (fix) failing parallel_Request_limiter test 2024-02-22 19:16:22 -08:00
ishaan-jaff
1fff8f8105 (fix) don't double check curr data and time 2024-02-22 18:50:02 -08:00
ishaan-jaff
b5900099af (feat) tpm/rpm limit by User 2024-02-22 18:44:03 -08:00
Krrish Dholakia
d706d3b672 fix(presidio_pii_masking.py): enable user to pass ad hoc recognizer for pii masking 2024-02-20 16:01:15 -08:00
Krrish Dholakia
72bcd5a4af fix(presidio_pii_masking.py): enable user to pass their own ad hoc recognizers to presidio 2024-02-20 15:19:31 -08:00
Krrish Dholakia
9fa4dfbdd3 test(test_presidio_pii_masking.py): add more unit tests 2024-02-19 16:30:44 -08:00
Krrish Dholakia
448537e684 feat(presidio_pii_masking.py): allow request level controls for turning on/off pii masking 
https://github.com/BerriAI/litellm/issues/2003
 2024-02-17 11:04:56 -08:00
Krrish Dholakia
3565f74338 docs(enterprise.md): add llama guard tutorial to enterprise docs 2024-02-17 09:25:49 -08:00
Krrish Dholakia
cd8d35107b fix: check key permissions for turning on/off pii masking 2024-02-15 20:16:15 -08:00
Krrish Dholakia
6b91f48c64 fix(presidio_pii_masking.py): fix conditional check 2024-02-13 22:11:03 -08:00
Krrish Dholakia
f68b656040 feat(presidio_pii_masking.py): enable output parsing for pii masking 2024-02-13 21:36:57 -08:00
Krrish Dholakia
2d845b12ed feat(proxy_server.py): support for pii masking with microsoft presidio 2024-02-10 20:21:12 -08:00
Krrish Dholakia
b9393fb769 fix(test_parallel_request_limiter.py): use mock responses for streaming 2024-02-08 21:45:38 -08:00
ishaan-jaff
13fe72d6d5 (fix) parallel_request_limiter debug 2024-02-06 12:43:28 -08:00
Krrish Dholakia
92058cbcd4 fix(utils.py): override default success callbacks with dynamic callbacks if set 2024-02-02 06:21:43 -08:00
Krrish Dholakia
f9acad87dc feat(proxy_server.py): enable cache controls per key + no-store cache flag 2024-01-30 20:46:50 -08:00
Krrish Dholakia
bbe71c8375 fix(test_parallel_request_limiter): increase time limit for waiting for success logging event to happen 2024-01-30 13:26:17 -08:00
Krrish Dholakia
f05aba1f85 fix(utils.py): add metadata to logging obj on setup, if exists 2024-01-19 17:29:47 -08:00
Krrish Dholakia
1a29272b47 fix(parallel_request_limiter.py): handle tpm/rpm limits being null 2024-01-19 10:22:27 -08:00
Krrish Dholakia
5dac2402ef test(test_parallel_request_limiter.py): unit testing for tpm/rpm rate limits 2024-01-18 15:28:28 -08:00
Krrish Dholakia
aef59c554f feat(parallel_request_limiter.py): add support for tpm/rpm limits 2024-01-18 13:52:15 -08:00
Krrish Dholakia
1ea3833ef7 fix(parallel_request_limiter.py): decrement count for failed llm calls 
https://github.com/BerriAI/litellm/issues/1477
 2024-01-18 12:42:14 -08:00
Krrish Dholakia
4905929de3 refactor: add black formatting 2023-12-25 14:11:20 +05:30
Krrish Dholakia
9f79f75635 fix(proxy/utils.py): return different exceptions if key is invalid vs. expired 
https://github.com/BerriAI/litellm/issues/1230
 2023-12-25 10:29:44 +05:30
Krrish Dholakia
402b2e5733 build(test_streaming.py): fix linting issues 2023-12-25 07:34:54 +05:30
Krrish Dholakia
89ee9fe400 fix(proxy_server.py): manage budget at user-level not key-level 
https://github.com/BerriAI/litellm/issues/1220
 2023-12-22 15:10:38 +05:30
Krrish Dholakia
1a32228da5 feat(proxy_server.py): support max budget on proxy 2023-12-21 16:07:20 +05:30
Krrish Dholakia
4791dda66f feat(proxy_server.py): enable infinite retries on rate limited requests 2023-12-15 20:03:41 -08:00