Krrish Dholakia
|
0d9c96bebf
|
test(test_prometheus_services.py): fix testing to handle caching ping in init
|
2024-04-19 16:15:29 -07:00 |
|
Krrish Dholakia
|
84685b5f34
|
fix(_redis.py): fix args passed to redis.from_url argument
|
2024-04-19 15:27:11 -07:00 |
|
Krrish Dholakia
|
08cf77623b
|
fix(caching.py): remove url parsing logic - causing redis ssl connections to fail
this reverts a change that was causing redis url w/ ssl to fail. this also adds unit testing for this sc
enario, to prevent future regressions
|
2024-04-19 14:01:13 -07:00 |
|
Krrish Dholakia
|
9c42c847a5
|
fix(router.py): instrument pre-call-checks for all openai endpoints
|
2024-04-18 21:54:25 -07:00 |
|
Krrish Dholakia
|
81573b2dd9
|
fix(test_lowest_tpm_rpm_routing_v2.py): unit testing for usage-based-routing-v2
|
2024-04-18 21:38:00 -07:00 |
|
Krrish Dholakia
|
a05f148c17
|
fix(tpm_rpm_routing_v2.py): fix tpm rpm routing
|
2024-04-18 20:01:22 -07:00 |
|
Krrish Dholakia
|
f65005f273
|
fix(caching.py): fix error logging
|
2024-04-18 17:21:10 -07:00 |
|
Krrish Dholakia
|
0afb5d6573
|
fix(caching.py): add call type to service logger object
|
2024-04-18 16:53:34 -07:00 |
|
Krrish Dholakia
|
0f95a824c4
|
feat(prometheus_services.py): emit proxy latency for successful llm api requests
uses prometheus histogram for this
|
2024-04-18 16:04:35 -07:00 |
|
Krrish Dholakia
|
bef24cd4ab
|
fix(caching.py): don't decode a string
|
2024-04-13 18:48:03 -07:00 |
|
Krrish Dholakia
|
4e81acf2c6
|
feat(prometheus_services.py): monitor health of proxy adjacent services (redis / postgres / etc.)
|
2024-04-13 18:15:02 -07:00 |
|
Krrish Dholakia
|
c03b0bbb24
|
fix(router.py): support pre_call_rpm_check for lowest_tpm_rpm_v2 routing
have routing strategies expose an ‘update rpm’ function; for checking + updating rpm pre call
|
2024-04-12 18:25:14 -07:00 |
|
Krrish Dholakia
|
76bd667b12
|
fix(caching.py): fix async batch redis get request
|
2024-04-10 21:37:45 -07:00 |
|
Krrish Dholakia
|
180cf9bd5c
|
feat(lowest_tpm_rpm_v2.py): move to using redis.incr and redis.mget for getting model usage from redis
makes routing work across multiple instances
|
2024-04-10 14:56:23 -07:00 |
|
Ishaan Jaff
|
c4cb0afa98
|
feat - delete cache key
|
2024-04-04 10:56:47 -07:00 |
|
Krish Dholakia
|
7233e5ab25
|
Merge pull request #2789 from BerriAI/litellm_set_ttl
fix(proxy_server.py): allow user to set in-memory + redis ttl
|
2024-04-02 08:53:34 -07:00 |
|
Krrish Dholakia
|
203e2776f8
|
fix(proxy_server.py): allow user to set in-memory + redis ttl
addresses - https://github.com/BerriAI/litellm/issues/2700
|
2024-04-01 19:13:23 -07:00 |
|
Ishaan Jaff
|
4ebb4570be
|
(fix) handle urls correctly
|
2024-04-01 17:08:48 -07:00 |
|
Ishaan Jaff
|
916e4032ee
|
redis show client_list, info
|
2024-04-01 16:54:42 -07:00 |
|
Krrish Dholakia
|
5926792de6
|
fix(caching.py): respect redis namespace for all redis get/set requests
|
2024-03-30 20:20:29 -07:00 |
|
Krrish Dholakia
|
f58fefd589
|
fix(tpm_rpm_limiter.py): enable redis caching for tpm/rpm checks on keys/user/teams
allows tpm/rpm checks to work across instances
https://github.com/BerriAI/litellm/issues/2730
|
2024-03-30 20:01:36 -07:00 |
|
Ishaan Jaff
|
098a03facc
|
(fix) undo changes from other branches
|
2024-03-26 09:22:19 -07:00 |
|
Ishaan Jaff
|
b8af946fb9
|
(feat) /cache/flushall
|
2024-03-26 09:18:58 -07:00 |
|
Ishaan Jaff
|
ade5d58331
|
(fix) in mem redis reads
|
2024-03-26 09:10:49 -07:00 |
|
Ishaan Jaff
|
553ac19f9f
|
(feat) improve cache debugging litellm
|
2024-03-25 18:26:58 -07:00 |
|
Ishaan Jaff
|
32d4756943
|
(fix) print verbose in batch writing redis
|
2024-03-25 18:02:31 -07:00 |
|
Ishaan Jaff
|
7134d66fae
|
Merge branch 'main' into litellm_batch_write_redis_cache
|
2024-03-25 16:41:29 -07:00 |
|
Ishaan Jaff
|
abce689862
|
(feat) batch write redis cache output
|
2024-03-25 16:39:47 -07:00 |
|
Ishaan Jaff
|
37aadba959
|
(feat) v0 batch redis cache writes
|
2024-03-25 15:20:10 -07:00 |
|
Krrish Dholakia
|
591a0a376e
|
fix(caching.py): support default ttl for caching
|
2024-03-25 13:40:17 -07:00 |
|
Krish Dholakia
|
007d439017
|
Merge pull request #2606 from BerriAI/litellm_jwt_auth_updates
fix(handle_jwt.py): track spend for user using jwt auth
|
2024-03-20 19:40:17 -07:00 |
|
Krrish Dholakia
|
8bb00c4ae8
|
fix(caching.py): enable async setting of cache for dual cache
|
2024-03-20 18:42:34 -07:00 |
|
Ishaan Jaff
|
cace0bd6fb
|
(fix) self.redis_version issue
|
2024-03-20 10:36:08 -07:00 |
|
Ishaan Jaff
|
7b29273b14
|
(fix) redis 6.2 version incompatibility issue
|
2024-03-20 09:38:21 -07:00 |
|
Ishaan Jaff
|
2256ece5a9
|
(feat) litellm cache ping
|
2024-03-20 08:24:13 -07:00 |
|
Krrish Dholakia
|
03acc07380
|
fix(caching.py): pass redis kwargs to connection pool init
|
2024-03-18 08:21:36 -07:00 |
|
Krrish Dholakia
|
48f96dd563
|
fix(caching.py): close redis connection pool upon proxy shutdown
|
2024-03-16 10:39:58 -07:00 |
|
Krrish Dholakia
|
3072137739
|
test(test_caching.py): fix async tests
|
2024-03-15 18:09:25 -07:00 |
|
Krrish Dholakia
|
226953e1d8
|
feat(batch_redis_get.py): batch redis GET requests for a given key + call type
reduces the number of GET requests we're making in high-throughput scenarios
|
2024-03-15 14:40:11 -07:00 |
|
Krrish Dholakia
|
7c72cc0ec9
|
fix(caching.py): support redis caching with namespaces
|
2024-03-14 13:35:17 -07:00 |
|
Krrish Dholakia
|
d6537a05ca
|
fix(caching.py): fix print statements
|
2024-03-14 12:58:34 -07:00 |
|
Krrish Dholakia
|
1b807fa3f5
|
fix(proxy_server.py): fix key caching logic
|
2024-03-13 19:10:24 -07:00 |
|
Krish Dholakia
|
c7d0af0a2e
|
Merge pull request #2426 from BerriAI/litellm_whisper_cost_tracking
feat: add cost tracking + caching for `/audio/transcription` calls
|
2024-03-09 19:12:06 -08:00 |
|
ishaan-jaff
|
1917ee7fcf
|
(feat) debug when in meory cache is used
|
2024-03-09 16:24:04 -08:00 |
|
Krrish Dholakia
|
9b7c8880c0
|
fix(caching.py): only add unique kwargs for transcription_only_kwargs in caching
|
2024-03-09 16:09:12 -08:00 |
|
Krrish Dholakia
|
fa45c569fd
|
feat: add cost tracking + caching for transcription calls
|
2024-03-09 15:43:38 -08:00 |
|
Krrish Dholakia
|
726dad5756
|
fix(caching.py): add s3 path as a top-level param
|
2024-03-06 18:07:28 -08:00 |
|
Krrish Dholakia
|
fb2ae3a032
|
fix(utils.py): only return cached streaming object for streaming calls
|
2024-02-21 21:27:40 -08:00 |
|
Krrish Dholakia
|
74381a2d06
|
fix(caching.py): use print verbose for logging error
|
2024-02-15 18:12:09 -08:00 |
|
Krrish Dholakia
|
66c7e35a97
|
fix(redis.py): fix instantiating redis client from url
|
2024-02-15 17:48:00 -08:00 |
|