litellm/litellm
Krish Dholakia 56e9047818
Litellm router max depth (#6501)
* feat(router.py): add check for max fallback depth

Prevent infinite loop for fallbacks

 Closes https://github.com/BerriAI/litellm/issues/6498

* test: update test

* (fix) Prometheus - Log Postgres DB latency, status on prometheus  (#6484)

* fix logging DB fails on prometheus

* unit testing log to otel wrapper

* unit testing for service logger + prometheus

* use LATENCY buckets for service logging

* fix service logging

* docs clarify vertex vs gemini

* (router_strategy/) ensure all async functions use async cache methods (#6489)

* fix router strat

* use async set / get cache in router_strategy

* add coverage for router strategy

* fix imports

* fix batch_get_cache

* use async methods for least busy

* fix least busy use async methods

* fix test_dual_cache_increment

* test async_get_available_deployment when routing_strategy="least-busy"

* (fix) proxy - fix when `STORE_MODEL_IN_DB` should be set (#6492)

* set store_model_in_db at the top

* correctly use store_model_in_db global

* (fix) `PrometheusServicesLogger` `_get_metric` should return metric in Registry  (#6486)

* fix logging DB fails on prometheus

* unit testing log to otel wrapper

* unit testing for service logger + prometheus

* use LATENCY buckets for service logging

* fix service logging

* fix _get_metric in prom services logger

* add clear doc string

* unit testing for prom service logger

* bump: version 1.51.0 → 1.51.1

* Add `azure/gpt-4o-mini-2024-07-18` to model_prices_and_context_window.json (#6477)

* Update utils.py (#6468)

Fixed missing keys

* (perf) Litellm redis router fix - ~100ms improvement (#6483)

* docs(exception_mapping.md): add missing exception types

Fixes https://github.com/Aider-AI/aider/issues/2120#issuecomment-2438971183

* fix(main.py): register custom model pricing with specific key

Ensure custom model pricing is registered to the specific model+provider key combination

* test: make testing more robust for custom pricing

* fix(redis_cache.py): instrument otel logging for sync redis calls

ensures complete coverage for all redis cache calls

* refactor: pass parent_otel_span for redis caching calls in router

allows for more observability into what calls are causing latency issues

* test: update tests with new params

* refactor: ensure e2e otel tracing for router

* refactor(router.py): add more otel tracing acrosss router

catch all latency issues for router requests

* fix: fix linting error

* fix(router.py): fix linting error

* fix: fix test

* test: fix tests

* fix(dual_cache.py): pass ttl to redis cache

* fix: fix param

* perf(cooldown_cache.py): improve cooldown cache, to store cache results in memory for 5s, prevents redis call from being made on each request

reduces 100ms latency per call with caching enabled on router

* fix: fix test

* fix(cooldown_cache.py): handle if a result is None

* fix(cooldown_cache.py): add debug statements

* refactor(dual_cache.py): move to using an in-memory check for batch get cache, to prevent redis from being hit for every call

* fix(cooldown_cache.py): fix linting erropr

* build: merge main

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Xingyao Wang <xingyao@all-hands.dev>
Co-authored-by: vibhanshu-ob <115142120+vibhanshu-ob@users.noreply.github.com>
2024-10-29 22:05:41 -07:00
..
adapters LiteLLM Minor Fixes & Improvements (09/27/2024) (#5938) 2024-09-27 22:52:57 -07:00
assistants Add pyright to ci/cd + Fix remaining type-checking errors (#6082) 2024-10-05 17:04:00 -04:00
batch_completion (refactor) caching use LLMCachingHandler for async_get_cache and set_cache (#6208) 2024-10-14 16:34:01 +05:30
batches Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
caching Litellm dev 10 29 2024 (#6502) 2024-10-29 22:04:16 -07:00
deprecated_litellm_server (refactor) caching use LLMCachingHandler for async_get_cache and set_cache (#6208) 2024-10-14 16:34:01 +05:30
files Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
fine_tuning Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
integrations Litellm dev 10 29 2024 (#6502) 2024-10-29 22:04:16 -07:00
litellm_core_utils Litellm dev 10 29 2024 (#6502) 2024-10-29 22:04:16 -07:00
llms Litellm dev 10 29 2024 (#6502) 2024-10-29 22:04:16 -07:00
proxy Litellm router max depth (#6501) 2024-10-29 22:05:41 -07:00
realtime_api LiteLLM Minor Fixes & Improvements (10/10/2024) (#6158) 2024-10-11 23:04:36 -07:00
rerank_api Add pyright to ci/cd + Fix remaining type-checking errors (#6082) 2024-10-05 17:04:00 -04:00
router_strategy (router_strategy/) ensure all async functions use async cache methods (#6489) 2024-10-29 21:07:17 +05:30
router_utils Litellm router max depth (#6501) 2024-10-29 22:05:41 -07:00
secret_managers (fix) proxy - fix when STORE_MODEL_IN_DB should be set (#6492) 2024-10-29 21:28:14 +05:30
tests Feat: Add Langtrace integration (#5341) 2024-10-11 19:19:53 +05:30
types Litellm router max depth (#6501) 2024-10-29 22:05:41 -07:00
__init__.py Litellm router max depth (#6501) 2024-10-29 22:05:41 -07:00
_logging.py Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
_redis.py Litellm dev 10 22 2024 (#6384) 2024-10-22 21:18:54 -07:00
_service_logger.py redis otel tracing + async support for latency routing (#6452) 2024-10-28 21:52:12 -07:00
_version.py Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
budget_manager.py Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
constants.py Litellm router max depth (#6501) 2024-10-29 22:05:41 -07:00
cost.json
cost_calculator.py LiteLLM Minor Fixes & Improvements (10/18/2024) (#6320) 2024-10-19 22:23:27 -07:00
exceptions.py Litellm dev 10 26 2024 (#6472) 2024-10-28 15:05:43 -07:00
main.py Litellm router max depth (#6501) 2024-10-29 22:05:41 -07:00
model_prices_and_context_window_backup.json LiteLLM Minor Fixes & Improvements (10/28/2024) (#6475) 2024-10-29 17:20:24 -07:00
py.typed feature - Types for mypy - #360 2024-05-30 14:14:41 -04:00
requirements.txt
router.py Litellm router max depth (#6501) 2024-10-29 22:05:41 -07:00
scheduler.py (refactor) caching use LLMCachingHandler for async_get_cache and set_cache (#6208) 2024-10-14 16:34:01 +05:30
timeout.py Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
utils.py Update utils.py (#6468) 2024-10-29 09:06:23 -07:00