Krish Dholakia
2fc6262675
fix(route_llm_request.py): move to using common router, even for clie… ( #8966 )
...
* fix(route_llm_request.py): move to using common router, even for client-side credentials
ensures fallbacks / cooldown logic still works
* test(test_route_llm_request.py): add unit test for route request
* feat(router.py): generate unique model id when clientside credential passed in
Prevents cooldowns for api key 1 from impacting api key 2
* test(test_router.py): update testing to ensure original litellm params not mutated
* fix(router.py): upsert clientside call into llm router model list
enables cooldown logic to work accurately
* fix: fix linting error
* test(test_router_utils.py): add direct test for new util on router
2025-03-03 22:57:08 -08:00
Ishaan Jaff
f6fa2399cc
(Router) - If allowed_fails or allowed_fail_policy set, use that for single deployment cooldown logic ( #8668 )
...
* fix cooldown 1 deployment
* test_single_deployment_cooldown_with_allowed_fail_policy
* fix docstring test
* test_single_deployment_no_cooldowns
* ci/cd run again
* test router cooldowns
2025-02-25 15:15:01 -08:00
Krish Dholakia
80f7af510b
Improve Proxy Resiliency: Cooldown single-deployment model groups if 100% calls failed in high traffic ( #7823 )
...
* refactor(_is_cooldown_required): move '_is_cooldown_required' into cooldown_handlers.py
* refactor(cooldown_handlers.py): move cooldown constants into `.constants.py`
* fix(cooldown_handlers.py): remove if single deployment don't cooldown logic
move to traffic based cooldown logic
Addresses https://github.com/BerriAI/litellm/issues/7822
* fix: add unit tests for '_should_cooldown_deployment'
* test: ensure all tests pass
* test: update test
* fix(cooldown_handlers.py): don't cooldown single deployment models for anything besides traffic related errors
* fix(cooldown_handlers.py): fix cooldown handler logic
* fix(cooldown_handlers.py): fix check
2025-01-17 20:17:02 -08:00
Ishaan Jaff
c7f14e936a
(code quality) run ruff rule to ban unused imports ( #7313 )
...
* remove unused imports
* fix AmazonConverseConfig
* fix test
* fix import
* ruff check fixes
* test fixes
* fix testing
* fix imports
2024-12-19 12:33:42 -08:00
Krish Dholakia
4f8a3fd4cf
redis otel tracing + async support for latency routing ( #6452 )
...
* docs(exception_mapping.md): add missing exception types
Fixes https://github.com/Aider-AI/aider/issues/2120#issuecomment-2438971183
* fix(main.py): register custom model pricing with specific key
Ensure custom model pricing is registered to the specific model+provider key combination
* test: make testing more robust for custom pricing
* fix(redis_cache.py): instrument otel logging for sync redis calls
ensures complete coverage for all redis cache calls
* refactor: pass parent_otel_span for redis caching calls in router
allows for more observability into what calls are causing latency issues
* test: update tests with new params
* refactor: ensure e2e otel tracing for router
* refactor(router.py): add more otel tracing acrosss router
catch all latency issues for router requests
* fix: fix linting error
* fix(router.py): fix linting error
* fix: fix test
* test: fix tests
* fix(dual_cache.py): pass ttl to redis cache
* fix: fix param
2024-10-28 21:52:12 -07:00
Ishaan Jaff
891e9001b5
(testing) add router unit testing for send_llm_exception_alert , router_cooldown_event_callback , cooldown utils ( #6258 )
...
* add router unit testing for send_llm_exception_alert
* test router_cooldown_event_callback
* test test_router_cooldown_event_callback_no_prometheus
* test_router_cooldown_event_callback_no_deployment
* test_router_cooldown_event_callback_no_deployment
* add testing for test_should_run_cooldown_logic
* test_increment_deployment_successes_for_current_minute_does_not_write_to_redis
* test test_should_cooldown_deployment_allowed_fails_set_on_router
* use helper for _is_allowed_fails_set_on_router
* add complete testing for cooldown utils
* move router unit tests
* move router handle error
* fix test_send_llm_exception_alert_no_logger
2024-10-16 23:19:51 +05:30
Ishaan Jaff
8530000b44
(testing) Router add testing coverage ( #6253 )
...
* test: add more router code coverage
* test: additional router testing coverage
* fix: fix linting error
* test: fix tests for ci/cd
* test: fix test
* test: handle flaky tests
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
2024-10-16 07:32:27 -07:00
Krish Dholakia
d57be47b0f
Litellm ruff linting enforcement ( #5992 )
...
* ci(config.yml): add a 'check_code_quality' step
Addresses https://github.com/BerriAI/litellm/issues/5991
* ci(config.yml): check why circle ci doesn't pick up this test
* ci(config.yml): fix to run 'check_code_quality' tests
* fix(__init__.py): fix unprotected import
* fix(__init__.py): don't remove unused imports
* build(ruff.toml): update ruff.toml to ignore unused imports
* fix: fix: ruff + pyright - fix linting + type-checking errors
* fix: fix linting errors
* fix(lago.py): fix module init error
* fix: fix linting errors
* ci(config.yml): cd into correct dir for checks
* fix(proxy_server.py): fix linting error
* fix(utils.py): fix bare except
causes ruff linting errors
* fix: ruff - fix remaining linting errors
* fix(clickhouse.py): use standard logging object
* fix(__init__.py): fix unprotected import
* fix: ruff - fix linting errors
* fix: fix linting errors
* ci(config.yml): cleanup code qa step (formatting handled in local_testing)
* fix(_health_endpoints.py): fix ruff linting errors
* ci(config.yml): just use ruff in check_code_quality pipeline for now
* build(custom_guardrail.py): include missing file
* style(embedding_handler.py): fix ruff check
2024-10-01 19:44:20 -04:00
Ishaan Jaff
c8eff2dc65
[Feat-Prometheus] Track exception status on litellm_deployment_failure_responses ( #5706 )
...
* add litellm_deployment_cooled_down
* track num cooldowns on prometheus
* track exception status
* fix linting
* docs prom metrics
* cleanup premium user checks
* prom track deployment failure state
* docs prometheus
2024-09-14 18:44:31 -07:00
Ishaan Jaff
c8d15544c8
[Fix] Router cooldown logic - use % thresholds instead of allowed fails to cooldown deployments ( #5698 )
...
* move cooldown logic to it's own helper
* add new track deployment metrics folder
* increment success, fails for deployment in current minute
* fix cooldown logic
* fix test_aaarouter_dynamic_cooldown_message_retry_time
* fix test_single_deployment_no_cooldowns_test_prod_mock_completion_calls
* clean up get from deployment test
* fix _async_get_healthy_deployments
* add mock InternalServerError
* test deployment failing 25% requests
* add test_high_traffic_cooldowns_one_bad_deployment
* fix vertex load test
* add test for rate limit error models in cool down
* change default cooldown time
* fix cooldown message time
* fix cooldown on 429 error
* fix doc string for _should_cooldown_deployment
* fix sync cooldown logic router
2024-09-14 18:01:19 -07:00
