Ishaan Jaff
d81ae45827
(Perf / latency improvement) improve pass through endpoint latency to ~50ms (before PR was 400ms) ( #6874 )
...
* use correct location for types
* fix types location
* perf improvement for pass through endpoints
* update lint check
* fix import
* fix ensure async clients test
* fix azure.py health check
* fix ollama
2024-11-22 18:47:26 -08:00
Ishaan Jaff
920f4c9f82
(fix) add linting check to ban creating AsyncHTTPHandler during LLM calling ( #6855 )
...
* fix triton
* fix TEXT_COMPLETION_CODESTRAL
* fix REPLICATE
* fix CLARIFAI
* fix HUGGINGFACE
* add test_no_async_http_handler_usage
* fix PREDIBASE
* fix anthropic use get_async_httpx_client
* fix vertex fine tuning
* fix dbricks get_async_httpx_client
* fix get_async_httpx_client vertex
* fix get_async_httpx_client
* fix get_async_httpx_client
* fix make_async_azure_httpx_request
* fix check_for_async_http_handler
* test: cleanup mistral model
* add check for AsyncClient
* fix check_for_async_http_handler
* fix get_async_httpx_client
* fix tests using in_memory_llm_clients_cache
* fix langfuse import
* fix import
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
2024-11-21 19:03:02 -08:00
Ishaan Jaff
ae385cfcdc
(QOL improvement) add unit testing for all static_methods in litellm_logging.py ( #6640 )
...
* add unit testing for standard logging payload
* unit testing for static methods in litellm_logging
* add code coverage check for litellm_logging
* litellm_logging_code_coverage
* test_get_final_response_obj
* fix validate_redacted_message_span_attributes
* test validate_redacted_message_span_attributes
2024-11-07 16:26:53 -08:00
Ishaan Jaff
441adad3ae
(router_strategy/) ensure all async functions use async cache methods ( #6489 )
...
* fix router strat
* use async set / get cache in router_strategy
* add coverage for router strategy
* fix imports
* fix batch_get_cache
* use async methods for least busy
* fix least busy use async methods
* fix test_dual_cache_increment
* test async_get_available_deployment when routing_strategy="least-busy"
2024-10-29 21:07:17 +05:30
Krish Dholakia
e22e8d24ef
Litellm router code coverage 3 ( #6274 )
...
* refactor(router.py): move assistants api endpoints to using 1 pass-through factory function
Reduces code, increases testing coverage
* refactor(router.py): reduce _common_check_available_deployment function size
make code more maintainable - reduce possible errors
* test(router_code_coverage.py): include batch_utils + pattern matching in enforced 100% code coverage
Improves reliability
* fix(router.py): fix model id match model dump
2024-10-16 21:30:25 -07:00
Ishaan Jaff
8530000b44
(testing) Router add testing coverage ( #6253 )
...
* test: add more router code coverage
* test: additional router testing coverage
* fix: fix linting error
* test: fix tests for ci/cd
* test: fix test
* test: handle flaky tests
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
2024-10-16 07:32:27 -07:00
Krish Dholakia
1eb435e50a
test(router_code_coverage.py): check if all router functions are dire… ( #6186 )
...
* test(router_code_coverage.py): check if all router functions are directly tested
prevent regressions
* docs(configs.md): document all environment variables (#6185 )
* docs: make it easier to find anthropic/openai prompt caching doc
* aded codecov yml (#6207 )
* fix codecov.yaml
* run ci/cd again
* (refactor) caching use LLMCachingHandler for async_get_cache and set_cache (#6208 )
* use folder for caching
* fix importing caching
* fix clickhouse pyright
* fix linting
* fix correctly pass kwargs and args
* fix test case for embedding
* fix linting
* fix embedding caching logic
* fix refactor handle utils.py
* fix test_embedding_caching_azure_individual_items_reordered
* (feat) prometheus have well defined latency buckets (#6211 )
* fix prometheus have well defined latency buckets
* use a well define latency bucket
* use types file for prometheus logging
* add test for LATENCY_BUCKETS
* fix prom testing
* fix config.yml
* (refactor caching) use LLMCachingHandler for caching streaming responses (#6210 )
* use folder for caching
* fix importing caching
* fix clickhouse pyright
* fix linting
* fix correctly pass kwargs and args
* fix test case for embedding
* fix linting
* fix embedding caching logic
* fix refactor handle utils.py
* refactor async set stream cache
* fix linting
* bump (#6187 )
* update code cov yaml
* fix config.yml
* add caching component to code cov
* fix config.yml ci/cd
* add coverage for proxy auth
* (refactor caching) use common `_retrieve_from_cache` helper (#6212 )
* use folder for caching
* fix importing caching
* fix clickhouse pyright
* fix linting
* fix correctly pass kwargs and args
* fix test case for embedding
* fix linting
* fix embedding caching logic
* fix refactor handle utils.py
* refactor async set stream cache
* fix linting
* refactor - use _retrieve_from_cache
* refactor use _convert_cached_result_to_model_response
* fix linting errors
* bump: version 1.49.2 → 1.49.3
* fix code cov components
* test(test_router_helpers.py): add router component unit tests
* test: add additional router tests
* test: add more router testing
* test: add more router testing + more mock functions
* ci(router_code_coverage.py): fix check
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: yujonglee <yujonglee.dev@gmail.com>
2024-10-14 22:44:00 -07:00
