Ishaan Jaff
835db6ae98
(load testing) add vertex_ai embeddings load test ( #6004 )
...
* use vertex llm as base class for embeddings
* use correct vertex class in main.py
* set_headers in vertex llm base
* add types for vertex embedding requests
* add embedding handler for vertex
* use async mode for vertex embedding tests
* use vertexAI textEmbeddingConfig
* fix linting
* add sync and async mode testing for vertex ai embeddings
* add basic load test
* add vertex ai load test on ci cd
2024-10-03 14:39:15 +05:30
Ishaan Jaff
b878a67a7c
fic otel load test %
2024-09-14 18:04:28 -07:00
Ishaan Jaff
c8d15544c8
[Fix] Router cooldown logic - use % thresholds instead of allowed fails to cooldown deployments ( #5698 )
...
* move cooldown logic to it's own helper
* add new track deployment metrics folder
* increment success, fails for deployment in current minute
* fix cooldown logic
* fix test_aaarouter_dynamic_cooldown_message_retry_time
* fix test_single_deployment_no_cooldowns_test_prod_mock_completion_calls
* clean up get from deployment test
* fix _async_get_healthy_deployments
* add mock InternalServerError
* test deployment failing 25% requests
* add test_high_traffic_cooldowns_one_bad_deployment
* fix vertex load test
* add test for rate limit error models in cool down
* change default cooldown time
* fix cooldown message time
* fix cooldown on 429 error
* fix doc string for _should_cooldown_deployment
* fix sync cooldown logic router
2024-09-14 18:01:19 -07:00
Ishaan Jaff
741c8e8a45
[Feat - Perf Improvement] DataDog Logger 91% lower latency ( #5687 )
...
* fix refactor dd to be an instance of custom logger
* migrate dd logger to be async
* clean up dd logging
* add datadog sync and async code
* use batching for datadog logger
* add doc string for dd logging
* add clear doc string
* fix doc string
* allow debugging intake url
* clean up requirements.txt
* allow setting custom batch size on logger
* fix dd logging to use compression
* fix linting
* add dd load test
* fix dd load test
* fix dd url
* add test_datadog_logging_http_request
* fix test_datadog_logging_http_request
2024-09-13 17:39:17 -07:00
Ishaan Jaff
cd8d7ca915
[Fix] Performance - use in memory cache when downloading images from a url ( #5657 )
...
* fix use in memory cache when getting images
* fix linting
* fix load testing
* fix load test size
* fix load test size
* trigger ci/cd again
2024-09-13 07:23:42 -07:00
Ishaan Jaff
88706488f9
fix otel load test
2024-09-11 21:27:31 -07:00
Ishaan Jaff
b80f27dce3
fix otel tests
2024-09-11 21:25:27 -07:00
Ishaan Jaff
97ecf86d3d
fix langsmith load tests
2024-09-11 21:19:03 -07:00
Ishaan Jaff
b01a42ef4f
fix langsmith load test
2024-09-11 21:16:16 -07:00
Ishaan Jaff
a1f8fcfeed
fix load test
2024-09-11 21:06:42 -07:00
Ishaan Jaff
850b5dbadc
add otel load test
2024-09-11 20:47:12 -07:00
Ishaan Jaff
e7b047223e
add langsmith logging test
2024-09-11 20:35:11 -07:00
