Commit graph

11 commits

Author SHA1 Message Date
Emilio Garcia
e8d20b9c50 fix(scripts): add metrics interval to env variables 2025-11-04 14:05:20 -05:00
Emilio Garcia
0a6c180631 fix(tests): metrics test improved to avoid race conditions 2025-11-04 14:04:16 -05:00
Emilio Garcia
23fce9718c fix(telemetry): token counters changed to histograms to reflect count per request 2025-11-04 14:04:16 -05:00
Emilio Garcia
0e0bc8aba7 Fix protobuf metric parsing in OTLP collector
- Add _create_metric_stub_from_protobuf method to correctly parse protobuf metrics
- Add _extract_attributes_from_data_point helper method
- Change metric handling to use protobuf-specific parsing instead of OpenTelemetry native parsing
- Add missing typing import
- Add OTEL_METRIC_EXPORT_INTERVAL environment variable for test configuration

This fixes the CI failure where metrics were not being properly extracted from
protobuf data in server mode tests.
2025-11-04 14:04:11 -05:00
Emilio Garcia
52a7784847 fix(tests): deduplicate code and simplify user experience for telemetry test code 2025-11-04 14:03:13 -05:00
Emilio Garcia
e4e8a59325 fix(tests): add methods to standardize test span data access across all tests 2025-11-04 14:03:13 -05:00
Emilio Garcia
79156bb08c fix(tests): improve structure of telemetry tests for consistency 2025-11-04 14:03:13 -05:00
Emilio Garcia
583df48479 fix(tests): resolve linting issues in telemetry tests 2025-11-04 14:03:13 -05:00
ehhuang
1aa8979050
test: enable telemetry tests in server mode (#3927)
# What does this PR do?
- added a server-based test OLTP collector

## Test Plan
CI
2025-10-28 16:33:48 -07:00
Ashwin Bharambe
f88416ef87
fix(inference): enable routing of models with provider_data alone (#3928)
This PR enables routing of fully qualified model IDs of the form
`provider_id/model_id` even when the models are not registered with the
Stack.

Here's the situation: assume a remote inference provider which works
only when users provide their own API keys via
`X-LlamaStack-Provider-Data` header. By definition, we cannot list
models and hence update our routing registry. But because we _require_ a
provider ID in the models now, we can identify which provider to route
to and let that provider decide.

Note that we still try to look up our registry since it may have a
pre-registered alias. Just that we don't outright fail when we are not
able to look it up.

Also, updated inference router so that the responses have the _exact_
model that the request had.

## Test Plan

Added an integration test

Closes #3929

---------

Co-authored-by: ehhuang <ehhuang@users.noreply.github.com>
2025-10-28 11:16:37 -07:00
Emilio Garcia
943558af36
test(telemetry): Telemetry Tests (#3805)
Some checks failed
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 0s
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 0s
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped
Python Package Build Test / build (3.12) (push) Failing after 10s
Python Package Build Test / build (3.13) (push) Failing after 10s
Integration Tests (Replay) / Integration Tests (, , , client=, ) (push) Failing after 14s
Unit Tests / unit-tests (3.13) (push) Failing after 11s
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 20s
Unit Tests / unit-tests (3.12) (push) Failing after 16s
Test External API and Providers / test-external (venv) (push) Failing after 28s
Vector IO Integration Tests / test-matrix (push) Failing after 30s
API Conformance Tests / check-schema-compatibility (push) Successful in 38s
UI Tests / ui-tests (22) (push) Successful in 1m32s
Pre-commit / pre-commit (push) Successful in 3m16s
# What does this PR do?
Adds a test and a standardized way to build future tests out for
telemetry in llama stack.
Contributes to https://github.com/llamastack/llama-stack/issues/3806

## Test Plan
This is the test plan 😎
2025-10-17 10:43:33 -07:00