phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-10-04 04:04:14 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat(tests): auto-merge all model list responses and unify recordings (#3320)
Some checks failed
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 1s
Details
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped
Details
Integration Tests (Replay) / Integration Tests (, , , client=, vision=) (push) Failing after 3s
Details
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 4s
Details
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 7s
Details
Update ReadTheDocs / update-readthedocs (push) Failing after 3s
Details
Test External API and Providers / test-external (venv) (push) Failing after 5s
Details
Vector IO Integration Tests / test-matrix (push) Failing after 7s
Details
Python Package Build Test / build (3.13) (push) Failing after 8s
Details
Python Package Build Test / build (3.12) (push) Failing after 8s
Details
Unit Tests / unit-tests (3.13) (push) Failing after 14s
Details
Unit Tests / unit-tests (3.12) (push) Failing after 14s
Details
UI Tests / ui-tests (22) (push) Successful in 1m7s
Details
Pre-commit / pre-commit (push) Successful in 2m34s
Details

Browse source 
One needed to specify record-replay related environment variables for
running integration tests. We could not use defaults because integration
tests could be run against Ollama instances which could be running
different models. For example, text vs vision tests needed separate
instances of Ollama because a single instance typically cannot serve
both of these models if you assume the standard CI worker configuration
on Github. As a result, `client.list()` as returned by the Ollama client
would be different between these runs and we'd end up overwriting
responses.

This PR "solves" it by adding a small amount of complexity -- we store
model list responses specially, keyed by the hashes of the models they
return. At replay time, we merge all of them and pretend that we have
the union of all models available.

## Test Plan

Re-recorded all the tests using `scripts/integration-tests.sh
--inference-mode record`, including the vision tests.
This commit is contained in:
Ashwin Bharambe 2025-09-03 11:33:03 -07:00 • committed by GitHub
parent d948e63340
commit c3d3a0b833
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
122 changed files with 17460 additions and 16129 deletions
Download patch file Download diff file Expand all files Collapse all files

32
tests/integration/recordings/responses/0b3f2e4754ff.json
View file

@ -24,7 +24,7 @@
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "chatcmpl-29",
"id": "chatcmpl-414",
"choices": [
{
"delta": {
@ -39,7 +39,7 @@
"logprobs": null
}
],
"created": 1754090031,
"created": 1756921333,
"model": "llama3.2:3b-instruct-fp16",
"object": "chat.completion.chunk",
"service_tier": null,
@ -50,7 +50,7 @@
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "chatcmpl-29",
"id": "chatcmpl-414",
"choices": [
{
"delta": {
@ -65,7 +65,7 @@
"logprobs": null
}
],
"created": 1754090031,
"created": 1756921333,
"model": "llama3.2:3b-instruct-fp16",
"object": "chat.completion.chunk",
"service_tier": null,
@ -76,7 +76,7 @@
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "chatcmpl-29",
"id": "chatcmpl-414",
"choices": [
{
"delta": {
@ -91,7 +91,7 @@
"logprobs": null
}
],
"created": 1754090031,
"created": 1756921333,
"model": "llama3.2:3b-instruct-fp16",
"object": "chat.completion.chunk",
"service_tier": null,
@ -102,7 +102,7 @@
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "chatcmpl-29",
"id": "chatcmpl-414",
"choices": [
{
"delta": {
@ -117,7 +117,7 @@
"logprobs": null
}
],
"created": 1754090031,
"created": 1756921333,
"model": "llama3.2:3b-instruct-fp16",
"object": "chat.completion.chunk",
"service_tier": null,
@ -128,7 +128,7 @@
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "chatcmpl-29",
"id": "chatcmpl-414",
"choices": [
{
"delta": {
@ -143,7 +143,7 @@
"logprobs": null
}
],
"created": 1754090031,
"created": 1756921334,
"model": "llama3.2:3b-instruct-fp16",
"object": "chat.completion.chunk",
"service_tier": null,
@ -154,7 +154,7 @@
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "chatcmpl-29",
"id": "chatcmpl-414",
"choices": [
{
"delta": {
@ -169,7 +169,7 @@
"logprobs": null
}
],
"created": 1754090031,
"created": 1756921334,
"model": "llama3.2:3b-instruct-fp16",
"object": "chat.completion.chunk",
"service_tier": null,
@ -180,7 +180,7 @@
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "chatcmpl-29",
"id": "chatcmpl-414",
"choices": [
{
"delta": {
@ -195,7 +195,7 @@
"logprobs": null
}
],
"created": 1754090031,
"created": 1756921334,
"model": "llama3.2:3b-instruct-fp16",
"object": "chat.completion.chunk",
"service_tier": null,
@ -206,7 +206,7 @@
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "chatcmpl-29",
"id": "chatcmpl-414",
"choices": [
{
"delta": {
@ -221,7 +221,7 @@
"logprobs": null
}
],
"created": 1754090031,
"created": 1756921334,
"model": "llama3.2:3b-instruct-fp16",
"object": "chat.completion.chunk",
"service_tier": null,