mirror of https://github.com/meta-llama/llama-stack.git synced 2025-12-03 09:53:45 +00:00

History

Ashwin Bharambe a8a8aa56c0 Some checks failed SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 0s Details Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 1s Details Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped Details Pre-commit / pre-commit (push) Failing after 3s Details Python Package Build Test / build (3.12) (push) Failing after 2s Details Python Package Build Test / build (3.13) (push) Failing after 2s Details Vector IO Integration Tests / test-matrix (push) Failing after 4s Details Integration Tests (Replay) / Integration Tests (, , , client=, ) (push) Failing after 5s Details Test External API and Providers / test-external (venv) (push) Failing after 5s Details SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 9s Details Unit Tests / unit-tests (3.13) (push) Failing after 5s Details Unit Tests / unit-tests (3.12) (push) Failing after 6s Details API Conformance Tests / check-schema-compatibility (push) Successful in 13s Details UI Tests / ui-tests (22) (push) Successful in 1m10s Details chore!: remove the agents (sessions and turns) API (#4055 ) - Removes the deprecated agents (sessions and turns) API that was marked alpha in 0.3.0 - Cleans up unused imports and orphaned types after the API removal - Removes `SessionNotFoundError` and `AgentTurnInputType` which are no longer needed The agents API is completely superseded by the Responses + Conversations APIs, and the client SDK Agent class already uses those implementations. Corresponding client-side PR: https://github.com/llamastack/llama-stack-client-python/pull/295		2025-11-04 09:38:39 -08:00
..
README.md	feat(tests): implement test isolation for inference recordings (#3681 )	2025-10-04 11:34:18 -07:00

README.md

Test Recording System

This directory contains recorded inference API responses used for deterministic testing without requiring live API access.

Structure

responses/ - JSON files containing request/response pairs for inference operations

Recording Format

Each JSON file contains:

request - The normalized request parameters (method, endpoint, body)
response - The response body (serialized from Pydantic models)

Normalization

To reduce noise in git diffs, the recording system automatically normalizes fields that vary between runs but don't affect test behavior:

OpenAI-style responses

id - Deterministic hash based on request: rec-{request_hash[:12]}
created - Normalized to epoch: 0

Ollama-style responses

created_at - Normalized to: "1970-01-01T00:00:00.000000Z"
total_duration - Normalized to: 0
load_duration - Normalized to: 0
prompt_eval_duration - Normalized to: 0
eval_duration - Normalized to: 0

These normalizations ensure that re-recording tests produces minimal git diffs, making it easier to review actual changes to test behavior.

Usage

Replay mode (default)

Responses are replayed from recordings:

LLAMA_STACK_TEST_INFERENCE_MODE=replay pytest tests/integration/

Record-if-missing mode (recommended for adding new tests)

Records only when no recording exists, otherwise replays. Use this for iterative development:

LLAMA_STACK_TEST_INFERENCE_MODE=record-if-missing pytest tests/integration/

Recording mode

Force-records all API interactions, overwriting existing recordings. Use with caution:

LLAMA_STACK_TEST_INFERENCE_MODE=record pytest tests/integration/

Live mode

Skip recordings entirely and use live APIs:

LLAMA_STACK_TEST_INFERENCE_MODE=live pytest tests/integration/

Re-normalizing Existing Recordings

If you need to apply normalization to existing recordings (e.g., after updating the normalization logic):

python scripts/normalize_recordings.py

Use --dry-run to preview changes without modifying files.