mirror of https://github.com/meta-llama/llama-stack.git synced 2025-10-08 04:54:38 +00:00

Ashwin Bharambe 045a0c1d57

feat(tests): implement test isolation for inference recordings (#3681 )

Uses test_id in request hashes and test-scoped subdirectories to prevent
cross-test contamination. Model list endpoints exclude test_id to enable
merging recordings from different servers.

Additionally, this PR adds a `record-if-missing` mode (which we will use
instead of `record` which records everything) which is very useful.

🤖 Co-authored with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>

2025-10-04 11:34:18 -07:00

2.1 KiB

Raw Blame History

Test Recording System

This directory contains recorded inference API responses used for deterministic testing without requiring live API access.

Structure

responses/ - JSON files containing request/response pairs for inference operations

Recording Format

Each JSON file contains:

request - The normalized request parameters (method, endpoint, body)
response - The response body (serialized from Pydantic models)

Normalization

To reduce noise in git diffs, the recording system automatically normalizes fields that vary between runs but don't affect test behavior:

OpenAI-style responses

id - Deterministic hash based on request: rec-{request_hash[:12]}
created - Normalized to epoch: 0

Ollama-style responses

created_at - Normalized to: "1970-01-01T00:00:00.000000Z"
total_duration - Normalized to: 0
load_duration - Normalized to: 0
prompt_eval_duration - Normalized to: 0
eval_duration - Normalized to: 0

These normalizations ensure that re-recording tests produces minimal git diffs, making it easier to review actual changes to test behavior.

Usage

Replay mode (default)

Responses are replayed from recordings:

LLAMA_STACK_TEST_INFERENCE_MODE=replay pytest tests/integration/

Record-if-missing mode (recommended for adding new tests)

Records only when no recording exists, otherwise replays. Use this for iterative development:

LLAMA_STACK_TEST_INFERENCE_MODE=record-if-missing pytest tests/integration/

Recording mode

Force-records all API interactions, overwriting existing recordings. Use with caution:

LLAMA_STACK_TEST_INFERENCE_MODE=record pytest tests/integration/

Live mode

Skip recordings entirely and use live APIs:

LLAMA_STACK_TEST_INFERENCE_MODE=live pytest tests/integration/

Re-normalizing Existing Recordings

If you need to apply normalization to existing recordings (e.g., after updating the normalization logic):

python scripts/normalize_recordings.py

Use --dry-run to preview changes without modifying files.

2.1 KiB Raw Blame History