chore(tests): normalize recording IDs and timestamps to reduce git diff noise

IDs are now deterministic hashes based on request content, and timestamps are normalized to constants, eliminating spurious changes when re-recording tests.
This commit is contained in:
Ashwin Bharambe 2025-10-03 13:54:49 -07:00
parent 61b4238912
commit 6f1afabcdc
348 changed files with 10358 additions and 8474 deletions

View file

@ -0,0 +1,60 @@
# Test Recording System
This directory contains recorded inference API responses used for deterministic testing without requiring live API access.
## Structure
- `responses/` - JSON files containing request/response pairs for inference operations
## Recording Format
Each JSON file contains:
- `request` - The normalized request parameters (method, endpoint, body)
- `response` - The response body (serialized from Pydantic models)
## Normalization
To reduce noise in git diffs, the recording system automatically normalizes fields that vary between runs but don't affect test behavior:
### OpenAI-style responses
- `id` - Deterministic hash based on request: `rec-{request_hash[:12]}`
- `created` - Normalized to epoch: `0`
### Ollama-style responses
- `created_at` - Normalized to: `"1970-01-01T00:00:00.000000Z"`
- `total_duration` - Normalized to: `0`
- `load_duration` - Normalized to: `0`
- `prompt_eval_duration` - Normalized to: `0`
- `eval_duration` - Normalized to: `0`
These normalizations ensure that re-recording tests produces minimal git diffs, making it easier to review actual changes to test behavior.
## Usage
### Recording mode
Set `LLAMA_STACK_TEST_INFERENCE_MODE=record` to capture new responses:
```bash
LLAMA_STACK_TEST_INFERENCE_MODE=record pytest tests/integration/
```
### Replay mode (default)
Responses are replayed from recordings:
```bash
LLAMA_STACK_TEST_INFERENCE_MODE=replay pytest tests/integration/
```
### Live mode
Skip recordings entirely and use live APIs:
```bash
LLAMA_STACK_TEST_INFERENCE_MODE=live pytest tests/integration/
```
## Re-normalizing Existing Recordings
If you need to apply normalization to existing recordings (e.g., after updating the normalization logic):
```bash
python scripts/normalize_recordings.py
```
Use `--dry-run` to preview changes without modifying files.