mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-10-12 05:54:38 +00:00
chore(tests): normalize recording IDs and timestamps to reduce git diff noise (#3676)
IDs are now deterministic hashes based on request content, and timestamps are normalized to constants, eliminating spurious changes when re-recording tests. ## Changes - Updated `inference_recorder.py` to normalize IDs and timestamps during recording - Added `scripts/normalize_recordings.py` utility to re-normalize existing recordings - Created documentation in `tests/integration/recordings/README.md` - Normalized 350 existing recording files
This commit is contained in:
parent
6bcd3e25f2
commit
3f36bfaeaa
348 changed files with 10154 additions and 8329 deletions
60
tests/integration/recordings/README.md
Normal file
60
tests/integration/recordings/README.md
Normal file
|
@ -0,0 +1,60 @@
|
|||
# Test Recording System
|
||||
|
||||
This directory contains recorded inference API responses used for deterministic testing without requiring live API access.
|
||||
|
||||
## Structure
|
||||
|
||||
- `responses/` - JSON files containing request/response pairs for inference operations
|
||||
|
||||
## Recording Format
|
||||
|
||||
Each JSON file contains:
|
||||
- `request` - The normalized request parameters (method, endpoint, body)
|
||||
- `response` - The response body (serialized from Pydantic models)
|
||||
|
||||
## Normalization
|
||||
|
||||
To reduce noise in git diffs, the recording system automatically normalizes fields that vary between runs but don't affect test behavior:
|
||||
|
||||
### OpenAI-style responses
|
||||
- `id` - Deterministic hash based on request: `rec-{request_hash[:12]}`
|
||||
- `created` - Normalized to epoch: `0`
|
||||
|
||||
### Ollama-style responses
|
||||
- `created_at` - Normalized to: `"1970-01-01T00:00:00.000000Z"`
|
||||
- `total_duration` - Normalized to: `0`
|
||||
- `load_duration` - Normalized to: `0`
|
||||
- `prompt_eval_duration` - Normalized to: `0`
|
||||
- `eval_duration` - Normalized to: `0`
|
||||
|
||||
These normalizations ensure that re-recording tests produces minimal git diffs, making it easier to review actual changes to test behavior.
|
||||
|
||||
## Usage
|
||||
|
||||
### Recording mode
|
||||
Set `LLAMA_STACK_TEST_INFERENCE_MODE=record` to capture new responses:
|
||||
```bash
|
||||
LLAMA_STACK_TEST_INFERENCE_MODE=record pytest tests/integration/
|
||||
```
|
||||
|
||||
### Replay mode (default)
|
||||
Responses are replayed from recordings:
|
||||
```bash
|
||||
LLAMA_STACK_TEST_INFERENCE_MODE=replay pytest tests/integration/
|
||||
```
|
||||
|
||||
### Live mode
|
||||
Skip recordings entirely and use live APIs:
|
||||
```bash
|
||||
LLAMA_STACK_TEST_INFERENCE_MODE=live pytest tests/integration/
|
||||
```
|
||||
|
||||
## Re-normalizing Existing Recordings
|
||||
|
||||
If you need to apply normalization to existing recordings (e.g., after updating the normalization logic):
|
||||
|
||||
```bash
|
||||
python scripts/normalize_recordings.py
|
||||
```
|
||||
|
||||
Use `--dry-run` to preview changes without modifying files.
|
Loading…
Add table
Add a link
Reference in a new issue