diff --git a/tests/README.md b/tests/README.md index abbfc6d60..4790ca458 100644 --- a/tests/README.md +++ b/tests/README.md @@ -60,7 +60,9 @@ FIREWORKS_API_KEY=your_key pytest -sv tests/integration/inference --stack-config ### Re-recording tests -If you want to re-record tests, you can do so with: +#### Local Re-recording (Manual Setup Required) + +If you want to re-record tests locally, you can do so with: ```bash LLAMA_STACK_TEST_INFERENCE_MODE=record \ @@ -71,7 +73,6 @@ LLAMA_STACK_TEST_INFERENCE_MODE=record \ This will record new API responses and overwrite the existing recordings. - ```{warning} You must be careful when re-recording. CI workflows assume a specific setup for running the replay-mode tests. You must re-record the tests in the same way as the CI workflows. This means @@ -79,6 +80,35 @@ You must be careful when re-recording. CI workflows assume a specific setup for - you are using the `starter` distribution. ``` +#### Remote Re-recording (Recommended) + +**For easier re-recording without local setup**, use the automated recording workflow: + +```bash +# Record tests for specific test subdirectories +./scripts/github/schedule-record-workflow.sh --test-subdirs "agents,inference" + +# Record with vision tests enabled +./scripts/github/schedule-record-workflow.sh --test-subdirs "inference" --run-vision-tests + +# Record with specific provider +./scripts/github/schedule-record-workflow.sh --test-subdirs "agents" --test-provider vllm +``` + +This script: +- 🚀 **Runs in GitHub Actions** - no local Ollama setup required +- 🔍 **Auto-detects your branch** and associated PR +- 🍴 **Works from forks** - handles repository context automatically +- ✅ **Commits recordings back** to your branch +- 🛡️ **Validates inputs** - prevents common mistakes + +**Prerequisites:** +- GitHub CLI: `brew install gh && gh auth login` +- jq: `brew install jq` +- Your branch pushed to a remote + +**Supported providers:** `vllm`, `ollama` + ### Next Steps diff --git a/tests/integration/README.md b/tests/integration/README.md index 427b905b4..46d66fd79 100644 --- a/tests/integration/README.md +++ b/tests/integration/README.md @@ -134,6 +134,15 @@ cat recordings/responses/abc123.json | jq '.' ``` ### Re-recording Tests + +#### Remote Re-recording (Recommended) +Use the automated workflow script for easier re-recording: +```bash +./scripts/github/schedule-record-workflow.sh --test-subdirs "inference,agents" +``` +See the [main testing guide](../README.md#remote-re-recording-recommended) for full details. + +#### Local Re-recording ```bash # Re-record specific tests LLAMA_STACK_TEST_INFERENCE_MODE=record \