diff --git a/docs/source/contributing/index.md b/docs/source/contributing/index.md index 228258cdf..296a49f24 100644 --- a/docs/source/contributing/index.md +++ b/docs/source/contributing/index.md @@ -31,5 +31,4 @@ For developers who need deeper understanding of the testing system internals: :maxdepth: 1 testing/record-replay -testing/troubleshooting ``` diff --git a/docs/source/contributing/testing/troubleshooting.md b/docs/source/contributing/testing/troubleshooting.md deleted file mode 100644 index 170f45b35..000000000 --- a/docs/source/contributing/testing/troubleshooting.md +++ /dev/null @@ -1,140 +0,0 @@ -# Common Testing Issues - -The most frequent problems when working with Llama Stack's testing system. - -## Missing Recordings - -**Error:** -``` -RuntimeError: No recorded response found for request hash: abc123def456 -Endpoint: /v1/chat/completions -Model: meta-llama/Llama-3.1-8B-Instruct -``` - -**Cause:** You're running a test that needs an API interaction that hasn't been recorded yet. - -**Solution:** -```bash -# Record the missing interaction -LLAMA_STACK_TEST_INFERENCE_MODE=record \ -LLAMA_STACK_TEST_RECORDING_DIR=./recordings \ -pytest tests/integration/inference/test_your_test.py -``` - -## API Key Issues - -**Error:** -``` -HTTP 401: Invalid API key -``` - -**Cause:** Missing or invalid API key for the provider you're testing. - -**Solution:** -```bash -# Set the required API key -export FIREWORKS_API_KEY=your_key_here -export OPENAI_API_KEY=your_key_here - -# Verify it's set -echo $FIREWORKS_API_KEY -``` - -## Model Not Found - -**Error:** -``` -Model 'meta-llama/Llama-3.1-8B-Instruct' not found -``` - -**Cause:** Model isn't available with the current provider or hasn't been downloaded locally. - -**For local providers (Ollama):** -```bash -# Download the model -ollama pull llama3.2:3b - -# Use the downloaded model -pytest tests/integration/ --text-model=llama3.2:3b -``` - -**For remote providers:** -```bash -# Check what models are available -uv run llama stack list-models - -# Use an available model -pytest tests/integration/ --text-model=available-model-id -``` - -## Server Connection Issues - -**Error:** -``` -requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5001) -``` - -**Cause:** Server isn't running or is on a different port. - -**Solution:** -```bash -# Check if server is running -curl http://localhost:5001/v1/health - -# Start server manually -llama stack run --template starter --port 5001 - -# Or use auto-server with custom port -pytest tests/integration/ --stack-config=server:starter:8322 -``` - -## Request Hash Mismatches - -**Problem:** Tests worked before but now fail with "No recorded response found" even though you didn't change the test. - -**Cause:** Request parameters changed slightly (different whitespace, float precision, etc.). The hashing is intentionally precise. - -**Solution:** -```bash -# Check what's in your recordings -sqlite3 recordings/index.sqlite "SELECT endpoint, model FROM recordings;" - -# Re-record if the request legitimately changed -rm recordings/responses/old_hash.json -LLAMA_STACK_TEST_INFERENCE_MODE=record pytest tests/integration/your_test.py -``` - -## No Tests Collected - -**Error:** -``` -collected 0 items -``` - -**Cause:** No models specified for tests that require model fixtures. - -**Solution:** -```bash -# Specify required models -pytest tests/integration/inference/ --text-model=llama3.2:3b -pytest tests/integration/embedding/ --embedding-model=all-MiniLM-L6-v2 -``` - -## Getting Help - -When reporting issues, include: - -```bash -# Environment info -uv run python --version -uv run python -c "import llama_stack; print(llama_stack.__version__)" - -# Test command that failed -pytest tests/integration/your_test.py -v - -# Stack configuration -echo $LLAMA_STACK_TEST_INFERENCE_MODE -ls -la recordings/ -``` - -Most issues are solved by re-recording interactions or checking API keys/model availability. \ No newline at end of file diff --git a/tests/README.md b/tests/README.md index 225cea944..abbfc6d60 100644 --- a/tests/README.md +++ b/tests/README.md @@ -58,6 +58,28 @@ If you don't specify LLAMA_STACK_TEST_INFERENCE_MODE, by default it will be in " FIREWORKS_API_KEY=your_key pytest -sv tests/integration/inference --stack-config=starter ``` +### Re-recording tests + +If you want to re-record tests, you can do so with: + +```bash +LLAMA_STACK_TEST_INFERENCE_MODE=record \ + LLAMA_STACK_TEST_RECORDING_DIR=tests/integration/recordings \ + uv run --group test \ + pytest -sv tests/integration/ --stack-config=starter -k "" +``` + +This will record new API responses and overwrite the existing recordings. + + +```{warning} + +You must be careful when re-recording. CI workflows assume a specific setup for running the replay-mode tests. You must re-record the tests in the same way as the CI workflows. This means +- you need Ollama running and serving some specific models. +- you are using the `starter` distribution. +``` + + ### Next Steps - [Integration Testing Guide](integration/README.md) - Detailed usage and configuration