mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-06-28 02:53:30 +00:00
2cbe9395b0
# What does this PR do? We have several places running tests for different purposes. - oss llama stack - provider tests - e2e tests - provider llama stack - unit tests - e2e tests It would be nice if they can *share the same set of test data*, so we maintain the consistency between spec and implementation. This is what this diff is about, isolating test data from test coding, so that we can reuse the same data at different places by writing different test coding. ## Test Plan == Set up Ollama local server == Run a provider test conda activate stack OLLAMA_URL="http://localhost:8321" \ pytest -v -s -k "ollama" --inference-model="llama3.2:3b-instruct-fp16" \ llama_stack/providers/tests/inference/test_text_inference.py::TestInference::test_completion_structured_output // test_structured_output should also work == Run an e2e test conda activate sherpa with-proxy pip install llama-stack export INFERENCE_MODEL=llama3.2:3b-instruct-fp16 export LLAMA_STACK_PORT=8322 with-proxy llama stack build --template ollama with-proxy llama stack run --env OLLAMA_URL=http://localhost:8321 ollama - Run test client, LLAMA_STACK_PORT=8322 LLAMA_STACK_BASE_URL="http://localhost:8322" \ pytest -v -s --inference-model="llama3.2:3b-instruct-fp16" \ tests/client-sdk/inference/test_text_inference.py::test_text_completion_structured_output // test_text_chat_completion_structured_output should also work ## Notes - This PR was automatically generated by oss_sync - Please refer to D69478008 for more details. |
||
|---|---|---|
| .. | ||
| agents | ||
| inference | ||
| safety | ||
| tool_runtime | ||
| vector_io | ||
| __init__.py | ||
| conftest.py | ||
| metadata.py | ||
| README.md | ||
| report.py | ||
Llama Stack Integration Tests
You can run llama stack integration tests on either a Llama Stack Library or a Llama Stack endpoint.
To test on a Llama Stack library with certain configuration, run
LLAMA_STACK_CONFIG=./llama_stack/templates/cerebras/run.yaml pytest -s -v tests/client-sdk/inference/
or just the template name
LLAMA_STACK_CONFIG=together pytest -s -v tests/client-sdk/inference/
To test on a Llama Stack endpoint, run
LLAMA_STACK_BASE_URL=http://localhost:8089 pytest -s -v tests/client-sdk/inference
Report Generation
To generate a report, run with --report option
LLAMA_STACK_CONFIG=together pytest -s -v report.md tests/client-sdk/ --report
Common options
Depending on the API, there are custom options enabled
- For tests in
inference/andagents/, we support--inference-model(to be used in text inference tests) and--vision-inference-model` (only used in image inference tests) overrides - For tests in
vector_io/, we support--embedding-modeloverride - For tests in
safety/, we support--safety-shieldoverride - The param can be
--reportor--report <path>If path is not provided, we do a best effort to infer based on the config / template name. For url endpoints, path is required.