From e0d10dd0b1e865119497ca134664224f2e3538af Mon Sep 17 00:00:00 2001 From: Nathan Weinberg <31703736+nathan-weinberg@users.noreply.github.com> Date: Tue, 13 May 2025 14:28:29 -0400 Subject: [PATCH] docs: revamp testing documentation (#2155) # What does this PR do? reduces duplication and centralizes information to be easier to find for contributors Signed-off-by: Nathan Weinberg --- CONTRIBUTING.md | 26 ++------------------ docs/source/contributing/new_api_provider.md | 3 ++- tests/README.md | 9 +++++++ tests/integration/README.md | 2 +- tests/unit/README.md | 21 ++++++++++++++++ tests/verifications/README.md | 2 +- 6 files changed, 36 insertions(+), 27 deletions(-) create mode 100644 tests/README.md create mode 100644 tests/unit/README.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bd63b31d4..d7c3e3e2f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -110,31 +110,9 @@ uv run pre-commit run --all-files > [!CAUTION] > Before pushing your changes, make sure that the pre-commit hooks have passed successfully. -## Running unit tests +## Running tests -You can run the unit tests by running: - -```bash -source .venv/bin/activate -./scripts/unit-tests.sh [PYTEST_ARGS] -``` - -Any additional arguments are passed to pytest. For example, you can specify a test directory, a specific test file, or any pytest flags (e.g., -vvv for verbosity). If no test directory is specified, it defaults to "tests/unit", e.g: - -```bash -./scripts/unit-tests.sh tests/unit/registry/test_registry.py -vvv -``` - -If you'd like to run for a non-default version of Python (currently 3.10), pass `PYTHON_VERSION` variable as follows: - -``` -source .venv/bin/activate -PYTHON_VERSION=3.13 ./scripts/unit-tests.sh -``` - -## Running integration tests - -You can run integration tests following the instructions [here](tests/integration/README.md). +You can find the Llama Stack testing documentation here [here](tests/README.md). ## Adding a new dependency to the project diff --git a/docs/source/contributing/new_api_provider.md b/docs/source/contributing/new_api_provider.md index c412a350b..83058896a 100644 --- a/docs/source/contributing/new_api_provider.md +++ b/docs/source/contributing/new_api_provider.md @@ -6,7 +6,7 @@ This guide will walk you through the process of adding a new API provider to Lla - Begin by reviewing the [core concepts](../concepts/index.md) of Llama Stack and choose the API your provider belongs to (Inference, Safety, VectorIO, etc.) - Determine the provider type ({repopath}`Remote::llama_stack/providers/remote` or {repopath}`Inline::llama_stack/providers/inline`). Remote providers make requests to external services, while inline providers execute implementation locally. - Add your provider to the appropriate {repopath}`Registry::llama_stack/providers/registry/`. Specify pip dependencies necessary. -- Update any distribution {repopath}`Templates::llama_stack/templates/` build.yaml and run.yaml files if they should include your provider by default. Run {repopath}`./scripts/distro_codegen.py` if necessary. Note that `distro_codegen.py` will fail if the new provider causes any distribution template to attempt to import provider-specific dependencies. This usually means the distribution's `get_distribution_template()` code path should only import any necessary Config or model alias definitions from each provider and not the provider's actual implementation. +- Update any distribution {repopath}`Templates::llama_stack/templates/` `build.yaml` and `run.yaml` files if they should include your provider by default. Run {repopath}`./scripts/distro_codegen.py` if necessary. Note that `distro_codegen.py` will fail if the new provider causes any distribution template to attempt to import provider-specific dependencies. This usually means the distribution's `get_distribution_template()` code path should only import any necessary Config or model alias definitions from each provider and not the provider's actual implementation. Here are some example PRs to help you get started: @@ -33,6 +33,7 @@ Note that each provider's `sample_run_config()` method (in the configuration cla Unit tests are located in {repopath}`tests/unit`. Provider-specific unit tests are located in {repopath}`tests/unit/providers`. These tests are all run automatically as part of the CI process. +Consult {repopath}`tests/unit/README.md` for more details on how to run the tests manually. ### 3. Additional end-to-end testing diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 000000000..ed7064bfb --- /dev/null +++ b/tests/README.md @@ -0,0 +1,9 @@ +# Llama Stack Tests + +Llama Stack has multiple layers of testing done to ensure continuous functionality and prevent regressions to the codebase. + +| Testing Type | Details | +|--------------|---------| +| Unit | [unit/README.md](unit/README.md) | +| Integration | [integration/README.md](integration/README.md) | +| Verification | [verifications/README.md](verifications/README.md) | diff --git a/tests/integration/README.md b/tests/integration/README.md index 92bcf7c51..8c1ee6355 100644 --- a/tests/integration/README.md +++ b/tests/integration/README.md @@ -11,7 +11,7 @@ pytest --help Here are the most important options: - `--stack-config`: specify the stack config to use. You have three ways to point to a stack: - a URL which points to a Llama Stack distribution server - - a template (e.g., `fireworks`, `together`) or a path to a run.yaml file + - a template (e.g., `fireworks`, `together`) or a path to a `run.yaml` file - a comma-separated list of api=provider pairs, e.g. `inference=fireworks,safety=llama-guard,agents=meta-reference`. This is most useful for testing a single API surface. - `--env`: set environment variables, e.g. --env KEY=value. this is a utility option to set environment variables required by various providers. diff --git a/tests/unit/README.md b/tests/unit/README.md new file mode 100644 index 000000000..db2114049 --- /dev/null +++ b/tests/unit/README.md @@ -0,0 +1,21 @@ +# Llama Stack Unit Tests + +You can run the unit tests by running: + +```bash +source .venv/bin/activate +./scripts/unit-tests.sh [PYTEST_ARGS] +``` + +Any additional arguments are passed to pytest. For example, you can specify a test directory, a specific test file, or any pytest flags (e.g., -vvv for verbosity). If no test directory is specified, it defaults to "tests/unit", e.g: + +```bash +./scripts/unit-tests.sh tests/unit/registry/test_registry.py -vvv +``` + +If you'd like to run for a non-default version of Python (currently 3.10), pass `PYTHON_VERSION` variable as follows: + +``` +source .venv/bin/activate +PYTHON_VERSION=3.13 ./scripts/unit-tests.sh +``` diff --git a/tests/verifications/README.md b/tests/verifications/README.md index 88762e0ba..19d122bec 100644 --- a/tests/verifications/README.md +++ b/tests/verifications/README.md @@ -4,7 +4,7 @@ Llama Stack Verifications provide standardized test suites to ensure API compati ## Overview -This framework allows you to run the same set of verification tests against different LLM providers' OpenAI-compatible endpoints (Fireworks, Together, Groq, Cerebras, etc., and OpenAI itself) to ensure they meet the expected behavior and interface standards. +This framework allows you to run the same set of verification tests against different LLM providers' OpenAI-compatible endpoints (Fireworks, Together, Groq, Cerebras, etc., and OpenAI itself) to ensure they meet the expected behavior and interface standards. ## Features