mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-10-05 04:17:32 +00:00
feat(tests): migrate to global "setups" system for test configuration
This PR refactors the integration test system from suite-specific "contexts" to global "setups". This provides better separation of concerns: **suites = what to test, setups = how to configure.** Key changes: - New `tests/integration/setups.py` with global, reusable configurations (ollama, vllm, gpt, claude) - Simplified `tests/integration/suites.py` to only define test collection roots + default setup references - Updated CLI from `--context` to `--setup` parameter that works with any suite - Modified `scripts/integration-tests.sh` to use `--test-setup` instead of `--test-context` - Updated documentation to reflect the new global setup system Benefits: - Setups can be reused across multiple suites (e.g., use "gpt" with any suite) - Clear separation between test selection (suites) and configuration (setups) - Easier to add new configurations without modifying existing suites - Centralized configuration management Usage examples: - `pytest tests/integration --suite=responses --setup=gpt` - `pytest tests/integration --suite=vision --setup=ollama` - `pytest tests/integration --suite=base --setup=vllm`
This commit is contained in:
parent
47b640370e
commit
c662d8aa31
10 changed files with 272 additions and 178 deletions
|
@ -14,7 +14,7 @@ set -euo pipefail
|
|||
# Default values
|
||||
BRANCH=""
|
||||
TEST_SUBDIRS=""
|
||||
TEST_PROVIDER="ollama"
|
||||
TEST_SETUP="ollama"
|
||||
TEST_SUITE="base"
|
||||
TEST_PATTERN=""
|
||||
|
||||
|
@ -27,7 +27,7 @@ Trigger the integration test recording workflow remotely. This way you do not ne
|
|||
|
||||
OPTIONS:
|
||||
-b, --branch BRANCH Branch to run the workflow on (defaults to current branch)
|
||||
-p, --test-provider PROVIDER Test provider to use: vllm or ollama (default: ollama)
|
||||
-p, --test-setup SETUP Test setup to use: vllm, ollama, gpt, etc. (default: ollama)
|
||||
-t, --test-suite SUITE Test suite to use: base, responses, vision, etc. (default: base)
|
||||
-s, --test-subdirs DIRS Comma-separated list of test subdirectories to run (overrides suite)
|
||||
-k, --test-pattern PATTERN Regex pattern to pass to pytest -k
|
||||
|
@ -40,8 +40,8 @@ EXAMPLES:
|
|||
# Record tests for specific branch with vision tests
|
||||
$0 -b my-feature-branch --test-suite vision
|
||||
|
||||
# Record multiple test subdirectories with specific provider
|
||||
$0 --test-subdirs "agents,inference" --test-provider vllm
|
||||
# Record multiple test subdirectories with specific setup
|
||||
$0 --test-subdirs "agents,inference" --test-setup vllm
|
||||
|
||||
# Record tests matching a specific pattern
|
||||
$0 --test-subdirs "inference" --test-pattern "test_streaming"
|
||||
|
@ -67,8 +67,8 @@ while [[ $# -gt 0 ]]; do
|
|||
TEST_SUBDIRS="$2"
|
||||
shift 2
|
||||
;;
|
||||
-p|--test-provider)
|
||||
TEST_PROVIDER="$2"
|
||||
-p|--test-setup)
|
||||
TEST_SETUP="$2"
|
||||
shift 2
|
||||
;;
|
||||
-t|--test-suite)
|
||||
|
@ -101,13 +101,8 @@ if [[ -z "$TEST_SUBDIRS" && -z "$TEST_SUITE" ]]; then
|
|||
exit 1
|
||||
fi
|
||||
|
||||
# Validate test provider
|
||||
if [[ "$TEST_PROVIDER" != "vllm" && "$TEST_PROVIDER" != "ollama" ]]; then
|
||||
echo "❌ Error: Invalid test provider '$TEST_PROVIDER'"
|
||||
echo " Supported providers: vllm, ollama"
|
||||
echo " Example: $0 --test-subdirs \"agents\" --test-provider vllm"
|
||||
exit 1
|
||||
fi
|
||||
# Validate test setup (optional - setups are validated by the workflow itself)
|
||||
# Common setups: ollama, vllm, gpt, etc.
|
||||
|
||||
# Check if required tools are installed
|
||||
if ! command -v gh &> /dev/null; then
|
||||
|
@ -237,7 +232,7 @@ fi
|
|||
# Build the workflow dispatch command
|
||||
echo "Triggering integration test recording workflow..."
|
||||
echo "Branch: $BRANCH"
|
||||
echo "Test provider: $TEST_PROVIDER"
|
||||
echo "Test setup: $TEST_SETUP"
|
||||
echo "Test subdirs: $TEST_SUBDIRS"
|
||||
echo "Test suite: $TEST_SUITE"
|
||||
echo "Test pattern: ${TEST_PATTERN:-"(none)"}"
|
||||
|
@ -245,16 +240,16 @@ echo ""
|
|||
|
||||
# Prepare inputs for gh workflow run
|
||||
if [[ -n "$TEST_SUBDIRS" ]]; then
|
||||
INPUTS="-f test-subdirs='$TEST_SUBDIRS'"
|
||||
INPUTS="-f subdirs='$TEST_SUBDIRS'"
|
||||
fi
|
||||
if [[ -n "$TEST_PROVIDER" ]]; then
|
||||
INPUTS="$INPUTS -f test-provider='$TEST_PROVIDER'"
|
||||
if [[ -n "$TEST_SETUP" ]]; then
|
||||
INPUTS="$INPUTS -f test-setup='$TEST_SETUP'"
|
||||
fi
|
||||
if [[ -n "$TEST_SUITE" ]]; then
|
||||
INPUTS="$INPUTS -f test-suite='$TEST_SUITE'"
|
||||
INPUTS="$INPUTS -f suite='$TEST_SUITE'"
|
||||
fi
|
||||
if [[ -n "$TEST_PATTERN" ]]; then
|
||||
INPUTS="$INPUTS -f test-pattern='$TEST_PATTERN'"
|
||||
INPUTS="$INPUTS -f pattern='$TEST_PATTERN'"
|
||||
fi
|
||||
|
||||
# Run the workflow
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue