phoenix-oss/llama-stack-mirror

mirror of https://github.com/meta-llama/llama-stack.git synced 2025-07-07 06:20:45 +00:00

History

ehhuang fc735a414e Some checks failed Integration Tests / test-matrix (http, 3.12, inference) (push) Failing after 4s Details Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 8s Details Integration Tests / test-matrix (http, 3.12, post_training) (push) Failing after 10s Details Integration Tests / test-matrix (http, 3.13, providers) (push) Failing after 8s Details Integration Tests / test-matrix (http, 3.13, inference) (push) Failing after 10s Details Integration Tests / test-matrix (http, 3.13, scoring) (push) Failing after 10s Details Integration Tests / test-matrix (http, 3.13, datasets) (push) Failing after 12s Details Integration Tests / test-matrix (http, 3.12, providers) (push) Failing after 13s Details Integration Tests / test-matrix (http, 3.13, vector_io) (push) Failing after 9s Details Integration Tests / test-matrix (http, 3.13, agents) (push) Failing after 12s Details Integration Tests / test-matrix (http, 3.13, post_training) (push) Failing after 22s Details Integration Tests / test-matrix (http, 3.12, tool_runtime) (push) Failing after 20s Details Integration Tests / test-matrix (http, 3.12, inspect) (push) Failing after 14s Details Integration Tests / test-matrix (library, 3.12, datasets) (push) Failing after 20s Details Integration Tests / test-matrix (library, 3.12, tool_runtime) (push) Failing after 19s Details Integration Tests / test-matrix (library, 3.12, vector_io) (push) Failing after 9s Details Integration Tests / test-matrix (library, 3.12, post_training) (push) Failing after 11s Details Integration Tests / test-matrix (http, 3.12, vector_io) (push) Failing after 9s Details Integration Tests / test-matrix (library, 3.12, providers) (push) Failing after 9s Details Integration Tests / test-matrix (http, 3.13, inspect) (push) Failing after 16s Details Integration Tests / test-matrix (library, 3.12, inference) (push) Failing after 8s Details Integration Tests / test-matrix (http, 3.12, datasets) (push) Failing after 22s Details Integration Tests / test-matrix (http, 3.12, agents) (push) Failing after 15s Details Integration Tests / test-matrix (library, 3.12, agents) (push) Failing after 14s Details Integration Tests / test-matrix (library, 3.12, inspect) (push) Failing after 9s Details Integration Tests / test-matrix (library, 3.13, agents) (push) Failing after 13s Details Integration Tests / test-matrix (library, 3.13, inference) (push) Failing after 10s Details Integration Tests / test-matrix (library, 3.13, inspect) (push) Failing after 10s Details Integration Tests / test-matrix (library, 3.13, post_training) (push) Failing after 9s Details Integration Tests / test-matrix (http, 3.12, scoring) (push) Failing after 18s Details Integration Tests / test-matrix (http, 3.13, tool_runtime) (push) Failing after 13s Details Integration Tests / test-matrix (library, 3.13, scoring) (push) Failing after 8s Details Integration Tests / test-matrix (library, 3.13, datasets) (push) Failing after 9s Details Integration Tests / test-matrix (library, 3.12, scoring) (push) Failing after 13s Details Integration Tests / test-matrix (library, 3.13, providers) (push) Failing after 9s Details Integration Tests / test-matrix (library, 3.13, tool_runtime) (push) Failing after 21s Details Integration Tests / test-matrix (library, 3.13, vector_io) (push) Failing after 20s Details Vector IO Integration Tests / test-matrix (3.12, inline::faiss) (push) Failing after 19s Details Vector IO Integration Tests / test-matrix (3.12, inline::milvus) (push) Failing after 18s Details Vector IO Integration Tests / test-matrix (3.12, inline::sqlite-vec) (push) Failing after 17s Details Vector IO Integration Tests / test-matrix (3.12, remote::chromadb) (push) Failing after 18s Details Vector IO Integration Tests / test-matrix (3.12, remote::pgvector) (push) Failing after 20s Details Vector IO Integration Tests / test-matrix (3.13, inline::milvus) (push) Failing after 20s Details Vector IO Integration Tests / test-matrix (3.13, inline::faiss) (push) Failing after 23s Details Vector IO Integration Tests / test-matrix (3.13, inline::sqlite-vec) (push) Failing after 19s Details Vector IO Integration Tests / test-matrix (3.13, remote::chromadb) (push) Failing after 11s Details Vector IO Integration Tests / test-matrix (3.13, remote::pgvector) (push) Failing after 12s Details Python Package Build Test / build (3.12) (push) Failing after 1m3s Details Python Package Build Test / build (3.13) (push) Failing after 1m3s Details Test External Providers / test-external-providers (venv) (push) Failing after 1m7s Details Unit Tests / unit-tests (3.12) (push) Failing after 1m15s Details Unit Tests / unit-tests (3.13) (push) Failing after 19s Details Pre-commit / pre-commit (push) Successful in 2m42s Details test: Add one-step integration testing with server auto-start (#2580 ) ## Summary Add support for `server:<config>` format in `--stack-config` option to enable seamless one-step integration testing. This eliminates the need to manually start servers in separate terminals before running tests. ## Key Features - Auto-start server: Automatically launches `llama stack run <config>` if target port is available - Smart reuse: Reuses existing server if port is already occupied - Health check polling: Waits up to 2 minutes for server readiness via `/v1/health` endpoint - Custom port support: Use `server:<config>:<port>` for non-default ports - Clean output: Server runs quietly in background without cluttering test output - Backward compatibility: All existing `--stack-config` formats continue to work ## Usage Examples ```bash # Auto-start server with default port 8321 pytest tests/integration/inference/ --stack-config=server:fireworks # Use custom port pytest tests/integration/safety/ --stack-config=server:together:8322 # Run multiple test suites seamlessly pytest tests/integration/inference/ tests/integration/agents/ --stack-config=server:starter ``` ## Implementation Details - Enhanced `llama_stack_client` fixture with server management - Updated documentation with cleaner organization and comprehensive examples - Added utility functions for port checking, server startup, and health verification ## Test Plan - Verified server auto-start when port 8321 is available - Verified server reuse when port 8321 is occupied - Tested health check polling via `/v1/health` endpoint - Confirmed custom port configuration works correctly - Verified backward compatibility with existing config formats ## Before/After Comparison Before (2 steps): ```bash # Terminal 1: Start server manually llama stack run fireworks --port 8321 # Terminal 2: Wait for startup, then run tests pytest tests/integration/inference/ --stack-config=http://localhost:8321 ``` After (1 step): ```bash # Single command handles everything pytest tests/integration/inference/ --stack-config=server:fireworks ```		2025-07-01 14:48:46 -07:00
..
client-sdk/post_training	feat: Add nemo customizer (#1448 )	2025-03-25 11:01:10 -07:00
common	feat(responses): implement full multi-turn support (#2295 )	2025-06-02 15:35:49 -07:00
external-provider/llama-stack-provider-ollama	refactor(env)!: enhanced environment variable substitution (#2490 )	2025-06-26 08:20:08 +05:30
integration	test: Add one-step integration testing with server auto-start (#2580 )	2025-07-01 14:48:46 -07:00
unit	fix: allow default empty vars for conditionals (#2570 )	2025-07-01 14:42:05 +02:00
verifications	fix(ollama): Download remote image URLs for Ollama (#2551 )	2025-06-30 20:36:11 +05:30
__init__.py	refactor(test): introduce --stack-config and simplify options (#1404 )	2025-03-05 17:02:02 -08:00
Containerfile	ci: use ollama container image with loaded models (#2410 )	2025-06-06 12:08:20 +02:00
README.md	docs: revamp testing documentation (#2155 )	2025-05-13 11:28:29 -07:00

README.md

Llama Stack Tests

Llama Stack has multiple layers of testing done to ensure continuous functionality and prevent regressions to the codebase.

Testing Type	Details
Unit	unit/README.md
Integration	integration/README.md
Verification	verifications/README.md