941f505eb0
# What does this PR do? This is an initial working prototype of wiring up the `file_search` builtin tool for the Responses API to our existing rag knowledge search tool. This is me seeing what I could pull together on top of the bits we already have merged. This may not be the ideal way to implement this, and things like how I shuffle the vector store ids from the original response API tool request to the actual tool execution feel a bit hacky (grep for `tool_kwargs["vector_db_ids"]` in `_execute_tool_call` to see what I mean). ## Test Plan I stubbed in some new tests to exercise this using text and pdf documents. Note that this is currently under tests/verification only because it sometimes flakes with tool calling of the small Llama-3.2-3B model we run in CI (and that I use as an example below). We'd want to make the test a bit more robust in some way if we moved this over to tests/integration and ran it in CI. ### OpenAI SaaS (to verify test correctness) ``` pytest -sv tests/verifications/openai_api/test_responses.py \ -k 'file_search' \ --base-url=https://api.openai.com/v1 \ --model=gpt-4o ``` ### Fireworks with faiss vector store ``` llama stack run llama_stack/templates/fireworks/run.yaml pytest -sv tests/verifications/openai_api/test_responses.py \ -k 'file_search' \ --base-url=http://localhost:8321/v1/openai/v1 \ --model=meta-llama/Llama-3.3-70B-Instruct ``` ### Ollama with faiss vector store This sometimes flakes on Ollama because the quantized small model doesn't always choose to call the tool to answer the user's question. But, it often works. ``` ollama run llama3.2:3b INFERENCE_MODEL="meta-llama/Llama-3.2-3B-Instruct" \ llama stack run ./llama_stack/templates/ollama/run.yaml \ --image-type venv \ --env OLLAMA_URL="http://0.0.0.0:11434" pytest -sv tests/verifications/openai_api/test_responses.py \ -k'file_search' \ --base-url=http://localhost:8321/v1/openai/v1 \ --model=meta-llama/Llama-3.2-3B-Instruct ``` ### OpenAI provider with sqlite-vec vector store ``` llama stack run ./llama_stack/templates/starter/run.yaml --image-type venv pytest -sv tests/verifications/openai_api/test_responses.py \ -k 'file_search' \ --base-url=http://localhost:8321/v1/openai/v1 \ --model=openai/gpt-4o-mini ``` ### Ensure existing vector store integration tests still pass ``` ollama run llama3.2:3b INFERENCE_MODEL="meta-llama/Llama-3.2-3B-Instruct" \ llama stack run ./llama_stack/templates/ollama/run.yaml \ --image-type venv \ --env OLLAMA_URL="http://0.0.0.0:11434" LLAMA_STACK_CONFIG=http://localhost:8321 \ pytest -sv tests/integration/vector_io \ --text-model "meta-llama/Llama-3.2-3B-Instruct" \ --embedding-model=all-MiniLM-L6-v2 ``` --------- Signed-off-by: Ben Browning <bbrownin@redhat.com> |
||
|---|---|---|
| .. | ||
| conf | ||
| openai_api | ||
| test_results | ||
| __init__.py | ||
| conftest.py | ||
| generate_report.py | ||
| openai-api-verification-run.yaml | ||
| README.md | ||
| REPORT.md | ||
Llama Stack Verifications
Llama Stack Verifications provide standardized test suites to ensure API compatibility and behavior consistency across different LLM providers. These tests help verify that different models and providers implement the expected interfaces and behaviors correctly.
Overview
This framework allows you to run the same set of verification tests against different LLM providers' OpenAI-compatible endpoints (Fireworks, Together, Groq, Cerebras, etc., and OpenAI itself) to ensure they meet the expected behavior and interface standards.
Features
The verification suite currently tests the following in both streaming and non-streaming modes:
- Basic chat completions
- Image input capabilities
- Structured JSON output formatting
- Tool calling functionality
Report
The lastest report can be found at REPORT.md.
To update the report, ensure you have the API keys set,
export OPENAI_API_KEY=<your_openai_api_key>
export FIREWORKS_API_KEY=<your_fireworks_api_key>
export TOGETHER_API_KEY=<your_together_api_key>
then run
uv run python tests/verifications/generate_report.py --run-tests
Running Tests
To run the verification tests, use pytest with the following parameters:
cd llama-stack
pytest tests/verifications/openai_api --provider=<provider-name>
Example:
# Run all tests
pytest tests/verifications/openai_api --provider=together
# Only run tests with Llama 4 models
pytest tests/verifications/openai_api --provider=together -k 'Llama-4'
Parameters
--provider: The provider name (openai, fireworks, together, groq, cerebras, etc.)--base-url: The base URL for the provider's API (optional - defaults to the standard URL for the specified provider)--api-key: Your API key for the provider (optional - defaults to the standard API_KEY name for the specified provider)
Supported Providers
The verification suite supports any provider with an OpenAI compatible endpoint.
See tests/verifications/conf/ for the list of supported providers.
To run on a new provider, simply add a new yaml file to the conf/ directory with the provider config. See tests/verifications/conf/together.yaml for an example.
Adding New Test Cases
To add new test cases, create appropriate JSON files in the openai_api/fixtures/test_cases/ directory following the existing patterns.
Structure
__init__.py- Marks the directory as a Python packageconf/- Provider-specific configuration filesopenai_api/- Tests specific to OpenAI-compatible APIsfixtures/- Test fixtures and utilitiesfixtures.py- Provider-specific fixturesload.py- Utilities for loading test casestest_cases/- JSON test case definitions
test_chat_completion.py- Tests for chat completion APIs