Testing - Memory provider fakes

Summary:
Implementing Memory provider fakes as discussed in this draft https://github.com/meta-llama/llama-stack/pull/490#issuecomment-2492877393.

High level changes:
* Fake provider is specified via the "fake" mark
* Test config will setup a fake fixture for the run of the test
* Test resolver checks fixtures and upon finding a fake provider it injects InlineProviderSpec for fake provider
* Fake provider gets resolved through path/naming convention
* Fake provider implementaion is contained to the tests/ directory and implements stubs and method fakes with minimal functionality to simulate real provider

Instructins to creating a fake
* Create the "fakes" module inside the provider test directory
* Inside the module implement `get_provider_impl` that will return fake implementation object
* Name the fake implementation class to match the fake provider id (e.g. memory_fake -> MemoryFakeImpl)
  * Same rule for the config (e.g. memory_fake -> MemoryFakeConfig)
* Add fake fixture (in the fixtures.py) and setup methods stubs there

Test Plan:
Run memory tests
```
pytest llama_stack/providers/tests/memory/test_memory.py -m "fake" -v -s --tb=short

====================================================================================================== test session starts ======================================================================================================
platform darwin -- Python 3.11.10, pytest-8.3.3, pluggy-1.5.0 -- /opt/homebrew/Caskroom/miniconda/base/envs/llama-stack/bin/python
cachedir: .pytest_cache
rootdir: /Users/vivic/Code/llama-stack
configfile: pyproject.toml
plugins: asyncio-0.24.0, anyio-4.6.2.post1
asyncio: mode=Mode.STRICT, default_loop_scope=None
collected 18 items / 15 deselected / 3 selected

llama_stack/providers/tests/memory/test_memory.py::TestMemory::test_banks_list[fake] PASSED
llama_stack/providers/tests/memory/test_memory.py::TestMemory::test_banks_register[fake] PASSED
llama_stack/providers/tests/memory/test_memory.py::TestMemory::test_query_documents[fake] The scores are: [0.5]
PASSED

========================================================================================= 3 passed, 15 deselected, 10 warnings in 0.46s =========================================================================================

```
This commit is contained in:
Vladimir Ivic 2024-11-25 10:24:52 -08:00
parent 4e6c984c26
commit e2d1b712e2
3 changed files with 189 additions and 3 deletions

View file

@ -4,8 +4,10 @@
# This source code is licensed under the terms described in the LICENSE file in
# the root directory of this source tree.
import importlib.util
import json
import tempfile
from typing import Any, Dict, List, Optional
from llama_stack.distribution.datatypes import * # noqa: F403
@ -51,8 +53,11 @@ async def construct_stack_for_test(
try:
remote_config = remote_provider_config(run_config)
if not remote_config:
# TODO: add to provider registry by creating interesting mocks or fakes
impls = await construct_stack(run_config, get_provider_registry())
# Here we create instance of registry with optional fake provider
provider_registry = setup_provider_registry_for_test(
run_config, get_provider_registry()
)
impls = await construct_stack(run_config, provider_registry)
else:
# we don't register resources for a remote stack as part of the fixture setup
# because the stack is already "up". if a test needs to register resources, it
@ -73,6 +78,60 @@ async def construct_stack_for_test(
return test_stack
def setup_provider_registry_for_test(
run_config: StackRunConfig, provider_registry: Dict[Api, Dict[str, ProviderSpec]]
) -> Dict[Api, Dict[str, ProviderSpec]]:
provider_registry = get_provider_registry()
for api_name, providers in run_config.providers.items():
for provider in providers:
if provider.provider_type == "test::fake":
# Check if the fake provider module exists for the API that is trying
# to use test::fake provider_type
provider_fake_module_name = (
f"llama_stack.providers.tests.{api_name}.fakes"
)
provider_fake_module_spec = importlib.util.find_spec(
provider_fake_module_name
)
if provider_fake_module_spec is None:
raise ValueError(
f"Fake provider module {provider_fake_module_name} does not exist. "
f"The module must be defined inside the providers/tests/{api_name}/fakes.py file."
)
# Import the module so we can validate that the config class exists
provider_fake_module = importlib.import_module(
provider_fake_module_name
)
# Check if the fake provider config class exists
# The class name is derived from the provider type e.g.
# provider_id: "example_provider" -> class_name: "ExampleProviderConfig"
provider_fake_config_class_name = (
f"{provider.provider_id}_config".title().replace("_", "")
)
if not hasattr(provider_fake_module, provider_fake_config_class_name):
raise ValueError(
f"Fake provider config class {provider_fake_config_class_name} "
f"does not exist in module {provider_fake_module_name}. "
f"The config class must be defined inside the providers/tests/{api_name}/fakes.py file."
)
provider_fake_config_class_path = f"llama_stack.providers.tests.{api_name}.fakes.{provider_fake_config_class_name}"
api = getattr(Api, api_name)
fake_api_spec = InlineProviderSpec(
api=api,
provider_type="test::fake",
pip_packages=[],
module=provider_fake_module_name,
config_class=provider_fake_config_class_path,
)
provider_registry[api].update({"test::fake": fake_api_spec})
return provider_registry
def remote_provider_config(
run_config: StackRunConfig,
) -> Optional[RemoteProviderConfig]: