add notes about batches development status to docs

this also captures other notes from agents, eval and inference apis
This commit is contained in:
Matthew Farrellee 2025-08-13 07:15:07 -04:00
parent 8e678912ec
commit 04a73c89ef
7 changed files with 56 additions and 1 deletions

View file

@ -18,3 +18,4 @@ We are working on adding a few more APIs to complete the application lifecycle.
- **Batch Inference**: run inference on a dataset of inputs - **Batch Inference**: run inference on a dataset of inputs
- **Batch Agents**: run agents on a dataset of inputs - **Batch Agents**: run agents on a dataset of inputs
- **Synthetic Data Generation**: generate synthetic data for model development - **Synthetic Data Generation**: generate synthetic data for model development
- **Batches**: OpenAI-compatible batch management for inference

View file

@ -2,6 +2,15 @@
## Overview ## Overview
Agents API for creating and interacting with agentic systems.
Main functionalities provided by this API:
- Create agents with specific instructions and ability to use tools.
- Interactions with agents are grouped into sessions ("threads"), and each interaction is called a "turn".
- Agents can be provided with various tools (see the ToolGroups and ToolRuntime APIs for more details).
- Agents can be provided with various shields (see the Safety API for more details).
- Agents can also use Memory to retrieve information from knowledge bases. See the RAG Tool and Vector IO APIs for more details.
This section contains documentation for all available providers for the **agents** API. This section contains documentation for all available providers for the **agents** API.
## Providers ## Providers

View file

@ -2,6 +2,14 @@
## Overview ## Overview
Protocol for batch processing API operations.
The Batches API enables efficient processing of multiple requests in a single operation,
particularly useful for processing large datasets, batch evaluation workflows, and
cost-effective inference at scale.
Note: This API is currently under active development and may undergo changes.
This section contains documentation for all available providers for the **batches** API. This section contains documentation for all available providers for the **batches** API.
## Providers ## Providers

View file

@ -2,6 +2,8 @@
## Overview ## Overview
Llama Stack Evaluation API for running evaluations on model and agent candidates.
This section contains documentation for all available providers for the **eval** API. This section contains documentation for all available providers for the **eval** API.
## Providers ## Providers

View file

@ -2,6 +2,12 @@
## Overview ## Overview
Llama Stack Inference API for generating completions, chat completions, and embeddings.
This API provides the raw interface to the underlying models. Two kinds of models are supported:
- LLM models: these models generate "raw" and "chat" (conversational) completions.
- Embedding models: these models generate embeddings to be used for semantic search.
This section contains documentation for all available providers for the **inference** API. This section contains documentation for all available providers for the **inference** API.
## Providers ## Providers

View file

@ -39,7 +39,14 @@ class ListBatchesResponse(BaseModel):
@runtime_checkable @runtime_checkable
class Batches(Protocol): class Batches(Protocol):
"""Protocol for batch processing API operations.""" """Protocol for batch processing API operations.
The Batches API enables efficient processing of multiple requests in a single operation,
particularly useful for processing large datasets, batch evaluation workflows, and
cost-effective inference at scale.
Note: This API is currently under active development and may undergo changes.
"""
@webmethod(route="/openai/v1/batches", method="POST") @webmethod(route="/openai/v1/batches", method="POST")
async def create_batch( async def create_batch(

View file

@ -18,6 +18,23 @@ from llama_stack.core.distribution import get_provider_registry
REPO_ROOT = Path(__file__).parent.parent REPO_ROOT = Path(__file__).parent.parent
def get_api_docstring(api_name: str) -> str | None:
"""Extract docstring from the API protocol class."""
try:
# Import the API module dynamically
api_module = __import__(f"llama_stack.apis.{api_name}", fromlist=[api_name.title()])
# Get the main protocol class (usually capitalized API name)
protocol_class_name = api_name.title()
if hasattr(api_module, protocol_class_name):
protocol_class = getattr(api_module, protocol_class_name)
return protocol_class.__doc__
except (ImportError, AttributeError):
pass
return None
class ChangedPathTracker: class ChangedPathTracker:
"""Track a list of paths we may have changed.""" """Track a list of paths we may have changed."""
@ -261,6 +278,11 @@ def process_provider_registry(progress, change_tracker: ChangedPathTracker) -> N
index_content.append(f"# {api_name.title()}\n") index_content.append(f"# {api_name.title()}\n")
index_content.append("## Overview\n") index_content.append("## Overview\n")
api_docstring = get_api_docstring(api_name)
if api_docstring:
cleaned_docstring = api_docstring.strip()
index_content.append(f"{cleaned_docstring}\n")
index_content.append( index_content.append(
f"This section contains documentation for all available providers for the **{api_name}** API.\n" f"This section contains documentation for all available providers for the **{api_name}** API.\n"
) )