phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-12-03 18:00:36 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge branch 'main' into feat/add-metadata

Browse source
This commit is contained in:
Ashwin Bharambe 2025-12-01 10:42:04 -08:00 • committed by GitHub
commit dc6d984f93
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
84 changed files with 489 additions and 4958 deletions
Download patch file Download diff file Expand all files Collapse all files

201
docs/docs/building_applications/telemetry.mdx
View file

@ -10,203 +10,34 @@ import TabItem from '@theme/TabItem';
# Telemetry
The Llama Stack uses OpenTelemetry to provide comprehensive tracing, metrics, and logging capabilities.
The preferred way to instrument Llama Stack is with OpenTelemetry. Llama Stack enriches the data
collected by OpenTelemetry to capture helpful information about the performance and behavior of your
application. Here is an example of how to forward your telemetry to an OTLP collector from Llama Stack:
```sh
export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:4318"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_SERVICE_NAME="llama-stack-server"
## Automatic Metrics Generation
uv pip install opentelemetry-distro opentelemetry-exporter-otlp
uv run opentelemetry-bootstrap -a requirements | uv pip install --requirement -
Llama Stack automatically generates metrics during inference operations. These metrics are aggregated at the **inference request level** and provide insights into token usage and model performance.
### Available Metrics
The following metrics are automatically generated for each inference request:
| Metric Name | Type | Unit | Description | Labels |
|-------------|------|------|-------------|--------|
| `llama_stack_prompt_tokens_total` | Counter | `tokens` | Number of tokens in the input prompt | `model_id`, `provider_id` |
| `llama_stack_completion_tokens_total` | Counter | `tokens` | Number of tokens in the generated response | `model_id`, `provider_id` |
| `llama_stack_tokens_total` | Counter | `tokens` | Total tokens used (prompt + completion) | `model_id`, `provider_id` |
### Metric Generation Flow
1. **Token Counting**: During inference operations (chat completion, completion, etc.), the system counts tokens in both input prompts and generated responses
2. **Metric Construction**: For each request, `MetricEvent` objects are created with the token counts
3. **Telemetry Logging**: Metrics are sent to the configured telemetry sinks
4. **OpenTelemetry Export**: When OpenTelemetry is enabled, metrics are exposed as standard OpenTelemetry counters
### Metric Aggregation Level
All metrics are generated and aggregated at the **inference request level**. This means:
- Each individual inference request generates its own set of metrics
- Metrics are not pre-aggregated across multiple requests
- Aggregation (sums, averages, etc.) can be performed by your observability tools (Prometheus, Grafana, etc.)
- Each metric includes labels for `model_id` and `provider_id` to enable filtering and grouping
### Example Metric Event
```python
MetricEvent(
trace_id="1234567890abcdef",
span_id="abcdef1234567890",
metric="total_tokens",
value=150,
timestamp=1703123456.789,
unit="tokens",
attributes={
"model_id": "meta-llama/Llama-3.2-3B-Instruct",
"provider_id": "tgi"
},
)
uv run opentelemetry-instrument llama stack run run.yaml
```
## Telemetry Sinks
Choose from multiple sink types based on your observability needs:
### Known issues
<Tabs>
<TabItem value="opentelemetry" label="OpenTelemetry">
Some database instrumentation libraries have a known bug where spans get wrapped twice, or do not get connected to a trace.
To prevent this, you can disable database specific tracing, and rely just on the SQLAlchemy tracing. If you are using
`sqlite3` as your database, for example, you can disable the additional tracing like this:
Send events to an OpenTelemetry Collector for integration with observability platforms:
**Use Cases:**
- Visualizing traces in tools like Jaeger
- Collecting metrics for Prometheus
- Integration with enterprise observability stacks
**Features:**
- Standard OpenTelemetry format
- Compatible with all OpenTelemetry collectors
- Supports both traces and metrics
</TabItem>
<TabItem value="console" label="Console">
Print events to the console for immediate debugging:
**Use Cases:**
- Development and testing
- Quick debugging sessions
- Simple logging without external tools
**Features:**
- Immediate output visibility
- No setup required
- Human-readable format
</TabItem>
</Tabs>
## Configuration
### Meta-Reference Provider
Currently, only the meta-reference provider is implemented. It can be configured to send events to multiple sink types:
```yaml
telemetry:
- provider_id: meta-reference
provider_type: inline::meta-reference
config:
service_name: "llama-stack-service"
sinks: ['console', 'otel_trace', 'otel_metric']
otel_exporter_otlp_endpoint: "http://localhost:4318"
```sh
export OTEL_PYTHON_DISABLED_INSTRUMENTATIONS="sqlite3"
```
### Environment Variables
Configure telemetry behavior using environment variables:
- **`OTEL_EXPORTER_OTLP_ENDPOINT`**: OpenTelemetry Collector endpoint (default: `http://localhost:4318`)
- **`OTEL_SERVICE_NAME`**: Service name for telemetry (default: empty string)
- **`TELEMETRY_SINKS`**: Comma-separated list of sinks (default: `[]`)
### Quick Setup: Complete Telemetry Stack
Use the automated setup script to launch the complete telemetry stack (Jaeger, OpenTelemetry Collector, Prometheus, and Grafana):
```bash
./scripts/telemetry/setup_telemetry.sh
```
This sets up:
- **Jaeger UI**: http://localhost:16686 (traces visualization)
- **Prometheus**: http://localhost:9090 (metrics)
- **Grafana**: http://localhost:3000 (dashboards with auto-configured data sources)
- **OTEL Collector**: http://localhost:4318 (OTLP endpoint)
Once running, you can visualize traces by navigating to [Grafana](http://localhost:3000/) and login with login `admin` and password `admin`.
## Querying Metrics
When using the OpenTelemetry sink, metrics are exposed in standard format and can be queried through various tools:
<Tabs>
<TabItem value="prometheus" label="Prometheus Queries">
Example Prometheus queries for analyzing token usage:
```promql
# Total tokens used across all models
sum(llama_stack_tokens_total)
# Tokens per model
sum by (model_id) (llama_stack_tokens_total)
# Average tokens per request over 5 minutes
rate(llama_stack_tokens_total[5m])
# Token usage by provider
sum by (provider_id) (llama_stack_tokens_total)
```
</TabItem>
<TabItem value="grafana" label="Grafana Dashboards">
Create dashboards using Prometheus as a data source:
- **Token Usage Over Time**: Line charts showing token consumption trends
- **Model Performance**: Comparison of different models by token efficiency
- **Provider Analysis**: Breakdown of usage across different providers
- **Request Patterns**: Understanding peak usage times and patterns
</TabItem>
<TabItem value="otlp" label="OpenTelemetry Collector">
Forward metrics to other observability systems:
- Export to multiple backends simultaneously
- Apply transformations and filtering
- Integrate with existing monitoring infrastructure
</TabItem>
</Tabs>
## Best Practices
### 🔍 **Monitoring Strategy**
- Use OpenTelemetry for production environments
- Set up alerts on key metrics like token usage and error rates
### 📊 **Metrics Analysis**
- Track token usage trends to optimize costs
- Monitor response times across different models
- Analyze usage patterns to improve resource allocation
### 🚨 **Alerting & Debugging**
- Set up alerts for unusual token consumption spikes
- Use trace data to debug performance issues
- Monitor error rates and failure patterns
### 🔧 **Configuration Management**
- Use environment variables for flexible deployment
- Ensure proper network access to OpenTelemetry collectors
## Related Resources
- **[Agents](./agent)** - Monitoring agent execution with telemetry
- **[Evaluations](./evals)** - Using telemetry data for performance evaluation
- **[Getting Started Notebook](https://github.com/meta-llama/llama-stack/blob/main/docs/getting_started.ipynb)** - Telemetry examples and queries
- **[OpenTelemetry Documentation](https://opentelemetry.io/)** - Comprehensive observability framework
- **[Jaeger Documentation](https://www.jaegertracing.io/)** - Distributed tracing visualization

1
docs/docs/concepts/apis/index.mdx
View file

@ -17,7 +17,6 @@ A Llama Stack API is described as a collection of REST endpoints following OpenA
- **Eval**: generate outputs (via Inference or Agents) and perform scoring
- **VectorIO**: perform operations on vector stores, such as adding documents, searching, and deleting documents
- **Files**: manage file uploads, storage, and retrieval
- **Telemetry**: collect telemetry data from the system
- **Post Training**: fine-tune a model
- **Tool Runtime**: interact with various tools and protocols
- **Responses**: generate responses from an LLM

7
docs/docs/distributions/k8s/stack-configmap.yaml
View file

@ -8,7 +8,6 @@ data:
- inference
- files
- safety
- telemetry
- tool_runtime
- vector_io
providers:
@ -73,12 +72,6 @@ data:
db: ${env.POSTGRES_DB:=llamastack}
user: ${env.POSTGRES_USER:=llamastack}
password: ${env.POSTGRES_PASSWORD:=llamastack}
telemetry:
- provider_id: meta-reference
provider_type: inline::meta-reference
config:
service_name: "${env.OTEL_SERVICE_NAME:=\u200B}"
sinks: ${env.TELEMETRY_SINKS:=console}
tool_runtime:
- provider_id: brave-search
provider_type: remote::brave-search

2
docs/docs/distributions/k8s/stack_run_config.yaml
View file

@ -140,8 +140,6 @@ server:
auth:
provider_config:
type: github_token
telemetry:
enabled: true
vector_stores:
default_provider_id: chromadb
default_embedding_model:

6
docs/docs/distributions/self_hosted_distro/starter.md
View file

@ -116,10 +116,6 @@ The following environment variables can be configured:
- `BRAVE_SEARCH_API_KEY`: Brave Search API key
- `TAVILY_SEARCH_API_KEY`: Tavily Search API key
### Telemetry Configuration
- `OTEL_SERVICE_NAME`: OpenTelemetry service name
- `OTEL_EXPORTER_OTLP_ENDPOINT`: OpenTelemetry collector endpoint URL
## Enabling Providers
You can enable specific providers by setting appropriate environment variables. For example,
@ -265,7 +261,7 @@ The starter distribution uses SQLite for local storage of various components:
2. **Flexible Configuration**: Easy to enable/disable providers based on your needs
3. **No Local GPU Required**: Most providers are cloud-based, making it accessible to developers without high-end hardware
4. **Easy Migration**: Start with hosted providers and gradually move to local ones as needed
5. **Production Ready**: Includes safety, evaluation, and telemetry components
5. **Production Ready**: Includes safety and evaluation
6. **Tool Integration**: Comes with web search, RAG, and model context protocol tools
The starter distribution is ideal for developers who want to experiment with different AI providers, build prototypes quickly, or create applications that can work with multiple AI backends.

2
docs/docs/distributions/starting_llama_stack_server.mdx
View file

@ -27,7 +27,7 @@ If you have built a container image and want to deploy it in a Kubernetes cluste
Control log output via environment variables before starting the server.
- `LLAMA_STACK_LOGGING` sets per-component levels, e.g. `LLAMA_STACK_LOGGING=server=debug;core=info`.
- `LLAMA_STACK_LOGGING` sets per-component levels, e.g. `LLAMA_STACK_LOGGING=server=debug,core=info`.
- Supported categories: `all`, `core`, `server`, `router`, `inference`, `agents`, `safety`, `eval`, `tools`, `client`.
- Levels: `debug`, `info`, `warning`, `error`, `critical` (default is `info`). Use `all=<level>` to apply globally.
- `LLAMA_STACK_LOG_FILE=/path/to/log` mirrors logs to a file while still printing to stdout.

26
docs/docs/references/python_sdk_reference/index.md
View file

@ -360,32 +360,6 @@ Methods:
- <code title="post /v1/synthetic-data-generation/generate">client.synthetic_data_generation.<a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/resources/synthetic_data_generation.py">generate</a>(\*\*<a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/types/synthetic_data_generation_generate_params.py">params</a>) -> <a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/types/synthetic_data_generation_response.py">SyntheticDataGenerationResponse</a></code>
## Telemetry
Types:
```python
from llama_stack_client.types import (
QuerySpansResponse,
SpanWithStatus,
Trace,
TelemetryGetSpanResponse,
TelemetryGetSpanTreeResponse,
TelemetryQuerySpansResponse,
TelemetryQueryTracesResponse,
)
```
Methods:
- <code title="get /v1/telemetry/traces/{trace_id}/spans/{span_id}">client.telemetry.<a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/resources/telemetry.py">get_span</a>(span_id, \*, trace_id) -> <a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/types/telemetry_get_span_response.py">TelemetryGetSpanResponse</a></code>
- <code title="get /v1/telemetry/spans/{span_id}/tree">client.telemetry.<a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/resources/telemetry.py">get_span_tree</a>(span_id, \*\*<a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/types/telemetry_get_span_tree_params.py">params</a>) -> <a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/types/telemetry_get_span_tree_response.py">TelemetryGetSpanTreeResponse</a></code>
- <code title="get /v1/telemetry/traces/{trace_id}">client.telemetry.<a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/resources/telemetry.py">get_trace</a>(trace_id) -> <a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/types/trace.py">Trace</a></code>
- <code title="post /v1/telemetry/events">client.telemetry.<a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/resources/telemetry.py">log_event</a>(\*\*<a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/types/telemetry_log_event_params.py">params</a>) -> None</code>
- <code title="get /v1/telemetry/spans">client.telemetry.<a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/resources/telemetry.py">query_spans</a>(\*\*<a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/types/telemetry_query_spans_params.py">params</a>) -> <a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/types/telemetry_query_spans_response.py">TelemetryQuerySpansResponse</a></code>
- <code title="get /v1/telemetry/traces">client.telemetry.<a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/resources/telemetry.py">query_traces</a>(\*\*<a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/types/telemetry_query_traces_params.py">params</a>) -> <a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/types/telemetry_query_traces_response.py">TelemetryQueryTracesResponse</a></code>
- <code title="post /v1/telemetry/spans/export">client.telemetry.<a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/resources/telemetry.py">save_spans_to_dataset</a>(\*\*<a href="https://github.com/meta-llama/llama-stack-client-python/tree/main/src/llama_stack_client/types/telemetry_save_spans_to_dataset_params.py">params</a>) -> None</code>
## Datasetio
Types:

4
docs/src/pages/index.js
View file

@ -13,7 +13,7 @@ function HomepageHeader() {
<div className={styles.heroContent}>
<h1 className={styles.heroTitle}>Build AI Applications with Llama Stack</h1>
<p className={styles.heroSubtitle}>
Unified APIs for Inference, RAG, Agents, Tools, Safety, and Telemetry
Unified APIs for Inference, RAG, Agents, Tools, and Safety
</p>
<div className={styles.buttons}>
<Link
@ -206,7 +206,7 @@ export default function Home() {
return (
<Layout
title="Build AI Applications"
description="The open-source framework for building generative AI applications with unified APIs for Inference, RAG, Agents, Tools, Safety, and Telemetry.">
description="The open-source framework for building generative AI applications with unified APIs for Inference, RAG, Agents, Tools, Safety, and Evals.">
<HomepageHeader />
<main>
<QuickStart />

451
docs/static/deprecated-llama-stack-spec.yaml vendored
View file

@ -9041,227 +9041,6 @@ components:
- $ref: '#/components/schemas/OpenAIResponseContentPartReasoningText'
title: OpenAIResponseContentPartReasoningText
title: OpenAIResponseContentPartOutputText | OpenAIResponseContentPartRefusal | OpenAIResponseContentPartReasoningText
SpanEndPayload:
description: Payload for a span end event.
properties:
type:
const: span_end
default: span_end
title: Type
type: string
status:
$ref: '#/components/schemas/SpanStatus'
required:
- status
title: SpanEndPayload
type: object
SpanStartPayload:
description: Payload for a span start event.
properties:
type:
const: span_start
default: span_start
title: Type
type: string
name:
title: Name
type: string
parent_span_id:
anyOf:
- type: string
- type: 'null'
nullable: true
required:
- name
title: SpanStartPayload
type: object
SpanStatus:
description: The status of a span indicating whether it completed successfully or with an error.
enum:
- ok
- error
title: SpanStatus
type: string
StructuredLogPayload:
discriminator:
mapping:
span_end: '#/components/schemas/SpanEndPayload'
span_start: '#/components/schemas/SpanStartPayload'
propertyName: type
oneOf:
- $ref: '#/components/schemas/SpanStartPayload'
title: SpanStartPayload
- $ref: '#/components/schemas/SpanEndPayload'
title: SpanEndPayload
title: SpanStartPayload | SpanEndPayload
LogSeverity:
description: The severity level of a log message.
enum:
- verbose
- debug
- info
- warn
- error
- critical
title: LogSeverity
type: string
MetricEvent:
description: A metric event containing a measured value.
properties:
trace_id:
title: Trace Id
type: string
span_id:
title: Span Id
type: string
timestamp:
format: date-time
title: Timestamp
type: string
attributes:
anyOf:
- additionalProperties:
anyOf:
- type: string
- type: integer
- type: number
- type: boolean
- type: 'null'
title: string | ... (4 variants)
type: object
- type: 'null'
type:
const: metric
default: metric
title: Type
type: string
metric:
title: Metric
type: string
value:
anyOf:
- type: integer
- type: number
title: integer | number
unit:
title: Unit
type: string
required:
- trace_id
- span_id
- timestamp
- metric
- value
- unit
title: MetricEvent
type: object
StructuredLogEvent:
description: A structured log event containing typed payload data.
properties:
trace_id:
title: Trace Id
type: string
span_id:
title: Span Id
type: string
timestamp:
format: date-time
title: Timestamp
type: string
attributes:
anyOf:
- additionalProperties:
anyOf:
- type: string
- type: integer
- type: number
- type: boolean
- type: 'null'
title: string | ... (4 variants)
type: object
- type: 'null'
type:
const: structured_log
default: structured_log
title: Type
type: string
payload:
discriminator:
mapping:
span_end: '#/components/schemas/SpanEndPayload'
span_start: '#/components/schemas/SpanStartPayload'
propertyName: type
oneOf:
- $ref: '#/components/schemas/SpanStartPayload'
title: SpanStartPayload
- $ref: '#/components/schemas/SpanEndPayload'
title: SpanEndPayload
title: SpanStartPayload | SpanEndPayload
required:
- trace_id
- span_id
- timestamp
- payload
title: StructuredLogEvent
type: object
UnstructuredLogEvent:
description: An unstructured log event containing a simple text message.
properties:
trace_id:
title: Trace Id
type: string
span_id:
title: Span Id
type: string
timestamp:
format: date-time
title: Timestamp
type: string
attributes:
anyOf:
- additionalProperties:
anyOf:
- type: string
- type: integer
- type: number
- type: boolean
- type: 'null'
title: string | ... (4 variants)
type: object
- type: 'null'
type:
const: unstructured_log
default: unstructured_log
title: Type
type: string
message:
title: Message
type: string
severity:
$ref: '#/components/schemas/LogSeverity'
required:
- trace_id
- span_id
- timestamp
- message
- severity
title: UnstructuredLogEvent
type: object
Event:
discriminator:
mapping:
metric: '#/components/schemas/MetricEvent'
structured_log: '#/components/schemas/StructuredLogEvent'
unstructured_log: '#/components/schemas/UnstructuredLogEvent'
propertyName: type
oneOf:
- $ref: '#/components/schemas/UnstructuredLogEvent'
title: UnstructuredLogEvent
- $ref: '#/components/schemas/MetricEvent'
title: MetricEvent
- $ref: '#/components/schemas/StructuredLogEvent'
title: StructuredLogEvent
title: UnstructuredLogEvent | MetricEvent | StructuredLogEvent
MetricInResponse:
description: A metric value included in API responses.
properties:
@ -10086,236 +9865,6 @@ components:
- logger_config
title: PostTrainingRLHFRequest
type: object
Span:
description: A span representing a single operation within a trace.
properties:
span_id:
title: Span Id
type: string
trace_id:
title: Trace Id
type: string
parent_span_id:
anyOf:
- type: string
- type: 'null'
nullable: true
name:
title: Name
type: string
start_time:
format: date-time
title: Start Time
type: string
end_time:
anyOf:
- format: date-time
type: string
- type: 'null'
nullable: true
attributes:
anyOf:
- additionalProperties: true
type: object
- type: 'null'
required:
- span_id
- trace_id
- name
- start_time
title: Span
type: object
Trace:
description: A trace representing the complete execution path of a request across multiple operations.
properties:
trace_id:
title: Trace Id
type: string
root_span_id:
title: Root Span Id
type: string
start_time:
format: date-time
title: Start Time
type: string
end_time:
anyOf:
- format: date-time
type: string
- type: 'null'
nullable: true
required:
- trace_id
- root_span_id
- start_time
title: Trace
type: object
EventType:
description: The type of telemetry event being logged.
enum:
- unstructured_log
- structured_log
- metric
title: EventType
type: string
StructuredLogType:
description: The type of structured log event payload.
enum:
- span_start
- span_end
title: StructuredLogType
type: string
EvalTrace:
description: A trace record for evaluation purposes.
properties:
session_id:
title: Session Id
type: string
step:
title: Step
type: string
input:
title: Input
type: string
output:
title: Output
type: string
expected_output:
title: Expected Output
type: string
required:
- session_id
- step
- input
- output
- expected_output
title: EvalTrace
type: object
SpanWithStatus:
description: A span that includes status information.
properties:
span_id:
title: Span Id
type: string
trace_id:
title: Trace Id
type: string
parent_span_id:
anyOf:
- type: string
- type: 'null'
nullable: true
name:
title: Name
type: string
start_time:
format: date-time
title: Start Time
type: string
end_time:
anyOf:
- format: date-time
type: string
- type: 'null'
nullable: true
attributes:
anyOf:
- additionalProperties: true
type: object
- type: 'null'
status:
anyOf:
- $ref: '#/components/schemas/SpanStatus'
title: SpanStatus
- type: 'null'
nullable: true
title: SpanStatus
required:
- span_id
- trace_id
- name
- start_time
title: SpanWithStatus
type: object
QueryConditionOp:
description: Comparison operators for query conditions.
enum:
- eq
- ne
- gt
- lt
title: QueryConditionOp
type: string
QueryCondition:
description: A condition for filtering query results.
properties:
key:
title: Key
type: string
op:
$ref: '#/components/schemas/QueryConditionOp'
value:
title: Value
required:
- key
- op
- value
title: QueryCondition
type: object
MetricLabel:
description: A label associated with a metric.
properties:
name:
title: Name
type: string
value:
title: Value
type: string
required:
- name
- value
title: MetricLabel
type: object
MetricDataPoint:
description: A single data point in a metric time series.
properties:
timestamp:
title: Timestamp
type: integer
value:
title: Value
type: number
unit:
title: Unit
type: string
required:
- timestamp
- value
- unit
title: MetricDataPoint
type: object
MetricSeries:
description: A time series of metric data points.
properties:
metric:
title: Metric
type: string
labels:
items:
$ref: '#/components/schemas/MetricLabel'
title: Labels
type: array
values:
items:
$ref: '#/components/schemas/MetricDataPoint'
title: Values
type: array
required:
- metric
- labels
- values
title: MetricSeries
type: object
responses:
BadRequest400:
description: The request was invalid or malformed

451
docs/static/experimental-llama-stack-spec.yaml vendored
View file

@ -7964,227 +7964,6 @@ components:
- $ref: '#/components/schemas/OpenAIResponseContentPartReasoningText'
title: OpenAIResponseContentPartReasoningText
title: OpenAIResponseContentPartOutputText | OpenAIResponseContentPartRefusal | OpenAIResponseContentPartReasoningText
SpanEndPayload:
description: Payload for a span end event.
properties:
type:
const: span_end
default: span_end
title: Type
type: string
status:
$ref: '#/components/schemas/SpanStatus'
required:
- status
title: SpanEndPayload
type: object
SpanStartPayload:
description: Payload for a span start event.
properties:
type:
const: span_start
default: span_start
title: Type
type: string
name:
title: Name
type: string
parent_span_id:
anyOf:
- type: string
- type: 'null'
nullable: true
required:
- name
title: SpanStartPayload
type: object
SpanStatus:
description: The status of a span indicating whether it completed successfully or with an error.
enum:
- ok
- error
title: SpanStatus
type: string
StructuredLogPayload:
discriminator:
mapping:
span_end: '#/components/schemas/SpanEndPayload'
span_start: '#/components/schemas/SpanStartPayload'
propertyName: type
oneOf:
- $ref: '#/components/schemas/SpanStartPayload'
title: SpanStartPayload
- $ref: '#/components/schemas/SpanEndPayload'
title: SpanEndPayload
title: SpanStartPayload | SpanEndPayload
LogSeverity:
description: The severity level of a log message.
enum:
- verbose
- debug
- info
- warn
- error
- critical
title: LogSeverity
type: string
MetricEvent:
description: A metric event containing a measured value.
properties:
trace_id:
title: Trace Id
type: string
span_id:
title: Span Id
type: string
timestamp:
format: date-time
title: Timestamp
type: string
attributes:
anyOf:
- additionalProperties:
anyOf:
- type: string
- type: integer
- type: number
- type: boolean
- type: 'null'
title: string | ... (4 variants)
type: object
- type: 'null'
type:
const: metric
default: metric
title: Type
type: string
metric:
title: Metric
type: string
value:
anyOf:
- type: integer
- type: number
title: integer | number
unit:
title: Unit
type: string
required:
- trace_id
- span_id
- timestamp
- metric
- value
- unit
title: MetricEvent
type: object
StructuredLogEvent:
description: A structured log event containing typed payload data.
properties:
trace_id:
title: Trace Id
type: string
span_id:
title: Span Id
type: string
timestamp:
format: date-time
title: Timestamp
type: string
attributes:
anyOf:
- additionalProperties:
anyOf:
- type: string
- type: integer
- type: number
- type: boolean
- type: 'null'
title: string | ... (4 variants)
type: object
- type: 'null'
type:
const: structured_log
default: structured_log
title: Type
type: string
payload:
discriminator:
mapping:
span_end: '#/components/schemas/SpanEndPayload'
span_start: '#/components/schemas/SpanStartPayload'
propertyName: type
oneOf:
- $ref: '#/components/schemas/SpanStartPayload'
title: SpanStartPayload
- $ref: '#/components/schemas/SpanEndPayload'
title: SpanEndPayload
title: SpanStartPayload | SpanEndPayload
required:
- trace_id
- span_id
- timestamp
- payload
title: StructuredLogEvent
type: object
UnstructuredLogEvent:
description: An unstructured log event containing a simple text message.
properties:
trace_id:
title: Trace Id
type: string
span_id:
title: Span Id
type: string
timestamp:
format: date-time
title: Timestamp
type: string
attributes:
anyOf:
- additionalProperties:
anyOf:
- type: string
- type: integer
- type: number
- type: boolean
- type: 'null'
title: string | ... (4 variants)
type: object
- type: 'null'
type:
const: unstructured_log
default: unstructured_log
title: Type
type: string
message:
title: Message
type: string
severity:
$ref: '#/components/schemas/LogSeverity'
required:
- trace_id
- span_id
- timestamp
- message
- severity
title: UnstructuredLogEvent
type: object
Event:
discriminator:
mapping:
metric: '#/components/schemas/MetricEvent'
structured_log: '#/components/schemas/StructuredLogEvent'
unstructured_log: '#/components/schemas/UnstructuredLogEvent'
propertyName: type
oneOf:
- $ref: '#/components/schemas/UnstructuredLogEvent'
title: UnstructuredLogEvent
- $ref: '#/components/schemas/MetricEvent'
title: MetricEvent
- $ref: '#/components/schemas/StructuredLogEvent'
title: StructuredLogEvent
title: UnstructuredLogEvent | MetricEvent | StructuredLogEvent
MetricInResponse:
description: A metric value included in API responses.
properties:
@ -9009,236 +8788,6 @@ components:
- logger_config
title: PostTrainingRLHFRequest
type: object
Span:
description: A span representing a single operation within a trace.
properties:
span_id:
title: Span Id
type: string
trace_id:
title: Trace Id
type: string
parent_span_id:
anyOf:
- type: string
- type: 'null'
nullable: true
name:
title: Name
type: string
start_time:
format: date-time
title: Start Time
type: string
end_time:
anyOf:
- format: date-time
type: string
- type: 'null'
nullable: true
attributes:
anyOf:
- additionalProperties: true
type: object
- type: 'null'
required:
- span_id
- trace_id
- name
- start_time
title: Span
type: object
Trace:
description: A trace representing the complete execution path of a request across multiple operations.
properties:
trace_id:
title: Trace Id
type: string
root_span_id:
title: Root Span Id
type: string
start_time:
format: date-time
title: Start Time
type: string
end_time:
anyOf:
- format: date-time
type: string
- type: 'null'
nullable: true
required:
- trace_id
- root_span_id
- start_time
title: Trace
type: object
EventType:
description: The type of telemetry event being logged.
enum:
- unstructured_log
- structured_log
- metric
title: EventType
type: string
StructuredLogType:
description: The type of structured log event payload.
enum:
- span_start
- span_end
title: StructuredLogType
type: string
EvalTrace:
description: A trace record for evaluation purposes.
properties:
session_id:
title: Session Id
type: string
step:
title: Step
type: string
input:
title: Input
type: string
output:
title: Output
type: string
expected_output:
title: Expected Output
type: string
required:
- session_id
- step
- input
- output
- expected_output
title: EvalTrace
type: object
SpanWithStatus:
description: A span that includes status information.
properties:
span_id:
title: Span Id
type: string
trace_id:
title: Trace Id
type: string
parent_span_id:
anyOf:
- type: string
- type: 'null'
nullable: true
name:
title: Name
type: string
start_time:
format: date-time
title: Start Time
type: string
end_time:
anyOf:
- format: date-time
type: string
- type: 'null'
nullable: true
attributes:
anyOf:
- additionalProperties: true
type: object
- type: 'null'
status:
anyOf:
- $ref: '#/components/schemas/SpanStatus'
title: SpanStatus
- type: 'null'
nullable: true
title: SpanStatus
required:
- span_id
- trace_id
- name
- start_time
title: SpanWithStatus
type: object
QueryConditionOp:
description: Comparison operators for query conditions.
enum:
- eq
- ne
- gt
- lt
title: QueryConditionOp
type: string
QueryCondition:
description: A condition for filtering query results.
properties:
key:
title: Key
type: string
op:
$ref: '#/components/schemas/QueryConditionOp'
value:
title: Value
required:
- key
- op
- value
title: QueryCondition
type: object
MetricLabel:
description: A label associated with a metric.
properties:
name:
title: Name
type: string
value:
title: Value
type: string
required:
- name
- value
title: MetricLabel
type: object
MetricDataPoint:
description: A single data point in a metric time series.
properties:
timestamp:
title: Timestamp
type: integer
value:
title: Value
type: number
unit:
title: Unit
type: string
required:
- timestamp
- value
- unit
title: MetricDataPoint
type: object
MetricSeries:
description: A time series of metric data points.
properties:
metric:
title: Metric
type: string
labels:
items:
$ref: '#/components/schemas/MetricLabel'
title: Labels
type: array
values:
items:
$ref: '#/components/schemas/MetricDataPoint'
title: Values
type: array
required:
- metric
- labels
- values
title: MetricSeries
type: object
responses:
BadRequest400:
description: The request was invalid or malformed

451
docs/static/llama-stack-spec.yaml vendored
View file

@ -10868,227 +10868,6 @@ components:
- $ref: '#/components/schemas/OpenAIResponseContentPartReasoningText'
title: OpenAIResponseContentPartReasoningText
title: OpenAIResponseContentPartOutputText | OpenAIResponseContentPartRefusal | OpenAIResponseContentPartReasoningText
SpanEndPayload:
description: Payload for a span end event.
properties:
type:
const: span_end
default: span_end
title: Type
type: string
status:
$ref: '#/components/schemas/SpanStatus'
required:
- status
title: SpanEndPayload
type: object
SpanStartPayload:
description: Payload for a span start event.
properties:
type:
const: span_start
default: span_start
title: Type
type: string
name:
title: Name
type: string
parent_span_id:
anyOf:
- type: string
- type: 'null'
nullable: true
required:
- name
title: SpanStartPayload
type: object
SpanStatus:
description: The status of a span indicating whether it completed successfully or with an error.
enum:
- ok
- error
title: SpanStatus
type: string
StructuredLogPayload:
discriminator:
mapping:
span_end: '#/components/schemas/SpanEndPayload'
span_start: '#/components/schemas/SpanStartPayload'
propertyName: type
oneOf:
- $ref: '#/components/schemas/SpanStartPayload'
title: SpanStartPayload
- $ref: '#/components/schemas/SpanEndPayload'
title: SpanEndPayload
title: SpanStartPayload | SpanEndPayload
LogSeverity:
description: The severity level of a log message.
enum:
- verbose
- debug
- info
- warn
- error
- critical
title: LogSeverity
type: string
MetricEvent:
description: A metric event containing a measured value.
properties:
trace_id:
title: Trace Id
type: string
span_id:
title: Span Id
type: string
timestamp:
format: date-time
title: Timestamp
type: string
attributes:
anyOf:
- additionalProperties:
anyOf:
- type: string
- type: integer
- type: number
- type: boolean
- type: 'null'
title: string | ... (4 variants)
type: object
- type: 'null'
type:
const: metric
default: metric
title: Type
type: string
metric:
title: Metric
type: string
value:
anyOf:
- type: integer
- type: number
title: integer | number
unit:
title: Unit
type: string
required:
- trace_id
- span_id
- timestamp
- metric
- value
- unit
title: MetricEvent
type: object
StructuredLogEvent:
description: A structured log event containing typed payload data.
properties:
trace_id:
title: Trace Id
type: string
span_id:
title: Span Id
type: string
timestamp:
format: date-time
title: Timestamp
type: string
attributes:
anyOf:
- additionalProperties:
anyOf:
- type: string
- type: integer
- type: number
- type: boolean
- type: 'null'
title: string | ... (4 variants)
type: object
- type: 'null'
type:
const: structured_log
default: structured_log
title: Type
type: string
payload:
discriminator:
mapping:
span_end: '#/components/schemas/SpanEndPayload'
span_start: '#/components/schemas/SpanStartPayload'
propertyName: type
oneOf:
- $ref: '#/components/schemas/SpanStartPayload'
title: SpanStartPayload
- $ref: '#/components/schemas/SpanEndPayload'
title: SpanEndPayload
title: SpanStartPayload | SpanEndPayload
required:
- trace_id
- span_id
- timestamp
- payload
title: StructuredLogEvent
type: object
UnstructuredLogEvent:
description: An unstructured log event containing a simple text message.
properties:
trace_id:
title: Trace Id
type: string
span_id:
title: Span Id
type: string
timestamp:
format: date-time
title: Timestamp
type: string
attributes:
anyOf:
- additionalProperties:
anyOf:
- type: string
- type: integer
- type: number
- type: boolean
- type: 'null'
title: string | ... (4 variants)
type: object
- type: 'null'
type:
const: unstructured_log
default: unstructured_log
title: Type
type: string
message:
title: Message
type: string
severity:
$ref: '#/components/schemas/LogSeverity'
required:
- trace_id
- span_id
- timestamp
- message
- severity
title: UnstructuredLogEvent
type: object
Event:
discriminator:
mapping:
metric: '#/components/schemas/MetricEvent'
structured_log: '#/components/schemas/StructuredLogEvent'
unstructured_log: '#/components/schemas/UnstructuredLogEvent'
propertyName: type
oneOf:
- $ref: '#/components/schemas/UnstructuredLogEvent'
title: UnstructuredLogEvent
- $ref: '#/components/schemas/MetricEvent'
title: MetricEvent
- $ref: '#/components/schemas/StructuredLogEvent'
title: StructuredLogEvent
title: UnstructuredLogEvent | MetricEvent | StructuredLogEvent
MetricInResponse:
description: A metric value included in API responses.
properties:
@ -11910,236 +11689,6 @@ components:
- logger_config
title: PostTrainingRLHFRequest
type: object
Span:
description: A span representing a single operation within a trace.
properties:
span_id:
title: Span Id
type: string
trace_id:
title: Trace Id
type: string
parent_span_id:
anyOf:
- type: string
- type: 'null'
nullable: true
name:
title: Name
type: string
start_time:
format: date-time
title: Start Time
type: string
end_time:
anyOf:
- format: date-time
type: string
- type: 'null'
nullable: true
attributes:
anyOf:
- additionalProperties: true
type: object
- type: 'null'
required:
- span_id
- trace_id
- name
- start_time
title: Span
type: object
Trace:
description: A trace representing the complete execution path of a request across multiple operations.
properties:
trace_id:
title: Trace Id
type: string
root_span_id:
title: Root Span Id
type: string
start_time:
format: date-time
title: Start Time
type: string
end_time:
anyOf:
- format: date-time
type: string
- type: 'null'
nullable: true
required:
- trace_id
- root_span_id
- start_time
title: Trace
type: object
EventType:
description: The type of telemetry event being logged.
enum:
- unstructured_log
- structured_log
- metric
title: EventType
type: string
StructuredLogType:
description: The type of structured log event payload.
enum:
- span_start
- span_end
title: StructuredLogType
type: string
EvalTrace:
description: A trace record for evaluation purposes.
properties:
session_id:
title: Session Id
type: string
step:
title: Step
type: string
input:
title: Input
type: string
output:
title: Output
type: string
expected_output:
title: Expected Output
type: string
required:
- session_id
- step
- input
- output
- expected_output
title: EvalTrace
type: object
SpanWithStatus:
description: A span that includes status information.
properties:
span_id:
title: Span Id
type: string
trace_id:
title: Trace Id
type: string
parent_span_id:
anyOf:
- type: string
- type: 'null'
nullable: true
name:
title: Name
type: string
start_time:
format: date-time
title: Start Time
type: string
end_time:
anyOf:
- format: date-time
type: string
- type: 'null'
nullable: true
attributes:
anyOf:
- additionalProperties: true
type: object
- type: 'null'
status:
anyOf:
- $ref: '#/components/schemas/SpanStatus'
title: SpanStatus
- type: 'null'
nullable: true
title: SpanStatus
required:
- span_id
- trace_id
- name
- start_time
title: SpanWithStatus
type: object
QueryConditionOp:
description: Comparison operators for query conditions.
enum:
- eq
- ne
- gt
- lt
title: QueryConditionOp
type: string
QueryCondition:
description: A condition for filtering query results.
properties:
key:
title: Key
type: string
op:
$ref: '#/components/schemas/QueryConditionOp'
value:
title: Value
required:
- key
- op
- value
title: QueryCondition
type: object
MetricLabel:
description: A label associated with a metric.
properties:
name:
title: Name
type: string
value:
title: Value
type: string
required:
- name
- value
title: MetricLabel
type: object
MetricDataPoint:
description: A single data point in a metric time series.
properties:
timestamp:
title: Timestamp
type: integer
value:
title: Value
type: number
unit:
title: Unit
type: string
required:
- timestamp
- value
- unit
title: MetricDataPoint
type: object
MetricSeries:
description: A time series of metric data points.
properties:
metric:
title: Metric
type: string
labels:
items:
$ref: '#/components/schemas/MetricLabel'
title: Labels
type: array
values:
items:
$ref: '#/components/schemas/MetricDataPoint'
title: Values
type: array
required:
- metric
- labels
- values
title: MetricSeries
type: object
responses:
BadRequest400:
description: The request was invalid or malformed

451
docs/static/stainless-llama-stack-spec.yaml vendored
View file

@ -12198,227 +12198,6 @@ components:
- $ref: '#/components/schemas/OpenAIResponseContentPartReasoningText'
title: OpenAIResponseContentPartReasoningText
title: OpenAIResponseContentPartOutputText | OpenAIResponseContentPartRefusal | OpenAIResponseContentPartReasoningText
SpanEndPayload:
description: Payload for a span end event.
properties:
type:
const: span_end
default: span_end
title: Type
type: string
status:
$ref: '#/components/schemas/SpanStatus'
required:
- status
title: SpanEndPayload
type: object
SpanStartPayload:
description: Payload for a span start event.
properties:
type:
const: span_start
default: span_start
title: Type
type: string
name:
title: Name
type: string
parent_span_id:
anyOf:
- type: string
- type: 'null'
nullable: true
required:
- name
title: SpanStartPayload
type: object
SpanStatus:
description: The status of a span indicating whether it completed successfully or with an error.
enum:
- ok
- error
title: SpanStatus
type: string
StructuredLogPayload:
discriminator:
mapping:
span_end: '#/components/schemas/SpanEndPayload'
span_start: '#/components/schemas/SpanStartPayload'
propertyName: type
oneOf:
- $ref: '#/components/schemas/SpanStartPayload'
title: SpanStartPayload
- $ref: '#/components/schemas/SpanEndPayload'
title: SpanEndPayload
title: SpanStartPayload | SpanEndPayload
LogSeverity:
description: The severity level of a log message.
enum:
- verbose
- debug
- info
- warn
- error
- critical
title: LogSeverity
type: string
MetricEvent:
description: A metric event containing a measured value.
properties:
trace_id:
title: Trace Id
type: string
span_id:
title: Span Id
type: string
timestamp:
format: date-time
title: Timestamp
type: string
attributes:
anyOf:
- additionalProperties:
anyOf:
- type: string
- type: integer
- type: number
- type: boolean
- type: 'null'
title: string | ... (4 variants)
type: object
- type: 'null'
type:
const: metric
default: metric
title: Type
type: string
metric:
title: Metric
type: string
value:
anyOf:
- type: integer
- type: number
title: integer | number
unit:
title: Unit
type: string
required:
- trace_id
- span_id
- timestamp
- metric
- value
- unit
title: MetricEvent
type: object
StructuredLogEvent:
description: A structured log event containing typed payload data.
properties:
trace_id:
title: Trace Id
type: string
span_id:
title: Span Id
type: string
timestamp:
format: date-time
title: Timestamp
type: string
attributes:
anyOf:
- additionalProperties:
anyOf:
- type: string
- type: integer
- type: number
- type: boolean
- type: 'null'
title: string | ... (4 variants)
type: object
- type: 'null'
type:
const: structured_log
default: structured_log
title: Type
type: string
payload:
discriminator:
mapping:
span_end: '#/components/schemas/SpanEndPayload'
span_start: '#/components/schemas/SpanStartPayload'
propertyName: type
oneOf:
- $ref: '#/components/schemas/SpanStartPayload'
title: SpanStartPayload
- $ref: '#/components/schemas/SpanEndPayload'
title: SpanEndPayload
title: SpanStartPayload | SpanEndPayload
required:
- trace_id
- span_id
- timestamp
- payload
title: StructuredLogEvent
type: object
UnstructuredLogEvent:
description: An unstructured log event containing a simple text message.
properties:
trace_id:
title: Trace Id
type: string
span_id:
title: Span Id
type: string
timestamp:
format: date-time
title: Timestamp
type: string
attributes:
anyOf:
- additionalProperties:
anyOf:
- type: string
- type: integer
- type: number
- type: boolean
- type: 'null'
title: string | ... (4 variants)
type: object
- type: 'null'
type:
const: unstructured_log
default: unstructured_log
title: Type
type: string
message:
title: Message
type: string
severity:
$ref: '#/components/schemas/LogSeverity'
required:
- trace_id
- span_id
- timestamp
- message
- severity
title: UnstructuredLogEvent
type: object
Event:
discriminator:
mapping:
metric: '#/components/schemas/MetricEvent'
structured_log: '#/components/schemas/StructuredLogEvent'
unstructured_log: '#/components/schemas/UnstructuredLogEvent'
propertyName: type
oneOf:
- $ref: '#/components/schemas/UnstructuredLogEvent'
title: UnstructuredLogEvent
- $ref: '#/components/schemas/MetricEvent'
title: MetricEvent
- $ref: '#/components/schemas/StructuredLogEvent'
title: StructuredLogEvent
title: UnstructuredLogEvent | MetricEvent | StructuredLogEvent
MetricInResponse:
description: A metric value included in API responses.
properties:
@ -13243,236 +13022,6 @@ components:
- logger_config
title: PostTrainingRLHFRequest
type: object
Span:
description: A span representing a single operation within a trace.
properties:
span_id:
title: Span Id
type: string
trace_id:
title: Trace Id
type: string
parent_span_id:
anyOf:
- type: string
- type: 'null'
nullable: true
name:
title: Name
type: string
start_time:
format: date-time
title: Start Time
type: string
end_time:
anyOf:
- format: date-time
type: string
- type: 'null'
nullable: true
attributes:
anyOf:
- additionalProperties: true
type: object
- type: 'null'
required:
- span_id
- trace_id
- name
- start_time
title: Span
type: object
Trace:
description: A trace representing the complete execution path of a request across multiple operations.
properties:
trace_id:
title: Trace Id
type: string
root_span_id:
title: Root Span Id
type: string
start_time:
format: date-time
title: Start Time
type: string
end_time:
anyOf:
- format: date-time
type: string
- type: 'null'
nullable: true
required:
- trace_id
- root_span_id
- start_time
title: Trace
type: object
EventType:
description: The type of telemetry event being logged.
enum:
- unstructured_log
- structured_log
- metric
title: EventType
type: string
StructuredLogType:
description: The type of structured log event payload.
enum:
- span_start
- span_end
title: StructuredLogType
type: string
EvalTrace:
description: A trace record for evaluation purposes.
properties:
session_id:
title: Session Id
type: string
step:
title: Step
type: string
input:
title: Input
type: string
output:
title: Output
type: string
expected_output:
title: Expected Output
type: string
required:
- session_id
- step
- input
- output
- expected_output
title: EvalTrace
type: object
SpanWithStatus:
description: A span that includes status information.
properties:
span_id:
title: Span Id
type: string
trace_id:
title: Trace Id
type: string
parent_span_id:
anyOf:
- type: string
- type: 'null'
nullable: true
name:
title: Name
type: string
start_time:
format: date-time
title: Start Time
type: string
end_time:
anyOf:
- format: date-time
type: string
- type: 'null'
nullable: true
attributes:
anyOf:
- additionalProperties: true
type: object
- type: 'null'
status:
anyOf:
- $ref: '#/components/schemas/SpanStatus'
title: SpanStatus
- type: 'null'
nullable: true
title: SpanStatus
required:
- span_id
- trace_id
- name
- start_time
title: SpanWithStatus
type: object
QueryConditionOp:
description: Comparison operators for query conditions.
enum:
- eq
- ne
- gt
- lt
title: QueryConditionOp
type: string
QueryCondition:
description: A condition for filtering query results.
properties:
key:
title: Key
type: string
op:
$ref: '#/components/schemas/QueryConditionOp'
value:
title: Value
required:
- key
- op
- value
title: QueryCondition
type: object
MetricLabel:
description: A label associated with a metric.
properties:
name:
title: Name
type: string
value:
title: Value
type: string
required:
- name
- value
title: MetricLabel
type: object
MetricDataPoint:
description: A single data point in a metric time series.
properties:
timestamp:
title: Timestamp
type: integer
value:
title: Value
type: number
unit:
title: Unit
type: string
required:
- timestamp
- value
- unit
title: MetricDataPoint
type: object
MetricSeries:
description: A time series of metric data points.
properties:
metric:
title: Metric
type: string
labels:
items:
$ref: '#/components/schemas/MetricLabel'
title: Labels
type: array
values:
items:
$ref: '#/components/schemas/MetricDataPoint'
title: Values
type: array
required:
- metric
- labels
- values
title: MetricSeries
type: object
responses:
BadRequest400:
description: The request was invalid or malformed