forked from phoenix-oss/llama-stack-mirror
Convert SamplingParams.strategy to a union (#767)
# What does this PR do?
Cleans up how we provide sampling params. Earlier, strategy was an enum
and all params (top_p, temperature, top_k) across all strategies were
grouped. We now have a strategy union object with each strategy (greedy,
top_p, top_k) having its corresponding params.
Earlier,
```
class SamplingParams:
strategy: enum ()
top_p, temperature, top_k and other params
```
However, the `strategy` field was not being used in any providers making
it confusing to know the exact sampling behavior purely based on the
params since you could pass temperature, top_p, top_k and how the
provider would interpret those would not be clear.
Hence we introduced -- a union where the strategy and relevant params
are all clubbed together to avoid this confusion.
Have updated all providers, tests, notebooks, readme and otehr places
where sampling params was being used to use the new format.
## Test Plan
`pytest llama_stack/providers/tests/inference/groq/test_groq_utils.py`
// inference on ollama, fireworks and together
`with-proxy pytest -v -s -k "ollama"
--inference-model="meta-llama/Llama-3.1-8B-Instruct"
llama_stack/providers/tests/inference/test_text_inference.py `
// agents on fireworks
`pytest -v -s -k 'fireworks and create_agent'
--inference-model="meta-llama/Llama-3.1-8B-Instruct"
llama_stack/providers/tests/agents/test_agents.py
--safety-shield="meta-llama/Llama-Guard-3-8B"`
## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the
other checks if that's the case).
- [X] Ran pre-commit to handle lint / formatting issues.
- [X] Read the [contributor
guideline](https://github.com/meta-llama/llama-stack/blob/main/CONTRIBUTING.md),
Pull Request section?
- [X] Updated relevant documentation.
- [X] Wrote necessary unit or integration tests.
---------
Co-authored-by: Hardik Shah <hjshah@fb.com>
This commit is contained in:
Hardik Shah
GitHub
parent
300e6e2702
commit
a51c8b4efc
29 changed files with 611 additions and 388 deletions
|
|
@ -3514,6 +3514,20 @@
|
|||
"tool_calls"
|
||||
]
|
||||
},
|
||||
"GreedySamplingStrategy": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"type": {
|
||||
"type": "string",
|
||||
"const": "greedy",
|
||||
"default": "greedy"
|
||||
}
|
||||
},
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"type"
|
||||
]
|
||||
},
|
||||
"ImageContentItem": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
|
|
@ -3581,20 +3595,17 @@
|
|||
"type": "object",
|
||||
"properties": {
|
||||
"strategy": {
|
||||
"$ref": "#/components/schemas/SamplingStrategy",
|
||||
"default": "greedy"
|
||||
},
|
||||
"temperature": {
|
||||
"type": "number",
|
||||
"default": 0.0
|
||||
},
|
||||
"top_p": {
|
||||
"type": "number",
|
||||
"default": 0.95
|
||||
},
|
||||
"top_k": {
|
||||
"type": "integer",
|
||||
"default": 0
|
||||
"oneOf": [
|
||||
{
|
||||
"$ref": "#/components/schemas/GreedySamplingStrategy"
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/schemas/TopPSamplingStrategy"
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/schemas/TopKSamplingStrategy"
|
||||
}
|
||||
]
|
||||
},
|
||||
"max_tokens": {
|
||||
"type": "integer",
|
||||
|
|
@ -3610,14 +3621,6 @@
|
|||
"strategy"
|
||||
]
|
||||
},
|
||||
"SamplingStrategy": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"greedy",
|
||||
"top_p",
|
||||
"top_k"
|
||||
]
|
||||
},
|
||||
"StopReason": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
|
|
@ -3871,6 +3874,45 @@
|
|||
"content"
|
||||
]
|
||||
},
|
||||
"TopKSamplingStrategy": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"type": {
|
||||
"type": "string",
|
||||
"const": "top_k",
|
||||
"default": "top_k"
|
||||
},
|
||||
"top_k": {
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"type",
|
||||
"top_k"
|
||||
]
|
||||
},
|
||||
"TopPSamplingStrategy": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"type": {
|
||||
"type": "string",
|
||||
"const": "top_p",
|
||||
"default": "top_p"
|
||||
},
|
||||
"temperature": {
|
||||
"type": "number"
|
||||
},
|
||||
"top_p": {
|
||||
"type": "number",
|
||||
"default": 0.95
|
||||
}
|
||||
},
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"type"
|
||||
]
|
||||
},
|
||||
"URL": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
|
|
@ -8887,6 +8929,10 @@
|
|||
"name": "GraphMemoryBankParams",
|
||||
"description": "<SchemaDefinition schemaRef=\"#/components/schemas/GraphMemoryBankParams\" />"
|
||||
},
|
||||
{
|
||||
"name": "GreedySamplingStrategy",
|
||||
"description": "<SchemaDefinition schemaRef=\"#/components/schemas/GreedySamplingStrategy\" />"
|
||||
},
|
||||
{
|
||||
"name": "HealthInfo",
|
||||
"description": "<SchemaDefinition schemaRef=\"#/components/schemas/HealthInfo\" />"
|
||||
|
|
@ -9136,10 +9182,6 @@
|
|||
"name": "SamplingParams",
|
||||
"description": "<SchemaDefinition schemaRef=\"#/components/schemas/SamplingParams\" />"
|
||||
},
|
||||
{
|
||||
"name": "SamplingStrategy",
|
||||
"description": "<SchemaDefinition schemaRef=\"#/components/schemas/SamplingStrategy\" />"
|
||||
},
|
||||
{
|
||||
"name": "SaveSpansToDatasetRequest",
|
||||
"description": "<SchemaDefinition schemaRef=\"#/components/schemas/SaveSpansToDatasetRequest\" />"
|
||||
|
|
@ -9317,6 +9359,14 @@
|
|||
{
|
||||
"name": "ToolRuntime"
|
||||
},
|
||||
{
|
||||
"name": "TopKSamplingStrategy",
|
||||
"description": "<SchemaDefinition schemaRef=\"#/components/schemas/TopKSamplingStrategy\" />"
|
||||
},
|
||||
{
|
||||
"name": "TopPSamplingStrategy",
|
||||
"description": "<SchemaDefinition schemaRef=\"#/components/schemas/TopPSamplingStrategy\" />"
|
||||
},
|
||||
{
|
||||
"name": "Trace",
|
||||
"description": "<SchemaDefinition schemaRef=\"#/components/schemas/Trace\" />"
|
||||
|
|
@ -9456,6 +9506,7 @@
|
|||
"GetSpanTreeRequest",
|
||||
"GraphMemoryBank",
|
||||
"GraphMemoryBankParams",
|
||||
"GreedySamplingStrategy",
|
||||
"HealthInfo",
|
||||
"ImageContentItem",
|
||||
"InferenceStep",
|
||||
|
|
@ -9513,7 +9564,6 @@
|
|||
"RunShieldResponse",
|
||||
"SafetyViolation",
|
||||
"SamplingParams",
|
||||
"SamplingStrategy",
|
||||
"SaveSpansToDatasetRequest",
|
||||
"ScoreBatchRequest",
|
||||
"ScoreBatchResponse",
|
||||
|
|
@ -9553,6 +9603,8 @@
|
|||
"ToolPromptFormat",
|
||||
"ToolResponse",
|
||||
"ToolResponseMessage",
|
||||
"TopKSamplingStrategy",
|
||||
"TopPSamplingStrategy",
|
||||
"Trace",
|
||||
"TrainingConfig",
|
||||
"Turn",
|
||||
|
|
|
|||
|
|
@ -937,6 +937,16 @@ components:
|
|||
required:
|
||||
- memory_bank_type
|
||||
type: object
|
||||
GreedySamplingStrategy:
|
||||
additionalProperties: false
|
||||
properties:
|
||||
type:
|
||||
const: greedy
|
||||
default: greedy
|
||||
type: string
|
||||
required:
|
||||
- type
|
||||
type: object
|
||||
HealthInfo:
|
||||
additionalProperties: false
|
||||
properties:
|
||||
|
|
@ -2064,26 +2074,13 @@ components:
|
|||
default: 1.0
|
||||
type: number
|
||||
strategy:
|
||||
$ref: '#/components/schemas/SamplingStrategy'
|
||||
default: greedy
|
||||
temperature:
|
||||
default: 0.0
|
||||
type: number
|
||||
top_k:
|
||||
default: 0
|
||||
type: integer
|
||||
top_p:
|
||||
default: 0.95
|
||||
type: number
|
||||
oneOf:
|
||||
- $ref: '#/components/schemas/GreedySamplingStrategy'
|
||||
- $ref: '#/components/schemas/TopPSamplingStrategy'
|
||||
- $ref: '#/components/schemas/TopKSamplingStrategy'
|
||||
required:
|
||||
- strategy
|
||||
type: object
|
||||
SamplingStrategy:
|
||||
enum:
|
||||
- greedy
|
||||
- top_p
|
||||
- top_k
|
||||
type: string
|
||||
SaveSpansToDatasetRequest:
|
||||
additionalProperties: false
|
||||
properties:
|
||||
|
|
@ -2931,6 +2928,34 @@ components:
|
|||
- tool_name
|
||||
- content
|
||||
type: object
|
||||
TopKSamplingStrategy:
|
||||
additionalProperties: false
|
||||
properties:
|
||||
top_k:
|
||||
type: integer
|
||||
type:
|
||||
const: top_k
|
||||
default: top_k
|
||||
type: string
|
||||
required:
|
||||
- type
|
||||
- top_k
|
||||
type: object
|
||||
TopPSamplingStrategy:
|
||||
additionalProperties: false
|
||||
properties:
|
||||
temperature:
|
||||
type: number
|
||||
top_p:
|
||||
default: 0.95
|
||||
type: number
|
||||
type:
|
||||
const: top_p
|
||||
default: top_p
|
||||
type: string
|
||||
required:
|
||||
- type
|
||||
type: object
|
||||
Trace:
|
||||
additionalProperties: false
|
||||
properties:
|
||||
|
|
@ -5587,6 +5612,9 @@ tags:
|
|||
- description: <SchemaDefinition schemaRef="#/components/schemas/GraphMemoryBankParams"
|
||||
/>
|
||||
name: GraphMemoryBankParams
|
||||
- description: <SchemaDefinition schemaRef="#/components/schemas/GreedySamplingStrategy"
|
||||
/>
|
||||
name: GreedySamplingStrategy
|
||||
- description: <SchemaDefinition schemaRef="#/components/schemas/HealthInfo" />
|
||||
name: HealthInfo
|
||||
- description: <SchemaDefinition schemaRef="#/components/schemas/ImageContentItem"
|
||||
|
|
@ -5753,9 +5781,6 @@ tags:
|
|||
name: SafetyViolation
|
||||
- description: <SchemaDefinition schemaRef="#/components/schemas/SamplingParams" />
|
||||
name: SamplingParams
|
||||
- description: <SchemaDefinition schemaRef="#/components/schemas/SamplingStrategy"
|
||||
/>
|
||||
name: SamplingStrategy
|
||||
- description: <SchemaDefinition schemaRef="#/components/schemas/SaveSpansToDatasetRequest"
|
||||
/>
|
||||
name: SaveSpansToDatasetRequest
|
||||
|
|
@ -5874,6 +5899,12 @@ tags:
|
|||
/>
|
||||
name: ToolResponseMessage
|
||||
- name: ToolRuntime
|
||||
- description: <SchemaDefinition schemaRef="#/components/schemas/TopKSamplingStrategy"
|
||||
/>
|
||||
name: TopKSamplingStrategy
|
||||
- description: <SchemaDefinition schemaRef="#/components/schemas/TopPSamplingStrategy"
|
||||
/>
|
||||
name: TopPSamplingStrategy
|
||||
- description: <SchemaDefinition schemaRef="#/components/schemas/Trace" />
|
||||
name: Trace
|
||||
- description: <SchemaDefinition schemaRef="#/components/schemas/TrainingConfig" />
|
||||
|
|
@ -5990,6 +6021,7 @@ x-tagGroups:
|
|||
- GetSpanTreeRequest
|
||||
- GraphMemoryBank
|
||||
- GraphMemoryBankParams
|
||||
- GreedySamplingStrategy
|
||||
- HealthInfo
|
||||
- ImageContentItem
|
||||
- InferenceStep
|
||||
|
|
@ -6047,7 +6079,6 @@ x-tagGroups:
|
|||
- RunShieldResponse
|
||||
- SafetyViolation
|
||||
- SamplingParams
|
||||
- SamplingStrategy
|
||||
- SaveSpansToDatasetRequest
|
||||
- ScoreBatchRequest
|
||||
- ScoreBatchResponse
|
||||
|
|
@ -6087,6 +6118,8 @@ x-tagGroups:
|
|||
- ToolPromptFormat
|
||||
- ToolResponse
|
||||
- ToolResponseMessage
|
||||
- TopKSamplingStrategy
|
||||
- TopPSamplingStrategy
|
||||
- Trace
|
||||
- TrainingConfig
|
||||
- Turn
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue