Remove the "ShieldType" concept (#430)

# What does this PR do?

This PR kills the notion of "ShieldType". The impetus for this is the
realization:

> Why is keyword llama-guard appearing so many times everywhere,
sometimes with hyphens, sometimes with underscores?

Now that we have a notion of "provider specific resource identifiers"
and "user specific aliases" for those and the fact that this works with
models ("Llama3.1-8B-Instruct" <> "fireworks/llama-3pv1-..."), we can
follow the same rules for Shields.

So each Safety provider can make up a notion of identifiers it has
registered. This already happens with Bedrock correctly. We just
generalize it for Llama Guard, Prompt Guard, etc.

For Llama Guard, we further simplify by just adopting the underlying
model name itself as the identifier! No confusion necessary.

While doing this, I noticed a bug in our DistributionRegistry where we
weren't scoping identifiers by type. Fixed.

## Feature/Issue validation/testing/test plan

Ran (inference, safety, memory, agents) tests with ollama and fireworks
providers.
This commit is contained in:
Ashwin Bharambe 2024-11-12 12:37:24 -08:00 committed by GitHub
parent 09269e2a44
commit 983d6ce2df
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
26 changed files with 150 additions and 209 deletions

View file

@ -21,7 +21,7 @@
"info": {
"title": "[DRAFT] Llama Stack Specification",
"version": "0.0.1",
"description": "This is the specification of the llama stack that provides\n a set of endpoints and their corresponding interfaces that are tailored to\n best leverage Llama Models. The specification is still in draft and subject to change.\n Generated at 2024-11-12 11:16:58.657871"
"description": "This is the specification of the llama stack that provides\n a set of endpoints and their corresponding interfaces that are tailored to\n best leverage Llama Models. The specification is still in draft and subject to change.\n Generated at 2024-11-12 11:39:48.665782"
},
"servers": [
{
@ -5743,9 +5743,6 @@
"const": "shield",
"default": "shield"
},
"shield_type": {
"$ref": "#/components/schemas/ShieldType"
},
"params": {
"type": "object",
"additionalProperties": {
@ -5777,20 +5774,10 @@
"identifier",
"provider_resource_id",
"provider_id",
"type",
"shield_type"
"type"
],
"title": "A safety shield resource that can be used to check content"
},
"ShieldType": {
"type": "string",
"enum": [
"generic_content_shield",
"llama_guard",
"code_scanner",
"prompt_guard"
]
},
"Trace": {
"type": "object",
"properties": {
@ -7262,9 +7249,6 @@
"shield_id": {
"type": "string"
},
"shield_type": {
"$ref": "#/components/schemas/ShieldType"
},
"provider_shield_id": {
"type": "string"
},
@ -7299,8 +7283,7 @@
},
"additionalProperties": false,
"required": [
"shield_id",
"shield_type"
"shield_id"
]
},
"RunEvalRequest": {
@ -7854,13 +7837,19 @@
],
"tags": [
{
"name": "Inference"
"name": "MemoryBanks"
},
{
"name": "BatchInference"
},
{
"name": "Agents"
},
{
"name": "Telemetry"
"name": "Inference"
},
{
"name": "DatasetIO"
},
{
"name": "Eval"
@ -7869,43 +7858,37 @@
"name": "Models"
},
{
"name": "Inspect"
},
{
"name": "EvalTasks"
"name": "PostTraining"
},
{
"name": "ScoringFunctions"
},
{
"name": "Memory"
},
{
"name": "Safety"
},
{
"name": "DatasetIO"
},
{
"name": "MemoryBanks"
"name": "Datasets"
},
{
"name": "Shields"
},
{
"name": "PostTraining"
"name": "Telemetry"
},
{
"name": "Datasets"
"name": "Inspect"
},
{
"name": "Scoring"
"name": "Safety"
},
{
"name": "SyntheticDataGeneration"
},
{
"name": "BatchInference"
"name": "Memory"
},
{
"name": "Scoring"
},
{
"name": "EvalTasks"
},
{
"name": "BuiltinTool",
@ -8255,10 +8238,6 @@
"name": "Shield",
"description": "A safety shield resource that can be used to check content\n\n<SchemaDefinition schemaRef=\"#/components/schemas/Shield\" />"
},
{
"name": "ShieldType",
"description": "<SchemaDefinition schemaRef=\"#/components/schemas/ShieldType\" />"
},
{
"name": "Trace",
"description": "<SchemaDefinition schemaRef=\"#/components/schemas/Trace\" />"
@ -8614,7 +8593,6 @@
"Session",
"Shield",
"ShieldCallStep",
"ShieldType",
"SpanEndPayload",
"SpanStartPayload",
"SpanStatus",