llama-stack-mirror/src/llama_stack/distributions/oci/doc_template.md

orphan
true

OCI Distribution

The llamastack/distribution-{{ name }} distribution consists of the following provider configurations.

{{ providers_table }}

{% if run_config_env_vars %}

Environment Variables

The following environment variables can be configured:

{% for var, (default_value, description) in run_config_env_vars.items() %}

• {{ var }}: {{ description }} (default: {{ default_value }}) {% endfor %} {% endif %}

{% if default_models %}

Models

The following models are available by default:

{% for model in default_models %}

• {{ model.model_id }} {{ model.doc_string }} {% endfor %} {% endif %}

Prerequisites

Oracle Cloud Infrastructure Setup

Before using the OCI Generative AI distribution, ensure you have:

1. Oracle Cloud Infrastructure Account: Sign up at Oracle Cloud Infrastructure
2. Generative AI Service Access: Enable the Generative AI service in your OCI tenancy
3. Compartment: Create or identify a compartment where you'll deploy Generative AI models
4. Authentication: Configure authentication using either:
   • Instance Principal (recommended for cloud-hosted deployments)
   • API Key (for on-premises or development environments)

Authentication Methods

Instance Principal Authentication (Recommended)

Instance Principal authentication allows OCI resources to authenticate using the identity of the compute instance they're running on. This is the most secure method for production deployments.

Requirements:

• Instance must be running in an Oracle Cloud Infrastructure compartment
• Instance must have appropriate IAM policies to access Generative AI services

API Key Authentication

For development or on-premises deployments, follow this doc to learn how to create your API signing key for your config file.

Required IAM Policies

Ensure your OCI user or instance has the following policy statements:

Allow group <group_name> to use generative-ai-inference-endpoints in compartment <compartment_name>
Allow group <group_name> to manage generative-ai-inference-endpoints in compartment <compartment_name>

Supported Services

Inference: OCI Generative AI

Oracle Cloud Infrastructure Generative AI provides access to high-performance AI models through OCI's Platform-as-a-Service offering. The service supports:

• Chat Completions: Conversational AI with context awareness
• Text Generation: Complete prompts and generate text content

Available Models

Common OCI Generative AI models include access to Meta, Cohere, OpenAI, Grok, and more models.

Safety: Llama Guard

For content safety and moderation, this distribution uses Meta's LlamaGuard model through the OCI Generative AI service to provide:

• Content filtering and moderation
• Policy compliance checking
• Harmful content detection

Vector Storage: Multiple Options

The distribution supports several vector storage providers:

• FAISS: Local in-memory vector search
• ChromaDB: Distributed vector database
• PGVector: PostgreSQL with vector extensions

Additional Services

• Dataset I/O: Local filesystem and Hugging Face integration
• Tool Runtime: Web search (Brave, Tavily) and RAG capabilities
• Evaluation: Meta reference evaluation framework

Running Llama Stack with OCI

You can run the OCI distribution via Docker or local virtual environment.

Via venv

If you've set up your local development environment, you can also build the image using your local virtual environment.

OCI_AUTH=$OCI_AUTH_TYPE OCI_REGION=$OCI_REGION OCI_COMPARTMENT_OCID=$OCI_COMPARTMENT_OCID llama stack run --port 8321 oci

Configuration Examples

Using Instance Principal (Recommended for Production)

export OCI_AUTH_TYPE=instance_principal
export OCI_REGION=us-chicago-1
export OCI_COMPARTMENT_OCID=ocid1.compartment.oc1..<your-compartment-id>

Using API Key Authentication (Development)

export OCI_AUTH_TYPE=config_file
export OCI_CONFIG_FILE_PATH=~/.oci/config
export OCI_CLI_PROFILE=DEFAULT
export OCI_REGION=us-chicago-1
export OCI_COMPARTMENT_OCID=ocid1.compartment.oc1..your-compartment-id

Regional Endpoints

OCI Generative AI is available in multiple regions. The service automatically routes to the appropriate regional endpoint based on your configuration. For a full list of regional model availability, visit:

https://docs.oracle.com/en-us/iaas/Content/generative-ai/overview.htm#regions

Troubleshooting

Common Issues

1. Authentication Errors: Verify your OCI credentials and IAM policies
2. Model Not Found: Ensure the model OCID is correct and the model is available in your region
3. Permission Denied: Check compartment permissions and Generative AI service access
4. Region Unavailable: Verify the specified region supports Generative AI services

Getting Help

For additional support:

• OCI Generative AI Documentation
• Llama Stack Issues