phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-08-02 08:44:44 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

pushed docker image, updated documentation

Browse source
This commit is contained in:
Jinan Zhou 2024-11-22 00:22:46 +00:00
parent cb82b1ee9e
commit e1c6a2c61c
4 changed files with 55 additions and 29 deletions
Download patch file Download diff file Expand all files Collapse all files

1
README.md
View file

@ -100,6 +100,7 @@ Additionally, we have designed every element of the Stack such that APIs as well
| TGI | [llamastack/distribution-tgi](https://hub.docker.com/repository/docker/llamastack/distribution-tgi/general) | [Guide](https://llama-stack.readthedocs.io/en/latest/distributions/self_hosted_distro/tgi.html) |
| Together | [llamastack/distribution-together](https://hub.docker.com/repository/docker/llamastack/distribution-together/general) | [Guide](https://llama-stack.readthedocs.io/en/latest/distributions/self_hosted_distro/together.html) |
| Fireworks | [llamastack/distribution-fireworks](https://hub.docker.com/repository/docker/llamastack/distribution-fireworks/general) | [Guide](https://llama-stack.readthedocs.io/en/latest/distributions/self_hosted_distro/fireworks.html) |
| Nutanix | [distribution-nutanix](https://hub.docker.com/repository/docker/jinanz/distribution-nutanix/general) | [Guide](https://llama-stack.readthedocs.io/en/latest/getting_started/distributions/remote_hosted_distro/nutanix.html) |
## Installation

1
docs/source/distributions/index.md
View file

@ -32,6 +32,7 @@ If so, we suggest:
- **Do you have an API key for a remote inference provider like Fireworks, Together, etc.?** If so, we suggest:
- {dockerhub}`distribution-together` ([Guide](remote_hosted_distro/index))
- {dockerhub}`distribution-fireworks` ([Guide](remote_hosted_distro/index))
- {dockerhub}`distribution-nutanix` ([Guide](remote_hosted_distro/index))
- **Do you want to run Llama Stack inference on your iOS / Android device** If so, we suggest:
- [iOS SDK](ondevice_distro/ios_sdk)

1
docs/source/distributions/remote_hosted_distro/index.md
View file

@ -9,6 +9,7 @@ Remote-Hosted distributions are available endpoints serving Llama Stack API that
|-------------|----------|-----------|---------|---------|---------|------------|
| Together | [https://llama-stack.together.ai](https://llama-stack.together.ai) | remote::together | meta-reference | remote::weaviate | meta-reference | meta-reference |
| Fireworks | [https://llamastack-preview.fireworks.ai](https://llamastack-preview.fireworks.ai) | remote::fireworks | meta-reference | remote::weaviate | meta-reference | meta-reference |
| Nutanix | [https://llamastack-preview.nutanix.ai](https://llamastack-preview.nutanix.ai) | remote::nutanix | meta-reference | meta-reference | meta-reference | meta-reference |
## Connecting to Remote-Hosted Distributions

81
llama_stack/templates/nutanix/doc_template.md
View file

@ -1,40 +1,63 @@
# Nutanix Distribution
The `llamastack/distribution-nutanix` distribution consists of the following provider configurations.
```{toctree}
:maxdepth: 2
:hidden:
| **API** | **Inference** | **Agents** | **Memory** | **Safety** | **Telemetry** |
|----------------- |--------------- |---------------- |-------------------------------------------------- |---------------- |---------------- |
| **Provider(s)** | remote::nutanix | meta-reference | meta-reference | meta-reference | meta-reference |
### Start the Distribution (Hosted remote)
> [!NOTE]
> This assumes you have an hosted Nutanix AI endpoint and an API Key.
1. Clone the repo
```
git clone git@github.com:meta-llama/llama-stack.git
cd llama-stack
self
```
2. Config the model name
The `llamastack/distribution-{{ name }}` distribution consists of the following provider configurations.
Please adjust the `NUTANIX_SUPPORTED_MODELS` variable at line 29 in `llama_stack/providers/adapters/inference/nutanix/nutanix.py` according to your deployment.
{{ providers_table }}
3. Build the distrbution
{% if run_config_env_vars %}
### Environment Variables
The following environment variables can be configured:
{% for var, (default_value, description) in run_config_env_vars.items() %}
- `{{ var }}`: {{ description }} (default: `{{ default_value }}`)
{% endfor %}
{% endif %}
{% if default_models %}
### Models
The following models are available by default:
{% for model in default_models %}
- `{{ model.model_id }} ({{ model.provider_model_id }})`
{% endfor %}
{% endif %}
### Prerequisite: API Keys
Make sure you have a Nutanix AI Endpoint deployed and a API key.
## Running Llama Stack with Nutanix
You can do this via Conda (build code) or Docker.
### Via Docker
```bash
llama stack build --template nutanix --image-type docker
LLAMA_STACK_PORT=1740
llama stack run nutanix \
--port $LLAMA_STACK_PORT \
--env NUTANIX_API_KEY=$NUTANIX_API_KEY
```
pip install -e .
### Via Conda
```bash
llama stack build --template nutanix --image-type conda
```
4. Edit the yaml file
```
vim
```
5. Serve and enjoy!
```
llama stack run ntnx --port 174
LLAMA_STACK_PORT=1740
llama stack run ./run.yaml \
--port $LLAMA_STACK_PORT \
--env NUTANIX_API_KEY=$NUTANIX_API_KEY
```