mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-08-02 08:44:44 +00:00
pushed docker image, updated documentation
This commit is contained in:
parent
cb82b1ee9e
commit
e1c6a2c61c
4 changed files with 55 additions and 29 deletions
|
@ -100,6 +100,7 @@ Additionally, we have designed every element of the Stack such that APIs as well
|
||||||
| TGI | [llamastack/distribution-tgi](https://hub.docker.com/repository/docker/llamastack/distribution-tgi/general) | [Guide](https://llama-stack.readthedocs.io/en/latest/distributions/self_hosted_distro/tgi.html) |
|
| TGI | [llamastack/distribution-tgi](https://hub.docker.com/repository/docker/llamastack/distribution-tgi/general) | [Guide](https://llama-stack.readthedocs.io/en/latest/distributions/self_hosted_distro/tgi.html) |
|
||||||
| Together | [llamastack/distribution-together](https://hub.docker.com/repository/docker/llamastack/distribution-together/general) | [Guide](https://llama-stack.readthedocs.io/en/latest/distributions/self_hosted_distro/together.html) |
|
| Together | [llamastack/distribution-together](https://hub.docker.com/repository/docker/llamastack/distribution-together/general) | [Guide](https://llama-stack.readthedocs.io/en/latest/distributions/self_hosted_distro/together.html) |
|
||||||
| Fireworks | [llamastack/distribution-fireworks](https://hub.docker.com/repository/docker/llamastack/distribution-fireworks/general) | [Guide](https://llama-stack.readthedocs.io/en/latest/distributions/self_hosted_distro/fireworks.html) |
|
| Fireworks | [llamastack/distribution-fireworks](https://hub.docker.com/repository/docker/llamastack/distribution-fireworks/general) | [Guide](https://llama-stack.readthedocs.io/en/latest/distributions/self_hosted_distro/fireworks.html) |
|
||||||
|
| Nutanix | [distribution-nutanix](https://hub.docker.com/repository/docker/jinanz/distribution-nutanix/general) | [Guide](https://llama-stack.readthedocs.io/en/latest/getting_started/distributions/remote_hosted_distro/nutanix.html) |
|
||||||
|
|
||||||
## Installation
|
## Installation
|
||||||
|
|
||||||
|
|
|
@ -32,6 +32,7 @@ If so, we suggest:
|
||||||
- **Do you have an API key for a remote inference provider like Fireworks, Together, etc.?** If so, we suggest:
|
- **Do you have an API key for a remote inference provider like Fireworks, Together, etc.?** If so, we suggest:
|
||||||
- {dockerhub}`distribution-together` ([Guide](remote_hosted_distro/index))
|
- {dockerhub}`distribution-together` ([Guide](remote_hosted_distro/index))
|
||||||
- {dockerhub}`distribution-fireworks` ([Guide](remote_hosted_distro/index))
|
- {dockerhub}`distribution-fireworks` ([Guide](remote_hosted_distro/index))
|
||||||
|
- {dockerhub}`distribution-nutanix` ([Guide](remote_hosted_distro/index))
|
||||||
|
|
||||||
- **Do you want to run Llama Stack inference on your iOS / Android device** If so, we suggest:
|
- **Do you want to run Llama Stack inference on your iOS / Android device** If so, we suggest:
|
||||||
- [iOS SDK](ondevice_distro/ios_sdk)
|
- [iOS SDK](ondevice_distro/ios_sdk)
|
||||||
|
|
|
@ -9,6 +9,7 @@ Remote-Hosted distributions are available endpoints serving Llama Stack API that
|
||||||
|-------------|----------|-----------|---------|---------|---------|------------|
|
|-------------|----------|-----------|---------|---------|---------|------------|
|
||||||
| Together | [https://llama-stack.together.ai](https://llama-stack.together.ai) | remote::together | meta-reference | remote::weaviate | meta-reference | meta-reference |
|
| Together | [https://llama-stack.together.ai](https://llama-stack.together.ai) | remote::together | meta-reference | remote::weaviate | meta-reference | meta-reference |
|
||||||
| Fireworks | [https://llamastack-preview.fireworks.ai](https://llamastack-preview.fireworks.ai) | remote::fireworks | meta-reference | remote::weaviate | meta-reference | meta-reference |
|
| Fireworks | [https://llamastack-preview.fireworks.ai](https://llamastack-preview.fireworks.ai) | remote::fireworks | meta-reference | remote::weaviate | meta-reference | meta-reference |
|
||||||
|
| Nutanix | [https://llamastack-preview.nutanix.ai](https://llamastack-preview.nutanix.ai) | remote::nutanix | meta-reference | meta-reference | meta-reference | meta-reference |
|
||||||
|
|
||||||
## Connecting to Remote-Hosted Distributions
|
## Connecting to Remote-Hosted Distributions
|
||||||
|
|
||||||
|
|
|
@ -1,40 +1,63 @@
|
||||||
# Nutanix Distribution
|
# Nutanix Distribution
|
||||||
|
|
||||||
The `llamastack/distribution-nutanix` distribution consists of the following provider configurations.
|
```{toctree}
|
||||||
|
:maxdepth: 2
|
||||||
|
:hidden:
|
||||||
|
|
||||||
|
self
|
||||||
| **API** | **Inference** | **Agents** | **Memory** | **Safety** | **Telemetry** |
|
|
||||||
|----------------- |--------------- |---------------- |-------------------------------------------------- |---------------- |---------------- |
|
|
||||||
| **Provider(s)** | remote::nutanix | meta-reference | meta-reference | meta-reference | meta-reference |
|
|
||||||
|
|
||||||
|
|
||||||
### Start the Distribution (Hosted remote)
|
|
||||||
|
|
||||||
> [!NOTE]
|
|
||||||
> This assumes you have an hosted Nutanix AI endpoint and an API Key.
|
|
||||||
|
|
||||||
1. Clone the repo
|
|
||||||
```
|
|
||||||
git clone git@github.com:meta-llama/llama-stack.git
|
|
||||||
cd llama-stack
|
|
||||||
```
|
```
|
||||||
|
|
||||||
2. Config the model name
|
The `llamastack/distribution-{{ name }}` distribution consists of the following provider configurations.
|
||||||
|
|
||||||
Please adjust the `NUTANIX_SUPPORTED_MODELS` variable at line 29 in `llama_stack/providers/adapters/inference/nutanix/nutanix.py` according to your deployment.
|
{{ providers_table }}
|
||||||
|
|
||||||
3. Build the distrbution
|
{% if run_config_env_vars %}
|
||||||
|
### Environment Variables
|
||||||
|
|
||||||
|
The following environment variables can be configured:
|
||||||
|
|
||||||
|
{% for var, (default_value, description) in run_config_env_vars.items() %}
|
||||||
|
- `{{ var }}`: {{ description }} (default: `{{ default_value }}`)
|
||||||
|
{% endfor %}
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
{% if default_models %}
|
||||||
|
### Models
|
||||||
|
|
||||||
|
The following models are available by default:
|
||||||
|
|
||||||
|
{% for model in default_models %}
|
||||||
|
- `{{ model.model_id }} ({{ model.provider_model_id }})`
|
||||||
|
{% endfor %}
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
|
||||||
|
### Prerequisite: API Keys
|
||||||
|
Make sure you have a Nutanix AI Endpoint deployed and a API key.
|
||||||
|
|
||||||
|
|
||||||
|
## Running Llama Stack with Nutanix
|
||||||
|
|
||||||
|
You can do this via Conda (build code) or Docker.
|
||||||
|
|
||||||
|
### Via Docker
|
||||||
|
|
||||||
|
```bash
|
||||||
|
llama stack build --template nutanix --image-type docker
|
||||||
|
|
||||||
|
LLAMA_STACK_PORT=1740
|
||||||
|
llama stack run nutanix \
|
||||||
|
--port $LLAMA_STACK_PORT \
|
||||||
|
--env NUTANIX_API_KEY=$NUTANIX_API_KEY
|
||||||
```
|
```
|
||||||
pip install -e .
|
|
||||||
|
### Via Conda
|
||||||
|
|
||||||
|
```bash
|
||||||
llama stack build --template nutanix --image-type conda
|
llama stack build --template nutanix --image-type conda
|
||||||
```
|
|
||||||
|
|
||||||
4. Edit the yaml file
|
LLAMA_STACK_PORT=1740
|
||||||
```
|
llama stack run ./run.yaml \
|
||||||
vim
|
--port $LLAMA_STACK_PORT \
|
||||||
```
|
--env NUTANIX_API_KEY=$NUTANIX_API_KEY
|
||||||
|
|
||||||
5. Serve and enjoy!
|
|
||||||
```
|
|
||||||
llama stack run ntnx --port 174
|
|
||||||
```
|
```
|
||||||
|
|
Loading…
Add table
Add a link
Reference in a new issue