forked from phoenix-oss/llama-stack-mirror
9436dd570d
# What does this PR do? We have support for embeddings in our Inference providers, but so far we haven't done the final step of actually registering the known embedding models and making sure they are extremely easy to use. This is one step towards that. ## Test Plan Run existing inference tests. ```bash $ cd llama_stack/providers/tests/inference $ pytest -s -v -k fireworks test_embeddings.py \ --inference-model nomic-ai/nomic-embed-text-v1.5 --env EMBEDDING_DIMENSION=784 $ pytest -s -v -k together test_embeddings.py \ --inference-model togethercomputer/m2-bert-80M-8k-retrieval --env EMBEDDING_DIMENSION=784 $ pytest -s -v -k ollama test_embeddings.py \ --inference-model all-minilm:latest --env EMBEDDING_DIMENSION=784 ``` The value of the EMBEDDING_DIMENSION isn't actually used in these tests, it is merely used by the test fixtures to check if the model is an LLM or Embedding.
83 lines
2.3 KiB
Markdown
83 lines
2.3 KiB
Markdown
---
|
|
orphan: true
|
|
---
|
|
<!-- This file was auto-generated by distro_codegen.py, please edit source -->
|
|
# Together Distribution
|
|
|
|
```{toctree}
|
|
:maxdepth: 2
|
|
:hidden:
|
|
|
|
self
|
|
```
|
|
|
|
The `llamastack/distribution-together` distribution consists of the following provider configurations.
|
|
|
|
| API | Provider(s) |
|
|
|-----|-------------|
|
|
| agents | `inline::meta-reference` |
|
|
| datasetio | `remote::huggingface`, `inline::localfs` |
|
|
| eval | `inline::meta-reference` |
|
|
| inference | `remote::together` |
|
|
| safety | `inline::llama-guard` |
|
|
| scoring | `inline::basic`, `inline::llm-as-judge`, `inline::braintrust` |
|
|
| telemetry | `inline::meta-reference` |
|
|
| tool_runtime | `remote::brave-search`, `remote::tavily-search`, `inline::code-interpreter`, `inline::rag-runtime`, `remote::model-context-protocol` |
|
|
| vector_io | `inline::faiss`, `remote::chromadb`, `remote::pgvector` |
|
|
|
|
|
|
### Environment Variables
|
|
|
|
The following environment variables can be configured:
|
|
|
|
- `LLAMA_STACK_PORT`: Port for the Llama Stack distribution server (default: `5001`)
|
|
- `TOGETHER_API_KEY`: Together.AI API Key (default: ``)
|
|
|
|
### Models
|
|
|
|
The following models are available by default:
|
|
|
|
- `meta-llama/Llama-3.1-8B-Instruct`
|
|
- `meta-llama/Llama-3.1-70B-Instruct`
|
|
- `meta-llama/Llama-3.1-405B-Instruct-FP8`
|
|
- `meta-llama/Llama-3.2-3B-Instruct`
|
|
- `meta-llama/Llama-3.2-11B-Vision-Instruct`
|
|
- `meta-llama/Llama-3.2-90B-Vision-Instruct`
|
|
- `meta-llama/Llama-3.3-70B-Instruct`
|
|
- `meta-llama/Llama-Guard-3-8B`
|
|
- `meta-llama/Llama-Guard-3-11B-Vision`
|
|
- `togethercomputer/m2-bert-80M-8k-retrieval`
|
|
- `togethercomputer/m2-bert-80M-32k-retrieval`
|
|
|
|
|
|
### Prerequisite: API Keys
|
|
|
|
Make sure you have access to a Together API Key. You can get one by visiting [together.xyz](https://together.xyz/).
|
|
|
|
|
|
## Running Llama Stack with Together
|
|
|
|
You can do this via Conda (build code) or Docker which has a pre-built image.
|
|
|
|
### Via Docker
|
|
|
|
This method allows you to get started quickly without having to build the distribution code.
|
|
|
|
```bash
|
|
LLAMA_STACK_PORT=5001
|
|
docker run \
|
|
-it \
|
|
-p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
|
|
llamastack/distribution-together \
|
|
--port $LLAMA_STACK_PORT \
|
|
--env TOGETHER_API_KEY=$TOGETHER_API_KEY
|
|
```
|
|
|
|
### Via Conda
|
|
|
|
```bash
|
|
llama stack build --template together --image-type conda
|
|
llama stack run ./run.yaml \
|
|
--port $LLAMA_STACK_PORT \
|
|
--env TOGETHER_API_KEY=$TOGETHER_API_KEY
|
|
```
|