refactor structure

docs/source/getting_started/distributions/meta-reference-gpu.md

@@ -0,0 +1,111 @@
+# Meta Reference Distribution
+
+The `llamastack/distribution-meta-reference-gpu` distribution consists of the following provider configurations.
+
+
+| **API**         	| **Inference** 	| **Agents**     	| **Memory**                                       	| **Safety**     	| **Telemetry**  	|
+|-----------------	|---------------	|----------------	|--------------------------------------------------	|----------------	|----------------	|
+| **Provider(s)** 	| meta-reference  	| meta-reference 	| meta-reference, remote::pgvector, remote::chroma 	| meta-reference 	| meta-reference 	|
+
+
+### Prerequisite
+Please make sure you have llama model checkpoints downloaded in `~/.llama` before proceeding. See [installation guide]() here to download the models.
+
+```
+$ ls ~/.llama/checkpoints
+Llama3.1-8B           Llama3.2-11B-Vision-Instruct  Llama3.2-1B-Instruct  Llama3.2-90B-Vision-Instruct  Llama-Guard-3-8B
+Llama3.1-8B-Instruct  Llama3.2-1B                   Llama3.2-3B-Instruct  Llama-Guard-3-1B              Prompt-Guard-86M
+```
+
+### Start the Distribution (Single Node GPU)
+
+```
+$ cd distributions/meta-reference-gpu
+$ ls
+build.yaml  compose.yaml  README.md  run.yaml
+$ docker compose up
+```
+
+> [!NOTE]
+> This assumes you have access to GPU to start a local server with access to your GPU.
+
+
+> [!NOTE]
+> `~/.llama` should be the path containing downloaded weights of Llama models.
+
+
+This will download and start running a pre-built docker container. Alternatively, you may use the following commands:
+
+```
+docker run -it -p 5000:5000 -v ~/.llama:/root/.llama -v ./run.yaml:/root/my-run.yaml --gpus=all distribution-meta-reference-gpu --yaml_config /root/my-run.yaml
+```
+
+### Alternative (Build and start distribution locally via conda)
+- You may checkout the [Getting Started](../../docs/getting_started.md) for more details on building locally via conda and starting up a meta-reference distribution.
+
+### Start Distribution With pgvector/chromadb Memory Provider
+##### pgvector
+1. Start running the pgvector server:
+
+```
+docker run --network host --name mypostgres -it -p 5432:5432 -e POSTGRES_PASSWORD=mysecretpassword -e POSTGRES_USER=postgres -e POSTGRES_DB=postgres pgvector/pgvector:pg16
+```
+
+2. Edit the `run.yaml` file to point to the pgvector server.
+```
+memory:
+  - provider_id: pgvector
+    provider_type: remote::pgvector
+    config:
+      host: 127.0.0.1
+      port: 5432
+      db: postgres
+      user: postgres
+      password: mysecretpassword
+```
+
+> [!NOTE]
+> If you get a `RuntimeError: Vector extension is not installed.`. You will need to run `CREATE EXTENSION IF NOT EXISTS vector;` to include the vector extension. E.g.
+
+```
+docker exec -it mypostgres ./bin/psql -U postgres
+postgres=# CREATE EXTENSION IF NOT EXISTS vector;
+postgres=# SELECT extname from pg_extension;
+ extname
+```
+
+3. Run `docker compose up` with the updated `run.yaml` file.
+
+##### chromadb
+1. Start running chromadb server
+```
+docker run -it --network host --name chromadb -p 6000:6000 -v ./chroma_vdb:/chroma/chroma -e IS_PERSISTENT=TRUE chromadb/chroma:latest
+```
+
+2. Edit the `run.yaml` file to point to the chromadb server.
+```
+memory:
+  - provider_id: remote::chromadb
+    provider_type: remote::chromadb
+    config:
+      host: localhost
+      port: 6000
+```
+
+3. Run `docker compose up` with the updated `run.yaml` file.
+
+### Serving a new model
+You may change the `config.model` in `run.yaml` to update the model currently being served by the distribution. Make sure you have the model checkpoint downloaded in your `~/.llama`.
+```
+inference:
+  - provider_id: meta0
+    provider_type: meta-reference
+    config:
+      model: Llama3.2-11B-Vision-Instruct
+      quantization: null
+      torch_seed: null
+      max_seq_len: 4096
+      max_batch_size: 1
+```
+
+Run `llama model list` to see the available models to download, and `llama model download` to download the checkpoints.