phoenix-oss/llama-stack
1
0
Fork 0
forked from phoenix-oss/llama-stack-mirror
Code Pull requests Releases Packages 2 Activity Actions

More simplification of the "Starting a Llama Stack" doc

Browse source
This commit is contained in:
Ashwin Bharambe 2024-11-22 22:38:53 -08:00
parent 76fc5d9f31
commit 1e6006c599
4 changed files with 17 additions and 54 deletions
Download patch file Download diff file Expand all files Collapse all files

28
docs/source/distributions/index.md
View file

@ -4,31 +4,33 @@
:hidden:
importing_as_library
self_hosted_distro/index
remote_hosted_distro/index
building_distro
ondevice_distro/index
```
You can start a Llama Stack server using "distributions" (see [Concepts](../concepts/index)) in one of the following ways:
- **Docker**: we provide a number of pre-built Docker containers allowing you to get started instantly. If you are focused on application development, we recommend this option. You can also build your own custom Docker container.
- **Conda**: the `llama` CLI provides a simple set of commands to build, configure and run a Llama Stack server containing the exact combination of providers you wish. We have provided various templates to make getting started easier.
<!-- self_hosted_distro/index -->
<!-- remote_hosted_distro/index -->
<!-- ondevice_distro/index -->
Which distribution to choose depends on the hardware you have for running LLM inference.
You can instantiate a Llama Stack in one of the following ways:
- **As a Library**: this is the simplest, especially if you are using an external inference service. See [Using Llama Stack as a Library](importing_as_library)
- **Docker**: we provide a number of pre-built Docker containers so you can start a Llama Stack server instantly. You can also build your own custom Docker container.
- **Conda**: finally, you can build a custom Llama Stack server using `llama stack build` containing the exact combination of providers you wish. We have provided various templates to make getting started easier.
Which templates / distributions to choose depends on the hardware you have for running LLM inference.
- **Do you have access to a machine with powerful GPUs?**
If so, we suggest:
- [distribution-remote-vllm](self_hosted_distro/remote-vllm)
- [distribution-meta-reference-gpu](self_hosted_distro/meta-reference-gpu)
- [distribution-tgi](self_hosted_distro/tgi)
- {dockerhub}`distribution-remote-vllm` ([Guide](self_hosted_distro/remote-vllm))
- {dockerhub}`distribution-meta-reference-gpu` ([Guide](self_hosted_distro/meta-reference-gpu))
- {dockerhub}`distribution-tgi` ([Guide](self_hosted_distro/tgi))
- **Are you running on a "regular" desktop machine?**
If so, we suggest:
- [distribution-ollama](self_hosted_distro/ollama)
- {dockerhub}`distribution-ollama` ([Guide](self_hosted_distro/ollama))
- **Do you have an API key for a remote inference provider like Fireworks, Together, etc.?** If so, we suggest:
- [distribution-together](remote_hosted_distro/index)
- [distribution-fireworks](remote_hosted_distro/index)
- {dockerhub}`distribution-together` ([Guide](remote_hosted_distro/index))
- {dockerhub}`distribution-fireworks` ([Guide](remote_hosted_distro/index))
- **Do you want to run Llama Stack inference on your iOS / Android device** If so, we suggest:
- [iOS](ondevice_distro/ios_sdk)