forked from phoenix-oss/llama-stack-mirror
Composable building blocks to build Llama Apps
be03cb7523
# What does this PR do?
If the generator fails, pre-commit logs will now show how it failed.
Note: stdout is still suppressed, so that regular informational messages
do not pollute pre-commit output when all the hook does is update
generated files.
[//]: # (If resolving an issue, uncomment and update the line below)
[//]: # (Closes #[issue-number])
## Test Plan
Inject a failure in the generator code and confirm it's seen in the
output.
```
$ git diff
diff --git a/docs/openapi_generator/pyopenapi/utility.py b/docs/openapi_generator/pyopenapi/utility.py
index f60a33bb..482e26ef 100644
--- a/docs/openapi_generator/pyopenapi/utility.py
+++ b/docs/openapi_generator/pyopenapi/utility.py
@@ -127,6 +127,7 @@ def is_optional_type(type_: Any) -> bool:
def validate_api_method_return_types() -> List[str]:
"""Validate that all API methods have proper return types."""
+ raise NotImplementedError("This function is not implemented yet")
errors = []
protocols = api_protocol_map()
```
```
$ pre-commit run --all-files
check for merge conflicts................................................Passed
trim trailing whitespace.................................................Passed
check for added large files..............................................Passed
fix end of files.........................................................Passed
Insert license in comments...............................................Passed
ruff.....................................................................Passed
ruff-format..............................................................Passed
blacken-docs.............................................................Passed
uv-lock..................................................................Passed
uv-export................................................................Passed
mypy.....................................................................Passed
Distribution Template Codegen............................................Passed
API Spec Codegen.........................................................Failed
- hook id: openapi-codegen
- exit code: 1
warning: `VIRTUAL_ENV=/Users/ihrachys/.cache/pre-commit/repo9p35zuhm/py_env-python3` does not match the project environment path `.venv` and will be ignored; use `--active` to target the active environment instead
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/Users/ihrachys/src/llama-stack/docs/openapi_generator/generate.py", line 91, in <module>
fire.Fire(main)
File "/Users/ihrachys/.cache/uv/archive-v0/FBgkcwcN-PaJ0NAur__7J/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ihrachys/.cache/uv/archive-v0/FBgkcwcN-PaJ0NAur__7J/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/Users/ihrachys/.cache/uv/archive-v0/FBgkcwcN-PaJ0NAur__7J/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ihrachys/src/llama-stack/docs/openapi_generator/generate.py", line 44, in main
return_type_errors = validate_api_method_return_types()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ihrachys/src/llama-stack/docs/openapi_generator/pyopenapi/utility.py", line 130, in validate_api_method_return_types
raise NotImplementedError("This function is not implemented yet")
NotImplementedError: This function is not implemented yet
```
Signed-off-by: Ihar Hrachyshka <ihar.hrachyshka@gmail.com>
|
||
|---|---|---|
| .github | ||
| distributions | ||
| docs | ||
| llama_stack | ||
| rfcs | ||
| scripts | ||
| tests | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| .readthedocs.yaml | ||
| CHANGELOG.md | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| LICENSE | ||
| MANIFEST.in | ||
| pyproject.toml | ||
| README.md | ||
| requirements.txt | ||
| SECURITY.md | ||
| uv.lock | ||
Llama Stack
Quick Start | Documentation | Colab Notebook
Llama Stack standardizes the core building blocks that simplify AI application development. It codifies best practices across the Llama ecosystem. More specifically, it provides
- Unified API layer for Inference, RAG, Agents, Tools, Safety, Evals, and Telemetry.
- Plugin architecture to support the rich ecosystem of different API implementations in various environments, including local development, on-premises, cloud, and mobile.
- Prepackaged verified distributions which offer a one-stop solution for developers to get started quickly and reliably in any environment.
- Multiple developer interfaces like CLI and SDKs for Python, Typescript, iOS, and Android.
- Standalone applications as examples for how to build production-grade AI applications with Llama Stack.
Llama Stack Benefits
- Flexible Options: Developers can choose their preferred infrastructure without changing APIs and enjoy flexible deployment choices.
- Consistent Experience: With its unified APIs, Llama Stack makes it easier to build, test, and deploy AI applications with consistent application behavior.
- Robust Ecosystem: Llama Stack is already integrated with distribution partners (cloud providers, hardware vendors, and AI-focused companies) that offer tailored infrastructure, software, and services for deploying Llama models.
By reducing friction and complexity, Llama Stack empowers developers to focus on what they do best: building transformative generative AI applications.
API Providers
Here is a list of the various API providers and available distributions that can help developers get started easily with Llama Stack.
| API Provider Builder | Environments | Agents | Inference | Memory | Safety | Telemetry |
|---|---|---|---|---|---|---|
| Meta Reference | Single Node | ✅ | ✅ | ✅ | ✅ | ✅ |
| SambaNova | Hosted | ✅ | ||||
| Cerebras | Hosted | ✅ | ||||
| Fireworks | Hosted | ✅ | ✅ | ✅ | ||
| AWS Bedrock | Hosted | ✅ | ✅ | |||
| Together | Hosted | ✅ | ✅ | ✅ | ||
| Groq | Hosted | ✅ | ||||
| Ollama | Single Node | ✅ | ||||
| TGI | Hosted and Single Node | ✅ | ||||
| NVIDIA NIM | Hosted and Single Node | ✅ | ||||
| Chroma | Single Node | ✅ | ||||
| PG Vector | Single Node | ✅ | ||||
| PyTorch ExecuTorch | On-device iOS | ✅ | ✅ | |||
| vLLM | Hosted and Single Node | ✅ | ||||
| OpenAI | Hosted | ✅ | ||||
| Anthropic | Hosted | ✅ | ||||
| Gemini | Hosted | ✅ |
Distributions
A Llama Stack Distribution (or "distro") is a pre-configured bundle of provider implementations for each API component. Distributions make it easy to get started with a specific deployment scenario - you can begin with a local development setup (eg. ollama) and seamlessly transition to production (eg. Fireworks) without changing your application code. Here are some of the distributions we support:
| Distribution | Llama Stack Docker | Start This Distribution |
|---|---|---|
| Meta Reference | llamastack/distribution-meta-reference-gpu | Guide |
| Meta Reference Quantized | llamastack/distribution-meta-reference-quantized-gpu | Guide |
| SambaNova | llamastack/distribution-sambanova | Guide |
| Cerebras | llamastack/distribution-cerebras | Guide |
| Ollama | llamastack/distribution-ollama | Guide |
| TGI | llamastack/distribution-tgi | Guide |
| Together | llamastack/distribution-together | Guide |
| Fireworks | llamastack/distribution-fireworks | Guide |
| vLLM | llamastack/distribution-remote-vllm | Guide |
Documentation
Please checkout our Documentation page for more details.
- CLI references
- llama (server-side) CLI Reference: Guide for using the
llamaCLI to work with Llama models (download, study prompts), and building/starting a Llama Stack distribution. - llama (client-side) CLI Reference: Guide for using the
llama-stack-clientCLI, which allows you to query information about the distribution.
- llama (server-side) CLI Reference: Guide for using the
- Getting Started
- Quick guide to start a Llama Stack server.
- Jupyter notebook to walk-through how to use simple text and vision inference llama_stack_client APIs
- The complete Llama Stack lesson Colab notebook of the new Llama 3.2 course on Deeplearning.ai.
- A Zero-to-Hero Guide that guide you through all the key components of llama stack with code samples.
- Contributing
- Adding a new API Provider to walk-through how to add a new API provider.
Llama Stack Client SDKs
| Language | Client SDK | Package |
|---|---|---|
| Python | llama-stack-client-python |  |
| Swift | llama-stack-client-swift |  |
| Typescript | llama-stack-client-typescript |  |
| Kotlin | llama-stack-client-kotlin |  |
Check out our client SDKs for connecting to a Llama Stack server in your preferred language, you can choose from python, typescript, swift, and kotlin programming languages to quickly build your applications.
You can find more example scripts with client SDKs to talk with the Llama Stack server in our llama-stack-apps repo.