forked from phoenix-oss/llama-stack-mirror

Composable building blocks to build Llama Apps

Find a file

Aidan Do 1c03ba239e [#342 ] RAG - fix PDF format in vector database (#551 ) # What does this PR do? Addresses issue (#342) - PDFs uploaded from url are being loaded into vector db as raw bytes - Instead this PR extracts text from PDF if mime_type is "application/json" - Adds tests to cover new cases ## Test Plan Ran these unit tests: ```bash llama stack build --template meta-reference-gpu --image-type conda conda activate llamastack-meta-reference-gpu pip install pytest pytest-asyncio pypdf pytest llama_stack/providers/tests/memory/test_vector_store.py -v ``` ``` platform linux -- Python 3.10.15, pytest-8.3.3, pluggy-1.5.0 -- /home/ubuntu/1xa100-2/llama-stack/envs/bin/python cachedir: .pytest_cache rootdir: /home/ubuntu/1xa100-2/llama-stack configfile: pyproject.toml plugins: anyio-4.6.2.post1, asyncio-0.24.0, httpx-0.35.0 asyncio: mode=strict, default_loop_scope=None collected 3 items llama_stack/providers/tests/memory/test_vector_store.py::TestVectorStore::test_returns_content_from_pdf_data_uri PASSED [ 33%] llama_stack/providers/tests/memory/test_vector_store.py::TestVectorStore::test_downloads_pdf_and_returns_content PASSED [ 66%] llama_stack/providers/tests/memory/test_vector_store.py::TestVectorStore::test_downloads_pdf_and_returns_content_with_url_object PASSED [100%] ======================================================= 3 passed, 1 warning in 0.62s ======================================================= ``` Tested manually via [this script](`afc8f8bebf/init.py`) to initialize and [this script](`afc8f8bebf/query.py`) to query ```bash # Ran with meta-reference-gpu with safety llama stack build --template meta-reference-gpu --image-type conda && llama stack run distributions/meta-reference-gpu/run-with-safety.yaml \ --port 5001 \ --env INFERENCE_MODEL=meta-llama/Llama-3.2-11B-Vision-Instruct # Run init.py script wget https://raw.githubusercontent.com/aidando73/llama-stack/afc8f8bebf70e1ad065d87e84692e1a3a45d9e19/init.py pip install httpx==0.27.2 # Due to issue https://github.com/meta-llama/llama-stack-client-python/issues/54 python init.py # Run query.py script wget https://raw.githubusercontent.com/aidando73/llama-stack/afc8f8bebf70e1ad065d87e84692e1a3a45d9e19/query.py python query.py ``` Should output valid text chunks ``` Chunk(content=' that it has a significantly\nlower violation rate than the competing standalone open source model, trading off a higher false refusal rate.\nLong-context safety. Long-context models are vulnerable to many-shot jailbreaking attacks without targeted\nmitigation (Anil et al., 2024). To address this, we finetune our models on SFT datasets that include examples\nof safe behavior in the presence of demonstrations of unsafe behavior in context. We develop a scalable\nmitigation strategy that significantly reduces VR, effectively neutralizing the impact of longer context attacks\neven for 256-shot attacks. This approach shows little to no impact on FRR and most helpfulness metrics.\nTo quantify the effectiveness of our long context safety mitigations, we use two additional benchmarking\nmethods: DocQA and Many-shot. For DocQA, short for “document question answering,” we use long documents\nwith information that could be utilized in adversarial ways. Models are provided both the document and a set\nof prompts related to the document in order to test whether the questions being related to information in the\ndocument affected the model’s ability to respond safely to the prompts. For Many-shot, following Anil et al.\n(2024), we construct a synthetic chat history composed of unsafe prompt-response pairs. A final prompt,\nunrelated to previous messages, is used to test whether the unsafe behavior in-context influenced the model\n45\nto response unsafely. The violation and false refusal rates for both DocQA and Many-shot are shown in\nFigure 20. We see that Llama 405B (with and without Llama Guard) is Pareto-better than the Comp. 2\nsystem across both violation rates and false refusal rates, across both DocQA and Many-shot. Relative to\nComp. 1, we find that Llama 405B is significantly safer, while coming at a trade off on false refusal.\nTool usage safety. The diversity of possible tools and the implementation of the tool usage call and integration\ninto the model make tool usage a challenging capability to fully mitigate (Wallace et al., 2024). We focus on\nthe search usecase. Violation and false refusal rates are shown in Figure 20. We tested against the Comp. 1\nsystem, where we find that Llama 405B is significantly safer, though has a slightly higher false refusal rate.\n5.4.5 Cybersecurity and Chemical/Biological Weapons Safety\nCyberSecurity evaluation results. To evaluate cybersecurity risk, we leverage the Cyber', document_id='num-0', token_count=512)0.7354530813978312 Chunk(content='.\nThrough careful ablations, we observe that mixing0.1% of synthetically generated long-context data with the\noriginal short-context data optimizes the performance across both short-context and long-context benchmarks.\nDPO. We observe that using only short context training data in DPO did not negatively impact long-context\nperformance as long as the SFT model is high quality in long context tasks. We suspect this is due to the\nfact that our DPO recipe has fewer optimizer steps than SFT. Given this finding, we keep the standard\nshort-context recipe for DPO on top of our long-context SFT checkpoints.\n4.3.5 Tool Use\nTeaching LLMs to use tools such as search engines or code interpreters hugely expands the range of tasks\nthey can solve, transforming them from pure chat models into more general assistants (Nakano et al., 2021;\nThoppilan et al., 2022; Parisi et al., 2022; Gao et al., 2023; Mialon et al., 2023a; Schick et al., 2024). We train\nLlama 3 to interact with the following tools:\n• Search engine. Llama 3 is trained to use Brave Search7 to answer questions about recent events that go\nbeyond its knowledge cutoff or that require retrieving a particular piece of information from the web.\n• Python interpreter. Llama 3 can generate and execute code to perform complex computations, read files\nuploaded by the user and solve tasks based on them such as question answering, summarization, data\nanalysis or visualization.\n7https://brave.com/search/api/\n24\n• Mathematical computational engine. Llama 3 can use the Wolfram Alpha API8 to more accurately solve\nmath, science problems, or retrieve accurate information from Wolfram’s database.\nThe resulting model is able to use these tools in a chat setup to solve the user’s queries, including in multi-turn\ndialogs. If a query requires multiple tool calls, the model can write a step-by-step plan, call the tools in\nsequence, and do reasoning after each tool call.\nWe also improve Llama 3’s zero-shot tool use capabilities — given in-context, potentially unseen tool definitions\nand a user query, we train the model to generate the correct tool call.\nImplementation. We implement our core tools as Python objects with different methods. Zero-shot tools can\nbe implemented as Python functions with descriptions, documentation (i.e., examples for', document_id='num-0', token_count=512)0.7350672465928054 Chunk(content=' Embeddings RoPE (θ = 500, 000)\nTable 3 Overview of the key hyperparameters of Llama 3. We display settings for 8B, 70B, and 405B language models.\n• We use a vocabulary with 128K tokens. Our token vocabulary combines 100K tokens from thetiktoken3\ntokenizer with 28K additional tokens to better support non-English languages. Compared to the Llama\n2 tokenizer, our new tokenizer improves compression rates on a sample of English data from 3.17 to\n3.94 characters per token. This enables the model to “read” more text for the same amount of training\ncompute. We also found that adding 28K tokens from select non-English languages improved both\ncompression ratios and downstream performance, with no impact on English tokenization.\n• We increase the RoPE base frequency hyperparameter to 500,000. This enables us to better support\nlonger contexts; Xiong et al. (2023) showed this value to be effective for context lengths up to 32,768.\nLlama 3 405B uses an architecture with 126 layers, a token representation dimension of 16,384, and 128\nattention heads; see Table 3 for details. This leads to a model size that is approximately compute-optimal\naccording to scaling laws on our data for our training budget of3.8 × 1025 FLOPs.\n3.2.1 Scaling Laws\nWe develop scaling laws (Hoffmann et al., 2022; Kaplan et al., 2020) to determine the optimal model size for\nour flagship model given our pre-training compute budget. In addition to determining the optimal model size,\na major challenge is to forecast the flagship model’s performance on downstream benchmark tasks, due to a\ncouple of issues: (1) Existing scaling laws typically predict only next-token prediction loss rather than specific\nbenchmark performance. (2) Scaling laws can be noisy and unreliable because they are developed based on\npre-training runs conducted with small compute budgets (Wei et al., 2022b).\nTo address these challenges, we implement a two-stage methodology to develop scaling laws that accurately\npredict downstream benchmark performance:\n1. We first establish a correlation between the compute-optimal model’s negative log-likelihood on down-\nstream tasks and the training FLOPs.\n2. Next, we correlate the negative log-likelihood on downstream tasks with task accuracy, utilizing both', document_id='num-0', token_count=512)0.7172908346230037 ``` ## Before submitting - [x] N/A - This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case). - [x] Ran pre-commit to handle lint / formatting issues. - [x] Read the [contributor guideline](https://github.com/meta-llama/llama-stack/blob/main/CONTRIBUTING.md), Pull Request section? - [x] N/A - Updated relevant documentation. - [x] Wrote necessary unit or integration tests.		2024-12-10 21:33:27 -08:00
.github	Introduce GitHub Actions Workflow for Llama Stack Tests (#523 )	2024-12-04 15:42:55 -08:00
distributions	Miscellaneous fixes around telemetry, library client and run yaml autogen	2024-12-08 20:40:22 -08:00
docs	add nvidia nim inference provider to docs (#534 )	2024-12-10 13:23:56 -08:00
llama_stack	[#342 ] RAG - fix PDF format in vector database (#551 )	2024-12-10 21:33:27 -08:00
rfcs	Update RFC-0001-llama-stack.md (#134 )	2024-09-27 09:14:36 -07:00
.flake8	ci: Run pre-commit checks in CI (#176 )	2024-10-10 11:21:59 -07:00
.gitignore	Move gitignore from docs/ to the main gitignore	2024-11-22 15:55:34 -08:00
.gitmodules	`impls` -> `inline`, `adapters` -> `remote` (#381 )	2024-11-06 14:54:05 -08:00
.pre-commit-config.yaml	Add a pre-commit for distro_codegen but it does not work yet	2024-11-18 15:21:13 -08:00
.readthedocs.yaml	first version of readthedocs (#278 )	2024-10-22 10:15:58 +05:30
CHANGELOG.md	add changelog (#487 )	2024-11-19 17:36:08 -08:00
CODE_OF_CONDUCT.md	Initial commit	2024-07-23 08:32:33 -07:00
CONTRIBUTING.md	Update CONTRIBUTING to include info about pre-commit	2024-11-18 18:17:54 -08:00
LICENSE	Update LICENSE (#47 )	2024-08-29 07:39:50 -07:00
MANIFEST.in	codegen per-distro dependencies; not hooked into setup.py yet	2024-11-19 09:54:30 -08:00
pyproject.toml	Initial commit	2024-07-23 08:32:33 -07:00
README.md	add nvidia nim inference provider to docs (#534 )	2024-12-10 13:23:56 -08:00
requirements.txt	Bump version to 0.0.61	2024-12-10 10:18:44 -08:00
SECURITY.md	Create SECURITY.md	2024-10-08 13:30:40 -04:00
setup.py	Bump version to 0.0.61	2024-12-10 10:18:44 -08:00

README.md

Llama Stack

Quick Start | Documentation | Zero-to-Hero Guide

Llama Stack defines and standardizes the set of core building blocks needed to bring generative AI applications to market. These building blocks are presented in the form of interoperable APIs with a broad set of Service Providers providing their implementations.

Our goal is to provide pre-packaged implementations which can be operated in a variety of deployment environments: developers start iterating with Desktops or their mobile devices and can seamlessly transition to on-prem or public cloud deployments. At every point in this transition, the same set of APIs and the same developer experience is available.

⚠️ Note The Stack APIs are rapidly improving, but still very much work in progress and we invite feedback as well as direct contributions.

APIs

We have working implementations of the following APIs today:

Inference
Safety
Memory
Agents
Eval
Telemetry

Alongside these APIs, we also related APIs for operating with associated resources (see Concepts):

Models
Shields
Memory Banks
EvalTasks
Datasets
Scoring Functions

We are also working on the following APIs which will be released soon:

Post Training
Synthetic Data Generation
Reward Scoring

Each of the APIs themselves is a collection of REST endpoints.

Philosophy

Service-oriented design

Unlike other frameworks, Llama Stack is built with a service-oriented, REST API-first approach. Such a design not only allows for seamless transitions from a local to remote deployments, but also forces the design to be more declarative. We believe this restriction can result in a much simpler, robust developer experience. This will necessarily trade-off against expressivity however if we get the APIs right, it can lead to a very powerful platform.

Composability

We expect the set of APIs we design to be composable. An Agent abstractly depends on { Inference, Memory, Safety } APIs but does not care about the actual implementation details. Safety itself may require model inference and hence can depend on the Inference API.

Turnkey one-stop solutions

We expect to provide turnkey solutions for popular deployment scenarios. It should be easy to deploy a Llama Stack server on AWS or on a private data center. Either of these should allow a developer to get started with powerful agentic apps, model evaluations or fine-tuning services in a matter of minutes. They should all result in the same uniform observability and developer experience.

Focus on Llama models

As a Meta initiated project, we have started by explicitly focusing on Meta's Llama series of models. Supporting the broad set of open models is no easy task and we want to start with models we understand best.

Supporting the Ecosystem

There is a vibrant ecosystem of Providers which provide efficient inference or scalable vector stores or powerful observability solutions. We want to make sure it is easy for developers to pick and choose the best implementations for their use cases. We also want to make sure it is easy for new Providers to onboard and participate in the ecosystem.

Additionally, we have designed every element of the Stack such that APIs as well as Resources (like Models) can be federated.

Supported Llama Stack Implementations

API Providers

API Provider Builder	Environments	Agents	Inference	Memory	Safety	Telemetry
Meta Reference	Single Node	✔️	✔️	✔️	✔️	✔️
Cerebras	Hosted		✔️
Fireworks	Hosted	✔️	✔️	✔️
AWS Bedrock	Hosted		✔️		✔️
Together	Hosted	✔️	✔️		✔️
Ollama	Single Node		✔️
TGI	Hosted and Single Node		✔️
NVIDIA NIM	Hosted and Single Node		✔️
Chroma	Single Node			✔️
PG Vector	Single Node			✔️
PyTorch ExecuTorch	On-device iOS	✔️	✔️

Distributions

Distribution	Llama Stack Docker	Start This Distribution
Meta Reference	llamastack/distribution-meta-reference-gpu	Guide
Meta Reference Quantized	llamastack/distribution-meta-reference-quantized-gpu	Guide
Cerebras	llamastack/distribution-cerebras	Guide
Ollama	llamastack/distribution-ollama	Guide
TGI	llamastack/distribution-tgi	Guide
Together	llamastack/distribution-together	Guide
Fireworks	llamastack/distribution-fireworks	Guide

Installation

You have two ways to install this repository:

Install as a package: You can install the repository directly from PyPI by running the following command:
```
pip install llama-stack
```

Install from source: If you prefer to install from the source code, make sure you have conda installed. Then, follow these steps:

 mkdir -p ~/local
 cd ~/local
 git clone git@github.com:meta-llama/llama-stack.git

 conda create -n stack python=3.10
 conda activate stack

 cd llama-stack
 $CONDA_PREFIX/bin/pip install -e .

Documentation

Please checkout our Documentation page for more details.

CLI reference
- Guide using llama CLI to work with Llama models (download, study prompts), and building/starting a Llama Stack distribution.
Getting Started
- Quick guide to start a Llama Stack server.
- Jupyter notebook to walk-through how to use simple text and vision inference llama_stack_client APIs
- The complete Llama Stack lesson Colab notebook of the new Llama 3.2 course on Deeplearning.ai.
- A Zero-to-Hero Guide that guide you through all the key components of llama stack with code samples.
Contributing
- Adding a new API Provider to walk-through how to add a new API provider.

Llama Stack Client SDKs

Language	Client SDK	Package
Python	llama-stack-client-python
Swift	llama-stack-client-swift
Node	llama-stack-client-node
Kotlin	llama-stack-client-kotlin

Check out our client SDKs for connecting to Llama Stack server in your preferred language, you can choose from python, node, swift, and kotlin programming languages to quickly build your applications.

You can find more example scripts with client SDKs to talk with the Llama Stack server in our llama-stack-apps repo.