feat: Add static file import system for docs (#3882)

# What does this PR do?

Add static file import system for docs

- Use `remark-code-import` plugin to embed code at build time
- Support importing Python code with syntax highlighting using
`raw-loader` + `ReactMarkdown`

One caveat is that currently when embedding markdown with code used the
syntax highlighting isn't behaving but I'll investigate that in a follow
up.

## Test Plan

Python Example:
<img width="1372" height="995" alt="Screenshot 2025-10-23 at 9 22 18 PM"
src="https://github.com/user-attachments/assets/656d2c78-4d9b-45a4-bd5e-3f8490352b85"
/>

Markdown example:
<img width="1496" height="1070" alt="Screenshot 2025-10-23 at 9 22
38 PM"
src="https://github.com/user-attachments/assets/6c0a07ec-ff7c-45aa-b05f-8c46acd4445c"
/>

---------

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
This commit is contained in:
Francisco Arceo 2025-10-24 14:01:33 -04:00 committed by GitHub
parent 8265d4efc8
commit 4566eebe05
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 683 additions and 914 deletions

View file

@ -13,6 +13,42 @@ npm run serve
```
You can open up the docs in your browser at http://localhost:3000
## File Import System
This documentation uses `remark-code-import` to import files directly from the repository, eliminating copy-paste maintenance. Files are automatically embedded during build time.
### Importing Code Files
To import Python code (or any code files) with syntax highlighting, use this syntax in `.mdx` files:
```markdown
```python file=./demo_script.py title="demo_script.py"
```
```
This automatically imports the file content and displays it as a formatted code block with Python syntax highlighting.
**Note:** Paths are relative to the current `.mdx` file location, not the repository root.
### Importing Markdown Files as Content
For importing and rendering markdown files (like CONTRIBUTING.md), use the raw-loader approach:
```jsx
import Contributing from '!!raw-loader!../../../CONTRIBUTING.md';
import ReactMarkdown from 'react-markdown';
<ReactMarkdown>{Contributing}</ReactMarkdown>
```
**Requirements:**
- Install dependencies: `npm install --save-dev raw-loader react-markdown`
**Path Resolution:**
- For `remark-code-import`: Paths are relative to the current `.mdx` file location
- For `raw-loader`: Paths are relative to the current `.mdx` file location
- Use `../` to navigate up directories as needed
## Content
Try out Llama Stack's capabilities through our detailed Jupyter notebooks:

View file

@ -1,232 +1,13 @@
# Contributing to Llama Stack
We want to make contributing to this project as easy and transparent as
possible.
---
title: Contributing
description: Contributing to Llama Stack
sidebar_label: Contributing to Llama Stack
sidebar_position: 3
hide_title: true
---
## Set up your development environment
import Contributing from '!!raw-loader!../../../CONTRIBUTING.md';
import ReactMarkdown from 'react-markdown';
We use [uv](https://github.com/astral-sh/uv) to manage python dependencies and virtual environments.
You can install `uv` by following this [guide](https://docs.astral.sh/uv/getting-started/installation/).
You can install the dependencies by running:
```bash
cd llama-stack
uv sync --group dev
uv pip install -e .
source .venv/bin/activate
```
```{note}
You can use a specific version of Python with `uv` by adding the `--python <version>` flag (e.g. `--python 3.12`).
Otherwise, `uv` will automatically select a Python version according to the `requires-python` section of the `pyproject.toml`.
For more info, see the [uv docs around Python versions](https://docs.astral.sh/uv/concepts/python-versions/).
```
Note that you can create a dotenv file `.env` that includes necessary environment variables:
```
LLAMA_STACK_BASE_URL=http://localhost:8321
LLAMA_STACK_CLIENT_LOG=debug
LLAMA_STACK_PORT=8321
LLAMA_STACK_CONFIG=<provider-name>
TAVILY_SEARCH_API_KEY=
BRAVE_SEARCH_API_KEY=
```
And then use this dotenv file when running client SDK tests via the following:
```bash
uv run --env-file .env -- pytest -v tests/integration/inference/test_text_inference.py --text-model=meta-llama/Llama-3.1-8B-Instruct
```
### Pre-commit Hooks
We use [pre-commit](https://pre-commit.com/) to run linting and formatting checks on your code. You can install the pre-commit hooks by running:
```bash
uv run pre-commit install
```
After that, pre-commit hooks will run automatically before each commit.
Alternatively, if you don't want to install the pre-commit hooks, you can run the checks manually by running:
```bash
uv run pre-commit run --all-files
```
```{caution}
Before pushing your changes, make sure that the pre-commit hooks have passed successfully.
```
## Discussions -> Issues -> Pull Requests
We actively welcome your pull requests. However, please read the following. This is heavily inspired by [Ghostty](https://github.com/ghostty-org/ghostty/blob/main/CONTRIBUTING.md).
If in doubt, please open a [discussion](https://github.com/meta-llama/llama-stack/discussions); we can always convert that to an issue later.
### Issues
We use GitHub issues to track public bugs. Please ensure your description is
clear and has sufficient instructions to be able to reproduce the issue.
Meta has a [bounty program](http://facebook.com/whitehat/info) for the safe
disclosure of security bugs. In those cases, please go through the process
outlined on that page and do not file a public issue.
### Contributor License Agreement ("CLA")
In order to accept your pull request, we need you to submit a CLA. You only need
to do this once to work on any of Meta's open source projects.
Complete your CLA here: [https://code.facebook.com/cla](https://code.facebook.com/cla)
**I'd like to contribute!**
If you are new to the project, start by looking at the issues tagged with "good first issue". If you're interested
leave a comment on the issue and a triager will assign it to you.
Please avoid picking up too many issues at once. This helps you stay focused and ensures that others in the community also have opportunities to contribute.
- Try to work on only 12 issues at a time, especially if youre still getting familiar with the codebase.
- Before taking an issue, check if its already assigned or being actively discussed.
- If youre blocked or cant continue with an issue, feel free to unassign yourself or leave a comment so others can step in.
**I have a bug!**
1. Search the issue tracker and discussions for similar issues.
2. If you don't have steps to reproduce, open a discussion.
3. If you have steps to reproduce, open an issue.
**I have an idea for a feature!**
1. Open a discussion.
**I've implemented a feature!**
1. If there is an issue for the feature, open a pull request.
2. If there is no issue, open a discussion and link to your branch.
**I have a question!**
1. Open a discussion or use [Discord](https://discord.gg/llama-stack).
**Opening a Pull Request**
1. Fork the repo and create your branch from `main`.
2. If you've changed APIs, update the documentation.
3. Ensure the test suite passes.
4. Make sure your code lints using `pre-commit`.
5. If you haven't already, complete the Contributor License Agreement ("CLA").
6. Ensure your pull request follows the [conventional commits format](https://www.conventionalcommits.org/en/v1.0.0/).
7. Ensure your pull request follows the [coding style](#coding-style).
Please keep pull requests (PRs) small and focused. If you have a large set of changes, consider splitting them into logically grouped, smaller PRs to facilitate review and testing.
```{tip}
As a general guideline:
- Experienced contributors should try to keep no more than 5 open PRs at a time.
- New contributors are encouraged to have only one open PR at a time until theyre familiar with the codebase and process.
```
## Repository guidelines
### Coding Style
* Comments should provide meaningful insights into the code. Avoid filler comments that simply
describe the next step, as they create unnecessary clutter, same goes for docstrings.
* Prefer comments to clarify surprising behavior and/or relationships between parts of the code
rather than explain what the next line of code does.
* Catching exceptions, prefer using a specific exception type rather than a broad catch-all like
`Exception`.
* Error messages should be prefixed with "Failed to ..."
* 4 spaces for indentation rather than tab
* When using `# noqa` to suppress a style or linter warning, include a comment explaining the
justification for bypassing the check.
* When using `# type: ignore` to suppress a mypy warning, include a comment explaining the
justification for bypassing the check.
* Don't use unicode characters in the codebase. ASCII-only is preferred for compatibility or
readability reasons.
* Providers configuration class should be Pydantic Field class. It should have a `description` field
that describes the configuration. These descriptions will be used to generate the provider
documentation.
* When possible, use keyword arguments only when calling functions.
* Llama Stack utilizes custom Exception classes for certain Resources that should be used where applicable.
### License
By contributing to Llama, you agree that your contributions will be licensed
under the LICENSE file in the root directory of this source tree.
## Common Tasks
Some tips about common tasks you work on while contributing to Llama Stack:
### Setup for development
```bash
git clone https://github.com/meta-llama/llama-stack.git
cd llama-stack
uv run llama stack list-deps <distro-name> | xargs -L1 uv pip install
# (Optional) If you are developing the llama-stack-client-python package, you can add it as an editable package.
git clone https://github.com/meta-llama/llama-stack-client-python.git
uv add --editable ../llama-stack-client-python
```
### Updating distribution configurations
If you have made changes to a provider's configuration in any form (introducing a new config key, or
changing models, etc.), you should run `./scripts/distro_codegen.py` to re-generate various YAML
files as well as the documentation. You should not change `docs/source/.../distributions/` files
manually as they are auto-generated.
### Updating the provider documentation
If you have made changes to a provider's configuration, you should run `./scripts/provider_codegen.py`
to re-generate the documentation. You should not change `docs/source/.../providers/` files manually
as they are auto-generated.
Note that the provider "description" field will be used to generate the provider documentation.
### Building the Documentation
If you are making changes to the documentation at [https://llamastack.github.io/](https://llamastack.github.io/), you can use the following command to build the documentation and preview your changes.
```bash
# This rebuilds the documentation pages and the OpenAPI spec.
npm install
npm run gen-api-docs all
npm run build
# This will start a local server (usually at http://127.0.0.1:3000).
npm run serve
```
### Update API Documentation
If you modify or add new API endpoints, update the API documentation accordingly. You can do this by running the following command:
```bash
uv run ./docs/openapi_generator/run_openapi_generator.sh
```
The generated API schema will be available in `docs/static/`. Make sure to review the changes before committing.
## Adding a New Provider
See:
- [Adding a New API Provider Page](./new_api_provider.mdx) which describes how to add new API providers to the Stack.
- [Vector Database Page](./new_vector_database.mdx) which describes how to add a new vector databases with Llama Stack.
- [External Provider Page](/docs/providers/external/) which describes how to add external providers to the Stack.
## Testing
See the [Testing README](https://github.com/meta-llama/llama-stack/blob/main/tests/README.md) for detailed testing information.
## Advanced Topics
For developers who need deeper understanding of the testing system internals:
- [Record-Replay Testing](./testing/record-replay.mdx)
### Benchmarking
See the [Benchmarking README](https://github.com/meta-llama/llama-stack/blob/main/benchmarking/k8s-benchmark/README.md) for benchmarking information.
<ReactMarkdown>{Contributing}</ReactMarkdown>

View file

@ -24,6 +24,9 @@ ollama run llama3.2:3b --keepalive 60m
#### Step 2: Run the Llama Stack server
```python file=./demo_script.py title="demo_script.py"
```
We will use `uv` to install dependencies and run the Llama Stack server.
```bash
# Install dependencies for the starter distribution
@ -35,27 +38,6 @@ OLLAMA_URL=http://localhost:11434 uv run --with llama-stack llama stack run star
#### Step 3: Run the demo
Now open up a new terminal and copy the following script into a file named `demo_script.py`.
```python
import io, requests
from openai import OpenAI
url="https://www.paulgraham.com/greatwork.html"
client = OpenAI(base_url="http://localhost:8321/v1/", api_key="none")
vs = client.vector_stores.create()
response = requests.get(url)
pseudo_file = io.BytesIO(str(response.content).encode('utf-8'))
uploaded_file = client.files.create(file=(url, pseudo_file, "text/html"), purpose="assistants")
client.vector_stores.files.create(vector_store_id=vs.id, file_id=uploaded_file.id)
resp = client.responses.create(
model="openai/gpt-4o",
input="How do you do great work? Use the existing knowledge_search tool.",
tools=[{"type": "file_search", "vector_store_ids": [vs.id]}],
include=["file_search_call.results"],
)
We will use `uv` to run the script
```
uv run --with llama-stack-client,fire,requests demo_script.py

View file

@ -71,6 +71,11 @@ const config: Config = {
docs: {
sidebarPath: require.resolve("./sidebars.ts"),
docItemComponent: "@theme/ApiItem", // Derived from docusaurus-theme-openapi
remarkPlugins: [
[require('remark-code-import'), {
rootDir: require('path').join(__dirname, '..') // Repository root
}]
],
},
blog: false,
theme: {

1044
docs/package-lock.json generated

File diff suppressed because it is too large Load diff

View file

@ -15,7 +15,8 @@
"gen-api-docs": "docusaurus gen-api-docs",
"clean-api-docs": "docusaurus clean-api-docs",
"gen-api-docs:version": "docusaurus gen-api-docs:version",
"clean-api-docs:version": "docusaurus clean-api-docs:version"
"clean-api-docs:version": "docusaurus clean-api-docs:version",
"sync-files": "node scripts/sync-files.js"
},
"dependencies": {
"@docusaurus/core": "3.8.1",
@ -27,7 +28,8 @@
"docusaurus-theme-openapi-docs": "4.3.7",
"prism-react-renderer": "^2.3.0",
"react": "^19.0.0",
"react-dom": "^19.0.0"
"react-dom": "^19.0.0",
"remark-code-import": "^1.2.0"
},
"browserslist": {
"production": [
@ -40,5 +42,9 @@
"last 1 firefox version",
"last 1 safari version"
]
},
"devDependencies": {
"raw-loader": "^4.0.2",
"react-markdown": "^10.1.0"
}
}

145
docs/scripts/sync-files.js Executable file
View file

@ -0,0 +1,145 @@
#!/usr/bin/env node
const fs = require('fs');
const path = require('path');
// Repository root is always one level up from docs
const repoRoot = path.join(__dirname, '..', '..');
// Get all requested files from the usage tracking file
function getRequestedFiles() {
const usageFile = path.join(__dirname, '..', 'static', 'imported-files', 'usage.json');
if (!fs.existsSync(usageFile)) {
return [];
}
try {
const usage = JSON.parse(fs.readFileSync(usageFile, 'utf8'));
return usage.files || [];
} catch (error) {
console.warn('Could not read usage file:', error.message);
return [];
}
}
// Track file usage
function trackFileUsage(filePath) {
const usageFile = path.join(__dirname, '..', 'static', 'imported-files', 'usage.json');
const usageDir = path.dirname(usageFile);
// Ensure directory exists
if (!fs.existsSync(usageDir)) {
fs.mkdirSync(usageDir, { recursive: true });
}
let usage = { files: [] };
if (fs.existsSync(usageFile)) {
try {
usage = JSON.parse(fs.readFileSync(usageFile, 'utf8'));
} catch (error) {
console.warn('Could not read existing usage file, creating new one');
}
}
if (!usage.files.includes(filePath)) {
usage.files.push(filePath);
fs.writeFileSync(usageFile, JSON.stringify(usage, null, 2));
}
}
// Filter content based on file type and options
function filterContent(content, filePath) {
let lines = content.split('\n');
// Skip copyright header for Python files
if (filePath.endsWith('.py')) {
// Read the license header file
const licenseHeaderPath = path.join(repoRoot, 'docs', 'license_header.txt');
if (fs.existsSync(licenseHeaderPath)) {
try {
const licenseText = fs.readFileSync(licenseHeaderPath, 'utf8');
const licenseLines = licenseText.trim().split('\n');
// Check if file starts with the license header (accounting for # comments)
if (lines.length >= licenseLines.length) {
let matches = true;
for (let i = 0; i < licenseLines.length; i++) {
const codeLine = lines[i]?.replace(/^#\s*/, '').trim();
const licenseLine = licenseLines[i]?.trim();
if (codeLine !== licenseLine) {
matches = false;
break;
}
}
if (matches) {
// Skip the license header and any trailing empty lines
let skipTo = licenseLines.length;
while (skipTo < lines.length && lines[skipTo].trim() === '') {
skipTo++;
}
lines = lines.slice(skipTo);
}
}
} catch (error) {
console.warn(`Could not read license header, skipping filtering for ${filePath}`);
}
}
}
// Trim empty lines from start and end
while (lines.length > 0 && lines[0].trim() === '') {
lines.shift();
}
while (lines.length > 0 && lines[lines.length - 1].trim() === '') {
lines.pop();
}
return lines.join('\n');
}
// Sync a file from repo root to static directory
function syncFile(filePath) {
const sourcePath = path.join(repoRoot, filePath);
const destPath = path.join(__dirname, '..', 'static', 'imported-files', filePath);
const destDir = path.dirname(destPath);
// Ensure destination directory exists
if (!fs.existsSync(destDir)) {
fs.mkdirSync(destDir, { recursive: true });
}
try {
if (fs.existsSync(sourcePath)) {
const content = fs.readFileSync(sourcePath, 'utf8');
const filteredContent = filterContent(content, filePath);
fs.writeFileSync(destPath, filteredContent);
console.log(`✅ Synced ${filePath}`);
trackFileUsage(filePath);
return true;
} else {
console.warn(`⚠️ Source file not found: ${sourcePath}`);
return false;
}
} catch (error) {
console.error(`❌ Error syncing ${filePath}:`, error.message);
return false;
}
}
// Main execution
console.log(`📁 Repository root: ${path.resolve(repoRoot)}`);
// Get files that are being requested by the documentation
const requestedFiles = getRequestedFiles();
console.log(`📄 Syncing ${requestedFiles.length} requested files...`);
if (requestedFiles.length === 0) {
console.log(' No files requested yet. Files will be synced when first referenced in documentation.');
} else {
requestedFiles.forEach(filePath => {
syncFile(filePath);
});
}
console.log('✅ File sync complete!');

View file

@ -0,0 +1,93 @@
import React, { useState, useEffect } from 'react';
import CodeBlock from '@theme/CodeBlock';
export default function CodeFromFile({
src,
language = 'python',
title,
startLine,
endLine,
highlightLines
}) {
const [content, setContent] = useState('');
const [error, setError] = useState(null);
useEffect(() => {
async function loadFile() {
try {
// File registration is now handled by the file-sync-plugin during build
// Load file from static/imported-files directory
const response = await fetch(`/imported-files/${src}`);
if (!response.ok) {
throw new Error(`Failed to fetch: ${response.status}`);
}
let text = await response.text();
// Handle line range if specified (filtering is done at build time)
if (startLine || endLine) {
const lines = text.split('\n');
const start = startLine ? Math.max(0, startLine - 1) : 0;
const end = endLine ? Math.min(lines.length, endLine) : lines.length;
text = lines.slice(start, end).join('\n');
}
setContent(text);
} catch (err) {
console.error('Failed to load file:', err);
setError(`Failed to load ${src}: ${err.message}`);
}
}
loadFile();
}, [src, startLine, endLine]);
if (error) {
return <div style={{ color: 'red', padding: '1rem', border: '1px solid red', borderRadius: '4px' }}>
Error: {error}
</div>;
}
if (!content) {
return <div>Loading {src}...</div>;
}
// Auto-detect language from file extension if not provided
const detectedLanguage = language || getLanguageFromExtension(src);
return (
<CodeBlock
language={detectedLanguage}
title={title || src}
metastring={highlightLines ? `{${highlightLines}}` : undefined}
>
{content}
</CodeBlock>
);
}
function getLanguageFromExtension(filename) {
const ext = filename.split('.').pop();
const languageMap = {
'py': 'python',
'js': 'javascript',
'jsx': 'jsx',
'ts': 'typescript',
'tsx': 'tsx',
'md': 'markdown',
'sh': 'bash',
'yaml': 'yaml',
'yml': 'yaml',
'json': 'json',
'css': 'css',
'html': 'html',
'cpp': 'cpp',
'c': 'c',
'java': 'java',
'go': 'go',
'rs': 'rust',
'php': 'php',
'rb': 'ruby',
};
return languageMap[ext] || 'text';
}