llama-stack-mirror/docs/source
ehhuang e980436a2e
chore: introduce write queue for inference_store (#3383)
# What does this PR do?
Adds a write worker queue for writes to inference store. This avoids
overwhelming request processing with slow inference writes.

## Test Plan

Benchmark:
```
cd /docs/source/distributions/k8s-benchmark
# start mock server
python openai-mock-server.py --port 8000
# start stack server
LLAMA_STACK_LOGGING="all=WARNING" uv run --with llama-stack python -m llama_stack.core.server.server docs/source/distributions/k8s-benchmark/stack_run_config.yaml
# run benchmark script
uv run python3 benchmark.py --duration 120 --concurrent 50 --base-url=http://localhost:8321/v1/openai/v1 --model=vllm-inference/meta-llama/Llama-3.2-3B-Instruct
```
## RPS from 21 -> 57
2025-09-10 11:57:42 -07:00
..
advanced_apis chore: remove absolute paths (#3263) 2025-08-27 12:04:25 -07:00
apis chore: bump min python version in docs and tests (#3103) 2025-08-12 08:52:57 -07:00
building_applications chore: remove absolute paths (#3263) 2025-08-27 12:04:25 -07:00
concepts chore: remove absolute paths (#3263) 2025-08-27 12:04:25 -07:00
contributing feat(tests): auto-merge all model list responses and unify recordings (#3320) 2025-09-03 11:33:03 -07:00
deploying chore(rename): move llama_stack.distribution to llama_stack.core (#2975) 2025-07-30 23:30:53 -07:00
distributions chore: introduce write queue for inference_store (#3383) 2025-09-10 11:57:42 -07:00
getting_started feat: Updating Rag Tool to use Files API and Vector Stores API (#3344) 2025-09-06 07:26:34 -06:00
providers docs: add MongoDB to external provider list (#3369) 2025-09-08 14:09:13 +02:00
references fix: Remove bfcl scoring function as not supported (#3281) 2025-08-29 11:03:52 -07:00
conf.py docs: Add documentation on how to contribute a Vector DB provider and update testing documentation (#3093) 2025-08-11 11:11:09 -07:00
index.md docs: Reorganize documentation on the webpage (#2651) 2025-07-15 14:19:35 -07:00