phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-08-21 09:23:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
llama-stack-mirror/docs/source/distributions
History
Exact
ehhuang 2c06b24c77
test: benchmark scripts (#3160) 
# What does this PR do?
1. Add our own benchmark script instead of locust (doesn't support
measuring streaming latency well)
2. Simplify k8s deployment
3. Add a simple profile script for locally running server

## Test Plan
❮ ./run-benchmark.sh --target stack --duration 180 --concurrent 10

============================================================
BENCHMARK RESULTS
============================================================
Total time: 180.00s
Concurrent users: 10
Total requests: 1636
Successful requests: 1636
Failed requests: 0
Success rate: 100.0%
Requests per second: 9.09

Response Time Statistics:
  Mean: 1.095s
  Median: 1.721s
  Min: 0.136s
  Max: 3.218s
  Std Dev: 0.762s

Percentiles:
  P50: 1.721s
  P90: 1.751s
  P95: 1.756s
  P99: 1.796s

Time to First Token (TTFT) Statistics:
  Mean: 0.037s
  Median: 0.037s
  Min: 0.023s
  Max: 0.211s
  Std Dev: 0.011s

TTFT Percentiles:
  P50: 0.037s
  P90: 0.040s
  P95: 0.044s
  P99: 0.055s

Streaming Statistics:
  Mean chunks per response: 64.0
  Total chunks received: 104775
2025-08-15 11:24:29 -07:00
..
eks fix: update k8s templates (#2645) 2025-07-08 15:57:01 -07:00
k8s chore: setup for performance benchmarking (#3096) 2025-08-13 10:58:22 -07:00
k8s-benchmark test: benchmark scripts (#3160) 2025-08-15 11:24:29 -07:00
ondevice_distro docs: remove pure venv references (#3047) 2025-08-06 10:42:34 -07:00
remote_hosted_distro refactor: remove Conda support from Llama Stack (#2969) 2025-08-02 15:52:59 -07:00
self_hosted_distro docs: fix the docs for NVIDIA Inference Provider (#3055) 2025-08-08 11:27:55 +02:00
building_distro.md fix(docs): update llama stack build CLI doc (#3050) 2025-08-06 09:32:09 -07:00
configuration.md refactor: remove Conda support from Llama Stack (#2969) 2025-08-02 15:52:59 -07:00
customizing_run_yaml.md docs: clarify run.yaml files are starting points for customization (#2746) 2025-07-14 09:53:13 -07:00
importing_as_library.md chore: rename templates to distributions (#3035) 2025-08-04 11:34:17 -07:00
index.md docs: part 1 - fix warnings in documentation generation (#2861) 2025-07-30 10:50:10 -07:00
list_of_distributions.md fix: Restore the nvidia distro (#2639) 2025-07-07 15:50:05 -07:00
starting_llama_stack_server.md refactor: remove Conda support from Llama Stack (#2969) 2025-08-02 15:52:59 -07:00