phoenix/litellm-mirror
0
0
Fork 1
mirror of https://github.com/BerriAI/litellm.git synced 2025-04-24 18:24:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

(docs) proxy performance

Browse source
This commit is contained in:
ishaan-jaff 2023-11-24 17:07:46 -08:00
parent ad1672b901
commit 111c7afaca
4 changed files with 113 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

12
docs/my-website/docs/simple_proxy.md
View file

@ -1077,6 +1077,18 @@ Expected output on Langfuse
<Image img={require('../img/langfuse_small.png')} />
## LiteLLM Proxy Performance
### Throughput - 30% Increase
LiteLLM proxy + Load Balancer gives **30% increase** in throughput compared to Raw OpenAI API
<Image img={require('../img/throughput.png')} />
### Latency Added - 0.00325 seconds
LiteLLM proxy adds **0.00325 seconds** latency as compared to using the Raw OpenAI API
<Image img={require('../img/latency.png')} />
## Proxy CLI Arguments

BIN
docs/my-website/img/latency.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 66 KiB

BIN
docs/my-website/img/throughput.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 71 KiB