phoenix/litellm
1
0
Fork 0
forked from phoenix/litellm-mirror
Code Pull requests Activity

docs add arch diagram

Browse source
This commit is contained in:
Ishaan Jaff 2024-09-07 16:21:29 -07:00
parent 54db564529
commit c2c63e4dbe
1 changed files with 17 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

28
docs/my-website/docs/proxy/architecture.md
View file

@ -13,19 +13,25 @@ import TabItem from '@theme/TabItem';
1. **User Sends Request**: The process begins when a user sends a request to the LiteLLM Proxy Server (Gateway).
2. [**Virtual Keys**](../virtual_keys): The request first passes through the Virtual Keys component
2. [**Virtual Keys**](../virtual_keys): At this stage the `Bearer` token in the request is checked to ensure it is valid and under it's budget
3. **Rate Limiting**: The MaxParallelRequestsHandler applies rate limiting to manage the flow of requests.
3. **Rate Limiting**: The [MaxParallelRequestsHandler](https://github.com/BerriAI/litellm/blob/main/litellm/proxy/hooks/parallel_request_limiter.py) checks the **rate limit (rpm/tpm)** for the the following components:
- Global Server Rate Limit
- Virtual Key Rate Limit
- User Rate Limit
- Team Limit
4. **Proxy Server Processing**: The request is then processed by the LiteLLM proxy_server.py, which handles the core logic of the proxy.
4. **LiteLLM `proxy_server.py`**: Contains the `/chat/completions` and `/embeddings` endpoints. Requests to these endpoints are sent through the LiteLLM Router
5. [**LiteLLM Router**](../routing): LiteLLM Router**: The LiteLLM Router determines where to send the request based on the configuration and request parameters.
5. [**LiteLLM Router**](../routing): The LiteLLM Router handles Load balancing, Fallbacks, Retries for LLM API deployments.
6. **Model Interaction**: The request is sent to the appropriate model API (litellm.completion() or litellm.embedding()) for processing.
6. [**litellm.completion() / litellm.embedding()**:](../index#litellm-python-sdk) The litellm Python SDK is used to call the LLM in the OpenAI API format (Translation and parameter mapping)
7. **Response**: The model's response is sent back through the same components to the user.
8. **Post-Request Processing**: After the response is sent, several asynchronous operations occur:
- The _PROXY_track_cost_callback updates spend in the database.
- Logging to LangFuse for analytics and monitoring.
- The MaxParallelRequestsHandler updates virtual key usage and performs post-request cleanup.
7. **Post-Request Processing**: After the response is sent back to the client, the following **asynchronous** tasks are performed:
- [Logging to LangFuse (logging destination is configurable)](./logging)
- The [MaxParallelRequestsHandler](https://github.com/BerriAI/litellm/blob/main/litellm/proxy/hooks/parallel_request_limiter.py) updates the rpm/tpm usage for the
- Global Server Rate Limit
- Virtual Key Rate Limit
- User Rate Limit
- Team Limit
- The `_PROXY_track_cost_callback` updates spend / usage in the LiteLLM database.