forked from phoenix/litellm-mirror
(Doc) Add section on what is stored in the DB + Add clear section on key/team based logging (#6769)
* fix clean up logging * add Default Login, Logout URLs * docs on tables in DB schema * docs add section on What is stored in the DB
This commit is contained in:
Ishaan Jaff
GitHub
parent
9cf3dcbbf3
commit
627d6a6614
4 changed files with 102 additions and 58 deletions
67
docs/my-website/docs/proxy/db_info.md
Normal file
67
docs/my-website/docs/proxy/db_info.md
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
# What is stored in the DB
|
||||
|
||||
The LiteLLM Proxy uses a PostgreSQL database to store various information. Here's are the main features the DB is used for:
|
||||
- Virtual Keys, Organizations, Teams, Users, Budgets, and more.
|
||||
- Per request Usage Tracking
|
||||
|
||||
## Link to DB Schema
|
||||
|
||||
You can see the full DB Schema [here](https://github.com/BerriAI/litellm/blob/main/schema.prisma)
|
||||
|
||||
## DB Tables
|
||||
|
||||
### Organizations, Teams, Users, End Users
|
||||
|
||||
| Table Name | Description | Row Insert Frequency |
|
||||
|------------|-------------|---------------------|
|
||||
| LiteLLM_OrganizationTable | Manages organization-level configurations. Tracks organization spend, model access, and metadata. Links to budget configurations and teams. | Low |
|
||||
| LiteLLM_TeamTable | Handles team-level settings within organizations. Manages team members, admins, and their roles. Controls team-specific budgets, rate limits, and model access. | Low |
|
||||
| LiteLLM_UserTable | Stores user information and their settings. Tracks individual user spend, model access, and rate limits. Manages user roles and team memberships. | Low |
|
||||
| LiteLLM_EndUserTable | Manages end-user configurations. Controls model access and regional requirements. Tracks end-user spend. | Low |
|
||||
| LiteLLM_TeamMembership | Tracks user participation in teams. Manages team-specific user budgets and spend. | Low |
|
||||
| LiteLLM_OrganizationMembership | Manages user roles within organizations. Tracks organization-specific user permissions and spend. | Low |
|
||||
| LiteLLM_InvitationLink | Handles user invitations. Manages invitation status and expiration. Tracks who created and accepted invitations. | Low |
|
||||
| LiteLLM_UserNotifications | Handles model access requests. Tracks user requests for model access. Manages approval status. | Low |
|
||||
|
||||
### Authentication
|
||||
|
||||
| Table Name | Description | Row Insert Frequency |
|
||||
|------------|-------------|---------------------|
|
||||
| LiteLLM_VerificationToken | Manages Virtual Keys and their permissions. Controls token-specific budgets, rate limits, and model access. Tracks key-specific spend and metadata. | **Medium** - stores all Virtual Keys |
|
||||
|
||||
### Model (LLM) Management
|
||||
|
||||
| Table Name | Description | Row Insert Frequency |
|
||||
|------------|-------------|---------------------|
|
||||
| LiteLLM_ProxyModelTable | Stores model configurations. Defines available models and their parameters. Contains model-specific information and settings. | Low - Configuration only |
|
||||
|
||||
### Budget Management
|
||||
|
||||
| Table Name | Description | Row Insert Frequency |
|
||||
|------------|-------------|---------------------|
|
||||
| LiteLLM_BudgetTable | Stores budget and rate limit configurations for organizations, keys, and end users. Tracks max budgets, soft budgets, TPM/RPM limits, and model-specific budgets. Handles budget duration and reset timing. | Low - Configuration only |
|
||||
|
||||
|
||||
### Tracking & Logging
|
||||
|
||||
| Table Name | Description | Row Insert Frequency |
|
||||
|------------|-------------|---------------------|
|
||||
| LiteLLM_SpendLogs | Detailed logs of all API requests. Records token usage, spend, and timing information. Tracks which models and keys were used. | **High - every LLM API request** |
|
||||
| LiteLLM_ErrorLogs | Captures failed requests and errors. Stores exception details and request information. Helps with debugging and monitoring. | **Medium - on errors only** |
|
||||
| LiteLLM_AuditLog | Tracks changes to system configuration. Records who made changes and what was modified. Maintains history of updates to teams, users, and models. | **Off by default**, **High - when enabled** |
|
||||
|
||||
## How to Disable `LiteLLM_SpendLogs`
|
||||
|
||||
You can disable spend_logs by setting `disable_spend_logs` to `True` on the `general_settings` section of your proxy_config.yaml file.
|
||||
|
||||
```yaml
|
||||
general_settings:
|
||||
disable_spend_logs: True
|
||||
```
|
||||
|
||||
|
||||
### What is the impact of disabling `LiteLLM_SpendLogs`?
|
||||
|
||||
- You **will not** be able to view Usage on the LiteLLM UI
|
||||
- You **will** continue seeing cost metrics on s3, Prometheus, Langfuse (any other Logging integration you are using)
|
||||
|
||||
|
|
@ -61,7 +61,7 @@ litellm_settings:
|
|||
|
||||
Removes any field with `user_api_key_*` from metadata.
|
||||
|
||||
## What gets logged? StandardLoggingPayload
|
||||
## What gets logged?
|
||||
|
||||
Found under `kwargs["standard_logging_object"]`. This is a standard payload, logged for every response.
|
||||
|
||||
|
|
@ -148,6 +148,11 @@ class StandardLoggingModelCostFailureDebugInformation(TypedDict, total=False):
|
|||
custom_pricing: Optional[bool]
|
||||
```
|
||||
|
||||
## Conditional Logging for Virtual Keys / Teams
|
||||
|
||||
[👉 Tutorial - Allow each team to use their own Langfuse Project / custom callbacks](team_logging)
|
||||
|
||||
|
||||
## Langfuse
|
||||
|
||||
We will use the `--config` to set `litellm.success_callback = ["langfuse"]` this will log all successfull LLM calls to langfuse. Make sure to set `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` in your environment
|
||||
|
|
@ -300,40 +305,7 @@ print(response)
|
|||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### Team based Logging to Langfuse
|
||||
|
||||
[👉 Tutorial - Allow each team to use their own Langfuse Project / custom callbacks](team_logging)
|
||||
<!--
|
||||
|
||||
**Example:**
|
||||
|
||||
This config would send langfuse logs to 2 different langfuse projects, based on the team id
|
||||
|
||||
```yaml
|
||||
litellm_settings:
|
||||
default_team_settings:
|
||||
- team_id: my-secret-project
|
||||
success_callback: ["langfuse"]
|
||||
langfuse_public_key: os.environ/LANGFUSE_PUB_KEY_1 # Project 1
|
||||
langfuse_secret: os.environ/LANGFUSE_PRIVATE_KEY_1 # Project 1
|
||||
- team_id: ishaans-secret-project
|
||||
success_callback: ["langfuse"]
|
||||
langfuse_public_key: os.environ/LANGFUSE_PUB_KEY_2 # Project 2
|
||||
langfuse_secret: os.environ/LANGFUSE_SECRET_2 # Project 2
|
||||
```
|
||||
|
||||
Now, when you [generate keys](./virtual_keys.md) for this team-id
|
||||
|
||||
```bash
|
||||
curl -X POST 'http://0.0.0.0:4000/key/generate' \
|
||||
-H 'Authorization: Bearer sk-1234' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"team_id": "ishaans-secret-project"}'
|
||||
```
|
||||
|
||||
All requests made with these keys will log data to their team-specific logging. -->
|
||||
|
||||
### Redacting Messages, Response Content from Langfuse Logging
|
||||
### Redact Messages, Response Content
|
||||
|
||||
Set `litellm.turn_off_message_logging=True` This will prevent the messages and responses from being logged to langfuse, but request metadata will still be logged.
|
||||
|
||||
|
|
@ -366,7 +338,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
|
|||
```
|
||||
|
||||
|
||||
### LiteLLM-specific Tags on Langfuse - `cache_hit`, `cache_key`
|
||||
### LiteLLM Tags - `cache_hit`, `cache_key`
|
||||
|
||||
Use this if you want to control which LiteLLM-specific fields are logged as tags by the LiteLLM proxy. By default LiteLLM Proxy logs no LiteLLM-specific fields
|
||||
|
||||
|
|
@ -401,7 +373,7 @@ litellm_settings:
|
|||
langfuse_default_tags: ["cache_hit", "cache_key", "proxy_base_url", "user_api_key_alias", "user_api_key_user_id", "user_api_key_user_email", "user_api_key_team_alias", "semantic-similarity", "proxy_base_url"]
|
||||
```
|
||||
|
||||
### 🔧 Debugging - Viewing RAW CURL sent from LiteLLM to provider
|
||||
### View POST sent from LiteLLM to provider
|
||||
|
||||
Use this when you want to view the RAW curl request sent from LiteLLM to the LLM API
|
||||
|
||||
|
|
@ -504,7 +476,7 @@ You will see `raw_request` in your Langfuse Metadata. This is the RAW CURL comma
|
|||
|
||||
<Image img={require('../../img/debug_langfuse.png')} />
|
||||
|
||||
## OpenTelemetry format
|
||||
## OpenTelemetry
|
||||
|
||||
:::info
|
||||
|
||||
|
|
@ -786,7 +758,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
|
|||
|
||||
** 🎉 Expect to see this trace logged in your OTEL collector**
|
||||
|
||||
### Redacting Messages, Response Content from OTEL Logging
|
||||
### Redacting Messages, Response Content
|
||||
|
||||
Set `message_logging=False` for `otel`, no messages / response will be logged
|
||||
|
||||
|
|
@ -800,7 +772,8 @@ callback_settings:
|
|||
message_logging: False
|
||||
```
|
||||
|
||||
### Context propagation across Services `Traceparent HTTP Header`
|
||||
### Traceparent Header
|
||||
##### Context propagation across Services `Traceparent HTTP Header`
|
||||
|
||||
❓ Use this when you want to **pass information about the incoming request in a distributed tracing system**
|
||||
|
||||
|
|
@ -850,7 +823,7 @@ Search for Trace=`80e1afed08e019fc1110464cfa66635c` on your OTEL Collector
|
|||
|
||||
<Image img={require('../../img/otel_parent.png')} />
|
||||
|
||||
### Forwarding `Traceparent HTTP Header` to LLM APIs
|
||||
##### Forwarding `Traceparent HTTP Header` to LLM APIs
|
||||
|
||||
Use this if you want to forward the traceparent headers to your self hosted LLMs like vLLM
|
||||
|
||||
|
|
@ -1095,7 +1068,7 @@ class MyCustomHandler(CustomLogger):
|
|||
{'mode': 'embedding', 'input_cost_per_token': 0.002}
|
||||
```
|
||||
|
||||
### Logging responses from proxy
|
||||
##### Logging responses from proxy
|
||||
|
||||
Both `/chat/completions` and `/embeddings` responses are available as `response_obj`
|
||||
|
||||
|
|
@ -1413,7 +1386,7 @@ export GALILEO_USERNAME=""
|
|||
export GALILEO_PASSWORD=""
|
||||
```
|
||||
|
||||
### Quick Start
|
||||
#### Quick Start
|
||||
|
||||
1. Add to Config.yaml
|
||||
|
||||
|
|
@ -1454,7 +1427,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
|
|||
|
||||
🎉 That's it - Expect to see your Logs on your Galileo Dashboard
|
||||
|
||||
## Logging Proxy Cost + Usage - OpenMeter
|
||||
## OpenMeter
|
||||
|
||||
Bill customers according to their LLM API usage with [OpenMeter](../observability/openmeter.md)
|
||||
|
||||
|
|
@ -1466,7 +1439,7 @@ export OPENMETER_API_ENDPOINT="" # defaults to https://openmeter.cloud
|
|||
export OPENMETER_API_KEY=""
|
||||
```
|
||||
|
||||
### Quick Start
|
||||
##### Quick Start
|
||||
|
||||
1. Add to Config.yaml
|
||||
|
||||
|
|
@ -1799,7 +1772,10 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
|
|||
}'
|
||||
```
|
||||
|
||||
## (BETA) Moderation with Azure Content Safety
|
||||
|
||||
<!-- ## (BETA) Moderation with Azure Content Safety
|
||||
|
||||
Note: This page is for logging callbacks and this is a moderation service. Commenting until we found a better location for this.
|
||||
|
||||
[Azure Content-Safety](https://azure.microsoft.com/en-us/products/ai-services/ai-content-safety) is a Microsoft Azure service that provides content moderation APIs to detect potential offensive, harmful, or risky content in text.
|
||||
|
||||
|
|
@ -1884,4 +1860,4 @@ litellm_settings:
|
|||
:::info
|
||||
`thresholds` are not required by default, but you can tune the values to your needs.
|
||||
Default values is `4` for all categories
|
||||
:::
|
||||
::: -->
|
||||
|
|
@ -64,7 +64,7 @@ Allow others to create/delete their own keys.
|
|||
Features here are behind a commercial license in our `/enterprise` folder. [**See Code**](https://github.com/BerriAI/litellm/tree/main/enterprise)
|
||||
|
||||
|
||||
### Setup SSO/Auth for UI
|
||||
### SSO for UI
|
||||
|
||||
#### Step 1: Set upperbounds for keys
|
||||
Control the upperbound that users can use for `max_budget`, `budget_duration` or any `key/generate` param per key.
|
||||
|
|
@ -88,12 +88,6 @@ litellm_settings:
|
|||
|
||||
#### Step 2: Setup Oauth Client
|
||||
|
||||
:::tip
|
||||
|
||||
Looking for how to use Oauth 2.0 for /chat, /completions API requests to the proxy? [Follow this doc](oauth2)
|
||||
|
||||
:::
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="okta" label="Okta SSO">
|
||||
|
||||
|
|
@ -196,6 +190,13 @@ GENERIC_SCOPE = "openid profile email" # default scope openid is sometimes not e
|
|||
|
||||
</Tabs>
|
||||
|
||||
### Default Login, Logout URLs
|
||||
|
||||
Some SSO providers require a specific redirect url for login and logout. You can input the following values.
|
||||
|
||||
- Login: `<your-proxy-base-url>/sso/key/generate`
|
||||
- Logout: `<your-proxy-base-url>`
|
||||
|
||||
#### Step 3. Set `PROXY_BASE_URL` in your .env
|
||||
|
||||
Set this in your .env (so the proxy can set the correct redirect url)
|
||||
|
|
@ -216,9 +217,9 @@ export ALLOWED_EMAIL_DOMAINS="berri.ai"
|
|||
|
||||
This will check if the user email we receive from SSO contains this domain, before allowing access.
|
||||
|
||||
### Set Admin view w/ SSO
|
||||
### Set Proxy Admin
|
||||
|
||||
You just need to set Proxy Admin ID
|
||||
Set a Proxy Admin when SSO is enabled. Once SSO is enabled, the `user_id` for users is retrieved from the SSO provider. In order to set a Proxy Admin, you need to copy the `user_id` from the UI and set it in your `.env` as `PROXY_ADMIN_ID`.
|
||||
|
||||
#### Step 1: Copy your ID from the UI
|
||||
|
||||
|
|
@ -256,7 +257,7 @@ general_settings:
|
|||
default_team_disabled: true # OR you can set env var PROXY_DEFAULT_TEAM_DISABLED="true"
|
||||
```
|
||||
|
||||
### Sign in with Username, Password when SSO is on
|
||||
### Use Username, Password when SSO is on
|
||||
|
||||
If you need to access the UI via username/password when SSO is on navigate to `/fallback/login`. This route will allow you to sign in with your username/password credentials.
|
||||
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ const sidebars = {
|
|||
{
|
||||
type: "category",
|
||||
label: "Architecture",
|
||||
items: ["proxy/architecture"],
|
||||
items: ["proxy/architecture", "proxy/db_info"],
|
||||
},
|
||||
{
|
||||
type: "link",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue