forked from phoenix/litellm-mirror
docs - fix doc build time errors
This commit is contained in:
parent
3a35657420
commit
69a20c94fd
5 changed files with 4 additions and 93 deletions
|
@ -162,7 +162,7 @@ def completion(
|
||||||
|
|
||||||
- `function`: *object* - Required.
|
- `function`: *object* - Required.
|
||||||
|
|
||||||
- `tool_choice`: *string or object (optional)* - Controls which (if any) function is called by the model. none means the model will not call a function and instead generates a message. auto means the model can pick between generating a message or calling a function. Specifying a particular function via {"type: "function", "function": {"name": "my_function"}} forces the model to call that function.
|
- `tool_choice`: *string or object (optional)* - Controls which (if any) function is called by the model. none means the model will not call a function and instead generates a message. auto means the model can pick between generating a message or calling a function. Specifying a particular function via `{"type: "function", "function": {"name": "my_function"}}` forces the model to call that function.
|
||||||
|
|
||||||
- `none` is the default when no functions are present. `auto` is the default if functions are present.
|
- `none` is the default when no functions are present. `auto` is the default if functions are present.
|
||||||
|
|
||||||
|
|
|
@ -1,90 +0,0 @@
|
||||||
import Image from '@theme/IdealImage';
|
|
||||||
import QueryParamReader from '../../src/components/queryParamReader.js'
|
|
||||||
|
|
||||||
# [Beta] Monitor Logs in Production
|
|
||||||
|
|
||||||
:::note
|
|
||||||
|
|
||||||
This is in beta. Expect frequent updates, as we improve based on your feedback.
|
|
||||||
|
|
||||||
:::
|
|
||||||
|
|
||||||
LiteLLM provides an integration to let you monitor logs in production.
|
|
||||||
|
|
||||||
👉 Jump to our sample LiteLLM Dashboard: https://admin.litellm.ai/
|
|
||||||
|
|
||||||
|
|
||||||
<Image img={require('../../img/alt_dashboard.png')} alt="Dashboard" />
|
|
||||||
|
|
||||||
## Debug your first logs
|
|
||||||
<a target="_blank" href="https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/liteLLM_OpenAI.ipynb">
|
|
||||||
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
|
|
||||||
</a>
|
|
||||||
|
|
||||||
|
|
||||||
### 1. Get your LiteLLM Token
|
|
||||||
|
|
||||||
Go to [admin.litellm.ai](https://admin.litellm.ai/) and copy the code snippet with your unique token
|
|
||||||
|
|
||||||
<Image img={require('../../img/hosted_debugger_usage_page.png')} alt="Usage" />
|
|
||||||
|
|
||||||
### 2. Set up your environment
|
|
||||||
|
|
||||||
**Add it to your .env**
|
|
||||||
|
|
||||||
```python
|
|
||||||
import os
|
|
||||||
|
|
||||||
os.env["LITELLM_TOKEN"] = "e24c4c06-d027-4c30-9e78-18bc3a50aebb" # replace with your unique token
|
|
||||||
|
|
||||||
```
|
|
||||||
|
|
||||||
**Turn on LiteLLM Client**
|
|
||||||
```python
|
|
||||||
import litellm
|
|
||||||
litellm.client = True
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Make a normal `completion()` call
|
|
||||||
```python
|
|
||||||
import litellm
|
|
||||||
from litellm import completion
|
|
||||||
import os
|
|
||||||
|
|
||||||
# set env variables
|
|
||||||
os.environ["LITELLM_TOKEN"] = "e24c4c06-d027-4c30-9e78-18bc3a50aebb" # replace with your unique token
|
|
||||||
os.environ["OPENAI_API_KEY"] = "openai key"
|
|
||||||
|
|
||||||
litellm.use_client = True # enable logging dashboard
|
|
||||||
messages = [{ "content": "Hello, how are you?","role": "user"}]
|
|
||||||
|
|
||||||
# openai call
|
|
||||||
response = completion(model="gpt-3.5-turbo", messages=messages)
|
|
||||||
```
|
|
||||||
|
|
||||||
Your `completion()` call print with a link to your session dashboard (https://admin.litellm.ai/<your_unique_token>)
|
|
||||||
|
|
||||||
In the above case it would be: [`admin.litellm.ai/e24c4c06-d027-4c30-9e78-18bc3a50aebb`](https://admin.litellm.ai/e24c4c06-d027-4c30-9e78-18bc3a50aebb)
|
|
||||||
|
|
||||||
Click on your personal dashboard link. Here's how you can find it 👇
|
|
||||||
|
|
||||||
<Image img={require('../../img/dash_output.png')} alt="Dashboard" />
|
|
||||||
|
|
||||||
[👋 Tell us if you need better privacy controls](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version?month=2023-08)
|
|
||||||
|
|
||||||
### 3. Review request log
|
|
||||||
|
|
||||||
Oh! Looks like our request was made successfully. Let's click on it and see exactly what got sent to the LLM provider.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Ah! So we can see that this request was made to a **Baseten** (see litellm_params > custom_llm_provider) for a model with ID - **7qQNLDB** (see model). The message sent was - `"Hey, how's it going?"` and the response received was - `"As an AI language model, I don't have feelings or emotions, but I can assist you with your queries. How can I assist you today?"`
|
|
||||||
|
|
||||||
<Image img={require('../../img/dashboard_log.png')} alt="Dashboard Log Row" />
|
|
||||||
|
|
||||||
:::info
|
|
||||||
|
|
||||||
🎉 Congratulations! You've successfully debugger your first log!
|
|
||||||
|
|
||||||
:::
|
|
|
@ -208,7 +208,7 @@ print(response)
|
||||||
|
|
||||||
Instead of using the `custom_llm_provider` arg to specify which provider you're using (e.g. together ai), you can just pass the provider name as part of the model name, and LiteLLM will parse it out.
|
Instead of using the `custom_llm_provider` arg to specify which provider you're using (e.g. together ai), you can just pass the provider name as part of the model name, and LiteLLM will parse it out.
|
||||||
|
|
||||||
Expected format: <custom_llm_provider>/<model_name>
|
Expected format: `<custom_llm_provider>/<model_name>`
|
||||||
|
|
||||||
e.g. completion(model="together_ai/togethercomputer/Llama-2-7B-32K-Instruct", ...)
|
e.g. completion(model="together_ai/togethercomputer/Llama-2-7B-32K-Instruct", ...)
|
||||||
|
|
||||||
|
|
|
@ -669,7 +669,7 @@ Once the stack is created, get the DatabaseURL of the Database resource, copy th
|
||||||
#### 3. Connect to the EC2 Instance and deploy litellm on the EC2 container
|
#### 3. Connect to the EC2 Instance and deploy litellm on the EC2 container
|
||||||
From the EC2 console, connect to the instance created by the stack (e.g., using SSH).
|
From the EC2 console, connect to the instance created by the stack (e.g., using SSH).
|
||||||
|
|
||||||
Run the following command, replacing <database_url> with the value you copied in step 2
|
Run the following command, replacing `<database_url>` with the value you copied in step 2
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
docker run --name litellm-proxy \
|
docker run --name litellm-proxy \
|
||||||
|
|
|
@ -26,6 +26,7 @@ print(response)
|
||||||
```
|
```
|
||||||
|
|
||||||
{'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'role': 'assistant', 'content': "\n\nI'm not able to provide real-time weather information. However, I can suggest"}}], 'created': 1691629657.9288375, 'model': 'togethercomputer/llama-2-70b-chat', 'usage': {'prompt_tokens': 9, 'completion_tokens': 17, 'total_tokens': 26}}
|
{'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'role': 'assistant', 'content': "\n\nI'm not able to provide real-time weather information. However, I can suggest"}}], 'created': 1691629657.9288375, 'model': 'togethercomputer/llama-2-70b-chat', 'usage': {'prompt_tokens': 9, 'completion_tokens': 17, 'total_tokens': 26}}
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
LiteLLM handles the prompt formatting for Together AI's Llama2 models as well, converting your message to the
|
LiteLLM handles the prompt formatting for Together AI's Llama2 models as well, converting your message to the
|
||||||
|
|
Loading…
Add table
Add a link
Reference in a new issue