phoenix/litellm-mirror
0
0
Fork 1
mirror of https://github.com/BerriAI/litellm.git synced 2025-04-26 11:14:04 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

(docs) Proxy add tutorial on using multiple llms

Browse source
This commit is contained in:
ishaan-jaff 2023-10-09 16:21:39 -07:00
parent b14bda6e1a
commit 2004b449e8
1 changed files with 34 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

41
docs/my-website/docs/proxy_server.md
View file

@ -5,7 +5,7 @@ import TabItem from '@theme/TabItem';
A local, fast, and lightweight OpenAI-compatible server to call 100+ LLM APIs. A local, fast, and lightweight OpenAI-compatible server to call 100+ LLM APIs.
## usage ## Usage
```shell ```shell
pip install litellm pip install litellm
``` ```
@ -15,13 +15,13 @@ $ litellm --model ollama/codellama
#INFO: Ollama running on http://0.0.0.0:8000 #INFO: Ollama running on http://0.0.0.0:8000
``` ```
### test ### Test
In a new shell, run: In a new shell, run:
```shell ```shell
$ litellm --test $ litellm --test
``` ```
### replace openai base ### Replace openai base
```python ```python
import openai import openai
@ -132,7 +132,7 @@ $ litellm --model command-nightly
[**Jump to Code**](https://github.com/BerriAI/litellm/blob/fef4146396d5d87006259e00095a62e3900d6bb4/litellm/proxy.py#L36) [**Jump to Code**](https://github.com/BerriAI/litellm/blob/fef4146396d5d87006259e00095a62e3900d6bb4/litellm/proxy.py#L36)
## [tutorial]: Use with Aider/AutoGen/Continue-Dev ## [Tutorial]: Use with Aider/AutoGen/Continue-Dev
Here's how to use the proxy to test codellama/mistral/etc. models for different github repos Here's how to use the proxy to test codellama/mistral/etc. models for different github repos
@ -207,7 +207,8 @@ Credits [@victordibia](https://github.com/microsoft/autogen/issues/45#issuecomme
::: :::
## Configure Model ## Advanced
### Configure Model
To save api keys and/or customize model prompt, run: To save api keys and/or customize model prompt, run:
```shell ```shell
@ -223,7 +224,33 @@ litellm --model ollama/llama2 \
--temperature 0.5 --temperature 0.5
``` ```
## Ollama Logs ### Create a proxy for multiple LLMs
```shell
$ litellm
#INFO: litellm proxy running on http://0.0.0.0:8000
```
#### Send a request to your proxy
```python
import openai
openai.api_key = "any-string-here"
openai.api_base = "http://0.0.0.0:8080" # your proxy url
# call gpt-3.5-turbo
response = openai.ChatCompletion.create(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey"}])
print(response)
# call ollama/llama2
response = openai.ChatCompletion.create(model="ollama/llama2", messages=[{"role": "user", "content": "Hey"}])
print(response)
```
### Ollama Logs
Ollama calls can sometimes fail (out-of-memory errors, etc.). Ollama calls can sometimes fail (out-of-memory errors, etc.).
To see your logs just call To see your logs just call
@ -234,7 +261,7 @@ $ curl 'http://0.0.0.0:8000/ollama_logs'
This will return your logs from `~/.ollama/logs/server.log`. This will return your logs from `~/.ollama/logs/server.log`.
## Deploy Proxy ### Deploy Proxy
<Tabs> <Tabs>
<TabItem value="self-hosted" label="Self-Hosted"> <TabItem value="self-hosted" label="Self-Hosted">