phoenix/litellm
1
0
Fork 0
forked from phoenix/litellm-mirror
Code Pull requests Activity

readme proxy

Browse source
This commit is contained in:
Ishaan Jaff 2023-11-08 13:28:39 -08:00 • committed by GitHub
parent 3498e881af
commit f00c2e6c16
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 13 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

20
README.md
View file

@ -74,17 +74,23 @@ result = completion('claude-2', messages, stream=True)
for chunk in result: for chunk in result:
print(chunk['choices'][0]['delta']) print(chunk['choices'][0]['delta'])
``` ```
## OpenAI Proxy
Use LiteLLM in any OpenAI API compatible project
## Reliability - Fallback LLMs ```shell
Never fail a request using LiteLLM $ litellm --model huggingface/bigcode/starcoder
#INFO: Proxy running on http://0.0.0.0:8000
```
### Replace openai base
```python ```python
from litellm import completion import openai
# if gpt-4 fails, retry the request with gpt-3.5-turbo->command-nightly->claude-instant-1
response = completion(model="gpt-4",messages=messages, fallbacks=["gpt-3.5-turbo", "command-nightly", "claude-instant-1"])
# if azure/gpt-4 fails, retry the request with fallback api_keys/api_base openai.api_base = "http://0.0.0.0:8000"
response = completion(model="azure/gpt-4", messages=messages, api_key=api_key, fallbacks=[{"api_key": "good-key-1"}, {"api_key": "good-key-2", "api_base": "good-api-base-2"}])
print(openai.ChatCompletion.create(model="test", messages=[{"role":"user", "content":"Hey!"}]))
``` ```
## Logging Observability - Log LLM Input/Output ([Docs](https://docs.litellm.ai/docs/observability/callbacks)) ## Logging Observability - Log LLM Input/Output ([Docs](https://docs.litellm.ai/docs/observability/callbacks))