mirror of
https://github.com/BerriAI/litellm.git
synced 2025-04-26 19:24:27 +00:00
docs(aws_sagemaker.md): cleanup sagemaker messages api docs
This commit is contained in:
parent
cd61ddc610
commit
74a85fac0e
1 changed files with 75 additions and 1 deletions
|
@ -415,7 +415,81 @@ response = completion(
|
||||||
You can also pass in your own [custom prompt template](../completion/prompt_formatting.md#format-prompt-yourself)
|
You can also pass in your own [custom prompt template](../completion/prompt_formatting.md#format-prompt-yourself)
|
||||||
|
|
||||||
|
|
||||||
### Completion Models
|
## Sagemaker Messages API
|
||||||
|
|
||||||
|
Use route `sagemaker_chat/*` to route to Sagemaker Messages API
|
||||||
|
|
||||||
|
```
|
||||||
|
model: sagemaker_chat/<your-endpoint-name>
|
||||||
|
```
|
||||||
|
|
||||||
|
<Tabs>
|
||||||
|
<TabItem value="sdk" label="SDK">
|
||||||
|
|
||||||
|
```python
|
||||||
|
import os
|
||||||
|
import litellm
|
||||||
|
from litellm import completion
|
||||||
|
|
||||||
|
litellm.set_verbose = True # 👈 SEE RAW REQUEST
|
||||||
|
|
||||||
|
os.environ["AWS_ACCESS_KEY_ID"] = ""
|
||||||
|
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
|
||||||
|
os.environ["AWS_REGION_NAME"] = ""
|
||||||
|
|
||||||
|
response = completion(
|
||||||
|
model="sagemaker_chat/<your-endpoint-name>",
|
||||||
|
messages=[{ "content": "Hello, how are you?","role": "user"}],
|
||||||
|
temperature=0.2,
|
||||||
|
max_tokens=80
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
</TabItem>
|
||||||
|
<TabItem value="proxy" label="PROXY">
|
||||||
|
|
||||||
|
#### 1. Setup config.yaml
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
model_list:
|
||||||
|
- model_name: "sagemaker-model"
|
||||||
|
litellm_params:
|
||||||
|
model: "sagemaker_chat/jumpstart-dft-hf-textgeneration1-mp-20240815-185614"
|
||||||
|
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
|
||||||
|
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
|
||||||
|
aws_region_name: os.environ/AWS_REGION_NAME
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 2. Start the proxy
|
||||||
|
|
||||||
|
```bash
|
||||||
|
litellm --config /path/to/config.yaml
|
||||||
|
```
|
||||||
|
#### 3. Test it
|
||||||
|
|
||||||
|
|
||||||
|
```shell
|
||||||
|
curl --location 'http://0.0.0.0:4000/chat/completions' \
|
||||||
|
--header 'Content-Type: application/json' \
|
||||||
|
--data ' {
|
||||||
|
"model": "sagemaker-model",
|
||||||
|
"messages": [
|
||||||
|
{
|
||||||
|
"role": "user",
|
||||||
|
"content": "what llm are you"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
'
|
||||||
|
```
|
||||||
|
|
||||||
|
[**👉 See OpenAI SDK/Langchain/Llamaindex/etc. examples**](../proxy/user_keys.md#chatcompletions)
|
||||||
|
|
||||||
|
</TabItem>
|
||||||
|
</Tabs>
|
||||||
|
|
||||||
|
|
||||||
|
## Completion Models
|
||||||
|
|
||||||
|
|
||||||
:::tip
|
:::tip
|
||||||
|
|
Loading…
Add table
Add a link
Reference in a new issue