phoenix/litellm-mirror
0
0
Fork 1
mirror of https://github.com/BerriAI/litellm.git synced 2025-04-27 11:43:54 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

docs(bedrock.md): update docs to show how to use converse like route for internal proxy usage

Browse source 
Resolves https://github.com/BerriAI/litellm/issues/8085
This commit is contained in:
Krrish Dholakia 2025-01-29 21:00:45 -08:00
parent b3ac24de59
commit 49f1b5f4f5
1 changed files with 18 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

28
docs/my-website/docs/providers/bedrock.md
View file

@ -4,6 +4,15 @@ import TabItem from '@theme/TabItem';
# AWS Bedrock # AWS Bedrock
ALL Bedrock models (Anthropic, Meta, Mistral, Amazon, etc.) are Supported ALL Bedrock models (Anthropic, Meta, Mistral, Amazon, etc.) are Supported
| Property | Details |
|-------|-------|
| Description | Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs). |
| Provider Route on LiteLLM | `bedrock/`, [`bedrock/converse/`](#set-converse--invoke-route), [`bedrock/invoke/`](#set-invoke-route), [`bedrock/converse_like/`](#calling-via-internal-proxy) |
| Provider Doc | [Amazon Bedrock ↗](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) |
| Supported OpenAI Endpoints | `/chat/completions`, `/completions`, `/embeddings`, `/images/generations` |
| Pass-through Endpoint | [Supported](../pass_through/bedrock.md) |
LiteLLM requires `boto3` to be installed on your system for Bedrock requests LiteLLM requires `boto3` to be installed on your system for Bedrock requests
```shell ```shell
pip install boto3>=1.28.57 pip install boto3>=1.28.57
@ -1201,11 +1210,9 @@ response = completion(
aws_bedrock_client=bedrock, aws_bedrock_client=bedrock,
) )
``` ```
## Calling via Proxy ## Calling via Internal Proxy
Here's how to call bedrock via your internal proxy. Use the `bedrock/converse_like/model` endpoint to call bedrock converse model via your internal proxy.
This example uses Cloudflare's AI Gateway.
<Tabs> <Tabs>
<TabItem value="sdk" label="SDK"> <TabItem value="sdk" label="SDK">
@ -1214,10 +1221,11 @@ This example uses Cloudflare's AI Gateway.
from litellm import completion from litellm import completion
response = completion( response = completion(
model="anthropic.claude-3-sonnet-20240229-v1:0", model="bedrock/converse_like/some-model",
messages=[{"role": "user", "content": "What's AWS?"}], messages=[{"role": "user", "content": "What's AWS?"}],
extra_headers={"test": "hello world", "Authorization": "my-test-key"}, api_key="sk-1234",
api_base="https://gateway.ai.cloudflare.com/v1/<some-id>/test/aws-bedrock/bedrock-runtime/us-east-1", api_base="https://some-api-url/models",
extra_headers={"test": "hello world"},
) )
``` ```
@ -1230,8 +1238,8 @@ response = completion(
model_list: model_list:
- model_name: anthropic-claude - model_name: anthropic-claude
litellm_params: litellm_params:
model: anthropic.claude-3-sonnet-20240229-v1:0 model: bedrock/converse_like/some-model
api_base: https://gateway.ai.cloudflare.com/v1/<some-id>/test/aws-bedrock/bedrock-runtime/us-east-1 api_base: https://some-api-url/models
``` ```
2. Start proxy server 2. Start proxy server
@ -1266,7 +1274,7 @@ curl -X POST 'http://0.0.0.0:4000/chat/completions' \
**Expected Output URL** **Expected Output URL**
```bash ```bash
https://gateway.ai.cloudflare.com/v1/<some-id>/test/aws-bedrock/bedrock-runtime/us-east-1/model/anthropic.claude-3-sonnet-20240229-v1:0/converse https://some-api-url/models
``` ```
## Provisioned throughput models ## Provisioned throughput models