forked from phoenix/litellm-mirror
72 lines
2.5 KiB
Markdown
72 lines
2.5 KiB
Markdown
import Image from '@theme/IdealImage';
|
|
|
|
# Deploy & Query Llama2-7B on Sagemaker
|
|
|
|
This tutorial has 2 major components:
|
|
1. Deploy Llama2-7B on Jumpstart
|
|
2. Use LiteLLM to Query Llama2-7B on Sagemaker
|
|
|
|
## Deploying Llama2-7B on AWS Sagemaker
|
|
### Pre-requisites
|
|
Ensure you have AWS quota for deploying your selected LLM. You can apply for a quota increase here: https://console.aws.amazon.com/servicequotas/home
|
|
* ml.g5.48xlarge
|
|
* ml.g5.2xlarge
|
|
|
|
### Create an Amazon SageMaker domain to use Studio and Studio Notebooks
|
|
|
|
- Head to AWS console https://aws.amazon.com/console/
|
|
- Navigate to AWS Sagemaker from the console
|
|
- On AWS Sagemaker select 'Create a Sagemaker Domain'
|
|
<Image img={require('../../img/sagemaker_domain.png')} />
|
|
|
|
### Deploying Llama2-7B using AWS Sagemaker Jumpstart
|
|
|
|
- After creating your sagemaker domain, click 'Open Studio', which should take you to AWS sagemaker studio
|
|
|
|
- On the left sidebar navigate to SageMaker Jumpstart -> Models, notebooks, solutions
|
|
- Now select the LLM you want to deploy by clicking 'View Model' - (in this case select Llama2-7B)
|
|
<Image img={require('../../img/sagemaker_jumpstart.png')} />
|
|
- Click `Deploy` for the Model you want to deploy
|
|
<Image img={require('../../img/sagemaker_deploy.png')} />
|
|
|
|
- After deploying Llama2, copy your model endpoint
|
|
<Image img={require('../../img/sagemaker_endpoint.png')} />
|
|
|
|
### Use LiteLLM to Query Llama2-7B on Sagemaker
|
|
|
|
#### Prerequisites
|
|
* `pip install boto3`
|
|
* `pip install litellm`
|
|
* Create your AWS Access Key, get your `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. You can create a new aws access key on the aws console under `Security Credentials` under your profile
|
|
|
|
#### Querying deployed Llama2-7b
|
|
Set `model` = `sagemaker/<your model endpoint>` for `completion`. Use the model endpoint you got after deploying llama2-7b on sagemaker. If you used jumpstart your model endpoint will look like this `jumpstart-dft-meta-textgeneration-llama-2-7b`
|
|
|
|
Code Example:
|
|
```python
|
|
from litellm import completion
|
|
os.environ['AWS_ACCESS_KEY_ID'] = "your-access-key-id"
|
|
os.environ['AWS_SECRET_ACCESS_KEY'] = "your-secret-key"
|
|
|
|
response = completion(
|
|
model="sagemaker/jumpstart-dft-meta-textgeneration-llama-2-7b",
|
|
messages=[{'role': 'user', 'content': 'are you a llama'}],
|
|
temperature=0.2, # optional params
|
|
max_tokens=80,
|
|
)
|
|
|
|
```
|
|
|
|
That's it! Happy completion()!
|
|
|
|
#### Next Steps:
|
|
- Add Caching: https://docs.litellm.ai/docs/caching/gpt_cache
|
|
- Add Logging and Observability to your deployed LLM: https://docs.litellm.ai/docs/observability/callbacks
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|