forked from phoenix/litellm-mirror
(feat) use @google-cloud/vertexai
js sdk with litellm (#6873)
* stash gemini JS test * add vertex js sdj example * handle vertex pass through separately * tes vertex JS sdk * fix vertex_proxy_route * use PassThroughStreamingHandler * fix PassThroughStreamingHandler * use common _create_vertex_response_logging_payload_for_generate_content * test vertex js * add working vertex jest tests * move basic bass through test * use good name for test * test vertex * test_chunk_processor_yields_raw_bytes * unit tests for streaming * test_convert_raw_bytes_to_str_lines * run unit tests 1st * simplify local * docs add usage example for js * use get_litellm_virtual_key * add unit tests for vertex pass through
This commit is contained in:
parent
5930c42e74
commit
b2b3e40d13
14 changed files with 680 additions and 89 deletions
|
@ -12,6 +12,71 @@ Looking for the Unified API (OpenAI format) for VertexAI ? [Go here - using vert
|
|||
|
||||
:::
|
||||
|
||||
Pass-through endpoints for Vertex AI - call provider-specific endpoint, in native format (no translation).
|
||||
|
||||
Just replace `https://REGION-aiplatform.googleapis.com` with `LITELLM_PROXY_BASE_URL/vertex-ai`
|
||||
|
||||
|
||||
#### **Example Usage**
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="curl" label="curl">
|
||||
|
||||
```bash
|
||||
curl http://localhost:4000/vertex-ai/publishers/google/models/gemini-1.0-pro:generateContent \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-d '{
|
||||
"contents":[{
|
||||
"role": "user",
|
||||
"parts":[{"text": "How are you doing today?"}]
|
||||
}]
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="js" label="Vertex Node.js SDK">
|
||||
|
||||
```javascript
|
||||
const { VertexAI } = require('@google-cloud/vertexai');
|
||||
|
||||
const vertexAI = new VertexAI({
|
||||
project: 'your-project-id', // enter your vertex project id
|
||||
location: 'us-central1', // enter your vertex region
|
||||
apiEndpoint: "localhost:4000/vertex-ai" // <proxy-server-url>/vertex-ai # note, do not include 'https://' in the url
|
||||
});
|
||||
|
||||
const model = vertexAI.getGenerativeModel({
|
||||
model: 'gemini-1.0-pro'
|
||||
}, {
|
||||
customHeaders: {
|
||||
"x-litellm-api-key": "sk-1234" // Your litellm Virtual Key
|
||||
}
|
||||
});
|
||||
|
||||
async function generateContent() {
|
||||
try {
|
||||
const prompt = {
|
||||
contents: [{
|
||||
role: 'user',
|
||||
parts: [{ text: 'How are you doing today?' }]
|
||||
}]
|
||||
};
|
||||
|
||||
const response = await model.generateContent(prompt);
|
||||
console.log('Response:', response);
|
||||
} catch (error) {
|
||||
console.error('Error:', error);
|
||||
}
|
||||
}
|
||||
|
||||
generateContent();
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
||||
## Supported API Endpoints
|
||||
|
||||
- Gemini API
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue