feat: Add missing Vector Store Files API surface (#2468)
Some checks failed
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 2s
Integration Tests / test-matrix (http, 3.11, tool_runtime) (push) Failing after 13s
Integration Tests / test-matrix (http, 3.11, providers) (push) Failing after 13s
Integration Tests / test-matrix (http, 3.11, datasets) (push) Failing after 14s
Integration Tests / test-matrix (http, 3.11, inspect) (push) Failing after 14s
Integration Tests / test-matrix (http, 3.12, agents) (push) Failing after 14s
Integration Tests / test-matrix (http, 3.12, scoring) (push) Failing after 13s
Integration Tests / test-matrix (library, 3.11, inspect) (push) Failing after 12s
Integration Tests / test-matrix (library, 3.11, tool_runtime) (push) Failing after 12s
Integration Tests / test-matrix (http, 3.12, post_training) (push) Failing after 15s
Integration Tests / test-matrix (http, 3.12, inference) (push) Failing after 19s
Integration Tests / test-matrix (http, 3.12, inspect) (push) Failing after 22s
Integration Tests / test-matrix (http, 3.12, vector_io) (push) Failing after 17s
Integration Tests / test-matrix (http, 3.11, post_training) (push) Failing after 23s
Integration Tests / test-matrix (library, 3.11, datasets) (push) Failing after 14s
Integration Tests / test-matrix (http, 3.11, vector_io) (push) Failing after 20s
Integration Tests / test-matrix (library, 3.11, inference) (push) Failing after 16s
Integration Tests / test-matrix (http, 3.11, agents) (push) Failing after 26s
Integration Tests / test-matrix (http, 3.12, tool_runtime) (push) Failing after 19s
Python Package Build Test / build (3.11) (push) Failing after 5s
Integration Tests / test-matrix (library, 3.12, post_training) (push) Failing after 6s
Python Package Build Test / build (3.12) (push) Failing after 3s
Integration Tests / test-matrix (http, 3.12, providers) (push) Failing after 18s
Integration Tests / test-matrix (library, 3.12, providers) (push) Failing after 10s
Integration Tests / test-matrix (library, 3.11, post_training) (push) Failing after 17s
Integration Tests / test-matrix (library, 3.11, vector_io) (push) Failing after 15s
Integration Tests / test-matrix (library, 3.11, scoring) (push) Failing after 18s
Integration Tests / test-matrix (library, 3.12, datasets) (push) Failing after 13s
Integration Tests / test-matrix (library, 3.12, scoring) (push) Failing after 8s
Python Package Build Test / build (3.13) (push) Failing after 5s
Integration Tests / test-matrix (http, 3.11, scoring) (push) Failing after 24s
Integration Tests / test-matrix (library, 3.11, agents) (push) Failing after 20s
Integration Tests / test-matrix (library, 3.12, inspect) (push) Failing after 10s
Integration Tests / test-matrix (library, 3.12, tool_runtime) (push) Failing after 7s
Integration Tests / test-matrix (library, 3.11, providers) (push) Failing after 15s
Integration Tests / test-matrix (http, 3.12, datasets) (push) Failing after 21s
Integration Tests / test-matrix (library, 3.12, inference) (push) Failing after 12s
Integration Tests / test-matrix (library, 3.12, agents) (push) Failing after 15s
Integration Tests / test-matrix (http, 3.11, inference) (push) Failing after 22s
Unit Tests / unit-tests (3.11) (push) Failing after 7s
Update ReadTheDocs / update-readthedocs (push) Failing after 4s
Unit Tests / unit-tests (3.12) (push) Failing after 7s
Integration Tests / test-matrix (library, 3.12, vector_io) (push) Failing after 48s
Test External Providers / test-external-providers (venv) (push) Failing after 43s
Unit Tests / unit-tests (3.13) (push) Failing after 52s
Pre-commit / pre-commit (push) Successful in 2m4s

# What does this PR do?

This adds the ability to list, retrieve, update, and delete Vector Store
Files. It implements these new APIs for the faiss and sqlite-vec
providers, since those are the two that also have the rest of the vector
store files implementation.

Closes #2445 

## Test Plan

### test_openai_vector_stores Integration Tests

There are a number of new integration tests added, which I ran for each
provider as outlined below.

faiss (from ollama distro):

```
INFERENCE_MODEL="meta-llama/Llama-3.2-3B-Instruct" \
llama stack run llama_stack/templates/ollama/run.yaml

LLAMA_STACK_CONFIG=http://localhost:8321 \
pytest -sv tests/integration/vector_io/test_openai_vector_stores.py \
  --embedding-model=all-MiniLM-L6-v2
```

sqlite-vec (from starter distro):

```
llama stack run llama_stack/templates/starter/run.yaml

LLAMA_STACK_CONFIG=http://localhost:8321 \
pytest -sv tests/integration/vector_io/test_openai_vector_stores.py \
  --embedding-model=all-MiniLM-L6-v2
```

### file_search verification tests

I also ensured the file_search verification tests continue to work, both
for faiss and sqlite-vec.

faiss (ollama distro):

```
INFERENCE_MODEL="meta-llama/Llama-3.2-3B-Instruct" \
llama stack run llama_stack/templates/ollama/run.yaml

pytest -sv tests/verifications/openai_api/test_responses.py \
  -k'file_search' \
  --base-url=http://localhost:8321/v1/openai/v1 \
  --model=meta-llama/Llama-3.2-3B-Instruct
```


sqlite-vec (starter distro):

```
llama stack run llama_stack/templates/starter/run.yaml

pytest -sv tests/verifications/openai_api/test_responses.py \
  -k'file_search' \
  --base-url=http://localhost:8321/v1/openai/v1 \
  --model=together/meta-llama/Llama-3.2-3B-Instruct-Turbo
```

---------

Signed-off-by: Ben Browning <bbrownin@redhat.com>
This commit is contained in:
Ben Browning 2025-06-19 11:08:24 -04:00 committed by GitHub
parent a2f054607d
commit f394c7f2d9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
11 changed files with 1991 additions and 122 deletions

View file

@ -3241,6 +3241,87 @@
}
},
"/v1/openai/v1/vector_stores/{vector_store_id}/files": {
"get": {
"responses": {
"200": {
"description": "A VectorStoreListFilesResponse containing the list of files.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/VectorStoreListFilesResponse"
}
}
}
},
"400": {
"$ref": "#/components/responses/BadRequest400"
},
"429": {
"$ref": "#/components/responses/TooManyRequests429"
},
"500": {
"$ref": "#/components/responses/InternalServerError500"
},
"default": {
"$ref": "#/components/responses/DefaultError"
}
},
"tags": [
"VectorIO"
],
"description": "List files in a vector store.",
"parameters": [
{
"name": "vector_store_id",
"in": "path",
"description": "The ID of the vector store to list files from.",
"required": true,
"schema": {
"type": "string"
}
},
{
"name": "limit",
"in": "query",
"required": false,
"schema": {
"type": "integer"
}
},
{
"name": "order",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
},
{
"name": "after",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
},
{
"name": "before",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
},
{
"name": "filter",
"in": "query",
"required": false,
"schema": {
"$ref": "#/components/schemas/VectorStoreFileStatus"
}
}
]
},
"post": {
"responses": {
"200": {
@ -3666,6 +3747,168 @@
]
}
},
"/v1/openai/v1/vector_stores/{vector_store_id}/files/{file_id}": {
"get": {
"responses": {
"200": {
"description": "A VectorStoreFileObject representing the file.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/VectorStoreFileObject"
}
}
}
},
"400": {
"$ref": "#/components/responses/BadRequest400"
},
"429": {
"$ref": "#/components/responses/TooManyRequests429"
},
"500": {
"$ref": "#/components/responses/InternalServerError500"
},
"default": {
"$ref": "#/components/responses/DefaultError"
}
},
"tags": [
"VectorIO"
],
"description": "Retrieves a vector store file.",
"parameters": [
{
"name": "vector_store_id",
"in": "path",
"description": "The ID of the vector store containing the file to retrieve.",
"required": true,
"schema": {
"type": "string"
}
},
{
"name": "file_id",
"in": "path",
"description": "The ID of the file to retrieve.",
"required": true,
"schema": {
"type": "string"
}
}
]
},
"post": {
"responses": {
"200": {
"description": "A VectorStoreFileObject representing the updated file.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/VectorStoreFileObject"
}
}
}
},
"400": {
"$ref": "#/components/responses/BadRequest400"
},
"429": {
"$ref": "#/components/responses/TooManyRequests429"
},
"500": {
"$ref": "#/components/responses/InternalServerError500"
},
"default": {
"$ref": "#/components/responses/DefaultError"
}
},
"tags": [
"VectorIO"
],
"description": "Updates a vector store file.",
"parameters": [
{
"name": "vector_store_id",
"in": "path",
"description": "The ID of the vector store containing the file to update.",
"required": true,
"schema": {
"type": "string"
}
},
{
"name": "file_id",
"in": "path",
"description": "The ID of the file to update.",
"required": true,
"schema": {
"type": "string"
}
}
],
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/OpenaiUpdateVectorStoreFileRequest"
}
}
},
"required": true
}
},
"delete": {
"responses": {
"200": {
"description": "A VectorStoreFileDeleteResponse indicating the deletion status.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/VectorStoreFileDeleteResponse"
}
}
}
},
"400": {
"$ref": "#/components/responses/BadRequest400"
},
"429": {
"$ref": "#/components/responses/TooManyRequests429"
},
"500": {
"$ref": "#/components/responses/InternalServerError500"
},
"default": {
"$ref": "#/components/responses/DefaultError"
}
},
"tags": [
"VectorIO"
],
"description": "Delete a vector store file.",
"parameters": [
{
"name": "vector_store_id",
"in": "path",
"description": "The ID of the vector store containing the file to delete.",
"required": true,
"schema": {
"type": "string"
}
},
{
"name": "file_id",
"in": "path",
"description": "The ID of the file to delete.",
"required": true,
"schema": {
"type": "string"
}
}
]
}
},
"/v1/openai/v1/embeddings": {
"post": {
"responses": {
@ -3909,6 +4152,58 @@
]
}
},
"/v1/openai/v1/vector_stores/{vector_store_id}/files/{file_id}/content": {
"get": {
"responses": {
"200": {
"description": "A list of InterleavedContent representing the file contents.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/VectorStoreFileContentsResponse"
}
}
}
},
"400": {
"$ref": "#/components/responses/BadRequest400"
},
"429": {
"$ref": "#/components/responses/TooManyRequests429"
},
"500": {
"$ref": "#/components/responses/InternalServerError500"
},
"default": {
"$ref": "#/components/responses/DefaultError"
}
},
"tags": [
"VectorIO"
],
"description": "Retrieves the contents of a vector store file.",
"parameters": [
{
"name": "vector_store_id",
"in": "path",
"description": "The ID of the vector store containing the file to retrieve.",
"required": true,
"schema": {
"type": "string"
}
},
{
"name": "file_id",
"in": "path",
"description": "The ID of the file to retrieve.",
"required": true,
"schema": {
"type": "string"
}
}
]
}
},
"/v1/openai/v1/vector_stores/{vector_store_id}/search": {
"post": {
"responses": {
@ -12102,24 +12397,7 @@
"$ref": "#/components/schemas/VectorStoreFileLastError"
},
"status": {
"oneOf": [
{
"type": "string",
"const": "completed"
},
{
"type": "string",
"const": "in_progress"
},
{
"type": "string",
"const": "cancelled"
},
{
"type": "string",
"const": "failed"
}
]
"$ref": "#/components/schemas/VectorStoreFileStatus"
},
"usage_bytes": {
"type": "integer",
@ -12143,6 +12421,26 @@
"title": "VectorStoreFileObject",
"description": "OpenAI Vector Store File object."
},
"VectorStoreFileStatus": {
"oneOf": [
{
"type": "string",
"const": "completed"
},
{
"type": "string",
"const": "in_progress"
},
{
"type": "string",
"const": "cancelled"
},
{
"type": "string",
"const": "failed"
}
]
},
"OpenAIJSONSchema": {
"type": "object",
"properties": {
@ -12969,6 +13267,35 @@
],
"title": "OpenaiCreateVectorStoreRequest"
},
"VectorStoreFileCounts": {
"type": "object",
"properties": {
"completed": {
"type": "integer"
},
"cancelled": {
"type": "integer"
},
"failed": {
"type": "integer"
},
"in_progress": {
"type": "integer"
},
"total": {
"type": "integer"
}
},
"additionalProperties": false,
"required": [
"completed",
"cancelled",
"failed",
"in_progress",
"total"
],
"title": "VectorStoreFileCounts"
},
"VectorStoreObject": {
"type": "object",
"properties": {
@ -12990,10 +13317,7 @@
"default": 0
},
"file_counts": {
"type": "object",
"additionalProperties": {
"type": "integer"
}
"$ref": "#/components/schemas/VectorStoreFileCounts"
},
"status": {
"type": "string",
@ -13120,6 +13444,30 @@
"title": "VectorStoreDeleteResponse",
"description": "Response from deleting a vector store."
},
"VectorStoreFileDeleteResponse": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"object": {
"type": "string",
"default": "vector_store.file.deleted"
},
"deleted": {
"type": "boolean",
"default": true
}
},
"additionalProperties": false,
"required": [
"id",
"object",
"deleted"
],
"title": "VectorStoreFileDeleteResponse",
"description": "Response from deleting a vector store file."
},
"OpenaiEmbeddingsRequest": {
"type": "object",
"properties": {
@ -13348,6 +13696,39 @@
"title": "OpenAIFileObject",
"description": "OpenAI File object as defined in the OpenAI Files API."
},
"VectorStoreListFilesResponse": {
"type": "object",
"properties": {
"object": {
"type": "string",
"default": "list"
},
"data": {
"type": "array",
"items": {
"$ref": "#/components/schemas/VectorStoreFileObject"
}
},
"first_id": {
"type": "string"
},
"last_id": {
"type": "string"
},
"has_more": {
"type": "boolean",
"default": false
}
},
"additionalProperties": false,
"required": [
"object",
"data",
"has_more"
],
"title": "VectorStoreListFilesResponse",
"description": "Response from listing vector stores."
},
"OpenAIModel": {
"type": "object",
"properties": {
@ -13429,6 +13810,75 @@
"type": "object",
"title": "Response"
},
"VectorStoreContent": {
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "text"
},
"text": {
"type": "string"
}
},
"additionalProperties": false,
"required": [
"type",
"text"
],
"title": "VectorStoreContent"
},
"VectorStoreFileContentsResponse": {
"type": "object",
"properties": {
"file_id": {
"type": "string"
},
"filename": {
"type": "string"
},
"attributes": {
"type": "object",
"additionalProperties": {
"oneOf": [
{
"type": "null"
},
{
"type": "boolean"
},
{
"type": "number"
},
{
"type": "string"
},
{
"type": "array"
},
{
"type": "object"
}
]
}
},
"content": {
"type": "array",
"items": {
"$ref": "#/components/schemas/VectorStoreContent"
}
}
},
"additionalProperties": false,
"required": [
"file_id",
"filename",
"attributes",
"content"
],
"title": "VectorStoreFileContentsResponse",
"description": "Response from retrieving the contents of a vector store file."
},
"OpenaiSearchVectorStoreRequest": {
"type": "object",
"properties": {
@ -13501,24 +13951,6 @@
],
"title": "OpenaiSearchVectorStoreRequest"
},
"VectorStoreContent": {
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "text"
},
"text": {
"type": "string"
}
},
"additionalProperties": false,
"required": [
"type",
"text"
],
"title": "VectorStoreContent"
},
"VectorStoreSearchResponse": {
"type": "object",
"properties": {
@ -13661,6 +14093,42 @@
"additionalProperties": false,
"title": "OpenaiUpdateVectorStoreRequest"
},
"OpenaiUpdateVectorStoreFileRequest": {
"type": "object",
"properties": {
"attributes": {
"type": "object",
"additionalProperties": {
"oneOf": [
{
"type": "null"
},
{
"type": "boolean"
},
{
"type": "number"
},
{
"type": "string"
},
{
"type": "array"
},
{
"type": "object"
}
]
},
"description": "The updated key-value attributes to store with the file."
}
},
"additionalProperties": false,
"required": [
"attributes"
],
"title": "OpenaiUpdateVectorStoreFileRequest"
},
"DPOAlignmentConfig": {
"type": "object",
"properties": {