feat: add dynamic model registration support to TGI inference (#3417)
Some checks failed
Vector IO Integration Tests / test-matrix (push) Failing after 4s
Update ReadTheDocs / update-readthedocs (push) Failing after 3s
UI Tests / ui-tests (22) (push) Successful in 43s
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 3s
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 1s
API Conformance Tests / check-schema-compatibility (push) Successful in 7s
Unit Tests / unit-tests (3.13) (push) Failing after 4s
Pre-commit / pre-commit (push) Successful in 1m21s
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped
Integration Tests (Replay) / Integration Tests (, , , client=, ) (push) Failing after 3s
Python Package Build Test / build (3.12) (push) Failing after 2s
Python Package Build Test / build (3.13) (push) Failing after 2s
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 5s
Unit Tests / unit-tests (3.12) (push) Failing after 3s
Test External API and Providers / test-external (venv) (push) Failing after 5s

# What does this PR do?

adds dynamic model support to TGI

add new overwrite_completion_id feature to OpenAIMixin to deal with TGI
always returning id=""

## Test Plan

tgi: `docker run --gpus all --shm-size 1g -p 8080:80 -v /data:/data
ghcr.io/huggingface/text-generation-inference --model-id
Qwen/Qwen3-0.6B`

stack: `TGI_URL=http://localhost:8080 uv run llama stack build
--image-type venv --distro ci-tests --run`

test: `./scripts/integration-tests.sh --stack-config
http://localhost:8321 --setup tgi --subdirs inference --pattern openai`
This commit is contained in:
Matthew Farrellee 2025-09-15 15:52:40 -04:00 committed by GitHub
parent ab321739f2
commit f4ab154ade
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
14 changed files with 12218 additions and 20 deletions

View file

@ -0,0 +1,56 @@
{
"request": {
"method": "POST",
"url": "http://localhost:8080/v1/v1/chat/completions",
"headers": {},
"body": {
"model": "Qwen/Qwen3-0.6B",
"messages": [
{
"role": "user",
"content": "Hello, world!"
}
],
"stream": false
},
"endpoint": "/v1/chat/completions",
"model": "Qwen/Qwen3-0.6B"
},
"response": {
"body": {
"__type__": "openai.types.chat.chat_completion.ChatCompletion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"logprobs": null,
"message": {
"content": "<think>\nOkay, the user just said \"Hello, world!\" so I need to respond in a friendly way. My prompt says to respond in the same style, so I should start with \"Hello, world!\" but maybe add some helpful information. Let me think. Since the user is probably testing or just sharing, a simple \"Hello, world!\" with a question would be best for user interaction. I'll make sure to keep it positive and open-ended.\n</think>\n\nHello, world! \ud83d\ude0a What do you need today?",
"refusal": null,
"role": "assistant",
"annotations": null,
"audio": null,
"function_call": null,
"tool_calls": null
}
}
],
"created": 1757550395,
"model": "Qwen/Qwen3-0.6B",
"object": "chat.completion",
"service_tier": null,
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": {
"completion_tokens": 108,
"prompt_tokens": 12,
"total_tokens": 120,
"completion_tokens_details": null,
"prompt_tokens_details": null
}
}
},
"is_streaming": false
}
}

File diff suppressed because one or more lines are too long

View file

@ -0,0 +1,56 @@
{
"request": {
"method": "POST",
"url": "http://localhost:8080/v1/v1/chat/completions",
"headers": {},
"body": {
"model": "Qwen/Qwen3-0.6B",
"messages": [
{
"role": "user",
"content": "Which planet has rings around it with a name starting with letter S?"
}
],
"stream": false
},
"endpoint": "/v1/chat/completions",
"model": "Qwen/Qwen3-0.6B"
},
"response": {
"body": {
"__type__": "openai.types.chat.chat_completion.ChatCompletion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"logprobs": null,
"message": {
"content": "<think>\nOkay, so the user is asking which planet has rings around it and its name starts with the letter S. Let me think... I know that the Sun is a star, not a planet. So the Moon is a natural satellite, which has the Moon's name and rings. But the Moon's name starts with M, not S. The Earth has the name Earth, but the rings aren't really around the Earth in any real sense. Mars has a thin ring of dust. Venus and Mercury don't have rings in the sense of planetary rings as we know. Wait, maybe the answer is the Moon, even though it's not the same as the name starting with S. But the question says a planet, so if there's a planet named S, that would be it. But actually, the only planet with rings is Jupiter. Wait, Jupiter has a famous system of rings. But why does the question mention a planet with a name starting with S? Maybe there's a trick. Let me double-check. Jupiter's name starts with J, so maybe the answer is Venus? But Venus doesn't have rings. Mercury, too, doesn't. The Moon, as a planet, a dwarf planet, and has rings. Despite the name, the rings are around it. So the answer would be the Moon. Therefore, the planet with rings and name starting with S is the Moon.\n</think>\n\nThe planet with rings around it and a name starting with the letter **S** is the **Moon**. Though its name doesn't start with an **S**, it is technically a dwarf planet and has the rings in its orbit. Oops Saturn!",
"refusal": null,
"role": "assistant",
"annotations": null,
"audio": null,
"function_call": null,
"tool_calls": null
}
}
],
"created": 1757550394,
"model": "Qwen/Qwen3-0.6B",
"object": "chat.completion",
"service_tier": null,
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": {
"completion_tokens": 336,
"prompt_tokens": 22,
"total_tokens": 358,
"completion_tokens_details": null,
"prompt_tokens_details": null
}
}
},
"is_streaming": false
}
}

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,366 @@
{
"request": {
"method": "POST",
"url": "http://localhost:8080/v1/v1/chat/completions",
"headers": {},
"body": {
"model": "Qwen/Qwen3-0.6B",
"messages": [
{
"role": "user",
"content": "What's the weather in Tokyo? Use the get_weather function to get the weather."
}
],
"stream": true,
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the weather in a given city",
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string",
"description": "The city to get the weather for"
}
}
}
}
}
]
},
"endpoint": "/v1/chat/completions",
"model": "Qwen/Qwen3-0.6B"
},
"response": {
"body": [
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "",
"choices": [
{
"delta": {
"content": null,
"function_call": null,
"refusal": null,
"role": "assistant",
"tool_calls": [
{
"index": 0,
"id": "0",
"function": {
"arguments": "{",
"name": "get_weather"
},
"type": "function"
}
]
},
"finish_reason": null,
"index": 0,
"logprobs": null
}
],
"created": 1757550392,
"model": "Qwen/Qwen3-0.6B",
"object": "chat.completion.chunk",
"service_tier": null,
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "",
"choices": [
{
"delta": {
"content": null,
"function_call": null,
"refusal": null,
"role": "assistant",
"tool_calls": [
{
"index": 0,
"id": "0",
"function": {
"arguments": " \"",
"name": null
},
"type": "function"
}
]
},
"finish_reason": null,
"index": 0,
"logprobs": null
}
],
"created": 1757550392,
"model": "Qwen/Qwen3-0.6B",
"object": "chat.completion.chunk",
"service_tier": null,
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "",
"choices": [
{
"delta": {
"content": null,
"function_call": null,
"refusal": null,
"role": "assistant",
"tool_calls": [
{
"index": 0,
"id": "0",
"function": {
"arguments": "c",
"name": null
},
"type": "function"
}
]
},
"finish_reason": null,
"index": 0,
"logprobs": null
}
],
"created": 1757550392,
"model": "Qwen/Qwen3-0.6B",
"object": "chat.completion.chunk",
"service_tier": null,
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "",
"choices": [
{
"delta": {
"content": null,
"function_call": null,
"refusal": null,
"role": "assistant",
"tool_calls": [
{
"index": 0,
"id": "0",
"function": {
"arguments": "ity",
"name": null
},
"type": "function"
}
]
},
"finish_reason": null,
"index": 0,
"logprobs": null
}
],
"created": 1757550392,
"model": "Qwen/Qwen3-0.6B",
"object": "chat.completion.chunk",
"service_tier": null,
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "",
"choices": [
{
"delta": {
"content": null,
"function_call": null,
"refusal": null,
"role": "assistant",
"tool_calls": [
{
"index": 0,
"id": "0",
"function": {
"arguments": "\":",
"name": null
},
"type": "function"
}
]
},
"finish_reason": null,
"index": 0,
"logprobs": null
}
],
"created": 1757550392,
"model": "Qwen/Qwen3-0.6B",
"object": "chat.completion.chunk",
"service_tier": null,
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "",
"choices": [
{
"delta": {
"content": null,
"function_call": null,
"refusal": null,
"role": "assistant",
"tool_calls": [
{
"index": 0,
"id": "0",
"function": {
"arguments": " \"",
"name": null
},
"type": "function"
}
]
},
"finish_reason": null,
"index": 0,
"logprobs": null
}
],
"created": 1757550392,
"model": "Qwen/Qwen3-0.6B",
"object": "chat.completion.chunk",
"service_tier": null,
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "",
"choices": [
{
"delta": {
"content": null,
"function_call": null,
"refusal": null,
"role": "assistant",
"tool_calls": [
{
"index": 0,
"id": "0",
"function": {
"arguments": "Tok",
"name": null
},
"type": "function"
}
]
},
"finish_reason": null,
"index": 0,
"logprobs": null
}
],
"created": 1757550392,
"model": "Qwen/Qwen3-0.6B",
"object": "chat.completion.chunk",
"service_tier": null,
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "",
"choices": [
{
"delta": {
"content": null,
"function_call": null,
"refusal": null,
"role": "assistant",
"tool_calls": [
{
"index": 0,
"id": "0",
"function": {
"arguments": "yo",
"name": null
},
"type": "function"
}
]
},
"finish_reason": null,
"index": 0,
"logprobs": null
}
],
"created": 1757550392,
"model": "Qwen/Qwen3-0.6B",
"object": "chat.completion.chunk",
"service_tier": null,
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.chat.chat_completion_chunk.ChatCompletionChunk",
"__data__": {
"id": "",
"choices": [
{
"delta": {
"content": null,
"function_call": null,
"refusal": null,
"role": "assistant",
"tool_calls": [
{
"index": 0,
"id": "0",
"function": {
"arguments": "\"}",
"name": null
},
"type": "function"
}
]
},
"finish_reason": null,
"index": 0,
"logprobs": null
}
],
"created": 1757550392,
"model": "Qwen/Qwen3-0.6B",
"object": "chat.completion.chunk",
"service_tier": null,
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
}
],
"is_streaming": true
}
}

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,976 @@
{
"request": {
"method": "POST",
"url": "http://localhost:8080/v1/v1/completions",
"headers": {},
"body": {
"model": "Qwen/Qwen3-0.6B",
"prompt": "Respond to this question and explain your answer. Complete the sentence using one word: Roses are red, violets are ",
"max_tokens": 50,
"stream": true
},
"endpoint": "/v1/completions",
"model": "Qwen/Qwen3-0.6B"
},
"response": {
"body": [
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " several"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " several"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " times"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " more"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " popular"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " than"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " ____"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": ".\n"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": "Answer"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": ":\n\n"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": "The"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " roses"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " are"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " red"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": ","
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " v"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": "io"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": "lets"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " are"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " several"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " several"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " times"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " more"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " popular"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " than"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " **"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": "numbers"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": "**"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": ".\n\n"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": "Explanation"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": ":"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " \""
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": "se"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": "veral"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " several"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " times"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " more"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " popular"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " than"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": "\""
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " can"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " be"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " replaced"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " with"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " \""
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": "numbers"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": "\""
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " as"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "",
"index": 0,
"logprobs": null,
"text": " the"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": null
}
},
{
"__type__": "openai.types.completion.Completion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"text": " number"
}
],
"created": 1757550367,
"model": "Qwen/Qwen3-0.6B",
"object": "text_completion",
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": {
"completion_tokens": 50,
"prompt_tokens": 25,
"total_tokens": 75,
"completion_tokens_details": null,
"prompt_tokens_details": null
}
}
}
],
"is_streaming": true
}
}

View file

@ -0,0 +1,84 @@
{
"request": {
"method": "POST",
"url": "http://localhost:8080/v1/v1/chat/completions",
"headers": {},
"body": {
"model": "Qwen/Qwen3-0.6B",
"messages": [
{
"role": "user",
"content": "What's the weather in Tokyo? Use the get_weather function to get the weather."
}
],
"stream": false,
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the weather in a given city",
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string",
"description": "The city to get the weather for"
}
}
}
}
}
]
},
"endpoint": "/v1/chat/completions",
"model": "Qwen/Qwen3-0.6B"
},
"response": {
"body": {
"__type__": "openai.types.chat.chat_completion.ChatCompletion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"logprobs": null,
"message": {
"content": null,
"refusal": null,
"role": "assistant",
"annotations": null,
"audio": null,
"function_call": null,
"tool_calls": [
{
"id": "0",
"function": {
"arguments": "{\"city\":\"Tokyo\"}",
"name": "get_weather",
"description": null
},
"type": "function"
}
]
}
}
],
"created": 1757550396,
"model": "Qwen/Qwen3-0.6B",
"object": "chat.completion",
"service_tier": null,
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": {
"completion_tokens": 19,
"prompt_tokens": 239,
"total_tokens": 258,
"completion_tokens_details": null,
"prompt_tokens_details": null
}
}
},
"is_streaming": false
}
}

View file

@ -0,0 +1,56 @@
{
"request": {
"method": "POST",
"url": "http://localhost:8080/v1/v1/chat/completions",
"headers": {},
"body": {
"model": "Qwen/Qwen3-0.6B",
"messages": [
{
"role": "user",
"content": "Which planet do humans live on?"
}
],
"stream": false
},
"endpoint": "/v1/chat/completions",
"model": "Qwen/Qwen3-0.6B"
},
"response": {
"body": {
"__type__": "openai.types.chat.chat_completion.ChatCompletion",
"__data__": {
"id": "",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"logprobs": null,
"message": {
"content": "<think>\nOkay, the user is asking which planet humans live on. I need to make sure I answer this accurately. First, I should recall what I know about our solar system. The Earth is our home, and it's in our solar system. There are eight planets in total, right? Let me check that. Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune. Yep, that's the list.\n\nBut wait, the user might be confusing Earth with Mars. I should clarify that Earth is the only planet known to support life. The other planets are mostly gas giants and have no liquid water, so they don't support life as Earth does. So the answer should be Earth. I should also mention that although there are other planets, none have liquid water, which makes the answer more complete.\n\nI need to make sure there are no alternatives. Maybe some people might think Mars, but I know that's not the case. Also, it's good to mention that life on Earth is closely linked to the presence of water, which is why Earth is our only planet with that characteristic. That way, the answer is not only accurate but also informative.\n</think>\n\nHumans live on **Earth**, the planet that supports life as we know it. The Earth is the only known planet in our solar system where liquid water exists and where life can occur. Other planets are considered \"gas giants\" or \"ice giants\" due to their extreme conditions and lack of liquid water, making them inhospitable for life.",
"refusal": null,
"role": "assistant",
"annotations": null,
"audio": null,
"function_call": null,
"tool_calls": null
}
}
],
"created": 1757550390,
"model": "Qwen/Qwen3-0.6B",
"object": "chat.completion",
"service_tier": null,
"system_fingerprint": "3.3.5-dev0-sha-1b90c50",
"usage": {
"completion_tokens": 312,
"prompt_tokens": 15,
"total_tokens": 327,
"completion_tokens_details": null,
"prompt_tokens_details": null
}
}
},
"is_streaming": false
}
}

File diff suppressed because it is too large Load diff