mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-08-15 14:08:00 +00:00
0caef40e0d
# What does this PR do? I found a few issues while adding new metrics for various APIs: currently metrics are only propagated in `chat_completion` and `completion` since most providers use the `openai_..` routes as the default in `llama-stack-client inference chat-completion`, metrics are currently not working as expected. in order to get them working the following had to be done: 1. get the completion as usual 2. use new `openai_` versions of the metric gathering functions which use `.usage` from the `OpenAI..` response types to gather the metrics which are already populated. 3. define a `stream_generator` which counts the tokens and computes the metrics (only for stream=True) 5. add metrics to response NOTE: I could not add metrics to `openai_completion` where stream=True because that ONLY returns an `OpenAICompletion` not an AsyncGenerator that we can manipulate. acquire the lock, and add event to the span as the other `_log_...` methods do some new output: `llama-stack-client inference chat-completion --message hi` <img width="2416" height="425" alt="Screenshot 2025-07-16 at 8 28 20 AM" src="https://github.com/user-attachments/assets/ccdf1643-a184-4ddd-9641-d426c4d51326" /> and in the client: <img width="763" height="319" alt="Screenshot 2025-07-16 at 8 28 32 AM" src="https://github.com/user-attachments/assets/6bceb811-5201-47e9-9e16-8130f0d60007" /> these were not previously being recorded nor were they being printed to the server due to the improper console sink handling --------- Signed-off-by: Charlie Doern <cdoern@redhat.com>
56 lines
3.8 KiB
JSON
56 lines
3.8 KiB
JSON
{
|
|
"request": {
|
|
"method": "POST",
|
|
"url": "http://localhost:11434/v1/v1/chat/completions",
|
|
"headers": {},
|
|
"body": {
|
|
"model": "llama3.2:3b-instruct-fp16",
|
|
"messages": [
|
|
{
|
|
"role": "user",
|
|
"content": "Test OpenAI telemetry creation"
|
|
}
|
|
],
|
|
"stream": false
|
|
},
|
|
"endpoint": "/v1/chat/completions",
|
|
"model": "llama3.2:3b-instruct-fp16"
|
|
},
|
|
"response": {
|
|
"body": {
|
|
"__type__": "openai.types.chat.chat_completion.ChatCompletion",
|
|
"__data__": {
|
|
"id": "chatcmpl-877",
|
|
"choices": [
|
|
{
|
|
"finish_reason": "stop",
|
|
"index": 0,
|
|
"logprobs": null,
|
|
"message": {
|
|
"content": "I'm not capable of directly testing the functionality of external systems like Telemetry. However, I can provide you with some general information about creating telemetry data and offer suggestions on how to set up a basic telemetry system.\r\n\r\nTelemetry is the automatic measurement, reporting, and transmission of data from sensors or other devices. In the context of OpenAI, telemetry refers to the collection and analysis of data related to the company's products and services.\r\n\r\nTo create telemetry creation using the OpenAI APIs you would need to follow these steps:\r\n\r\n1. Register for an OpenAI account and get an access token.\r\n2. Choose the OpenAI API that you want to use (e.g., GPT-3).\r\n3. Create a new file or project in your preferred programming language or framework.\r\n4. Import the necessary libraries and modules to interact with the OpenAI API.\r\n5. Use the OpenAI API to create and send telemetry data.\r\n\r\nHere is an example of how you might create a basic telemetry system using Python and the OpenAI GPT-3 API:\r\n\r\n```python\r\nimport os\r\nimport json\r\n\r\n# Set your OpenAI access token\r\naccess_token = \"YOUR_OPENAI_ACCESS_TOKEN\"\r\n\r\n# Define the telemetry data\r\ntelemetry_data = {\r\n \"name\": \"example-telemetry\",\r\n \"description\": \"Example telemetry data.\r\n\r\n # Define the telemetry metrics\r\n \"metrics\": [\r\n {\"key\": \"users\", \"value\": 100},\r\n {\"key\": \" engagement\", \"value\": 20}\r\n ]\r\n}\r\n\r\n# Convert the telemetry data to JSON\r\ntelemetry_json = json.dumps(telemetry_data)\r\n\r\n# Set the OpenAI API endpoint and headers\r\napi_endpoint = \"https://api.openai.com/v1/telemetry\"\r\nheaders = {\r\n \"Authorization\": f\"Bearer {access_token}\",\r\n \"Content-Type\": \"application/json\"\r\n}\r\n\r\n# Send the telemetry data to the OpenAI API\r\nimport requests\r\n\r\nresponse = requests.post(api_endpoint, headers=headers, data=telemetry_json)\r\n\r\n# Check if the request was successful\r\nif response.status_code == 200:\r\n print(\"Telemetry data sent successfully\")\r\nelse:\r\n print(\"Error sending telemetry data\")\r\n```\n\nPlease note that this is a basic example and you should adjust it according to your needs. Also, the specific implementation details may vary depending on the OpenAI API you're using and the programming language or framework you're working with.\r\n\r\nI hope this helps! Let me know if you have any further questions.",
|
|
"refusal": null,
|
|
"role": "assistant",
|
|
"annotations": null,
|
|
"audio": null,
|
|
"function_call": null,
|
|
"tool_calls": null
|
|
}
|
|
}
|
|
],
|
|
"created": 1754510083,
|
|
"model": "llama3.2:3b-instruct-fp16",
|
|
"object": "chat.completion",
|
|
"service_tier": null,
|
|
"system_fingerprint": "fp_ollama",
|
|
"usage": {
|
|
"completion_tokens": 505,
|
|
"prompt_tokens": 30,
|
|
"total_tokens": 535,
|
|
"completion_tokens_details": null,
|
|
"prompt_tokens_details": null
|
|
}
|
|
}
|
|
},
|
|
"is_streaming": false
|
|
}
|
|
}
|