phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-08-12 13:00:39 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
llama-stack-mirror/tests/integration/recordings/responses
History
Exact
Charlie Doern 0caef40e0d
fix: telemetry fixes (inference and core telemetry) (#2733) 
# What does this PR do?

I found a few issues while adding new metrics for various APIs:

currently metrics are only propagated in `chat_completion` and
`completion`

since most providers use the `openai_..` routes as the default in
`llama-stack-client inference chat-completion`, metrics are currently
not working as expected.

in order to get them working the following had to be done:

1. get the completion as usual
2. use new `openai_` versions of the metric gathering functions which
use `.usage` from the `OpenAI..` response types to gather the metrics
which are already populated.
3. define a `stream_generator` which counts the tokens and computes the
metrics (only for stream=True)
5. add metrics to response


NOTE: I could not add metrics to `openai_completion` where stream=True
because that ONLY returns an `OpenAICompletion` not an AsyncGenerator
that we can manipulate.


acquire the lock, and add event to the span as the other `_log_...`
methods do

some new output:

`llama-stack-client inference chat-completion --message hi`

<img width="2416" height="425" alt="Screenshot 2025-07-16 at 8 28 20 AM"
src="https://github.com/user-attachments/assets/ccdf1643-a184-4ddd-9641-d426c4d51326"
/>


and in the client:

<img width="763" height="319" alt="Screenshot 2025-07-16 at 8 28 32 AM"
src="https://github.com/user-attachments/assets/6bceb811-5201-47e9-9e16-8130f0d60007"
/>

these were not previously being recorded nor were they being printed to
the server due to the improper console sink handling

---------

Signed-off-by: Charlie Doern <cdoern@redhat.com>
2025-08-06 13:37:40 -07:00
..
0b3f2e4754ff.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
0b27fd737699.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
00ba04f74a96.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
0e8f2b001dd9.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
0ff78129bb3a.json feat(ci): use replay mode, setup ollama if specific label exists on PR (#2955) 2025-07-29 16:50:26 -07:00
1adfaa0e062e.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
1b92be674e2a.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
1b8394f90636.json chore: fix: integration tests failures marked as successful (#3039) 2025-08-04 17:06:28 -07:00
1b45391880c6.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
1f48f4b2ae33.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
2afe3b38ca01.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
2d187a11704c.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
3c3f13cb7794.json chore: fix: integration tests failures marked as successful (#3039) 2025-08-04 17:06:28 -07:00
3ca695048bee.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
4a3a4447b16b.json chore(misc): make tests and starter faster (#3042) 2025-08-05 14:55:05 -07:00
5f5d16afadb4.json chore: fix: integration tests failures marked as successful (#3039) 2025-08-04 17:06:28 -07:00
6cc063bbd7d3.json chore: fix: integration tests failures marked as successful (#3039) 2025-08-04 17:06:28 -07:00
6d35c91287e2.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
6f96090aa955.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
6fbea1abca7c.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
6fe1d4fedf12.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
7b4815aba6c5.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
7e6806cba34a.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
8bba71367e87.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
9b812cbcb88d.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
9c4bc9c3e7ac.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
9c007f300365.json fix: telemetry fixes (inference and core telemetry) (#2733) 2025-08-06 13:37:40 -07:00
9c28ec9ac338.json feat(ci): use replay mode, setup ollama if specific label exists on PR (#2955) 2025-07-29 16:50:26 -07:00
9c140a29ae09.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
9e7a83d3d596.json chore: fix: integration tests failures marked as successful (#3039) 2025-08-04 17:06:28 -07:00
9fadf5a3d68f.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
10eea8c15ddc.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
011f70e24ce4.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
35db283fef1d.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
40f524d1934a.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
43e106de6736.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
44fb9cf5875f.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
48d2fb183a2a.json chore: fix: integration tests failures marked as successful (#3039) 2025-08-04 17:06:28 -07:00
56ac6a7c6df0.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
61be36ad8ccd.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
70adef2c30c4.json chore: fix: integration tests failures marked as successful (#3039) 2025-08-04 17:06:28 -07:00
75d0dd9d0fa3.json chore: fix: integration tests failures marked as successful (#3039) 2025-08-04 17:06:28 -07:00
76b89a84cd6f.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
80e4404d8987.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
80f09f27dd61.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
81a91f79c51d.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
83c2ffb72daa.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
84cab42e1f5c.json chore: fix: integration tests failures marked as successful (#3039) 2025-08-04 17:06:28 -07:00
90fec951fdb9.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
97d3812bfccb.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
97e259c0d3e5.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
173ecb3aab28.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
177ba8517262.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
198ef7208389.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
211b1562d4e6.json chore: fix: integration tests failures marked as successful (#3039) 2025-08-04 17:06:28 -07:00
325a72db5755.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
382c2f22274c.json fix: telemetry fixes (inference and core telemetry) (#2733) 2025-08-06 13:37:40 -07:00
477f8946bf7d.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
545d86510a80.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
554de3cd986f.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
563b994bb7d1.json chore: fix: integration tests failures marked as successful (#3039) 2025-08-04 17:06:28 -07:00
830a1fe14938.json fix: telemetry fixes (inference and core telemetry) (#2733) 2025-08-06 13:37:40 -07:00
836f51dfb3c5.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
840fbb380b73.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
3877ecf1bc62.json chore: move recordings one directory upwards 2025-07-29 12:46:19 -07:00
4014dd44c15f.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
6906a6e71988.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
7354ec181984.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
12913f20f6ac.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
17253d7cc667.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
31407e035752.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
50340cd4d253.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
67198cbad48f.json fix: telemetry fixes (inference and core telemetry) (#2733) 2025-08-06 13:37:40 -07:00
85594a69d74a.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
140187e305dc.json fix: telemetry fixes (inference and core telemetry) (#2733) 2025-08-06 13:37:40 -07:00
174458ad71b2.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
178016edef0e.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
197228e26971.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
561746e1c8de.json chore: fix: integration tests failures marked as successful (#3039) 2025-08-04 17:06:28 -07:00
4096743baf8e.json fix: telemetry fixes (inference and core telemetry) (#2733) 2025-08-06 13:37:40 -07:00
04172112ffbb.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
4597743bcd2a.json feat(ci): use replay mode, setup ollama if specific label exists on PR (#2955) 2025-07-29 16:50:26 -07:00
8295382a8e7c.json fix: telemetry fixes (inference and core telemetry) (#2733) 2025-08-06 13:37:40 -07:00
a0c4df33879f.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
a4c8d19bb1eb.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
a59d0d7c1485.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
a410d4840402.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
a5187d9d5057.json fix: telemetry fixes (inference and core telemetry) (#2733) 2025-08-06 13:37:40 -07:00
a6810c23eda8.json chore: fix: integration tests failures marked as successful (#3039) 2025-08-04 17:06:28 -07:00
a97477559b10.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
ae1c22f18ecc.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
ae6835cfe70e.json chore: fix: integration tests failures marked as successful (#3039) 2025-08-04 17:06:28 -07:00
afb33182f365.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
b4cda53cd04f.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
b14ff438ca99.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
b44cc7a7afc8.json chore(misc): make tests and starter faster (#3042) 2025-08-05 14:55:05 -07:00
b91f1fb4aedb.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
b24590574a85.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
bbd0637dce16.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
bd356b27a085.json feat(ci): use replay mode, setup ollama if specific label exists on PR (#2955) 2025-07-29 16:50:26 -07:00
c9cba6f3ee38.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
c7582fa7c2c4.json feat(ci): use replay mode, setup ollama if specific label exists on PR (#2955) 2025-07-29 16:50:26 -07:00
c9667519ad7c.json fix: telemetry fixes (inference and core telemetry) (#2733) 2025-08-06 13:37:40 -07:00
cb3df2a1dc22.json fix: telemetry fixes (inference and core telemetry) (#2733) 2025-08-06 13:37:40 -07:00
cd094caaf1c0.json chore: fix: integration tests failures marked as successful (#3039) 2025-08-04 17:06:28 -07:00
d0ac68cbde69.json fix: telemetry fixes (inference and core telemetry) (#2733) 2025-08-06 13:37:40 -07:00
d4c86ac355fb.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
d4f56d7d1996.json fix: telemetry fixes (inference and core telemetry) (#2733) 2025-08-06 13:37:40 -07:00
da531c71e64f.json chore(misc): make tests and starter faster (#3042) 2025-08-05 14:55:05 -07:00
dac7a32e5db9.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
dbc41d2417e1.json chore(misc): make tests and starter faster (#3042) 2025-08-05 14:55:05 -07:00
dd9e7d5913e9.json chore: fix: integration tests failures marked as successful (#3039) 2025-08-04 17:06:28 -07:00
dd226d71f844.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
e2c9b07709fe.json fix: telemetry fixes (inference and core telemetry) (#2733) 2025-08-06 13:37:40 -07:00
e9c8a0e4f0e0.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
e08848bfcd28.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
e29300494763.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
e96152610712.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
ed9e9b34008d.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
eee6a163b837.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00
eee47930e3ae.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
ef59cbff54d0.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
ef757a75ed08.json feat(ci): use replay mode, setup ollama if specific label exists on PR (#2955) 2025-07-29 16:50:26 -07:00
f1ea938b0b0d.json chore(misc): make tests and starter faster (#3042) 2025-08-05 14:55:05 -07:00
f3c3afbd9b7e.json feat(ci): use replay mode, setup ollama if specific label exists on PR (#2955) 2025-07-29 16:50:26 -07:00
f70f30f54211.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
f477c2fe1332.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
fcdef245da95.json fix(recording): endpoint resolution (#3013) 2025-08-01 16:23:54 -07:00
fe140befeba4.json feat(ci): add support for running vision inference tests (#2972) 2025-07-31 11:50:42 -07:00