dependabot[bot]
e86cf2c153
chore(github-deps): bump actions/checkout from 5.0.0 to 6.0.0 ( #4217 )
...
Bumps [actions/checkout](https://github.com/actions/checkout ) from 5.0.0
to 6.0.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/checkout/releases ">actions/checkout's
releases</a>.</em></p>
<blockquote>
<h2>v6.0.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Update README to include Node.js 24 support details and requirements
by <a href="https://github.com/salmanmkc "><code>@salmanmkc</code></a>
in <a
href="https://redirect.github.com/actions/checkout/pull/2248 ">actions/checkout#2248</a></li>
<li>Persist creds to a separate file by <a
href="https://github.com/ericsciple "><code>@ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2286 ">actions/checkout#2286</a></li>
<li>v6-beta by <a
href="https://github.com/ericsciple "><code>@ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2298 ">actions/checkout#2298</a></li>
<li>update readme/changelog for v6 by <a
href="https://github.com/ericsciple "><code>@ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2311 ">actions/checkout#2311</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/checkout/compare/v5.0.0...v6.0.0 ">https://github.com/actions/checkout/compare/v5.0.0...v6.0.0 </a></p>
<h2>v6-beta</h2>
<h2>What's Changed</h2>
<p>Updated persist-credentials to store the credentials under
<code>$RUNNER_TEMP</code> instead of directly in the local git
config.</p>
<p>This requires a minimum Actions Runner version of <a
href="https://github.com/actions/runner/releases/tag/v2.329.0 ">v2.329.0</a>
to access the persisted credentials for <a
href="https://docs.github.com/en/actions/tutorials/use-containerized-services/create-a-docker-container-action ">Docker
container action</a> scenarios.</p>
<h2>v5.0.1</h2>
<h2>What's Changed</h2>
<ul>
<li>Port v6 cleanup to v5 by <a
href="https://github.com/ericsciple "><code>@ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2301 ">actions/checkout#2301</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/checkout/compare/v5...v5.0.1 ">https://github.com/actions/checkout/compare/v5...v5.0.1 </a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/actions/checkout/blob/main/CHANGELOG.md ">actions/checkout's
changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<h2>V6.0.0</h2>
<ul>
<li>Persist creds to a separate file by <a
href="https://github.com/ericsciple "><code>@ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2286 ">actions/checkout#2286</a></li>
<li>Update README to include Node.js 24 support details and requirements
by <a href="https://github.com/salmanmkc "><code>@salmanmkc</code></a>
in <a
href="https://redirect.github.com/actions/checkout/pull/2248 ">actions/checkout#2248</a></li>
</ul>
<h2>V5.0.1</h2>
<ul>
<li>Port v6 cleanup to v5 by <a
href="https://github.com/ericsciple "><code>@ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2301 ">actions/checkout#2301</a></li>
</ul>
<h2>V5.0.0</h2>
<ul>
<li>Update actions checkout to use node 24 by <a
href="https://github.com/salmanmkc "><code>@salmanmkc</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2226 ">actions/checkout#2226</a></li>
</ul>
<h2>V4.3.1</h2>
<ul>
<li>Port v6 cleanup to v4 by <a
href="https://github.com/ericsciple "><code>@ericsciple</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2305 ">actions/checkout#2305</a></li>
</ul>
<h2>V4.3.0</h2>
<ul>
<li>docs: update README.md by <a
href="https://github.com/motss "><code>@motss</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1971 ">actions/checkout#1971</a></li>
<li>Add internal repos for checking out multiple repositories by <a
href="https://github.com/mouismail "><code>@mouismail</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1977 ">actions/checkout#1977</a></li>
<li>Documentation update - add recommended permissions to Readme by <a
href="https://github.com/benwells "><code>@benwells</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2043 ">actions/checkout#2043</a></li>
<li>Adjust positioning of user email note and permissions heading by <a
href="https://github.com/joshmgross "><code>@joshmgross</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2044 ">actions/checkout#2044</a></li>
<li>Update README.md by <a
href="https://github.com/nebuk89 "><code>@nebuk89</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2194 ">actions/checkout#2194</a></li>
<li>Update CODEOWNERS for actions by <a
href="https://github.com/TingluoHuang "><code>@TingluoHuang</code></a>
in <a
href="https://redirect.github.com/actions/checkout/pull/2224 ">actions/checkout#2224</a></li>
<li>Update package dependencies by <a
href="https://github.com/salmanmkc "><code>@salmanmkc</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2236 ">actions/checkout#2236</a></li>
</ul>
<h2>v4.2.2</h2>
<ul>
<li><code>url-helper.ts</code> now leverages well-known environment
variables by <a href="https://github.com/jww3 "><code>@jww3</code></a>
in <a
href="https://redirect.github.com/actions/checkout/pull/1941 ">actions/checkout#1941</a></li>
<li>Expand unit test coverage for <code>isGhes</code> by <a
href="https://github.com/jww3 "><code>@jww3</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1946 ">actions/checkout#1946</a></li>
</ul>
<h2>v4.2.1</h2>
<ul>
<li>Check out other refs/* by commit if provided, fall back to ref by <a
href="https://github.com/orhantoy "><code>@orhantoy</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1924 ">actions/checkout#1924</a></li>
</ul>
<h2>v4.2.0</h2>
<ul>
<li>Add Ref and Commit outputs by <a
href="https://github.com/lucacome "><code>@lucacome</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1180 ">actions/checkout#1180</a></li>
<li>Dependency updates by <a
href="https://github.com/dependabot "><code>@dependabot</code></a>- <a
href="https://redirect.github.com/actions/checkout/pull/1777 ">actions/checkout#1777</a>,
<a
href="https://redirect.github.com/actions/checkout/pull/1872 ">actions/checkout#1872</a></li>
</ul>
<h2>v4.1.7</h2>
<ul>
<li>Bump the minor-npm-dependencies group across 1 directory with 4
updates by <a
href="https://github.com/dependabot "><code>@dependabot</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1739 ">actions/checkout#1739</a></li>
<li>Bump actions/checkout from 3 to 4 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1697 ">actions/checkout#1697</a></li>
<li>Check out other refs/* by commit by <a
href="https://github.com/orhantoy "><code>@orhantoy</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1774 ">actions/checkout#1774</a></li>
<li>Pin actions/checkout's own workflows to a known, good, stable
version. by <a href="https://github.com/jww3 "><code>@jww3</code></a> in
<a
href="https://redirect.github.com/actions/checkout/pull/1776 ">actions/checkout#1776</a></li>
</ul>
<h2>v4.1.6</h2>
<ul>
<li>Check platform to set archive extension appropriately by <a
href="https://github.com/cory-miller "><code>@cory-miller</code></a> in
<a
href="https://redirect.github.com/actions/checkout/pull/1732 ">actions/checkout#1732</a></li>
</ul>
<h2>v4.1.5</h2>
<ul>
<li>Update NPM dependencies by <a
href="https://github.com/cory-miller "><code>@cory-miller</code></a> in
<a
href="https://redirect.github.com/actions/checkout/pull/1703 ">actions/checkout#1703</a></li>
<li>Bump github/codeql-action from 2 to 3 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1694 ">actions/checkout#1694</a></li>
<li>Bump actions/setup-node from 1 to 4 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1696 ">actions/checkout#1696</a></li>
<li>Bump actions/upload-artifact from 2 to 4 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1695 ">actions/checkout#1695</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="1af3b93b68https://redirect.github.com/actions/checkout/issues/2311 ">#2311</a>)</li>
<li><a
href="71cf2267d8https://redirect.github.com/actions/checkout/issues/2298 ">#2298</a>)</li>
<li><a
href="069c695914https://redirect.github.com/actions/checkout/issues/2286 ">#2286</a>)</li>
<li><a
href="ff7abcd0c3https://redirect.github.com/actions/checkout/issues/2248 ">#2248</a>)</li>
<li>See full diff in <a
href="08c6903cd8...1af3b93b68https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/checkout&package-manager=github_actions&previous-version=5.0.0&new-version=6.0.0 )](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores )
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-24 09:32:41 -08:00
dependabot[bot]
3434c92a14
chore(github-deps): bump actions/setup-node from 4.1.0 to 6.0.0 ( #4216 )
...
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 2s
Integration Tests (Replay) / generate-matrix (push) Successful in 3s
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 4s
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 6s
Python Package Build Test / build (3.12) (push) Failing after 5s
Python Package Build Test / build (3.13) (push) Failing after 4s
API Conformance Tests / check-schema-compatibility (push) Successful in 12s
Test External API and Providers / test-external (venv) (push) Failing after 29s
UI Tests / ui-tests (22) (push) Successful in 36s
Vector IO Integration Tests / test-matrix (push) Failing after 44s
Unit Tests / unit-tests (3.13) (push) Failing after 1m35s
Unit Tests / unit-tests (3.12) (push) Failing after 2m13s
Pre-commit / pre-commit (push) Successful in 3m4s
Integration Tests (Replay) / Integration Tests (, , , client=, ) (push) Failing after 3m6s
Bumps [actions/setup-node](https://github.com/actions/setup-node ) from
4.1.0 to 6.0.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/setup-node/releases ">actions/setup-node's
releases</a>.</em></p>
<blockquote>
<h2>v6.0.0</h2>
<h2>What's Changed</h2>
<p><strong>Breaking Changes</strong></p>
<ul>
<li>Limit automatic caching to npm, update workflows and documentation
by <a
href="https://github.com/priyagupta108 "><code>@priyagupta108</code></a>
in <a
href="https://redirect.github.com/actions/setup-node/pull/1374 ">actions/setup-node#1374</a></li>
</ul>
<p><strong>Dependency Upgrades</strong></p>
<ul>
<li>Upgrade ts-jest from 29.1.2 to 29.4.1 and document breaking changes
in v5 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/setup-node/pull/1336 ">#1336</a></li>
<li>Upgrade prettier from 2.8.8 to 3.6.2 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/setup-node/pull/1334 ">#1334</a></li>
<li>Upgrade actions/publish-action from 0.3.0 to 0.4.0 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/setup-node/pull/1362 ">#1362</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/setup-node/compare/v5...v6.0.0 ">https://github.com/actions/setup-node/compare/v5...v6.0.0 </a></p>
<h2>v5.0.0</h2>
<h2>What's Changed</h2>
<h3>Breaking Changes</h3>
<ul>
<li>Enhance caching in setup-node with automatic package manager
detection by <a
href="https://github.com/priya-kinthali "><code>@priya-kinthali</code></a>
in <a
href="https://redirect.github.com/actions/setup-node/pull/1348 ">actions/setup-node#1348</a></li>
</ul>
<p>This update, introduces automatic caching when a valid
<code>packageManager</code> field is present in your
<code>package.json</code>. This aims to improve workflow performance and
make dependency management more seamless.
To disable this automatic caching, set <code>package-manager-cache:
false</code></p>
<pre lang="yaml"><code>steps:
- uses: actions/checkout@v5
- uses: actions/setup-node@v5
with:
package-manager-cache: false
</code></pre>
<ul>
<li>Upgrade action to use node24 by <a
href="https://github.com/salmanmkc "><code>@salmanmkc</code></a> in <a
href="https://redirect.github.com/actions/setup-node/pull/1325 ">actions/setup-node#1325</a></li>
</ul>
<p>Make sure your runner is on version v2.327.1 or later to ensure
compatibility with this release. <a
href="https://github.com/actions/runner/releases/tag/v2.327.1 ">See
Release Notes</a></p>
<h3>Dependency Upgrades</h3>
<ul>
<li>Upgrade <code>@octokit/request-error</code> and
<code>@actions/github</code> by <a
href="https://github.com/dependabot "><code>@dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/setup-node/pull/1227 ">actions/setup-node#1227</a></li>
<li>Upgrade uuid from 9.0.1 to 11.1.0 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/setup-node/pull/1273 ">actions/setup-node#1273</a></li>
<li>Upgrade undici from 5.28.5 to 5.29.0 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/setup-node/pull/1295 ">actions/setup-node#1295</a></li>
<li>Upgrade form-data to bring in fix for critical vulnerability by <a
href="https://github.com/gowridurgad "><code>@gowridurgad</code></a> in
<a
href="https://redirect.github.com/actions/setup-node/pull/1332 ">actions/setup-node#1332</a></li>
<li>Upgrade actions/checkout from 4 to 5 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/actions/setup-node/pull/1345 ">actions/setup-node#1345</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/priya-kinthali "><code>@priya-kinthali</code></a>
made their first contribution in <a
href="https://redirect.github.com/actions/setup-node/pull/1348 ">actions/setup-node#1348</a></li>
<li><a href="https://github.com/salmanmkc "><code>@salmanmkc</code></a>
made their first contribution in <a
href="https://redirect.github.com/actions/setup-node/pull/1325 ">actions/setup-node#1325</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/setup-node/compare/v4...v5.0.0 ">https://github.com/actions/setup-node/compare/v4...v5.0.0 </a></p>
<h2>v4.4.0</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="2028fbc5c2https://redirect.github.com/actions/setup-node/issues/1374 ">#1374</a>)</li>
<li><a
href="13427813f7https://redirect.github.com/actions/setup-node/issues/1362 ">#1362</a>)</li>
<li><a
href="89d709d423https://redirect.github.com/actions/setup-node/issues/1334 ">#1334</a>)</li>
<li><a
href="cd2651c462https://redirect.github.com/actions/setup-node/issues/1336 ">#1336</a>)</li>
<li><a
href="a0853c2454https://redirect.github.com/actions/setup-node/issues/1345 ">#1345</a>)</li>
<li><a
href="b7234cc9fehttps://redirect.github.com/actions/setup-node/issues/1325 ">#1325</a>)</li>
<li><a
href="d7a11313b5https://redirect.github.com/actions/setup-node/issues/1348 ">#1348</a>)</li>
<li><a
href="5e2628c959https://redirect.github.com/actions/setup-node/issues/1332 ">#1332</a>)</li>
<li><a
href="65beceff8ehttps://redirect.github.com/actions/setup-node/issues/1295 ">#1295</a>)</li>
<li><a
href="7e24a656e1https://redirect.github.com/actions/setup-node/issues/1273 ">#1273</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/actions/setup-node/compare/v4.1.0...2028fbc5c25fe9cf00d9f06a71cc4710d4507903 ">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores )
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-23 22:32:58 -05:00
Ashwin Bharambe
40b11efac4
feat(tests): add TypeScript client integration test support ( #4185 )
...
Integration tests can now validate the TypeScript SDK alongside Python
tests when running against server-mode stacks. Currently, this only adds
a _small_ number of tests. We should extend only if truly needed -- this
smoke check may be sufficient.
When `RUN_CLIENT_TS_TESTS=1` is set, the test script runs TypeScript
tests after Python tests pass. Tests are mapped via
`tests/integration/client-typescript/suites.json` which defines which
TypeScript test files correspond to each Python suite/setup combination.
The fact that we need exact "test_id"s (which are actually generated by
pytest) to be hardcoded inside the Typescript tests (so we hit the
recorded paths) is a big smell and it might become grating, but maybe
the benefit is worth it if we keep this test suite _small_ and targeted.
## Test Plan
Run with TypeScript tests enabled:
```bash
OPENAI_API_KEY=dummy RUN_CLIENT_TS_TESTS=1 \
scripts/integration-tests.sh --stack-config server:ci-tests --suite responses --setup gpt
```
2025-11-19 10:07:53 -08:00
Ashwin Bharambe
492f79ca9b
fix: harden storage semantics ( #4118 )
...
Fixes issues in the storage system by guaranteeing immediate durability
for responses and ensuring background writers stay alive. Three related
fixes:
* Responses to the OpenAI-compatible API now write directly to
Postgres/SQLite inside the request instead of detouring through an async
queue that might never drain; this restores the expected
read-after-write behavior and removes the "response not found" races
reported by users.
* The access-control shim was stamping owner_principal/access_attributes
as SQL NULL, which Postgres interprets as non-public rows; fixing it to
use the empty-string/JSON-null pattern means conversations and responses
stored without an authenticated user stay queryable (matching SQLite).
* The inference-store queue remains for batching, but its worker tasks
now start lazily on the live event loop so server startup doesn't cancel
them—writes keep flowing even when the stack is launched via llama stack
run.
Closes #4115
### Test Plan
Added a matrix entry to test our "base" suite against Postgres as the
store.
2025-11-12 10:35:39 -08:00
Derek Higgins
c62a09ab76
ci: Add vLLM support to integration testing infrastructure (with qwen) ( #3545 )
...
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 1s
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 2s
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 1s
Integration Tests (Replay) / generate-matrix (push) Successful in 4s
Python Package Build Test / build (3.13) (push) Failing after 2s
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped
Vector IO Integration Tests / test-matrix (push) Failing after 6s
Pre-commit / pre-commit (push) Failing after 6s
Test External API and Providers / test-external (venv) (push) Failing after 5s
API Conformance Tests / check-schema-compatibility (push) Successful in 14s
Integration Tests (Replay) / Integration Tests (, , , client=, ) (push) Failing after 5s
Python Package Build Test / build (3.12) (push) Failing after 22s
UI Tests / ui-tests (22) (push) Successful in 57s
o Introduces vLLM provider support to the record/replay testing
framework
o Enabling both recording and replay of vLLM API interactions alongside
existing Ollama support.
The changes enable testing of vLLM functionality. vLLM tests focus on
inference capabilities, while Ollama continues to exercise the full API
surface
including vision features.
--
This is an alternative to #3128 , using qwen3 instead of llama 3.2 1B
appears to be more capable at structure output and tool calls.
---------
Signed-off-by: Derek Higgins <derekh@redhat.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-11-06 10:36:40 +01:00
Ashwin Bharambe
4d3069bfa5
chore(ci): remove unused recordings ( #4074 )
...
Added a script to cleanup recordings. While doing this, moved the CI
matrix generation to a separate script so there is a single source of
truth for the matrix.
Ran the cleanup script as:
```
PYTHONPATH=. python scripts/cleanup_recordings.py
```
Also added this as part of the pre-commit workflow to ensure that the
recordings are always up to date and that no stale recordings are left
in the repo.
2025-11-05 09:21:58 -08:00
ehhuang
95b0493fae
chore: move src/llama_stack/ui to src/llama_stack_ui ( #4068 )
...
# What does this PR do?
This better separates UI from backend code, which was a point of
confusion often for our beloved AI friends.
## Test Plan
CI
2025-11-04 15:21:49 -08:00
ehhuang
628e38b3d5
test: always start a new server in integration-tests.sh ( #4050 )
...
# What does this PR do?
This prevents interference from already running servers, and allows
multiple concurrent integration test runs. Unleash the AIs!
## Test Plan
start a LS server at port 8321
Then observe test uses port 8322:
❯ uv run --no-sync ./scripts/integration-tests.sh --stack-config
server:ci-tests --inference-mode replay --setup ollama --suite base
--pattern '(telemetry or safety)'
=== Llama Stack Integration Test Runner ===
Stack Config: server:ci-tests
Setup: ollama
Inference Mode: replay
Test Suite: base
Test Subdirs:
Test Pattern: (telemetry or safety)
Checking llama packages
llama-stack 0.4.0.dev0 /Users/erichuang/projects/new_test_server
llama-stack-client 0.3.0
ollama 0.6.0
=== Applying Setup Environment Variables ===
Setting SQLITE_STORE_DIR:
/var/folders/cz/vyh7y1d11xg881lsxsshnc5c0000gn/T/tmp.bKLsaVAxyU
Setting stack config type: server
Setting up environment variables:
export OLLAMA_URL='http://0.0.0.0:11434 '
export SAFETY_MODEL='ollama/llama-guard3:1b'
Will use port: 8322
=== Starting Llama Stack Server ===
Waiting for Llama Stack Server to start on port 8322...
✅ Llama Stack Server started successfully
2025-11-03 15:23:10 -08:00
Ashwin Bharambe
c396de57a4
ci: standardize release branch pattern to release-X.Y.x ( #3999 )
...
Standardize CI workflows to use `release-X.Y.x` branch pattern instead
of multiple numeric variants.
That's the pattern we are settling on. See
https://github.com/llamastack/llama-stack-ops/pull/20 for reference.
2025-10-30 21:33:32 -07:00
Ashwin Bharambe
6f90a7af4b
ci: target release-X.Y.x branches instead of release-X.Y.x-maint ( #3995 )
...
We will be updating our release procedure to be more "normal" or "sane".
We will
- create release branches like normal people
- land cherry-picks onto those branches
- run releases off of those branches
- no more "rc" branch pollution either
Given that, this PR cleans things up a bit
- Remove `-maint` suffix from release branch patterns in CI workflows
- Update branch matching to `release-X.Y.x` format
2025-10-30 16:27:13 -07:00
Ashwin Bharambe
90234d6973
ci: support release branches and match client branch ( #3990 )
...
- Update workflows to trigger on release-X.Y.x-maint branches
- When PR targets release branch, fetch matching branch from
llama-stack-client-python
- Falls back to main if matching client branch doesn't exist
- Updated workflows:
- integration-tests.yml
- integration-auth-tests.yml
- integration-sql-store-tests.yml
- integration-vector-io-tests.yml
- unit-tests.yml
- backward-compat.yml
- pre-commit.yml
2025-10-30 15:20:34 -07:00
Ashwin Bharambe
77c8bc6fa7
fix(ci): add back server:ci-tests to replay tests ( #3976 )
...
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 0s
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 1s
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped
Integration Tests (Replay) / Integration Tests (, , , client=, ) (push) Failing after 3s
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 4s
Pre-commit / pre-commit (push) Failing after 4s
Python Package Build Test / build (3.13) (push) Failing after 5s
Test External API and Providers / test-external (venv) (push) Failing after 6s
Vector IO Integration Tests / test-matrix (push) Failing after 7s
Unit Tests / unit-tests (3.13) (push) Failing after 8s
API Conformance Tests / check-schema-compatibility (push) Successful in 15s
Python Package Build Test / build (3.12) (push) Failing after 39s
Unit Tests / unit-tests (3.12) (push) Failing after 40s
UI Tests / ui-tests (22) (push) Successful in 42s
It is useful for local debugging. If both server and docker are failing,
you can just run server locally to debug which is much easier to do.
2025-10-30 11:02:59 -07:00
Ashwin Bharambe
471b1b248b
chore(package): migrate to src/ layout ( #3920 )
...
Migrates package structure to src/ layout following Python packaging
best practices.
All code moved from `llama_stack/` to `src/llama_stack/`. Public API
unchanged - imports remain `import llama_stack.*`.
Updated build configs, pre-commit hooks, scripts, and GitHub workflows
accordingly. All hooks pass, package builds cleanly.
**Developer note**: Reinstall after pulling: `pip install -e .`
2025-10-27 12:02:21 -07:00
Ashwin Bharambe
7918188f1e
fix(ci): enable responses tests in CI; suppress expected MCP auth error logs ( #3889 )
...
Let us enable responses suite in CI now.
Also a minor fix: MCP tool tests intentionally trigger authentication
failures to verify error handling, but the resulting error logs clutter
test output.
2025-10-22 14:59:42 -07:00
Ashwin Bharambe
c582654d70
fix(ci): dont need server: anymore, docker: is sufficient
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 0s
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 1s
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 2s
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped
Python Package Build Test / build (3.13) (push) Failing after 0s
Python Package Build Test / build (3.12) (push) Failing after 1s
Integration Tests (Replay) / Integration Tests (, , , client=, ) (push) Failing after 5s
Test External API and Providers / test-external (venv) (push) Failing after 4s
Vector IO Integration Tests / test-matrix (push) Failing after 6s
Unit Tests / unit-tests (3.12) (push) Failing after 4s
Unit Tests / unit-tests (3.13) (push) Failing after 3s
API Conformance Tests / check-schema-compatibility (push) Successful in 12s
UI Tests / ui-tests (22) (push) Successful in 44s
Pre-commit / pre-commit (push) Successful in 2m14s
2025-10-22 09:13:39 -07:00
Ashwin Bharambe
a701f68bd7
feat(ci): enable docker based server tests ( #3833 )
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 1s
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped
Python Package Build Test / build (3.12) (push) Failing after 3s
Integration Tests (Replay) / Integration Tests (, , , client=, ) (push) Failing after 3s
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 7s
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 9s
Unit Tests / unit-tests (3.12) (push) Failing after 7s
Python Package Build Test / build (3.13) (push) Failing after 12s
Unit Tests / unit-tests (3.13) (push) Failing after 13s
Test External API and Providers / test-external (venv) (push) Failing after 19s
Vector IO Integration Tests / test-matrix (push) Failing after 22s
API Conformance Tests / check-schema-compatibility (push) Successful in 31s
UI Tests / ui-tests (22) (push) Successful in 1m35s
Pre-commit / pre-commit (push) Successful in 2m27s
2025-10-17 09:19:25 +02:00
Ashwin Bharambe
932fea813a
fix(ci): remove responses from CI for now ( #3773 )
...
There are many changes to responses which are landing. They are
introducing fundamental new types. This means re-recordings even from
the inference calls. Let's avoid that for now.
Once everything lands I will re-record everything, make things pass and
re-enable.
2025-10-10 11:52:17 -07:00
Ashwin Bharambe
f50ce11a3b
feat(tests): make inference_recorder into api_recorder (include tool_invoke) ( #3403 )
...
Renames `inference_recorder.py` to `api_recorder.py` and extends it to
support recording/replaying tool invocations in addition to inference
calls.
This allows us to record web-search, etc. tool calls and thereafter
apply recordings for `tests/integration/responses`
## Test Plan
```
export OPENAI_API_KEY=...
export TAVILY_SEARCH_API_KEY=...
./scripts/integration-tests.sh --stack-config ci-tests \
--suite responses --inference-mode record-if-missing
```
2025-10-09 14:27:51 -07:00
Ashwin Bharambe
045a0c1d57
feat(tests): implement test isolation for inference recordings ( #3681 )
...
Uses test_id in request hashes and test-scoped subdirectories to prevent
cross-test contamination. Model list endpoints exclude test_id to enable
merging recordings from different servers.
Additionally, this PR adds a `record-if-missing` mode (which we will use
instead of `record` which records everything) which is very useful.
🤖 Co-authored with [Claude Code](https://claude.com/claude-code )
---------
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-04 11:34:18 -07:00
Ashwin Bharambe
a8aa815b6a
feat(tests): migrate to global "setups" system for test configuration ( #3390 )
...
This PR refactors the integration test system to use global "setups"
which provides better separation of concerns:
**suites = what to test, setups = how to configure.**
NOTE: if you naming suggestions, please provide feedback
Changes:
- New `tests/integration/setups.py` with global, reusable configurations
(ollama, vllm, gpt, claude)
- Modified `scripts/integration-tests.sh` options to match with the
underlying pytest options
- Updated documentation to reflect the new global setup system
The main benefit is that setups can be reused across multiple suites
(e.g., use "gpt" with any suite) even though sometimes they could
specifically tailored for a suite (vision <> ollama-vision). It is now
easier to add new configurations without modifying existing suites.
Usage examples:
- `pytest tests/integration --suite=responses --setup=gpt`
- `pytest tests/integration --suite=vision` # auto-selects
"ollama-vision" setup
- `pytest tests/integration --suite=base --setup=vllm`
2025-09-09 15:50:56 -07:00
Ashwin Bharambe
47b640370e
feat(tests): introduce a test "suite" concept to encompass dirs, options ( #3339 )
...
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 1s
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 0s
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped
Python Package Build Test / build (3.13) (push) Failing after 1s
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 4s
Integration Tests (Replay) / Integration Tests (, , , client=, ) (push) Failing after 4s
Vector IO Integration Tests / test-matrix (push) Failing after 4s
Python Package Build Test / build (3.12) (push) Failing after 3s
Test External API and Providers / test-external (venv) (push) Failing after 4s
Unit Tests / unit-tests (3.12) (push) Failing after 4s
Unit Tests / unit-tests (3.13) (push) Failing after 3s
UI Tests / ui-tests (22) (push) Successful in 33s
Pre-commit / pre-commit (push) Successful in 1m15s
Our integration tests need to be 'grouped' because each group often
needs a specific set of models it works with. We separated vision tests
due to this, and we have a separate set of tests which test "Responses"
API.
This PR makes this system a bit more official so it is very easy to
target these groups and apply all testing infrastructure towards all the
groups (for example, record-replay) uniformly.
There are three suites declared:
- base
- vision
- responses
Note that our CI currently runs the "base" and "vision" suites.
You can use the `--suite` option when running pytest (or any of the
testing scripts or workflows.) For example:
```
OLLAMA_URL=http://localhost:11434 \
pytest -s -v tests/integration/ --stack-config starter --suite vision
```
2025-09-05 13:58:49 -07:00
dependabot[bot]
6a719716f2
chore(github-deps): bump actions/checkout from 4.2.2 to 5.0.0 ( #3178 )
...
[//]: # (dependabot-start)
⚠️ **Dependabot is rebasing this PR** ⚠️
Rebasing might not happen immediately, so don't worry if this takes some
time.
Note: if you make any changes to this PR yourself, they will take
precedence over the rebase.
---
[//]: # (dependabot-end)
Bumps [actions/checkout](https://github.com/actions/checkout ) from 4.2.2
to 5.0.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/checkout/releases ">actions/checkout's
releases</a>.</em></p>
<blockquote>
<h2>v5.0.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Update actions checkout to use node 24 by <a
href="https://github.com/salmanmkc "><code>@salmanmkc</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2226 ">actions/checkout#2226</a></li>
<li>Prepare v5.0.0 release by <a
href="https://github.com/salmanmkc "><code>@salmanmkc</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2238 ">actions/checkout#2238</a></li>
</ul>
<h2>⚠️ Minimum Compatible Runner Version</h2>
<p><strong>v2.327.1</strong><br />
<a
href="https://github.com/actions/runner/releases/tag/v2.327.1 ">Release
Notes</a></p>
<p>Make sure your runner is updated to this version or newer to use this
release.</p>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/checkout/compare/v4...v5.0.0 ">https://github.com/actions/checkout/compare/v4...v5.0.0 </a></p>
<h2>v4.3.0</h2>
<h2>What's Changed</h2>
<ul>
<li>docs: update README.md by <a
href="https://github.com/motss "><code>@motss</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1971 ">actions/checkout#1971</a></li>
<li>Add internal repos for checking out multiple repositories by <a
href="https://github.com/mouismail "><code>@mouismail</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1977 ">actions/checkout#1977</a></li>
<li>Documentation update - add recommended permissions to Readme by <a
href="https://github.com/benwells "><code>@benwells</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2043 ">actions/checkout#2043</a></li>
<li>Adjust positioning of user email note and permissions heading by <a
href="https://github.com/joshmgross "><code>@joshmgross</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2044 ">actions/checkout#2044</a></li>
<li>Update README.md by <a
href="https://github.com/nebuk89 "><code>@nebuk89</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2194 ">actions/checkout#2194</a></li>
<li>Update CODEOWNERS for actions by <a
href="https://github.com/TingluoHuang "><code>@TingluoHuang</code></a>
in <a
href="https://redirect.github.com/actions/checkout/pull/2224 ">actions/checkout#2224</a></li>
<li>Update package dependencies by <a
href="https://github.com/salmanmkc "><code>@salmanmkc</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2236 ">actions/checkout#2236</a></li>
<li>Prepare release v4.3.0 by <a
href="https://github.com/salmanmkc "><code>@salmanmkc</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2237 ">actions/checkout#2237</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/motss "><code>@motss</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/checkout/pull/1971 ">actions/checkout#1971</a></li>
<li><a href="https://github.com/mouismail "><code>@mouismail</code></a>
made their first contribution in <a
href="https://redirect.github.com/actions/checkout/pull/1977 ">actions/checkout#1977</a></li>
<li><a href="https://github.com/benwells "><code>@benwells</code></a>
made their first contribution in <a
href="https://redirect.github.com/actions/checkout/pull/2043 ">actions/checkout#2043</a></li>
<li><a href="https://github.com/nebuk89 "><code>@nebuk89</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/checkout/pull/2194 ">actions/checkout#2194</a></li>
<li><a href="https://github.com/salmanmkc "><code>@salmanmkc</code></a>
made their first contribution in <a
href="https://redirect.github.com/actions/checkout/pull/2236 ">actions/checkout#2236</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/checkout/compare/v4...v4.3.0 ">https://github.com/actions/checkout/compare/v4...v4.3.0 </a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/actions/checkout/blob/main/CHANGELOG.md ">actions/checkout's
changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<h2>V5.0.0</h2>
<ul>
<li>Update actions checkout to use node 24 by <a
href="https://github.com/salmanmkc "><code>@salmanmkc</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2226 ">actions/checkout#2226</a></li>
</ul>
<h2>V4.3.0</h2>
<ul>
<li>docs: update README.md by <a
href="https://github.com/motss "><code>@motss</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1971 ">actions/checkout#1971</a></li>
<li>Add internal repos for checking out multiple repositories by <a
href="https://github.com/mouismail "><code>@mouismail</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1977 ">actions/checkout#1977</a></li>
<li>Documentation update - add recommended permissions to Readme by <a
href="https://github.com/benwells "><code>@benwells</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2043 ">actions/checkout#2043</a></li>
<li>Adjust positioning of user email note and permissions heading by <a
href="https://github.com/joshmgross "><code>@joshmgross</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2044 ">actions/checkout#2044</a></li>
<li>Update README.md by <a
href="https://github.com/nebuk89 "><code>@nebuk89</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2194 ">actions/checkout#2194</a></li>
<li>Update CODEOWNERS for actions by <a
href="https://github.com/TingluoHuang "><code>@TingluoHuang</code></a>
in <a
href="https://redirect.github.com/actions/checkout/pull/2224 ">actions/checkout#2224</a></li>
<li>Update package dependencies by <a
href="https://github.com/salmanmkc "><code>@salmanmkc</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/2236 ">actions/checkout#2236</a></li>
</ul>
<h2>v4.2.2</h2>
<ul>
<li><code>url-helper.ts</code> now leverages well-known environment
variables by <a href="https://github.com/jww3 "><code>@jww3</code></a>
in <a
href="https://redirect.github.com/actions/checkout/pull/1941 ">actions/checkout#1941</a></li>
<li>Expand unit test coverage for <code>isGhes</code> by <a
href="https://github.com/jww3 "><code>@jww3</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1946 ">actions/checkout#1946</a></li>
</ul>
<h2>v4.2.1</h2>
<ul>
<li>Check out other refs/* by commit if provided, fall back to ref by <a
href="https://github.com/orhantoy "><code>@orhantoy</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1924 ">actions/checkout#1924</a></li>
</ul>
<h2>v4.2.0</h2>
<ul>
<li>Add Ref and Commit outputs by <a
href="https://github.com/lucacome "><code>@lucacome</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1180 ">actions/checkout#1180</a></li>
<li>Dependency updates by <a
href="https://github.com/dependabot "><code>@dependabot</code></a>- <a
href="https://redirect.github.com/actions/checkout/pull/1777 ">actions/checkout#1777</a>,
<a
href="https://redirect.github.com/actions/checkout/pull/1872 ">actions/checkout#1872</a></li>
</ul>
<h2>v4.1.7</h2>
<ul>
<li>Bump the minor-npm-dependencies group across 1 directory with 4
updates by <a
href="https://github.com/dependabot "><code>@dependabot</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1739 ">actions/checkout#1739</a></li>
<li>Bump actions/checkout from 3 to 4 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1697 ">actions/checkout#1697</a></li>
<li>Check out other refs/* by commit by <a
href="https://github.com/orhantoy "><code>@orhantoy</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1774 ">actions/checkout#1774</a></li>
<li>Pin actions/checkout's own workflows to a known, good, stable
version. by <a href="https://github.com/jww3 "><code>@jww3</code></a> in
<a
href="https://redirect.github.com/actions/checkout/pull/1776 ">actions/checkout#1776</a></li>
</ul>
<h2>v4.1.6</h2>
<ul>
<li>Check platform to set archive extension appropriately by <a
href="https://github.com/cory-miller "><code>@cory-miller</code></a> in
<a
href="https://redirect.github.com/actions/checkout/pull/1732 ">actions/checkout#1732</a></li>
</ul>
<h2>v4.1.5</h2>
<ul>
<li>Update NPM dependencies by <a
href="https://github.com/cory-miller "><code>@cory-miller</code></a> in
<a
href="https://redirect.github.com/actions/checkout/pull/1703 ">actions/checkout#1703</a></li>
<li>Bump github/codeql-action from 2 to 3 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1694 ">actions/checkout#1694</a></li>
<li>Bump actions/setup-node from 1 to 4 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1696 ">actions/checkout#1696</a></li>
<li>Bump actions/upload-artifact from 2 to 4 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1695 ">actions/checkout#1695</a></li>
<li>README: Suggest <code>user.email</code> to be
<code>41898282+github-actions[bot]@users.noreply.github.com</code> by <a
href="https://github.com/cory-miller "><code>@cory-miller</code></a> in
<a
href="https://redirect.github.com/actions/checkout/pull/1707 ">actions/checkout#1707</a></li>
</ul>
<h2>v4.1.4</h2>
<ul>
<li>Disable <code>extensions.worktreeConfig</code> when disabling
<code>sparse-checkout</code> by <a
href="https://github.com/jww3 "><code>@jww3</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1692 ">actions/checkout#1692</a></li>
<li>Add dependabot config by <a
href="https://github.com/cory-miller "><code>@cory-miller</code></a> in
<a
href="https://redirect.github.com/actions/checkout/pull/1688 ">actions/checkout#1688</a></li>
<li>Bump the minor-actions-dependencies group with 2 updates by <a
href="https://github.com/dependabot "><code>@dependabot</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1693 ">actions/checkout#1693</a></li>
<li>Bump word-wrap from 1.2.3 to 1.2.5 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a> in <a
href="https://redirect.github.com/actions/checkout/pull/1643 ">actions/checkout#1643</a></li>
</ul>
<h2>v4.1.3</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="08c6903cd8https://redirect.github.com/actions/checkout/issues/2238 ">#2238</a>)</li>
<li><a
href="9f265659d3https://redirect.github.com/actions/checkout/issues/2226 ">#2226</a>)</li>
<li><a
href="08eba0b27ehttps://redirect.github.com/actions/checkout/issues/2237 ">#2237</a>)</li>
<li><a
href="631c7dc4f8https://redirect.github.com/actions/checkout/issues/2236 ">#2236</a>)</li>
<li><a
href="8edcb1bdb4https://redirect.github.com/actions/checkout/issues/2224 ">#2224</a>)</li>
<li><a
href="09d2acae67https://redirect.github.com/actions/checkout/issues/2194 ">#2194</a>)</li>
<li><a
href="85e6279cechttps://redirect.github.com/actions/checkout/issues/2044 ">#2044</a>)</li>
<li><a
href="009b9ae9e4https://redirect.github.com/actions/checkout/issues/2043 ">#2043</a>)</li>
<li><a
href="cbb722410chttps://redirect.github.com/actions/checkout/issues/1977 ">#1977</a>)</li>
<li><a
href="3b9b8c884fhttps://redirect.github.com/actions/checkout/issues/1971 ">#1971</a>)</li>
<li>See full diff in <a
href="11bd71901b...08c6903cd8https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/checkout&package-manager=github_actions&previous-version=4.2.2&new-version=5.0.0 )](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores )
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-20 16:51:40 -07:00
Francisco Arceo
ac78e9f66a
chore: Adding UI unit tests in CI ( #3191 )
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 1s
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped
Test Llama Stack Build / generate-matrix (push) Successful in 6s
Python Package Build Test / build (3.12) (push) Failing after 9s
Test Llama Stack Build / build-ubi9-container-distribution (push) Failing after 12s
Integration Tests (Replay) / Integration Tests (, , , client=, vision=) (push) Failing after 14s
Unit Tests / unit-tests (3.12) (push) Failing after 12s
Vector IO Integration Tests / test-matrix (push) Failing after 16s
Test Llama Stack Build / build-single-provider (push) Failing after 15s
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 16s
Test Llama Stack Build / build-custom-container-distribution (push) Failing after 14s
Test External API and Providers / test-external (venv) (push) Failing after 14s
Test Llama Stack Build / build (push) Failing after 9s
Unit Tests / unit-tests (3.13) (push) Failing after 14s
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 21s
Update ReadTheDocs / update-readthedocs (push) Failing after 1m2s
Python Package Build Test / build (3.13) (push) Failing after 1m4s
UI Tests / ui-tests (22) (push) Successful in 1m33s
Pre-commit / pre-commit (push) Successful in 2m38s
2025-08-18 16:48:21 -06:00
Ashwin Bharambe
0e8bb94bf3
feat(ci): make recording workflow simpler, more parameterizable ( #3169 )
...
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 1s
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped
Python Package Build Test / build (3.13) (push) Failing after 4s
Integration Tests (Replay) / Integration Tests (, , , client=, vision=) (push) Failing after 7s
Python Package Build Test / build (3.12) (push) Failing after 12s
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 14s
Update ReadTheDocs / update-readthedocs (push) Failing after 12s
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 17s
Test External API and Providers / test-external (venv) (push) Failing after 15s
Vector IO Integration Tests / test-matrix (push) Failing after 28s
Unit Tests / unit-tests (3.12) (push) Failing after 27s
Unit Tests / unit-tests (3.13) (push) Failing after 51s
Pre-commit / pre-commit (push) Successful in 2m6s
# What does this PR do?
Recording tests has become a nightmare. This is the first part of making
that process simpler by making it _less_ automatic. I tried to be too
clever earlier.
It simplifies the record-integration-tests workflow to use workflow
dispatch inputs instead of PR labels. No more opaque stuff. Just go to
the GitHub UI and run the workflow with inputs. I will soon add a helper
script for this also.
Other things to aid re-running just the small set of things you need to
re-record:
- Replaces the `test-types` JSON array parameter with a more intuitive
`test-subdirs` comma-separated list. The whole JSON array crap was for
matrix.
- Adds a new `test-pattern` parameter to allow filtering tests using
pytest's `-k` option
## Test Plan
Note that this PR is in a fork not the source repository.
- Replay tests on this PR are green
- Manually
[ran](16998562924819508034
2025-08-15 14:47:20 -07:00
Ashwin Bharambe
61582f327c
fix(ci): update triggers for the workflows ( #3152 )
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped
Integration Tests (Replay) / discover-tests (push) Successful in 8s
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 10s
Python Package Build Test / build (3.12) (push) Failing after 6s
Vector IO Integration Tests / test-matrix (3.13, remote::chromadb) (push) Failing after 11s
Vector IO Integration Tests / test-matrix (3.12, remote::qdrant) (push) Failing after 12s
Vector IO Integration Tests / test-matrix (3.12, inline::faiss) (push) Failing after 14s
Unit Tests / unit-tests (3.12) (push) Failing after 8s
Vector IO Integration Tests / test-matrix (3.12, inline::sqlite-vec) (push) Failing after 17s
Python Package Build Test / build (3.13) (push) Failing after 14s
Vector IO Integration Tests / test-matrix (3.12, remote::weaviate) (push) Failing after 20s
Integration Tests (Replay) / Integration Tests (, , , client=, vision=) (push) Failing after 12s
Unit Tests / unit-tests (3.13) (push) Failing after 12s
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 23s
Update ReadTheDocs / update-readthedocs (push) Failing after 13s
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 21s
Vector IO Integration Tests / test-matrix (3.12, remote::chromadb) (push) Failing after 20s
Vector IO Integration Tests / test-matrix (3.12, inline::milvus) (push) Failing after 21s
Test External API and Providers / test-external (venv) (push) Failing after 17s
Vector IO Integration Tests / test-matrix (3.13, remote::pgvector) (push) Failing after 20s
Vector IO Integration Tests / test-matrix (3.13, remote::qdrant) (push) Failing after 14s
Vector IO Integration Tests / test-matrix (3.13, remote::weaviate) (push) Failing after 26s
Vector IO Integration Tests / test-matrix (3.13, inline::milvus) (push) Failing after 25s
Vector IO Integration Tests / test-matrix (3.13, inline::faiss) (push) Failing after 15s
Vector IO Integration Tests / test-matrix (3.12, remote::pgvector) (push) Failing after 19s
Vector IO Integration Tests / test-matrix (3.13, inline::sqlite-vec) (push) Failing after 19s
Pre-commit / pre-commit (push) Successful in 1m39s
2025-08-14 10:27:25 -07:00
Ashwin Bharambe
a9081d87b9
feat(ci): update Recording workflow trigger and concurrency group
2025-08-13 09:36:13 -07:00
Ashwin Bharambe
3d90117891
chore(tests): fix responses and vector_io tests ( #3119 )
...
Some fixes to MCP tests. And a bunch of fixes for Vector providers.
I also enabled a bunch of Vector IO tests to be used with
`LlamaStackLibraryClient`
## Test Plan
Run Responses tests with llama stack library client:
```
pytest -s -v tests/integration/non_ci/responses/ --stack-config=server:starter \
--text-model openai/gpt-4o \
--embedding-model=sentence-transformers/all-MiniLM-L6-v2 \
-k "client_with_models"
```
Do the same with `-k openai_client`
The rest should be taken care of by CI.
2025-08-12 16:15:53 -07:00
Ashwin Bharambe
5f1ddd35e4
chore(tests): refactor and move responses tests away from verifications ( #3068 )
...
This PR kills the verifications infrastructure which is no longer used.
It was relocated to the `llama-stack-evals`
(https://github.com/meta-llama/llama-stack-evals ) repository previously.
Responses tests used this infrastructure but that wasn't quite
necessary, just a little useful back when @bbrownin introduced the
tests. On Discord, we agreed that tests can be moved to our regular
integrations test infra.
## Test Plan
Some tests currently do fail (although they run!) I will send a
follow-up PR which makes them all pass.
2025-08-07 13:48:16 -07:00
ehhuang
1e3b5aa9b8
chore: CI action names ( #3014 )
...
# What does this PR do?
## Test Plan
CI
<img width="795" height="162" alt="image"
src="https://github.com/user-attachments/assets/78dedfa6-809c-4d82-9eb3-6479234dd657 "
/>
2025-08-02 15:56:42 -07:00
Ashwin Bharambe
89ff93182c
feat(ci): only run on 3.12, run on both 3.12 and 3.13 nightly ( #3000 )
...
We don't need to run on all python versions all the time
2025-07-31 15:32:05 -07:00
Ashwin Bharambe
f4489eeb83
fix(ci): simplify integration tests replay mode ( #2997 )
...
We are going to split record and replay workflows completely to simplify
the concurrency key design.
We can add vision tests by just adding to our matrix.
2025-07-31 15:18:18 -07:00
Ashwin Bharambe
22f79bdb9e
fix(ci): lets attempt another fix for concurrency
2025-07-31 13:22:24 -07:00
Ashwin Bharambe
18576349ca
fix(ci): simplified concurrency and job eligibility criteria
2025-07-31 13:11:04 -07:00
Ashwin Bharambe
752fd3b1c1
fix(ci): use single quotes please
2025-07-31 11:56:25 -07:00
Ashwin Bharambe
5ba25efd54
fix(ci): ensure workflow runs when manually run or scheduled
2025-07-31 11:54:51 -07:00
Ashwin Bharambe
27d866795c
feat(ci): add support for running vision inference tests ( #2972 )
...
This PR significantly refactors the Integration Tests workflow. The main
goal behind the PR was to enable recording of vision tests which were
never run as part of our CI ever before. During debugging, I ended up
making several other changes refactoring and hopefully increasing the
robustness of the workflow.
After doing the experiments, I have updated the trigger event to be
`pull_request_target` so this workflow can get write permissions by
default but it will run with source code from the base (main) branch in
the source repository only. If you do change the workflow, you'd need to
experiment using the `workflow_dispatch` triggers. This should not be
news to anyone using Github Actions (except me!)
It is likely to be a little rocky though while I learn more about GitHub
Actions, etc. Please be patient :)
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-07-31 11:50:42 -07:00
Charlie Doern
709c974bd8
fix: integration tests not triggering on PR open ( #2985 )
...
# What does this PR do?
I realized that when a new PR is opened, the integration tests aren't
triggering (or aren't always?) since the replay logic was introduced
amend the concurrency logic a bit to trigger on opened PRs
---------
Signed-off-by: Charlie Doern <cdoern@redhat.com>
Co-authored-by: Ashwin Bharambe <ashwin.bharambe@gmail.com>
2025-07-31 11:36:44 -07:00
Ashwin Bharambe
d6ae2b0f47
fix(ci): more correct concurrency key for workflows ( #2973 )
...
See comment inline. We don't want a random label to pre-empt an existing
workflow which had gone ahead.
2025-07-30 18:23:14 -07:00
Ashwin Bharambe
266e2afb9c
fix(ci): slightly update workflow trigger ( #2966 )
...
We want to avoid re-triggering the workflow when random other labels are
added (e.g., `meta-cla`, etc.) Also no point restarting the workflow
when someone _unlabels_.
2025-07-30 12:04:13 -07:00
Ashwin Bharambe
f6afb3c26b
feat(ci): keep only one re-recording job because independent recordings will conflict ( #2956 )
...
A couple of important updates:
- When recording tests, we cannot be generating a matrix because all the
independent recordings will conflict.
- In fact, we just don't need a matrix on test types any more because
things are very fast and the overhead of `llama stack build` and setting
up `uv` etc. is much more.
- Refactored the running of tests into an independent action
2025-07-29 17:48:04 -07:00
Ashwin Bharambe
b237df8f18
feat(ci): use replay mode, setup ollama if specific label exists on PR ( #2955 )
...
This PR makes setting up Ollama optional for CI. By default, we use
`replay` mode for inference requests and use the stored results from the
`tests/integration/recordings/` directory.
Every so often, users will update tests which will need us to re-record.
To do this, we check for the existence of a label `re-record-tests` on
the PR. If detected,
- ollama is spun up
- inference mode is set to record
- after the tests are done, if any new changes are detected, they are
pushed back to the PR
## Test Plan
This is GitHub CI. Gotta test it live.
2025-07-29 16:50:26 -07:00
Ashwin Bharambe
072d20a124
feat(test): record agents, safety and vector_io integration tests ( #2952 )
...
Continue to build on top of
https://github.com/meta-llama/llama-stack/pull/2941
## Test Plan
Run server with `LLAMA_STACK_TEST_INFERENCE_MODE=record` and then run
the integration tests with `--stack-config=server:starter`. Then restart
the server with `LLAMA_STACK_TEST_INFERENCE_MODE=replay` and re-run the
tests. Verify that no request hit Ollama at any point.
2025-07-29 14:02:14 -07:00
Ashwin Bharambe
9583f468f8
feat(starter)!: simplify starter distro; litellm model registry changes ( #2916 )
2025-07-25 15:02:04 -07:00
Nathan Weinberg
025163d8e6
feat: add auto-generated CI documentation pre-commit hook ( #2890 )
...
# What does this PR do?
Our CI is entirely undocumented, this commit adds a README.md file with
a table of the current CI and what is does
---------
Signed-off-by: Nathan Weinberg <nweinber@redhat.com>
2025-07-25 17:57:01 +02:00
Derek Higgins
4ea1f2aa9f
test: Add VLLM provider support to integration tests ( #2757 )
...
- Add setup-vllm GitHub action to start VLLM container
- Extend integration test matrix to support both ollama and vllm
providers
- Make test setup conditional based on provider type
- Add provider-specific environment variables and configurations
- vllm tests setup to run weekly or can be triggered manually (only
ollama on PR)
TODO:
investigate failing tests for vllm provider (safety and post_training)
Also need a proper fix for #2713 (tmp fix for this in the first commit
in this PR)
Closes : #1648
---------
Signed-off-by: Derek Higgins <derekh@redhat.com>
2025-07-24 09:42:26 -07:00
Ashwin Bharambe
3b83032555
feat(registry): more flexible model lookup ( #2859 )
...
This PR updates model registration and lookup behavior to be slightly
more general / flexible. See
https://github.com/meta-llama/llama-stack/issues/2843 for more details.
Note that this change is backwards compatible given the design of the
`lookup_model()` method.
## Test Plan
Added unit tests
2025-07-22 15:22:48 -07:00
Ashwin Bharambe
dd303327f3
feat(ci): add a ci-tests distro ( #2826 )
2025-07-18 17:11:06 -07:00
Charlie Doern
0eb0583cdf
fix: amend integration test workflow ( #2812 )
...
# What does this PR do?
trigger integration tests on ALL changes to `tests/` to catch failures
before they merge into main
Signed-off-by: Charlie Doern <cdoern@redhat.com>
2025-07-18 15:23:36 +02:00
Sébastien Han
ff9d4d8a9d
ci: do not pull model ( #2776 )
...
the model is now available in the container image
Signed-off-by: Sébastien Han <seb@redhat.com>
2025-07-16 04:58:05 -07:00
ehhuang
e64e4fc5a2
test: add tests against published client ( #2752 )
...
# What does this PR do?
closes #2751
## Test Plan
---------
Co-authored-by: Nathan Weinberg <31703736+nathan-weinberg@users.noreply.github.com>
2025-07-15 12:25:31 -07:00
