The CI integration tests were failing with a signature mismatch error, but the root cause was missing dependencies (specifically the 'together' package). The signature mismatch was a misleading error that occurred because the provider modules failed to load properly due to missing dependencies.
This fix adds a step to install all ci-tests distribution dependencies using:
llama stack list-deps ci-tests | xargs -L1 uv pip install
This ensures all required provider dependencies are installed before running tests.
The issue was timing - we were clearing cache before installations,
but uv sync/pip install were creating new .pyc files. This commit:
1. Adds PYTHONDONTWRITEBYTECODE=1 to prevent .pyc generation
2. Clears bytecode cache AFTER all installations complete
3. Ensures no stale .pyc files exist before tests run
For editable installs (-e .), Python loads from source directory,
so clearing cache after installation ensures the resolver sees the
latest method signatures with the authorization parameter.
The GitHub Actions cache was restoring a cached virtual environment
(.venv) with old code. This commit clears all caching layers:
1. Removes cached .venv directory (the main culprit)
2. Clears Python bytecode cache (.pyc files)
3. Clears UV cache directory
This forces uv sync to create a completely fresh virtual environment
with the latest source code changes, ensuring the authorization
parameter is picked up across all tool runtime providers.
The previous approach of removing uv.lock caused dependency resolution
failures. The real issue is the UV_CACHE_DIR that contains pre-built
wheels with old code. This commit:
1. Keeps uv.lock (it's part of the project)
2. Clears UV_CACHE_DIR (where compiled wheels are cached)
3. Forces uv to rebuild wheels from source
This ensures the latest source code changes are picked up without
breaking dependency resolution.
The uv.lock file contains cached dependency resolutions that prevent
source code changes from being picked up. By removing it before uv sync,
we force a fresh resolution and rebuild of dependencies.
This should fix the 73 CI test failures where the resolver was loading
stale method signatures without the authorization parameter.
The real issue was stale .pyc bytecode files in __pycache__ directories.
These cached files contained the old method signatures without the
authorization parameter, causing signature mismatch errors even though
the source .py files were correct.
Now clearing all __pycache__ directories and .pyc files before the
force-reinstall to ensure Python loads fresh bytecode from the updated
source files.
The CI was using a cached/stale version of the package that didn't
include our authorization parameter changes. Add explicit force
reinstall step to ensure the latest source code is used.
Fixes latent bug where UV_INDEX_STRATEGY was only exported to GITHUB_ENV
but not to the current shell.
While this bug doesn't currently affect main (since UV_EXTRA_INDEX_URL
is only set on release branches), it's a latent bug that could cause
issues if the logic changes in the future or if someone tests with
UV_EXTRA_INDEX_URL set.
The setup-runner action only exported UV_INDEX_STRATEGY to GITHUB_ENV
(for subsequent steps), not to the current shell environment. Since uv
sync runs in the same step, it would never see the variable if it were
set.
This fix adds `export UV_INDEX_STRATEGY=unsafe-best-match` to make the
variable available in the current shell before running uv commands.
Related: #4019 (same fix for release-0.3.x where the bug is actively
triggered)
Backports UV index configuration fixes from `release-0.3.x` (PR #4002).
The main issue: when we created the release branch infrastructure, we
configured UV to use `test.pypi` as the PRIMARY index to resolve RC
dependencies. This caused UV to look for ALL packages there first, which
led to problems - some packages don't have binary wheels on `test.pypi`,
so UV tried building from source and failed (like the `psycopg2-binary`
issue we hit).
The fix is simple: use PyPI as primary (default) and `test.pypi` as an
EXTRA index. UV will check PyPI first for everything, and only fall back
to `test.pypi` for packages not found there (like our RC client
versions).
This PR includes:
- Fixed `install-llama-stack-client` action to output
`UV_EXTRA_INDEX_URL` instead of `UV_INDEX_URL`
- New `uv-run-with-index.sh` wrapper that auto-detects release branches
and sets UV env vars
- Updated pre-commit hooks (`uv-lock`, codegen, etc.) to use the wrapper
- Pass UV env vars as Docker build args in all locations
- Scope UV env vars properly in Containerfile (inline for llama-stack
install, explicitly unset before distribution deps)
- Export UV env vars to `GITHUB_ENV` in setup-runner for cross-step
persistence
The wrapper detects release branches automatically in both CI and local
environments, so this "just works" without manual configuration. On main
(non-release branch), the wrapper becomes a no-op.
Tested and validated on `release-0.3.x` where all CI checks pass.
Fixes CI failures on release branches where uv sync can't resolve RC
dependencies.
The problem: on release branches like `release-0.3.x`, pyproject.toml
requires `llama-stack-client>=0.3.1rc1`. But RC versions only exist on
test.pypi, not PyPI. So uv sync fails before we even get a chance to
install the client from git.
The fix is simple - on release branches, pre-install the client from the
matching git branch first, then run uv sync. This satisfies the RC
requirement and lets dependency resolution succeed.
Modified setup-runner and pre-commit workflows to do this. Also cleaned
up some duplicate logic in setup-test-environment that's now handled
centrally.
Example failure:
5415478835
Reverts llamastack/llama-stack#3576
When I edit Stainless and codegen succeeds, the `next` branch is updated
directly. It provides us no chance to see if there might be something
unideal going on. If something is wrong, all CI will start breaking
immediately. This is not ideal. I will likely create another staging
branch `next-release` or something to accomodate the special workflow
that Stainless requires.
When we update Stainless (editor changes), the `next` branch gets
updated. Eventually when one decides on a release, you land changes into
`main`. This is the Stainless workflow.
This PR makes sure we follow that workflow by pulling from the `next`
branch for our integration tests.
I started this PR trying to unbreak a newly broken test
`test_agent_name`. This test was broken all along but did not show up
because during testing we were pulling the "non-updated" llama stack
client. See this comment:
https://github.com/llamastack/llama-stack/pull/3119#discussion_r2270988205
While fixing this, I encountered a large amount of badness in our CI
workflow definitions.
- We weren't passing `LLAMA_STACK_DIR` or `LLAMA_STACK_CLIENT_DIR`
overrides to `llama stack build` at all in some cases.
- Even when we did, we used `uv run` liberally. The first thing `uv run`
does is "syncs" the project environment. This means, it is going to undo
any mutations we might have done ourselves. But we make many mutations
in our CI runners to these environments. The most important of which is
why `llama stack build` where we install distro dependencies. As a
result, when you tried to run the integration tests, you would see old,
strange versions.
## Test Plan
Re-record using:
```
sh scripts/integration-tests.sh --stack-config ci-tests \
--provider ollama --test-pattern test_agent_name --inference-mode record
```
Then re-run with `--inference-mode replay`. But:
Eventually, this test turned out to be quite flaky for telemetry
reasons. I haven't investigated it for now and just disabled it sadly
since we have a release to push out.
Some fixes to MCP tests. And a bunch of fixes for Vector providers.
I also enabled a bunch of Vector IO tests to be used with
`LlamaStackLibraryClient`
## Test Plan
Run Responses tests with llama stack library client:
```
pytest -s -v tests/integration/non_ci/responses/ --stack-config=server:starter \
--text-model openai/gpt-4o \
--embedding-model=sentence-transformers/all-MiniLM-L6-v2 \
-k "client_with_models"
```
Do the same with `-k openai_client`
The rest should be taken care of by CI.
# What does this PR do?
dropped python3.10, updated pyproject and dependencies, and also removed
some blocks of code with special handling for enum.StrEnum
Closes#2458
Signed-off-by: Charlie Doern <cdoern@redhat.com>
# What does this PR do?
Expand the test matrix to include Python 3.10, 3.11, and 3.12 to ensure
the project runs correctly on these versions. This will give us
confidence to begin considering an increase to the project's minimum
supported Python version.
Signed-off-by: Sébastien Han <seb@redhat.com>
# What does this PR do?
The previous `[project.optional-dependencies]` was misrepresenting what
the packages were. They were NOT optional dependencies to the project
but development dependencies. Unlike optional dependencies, development
dependencies are local-only and will not be included in the project
requirements when published to PyPI or other indexes. As such,
development dependencies are not included in the [project] table.
Additionally, the dev group is synced by default.
Source:
https://docs.astral.sh/uv/concepts/projects/dependencies/#development-dependencies
Signed-off-by: Sébastien Han <seb@redhat.com>
# What does this PR do?
Use a composite action to avoid similar steps repetitions and
centralization of the defaults.
Signed-off-by: Sébastien Han <seb@redhat.com>
