phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-10-04 04:04:14 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

test: introduce api conformance test (#3257)
Some checks failed
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 1s
Details
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 0s
Details
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 0s
Details
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped
Details
Python Package Build Test / build (3.12) (push) Failing after 1s
Details
Vector IO Integration Tests / test-matrix (push) Failing after 4s
Details
Python Package Build Test / build (3.13) (push) Failing after 3s
Details
Integration Tests (Replay) / Integration Tests (, , , client=, ) (push) Failing after 7s
Details
Test External API and Providers / test-external (venv) (push) Failing after 4s
Details
Unit Tests / unit-tests (3.12) (push) Failing after 4s
Details
Unit Tests / unit-tests (3.13) (push) Failing after 4s
Details
API Conformance Tests / check-schema-compatibility (push) Successful in 12s
Details
UI Tests / ui-tests (22) (push) Successful in 40s
Details
Pre-commit / pre-commit (push) Successful in 1m48s
Details

Browse source 
# What does this PR do?

this test runs on each PR and uses a new conformance workflow to compare
the base (main) branch openapi spec to the one on this PR if one of our
"stable" APIs change, the test will fail.

this workflow uses `oasdiff` to identify breaking changes for paths we
want to ensure comptability for.

specifically this is using `oasdiff breaking` with `--match-path` which
only checks breaking changes for the specified paths.

As a follow up to this, we can add an optional way to make it so that it
is OK to make these change if properly documented or in a changelog or
something. or by using a label on the PR to override the failing test.

related to #3237


## Test Plan

conformance test should pass given there are no changes

Signed-off-by: Charlie Doern <cdoern@redhat.com>
This commit is contained in:
Charlie Doern 2025-09-06 15:40:33 -04:00 • committed by GitHub
parent 9252d9fc01
commit ecd9d8dc1a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 58 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.github/workflows/README.md vendored
View file

@ -5,6 +5,7 @@ Llama Stack uses GitHub Actions for Continuous Integration (CI). Below is a tabl
| Name | File | Purpose |
| ---- | ---- | ------- |
| Update Changelog | [changelog.yml](changelog.yml) | Creates PR for updating the CHANGELOG.md |
| API Conformance Tests | [conformance.yml](conformance.yml) | Run the API Conformance test suite on the changes. |
| Installer CI | [install-script-ci.yml](install-script-ci.yml) | Test the installation script |
| Integration Auth Tests | [integration-auth-tests.yml](integration-auth-tests.yml) | Run the integration test suite with Kubernetes authentication |
| SqlStore Integration Tests | [integration-sql-store-tests.yml](integration-sql-store-tests.yml) | Run the integration test suite with SqlStore |