feat: add backward compatibility tests for run.yaml (#3952)

This adds automated backward compatibility testing for `run.yaml` files.
As we evolve `StackRunConfig`, changes can inadvertently break existing
user configurations. This workflow catches those breaks before merge.

We test old run.yaml files (from main and the latest release) against
the PR's new code. If configs that worked before now fail, the PR is
blocked unless explicitly acknowledged as a breaking change.

**Two test layers:**
- Schema validation: Quick pytest checks that configs parse without
errors
- Integration tests: Full test suite execution to catch runtime semantic
issues (cross-field validations, provider initialization, etc.)

**What we test against:**
- main branch: Breaking changes here block the PR (this is the gate)
- Latest release: Informational only - shows if we've drifted from what
users have

If tests fail, the PR author must acknowledge the breaking change by
adding `!:` to the PR title (e.g., `feat!: change xyz`) or including
`BREAKING CHANGE:` in a commit message. Once acknowledged, the check
passes with a warning.

These jobs are run:
1. `check-main-compatibility` - Schema validation of all distribution
run.yaml files from main
2. `test-integration-main` - Full integration test suite using main's
ci-tests run.yaml
3. `test-integration-release` - Integration tests with latest release
config (informational)
4. `check-schema-release-compatibility` - Schema checks against release
(informational)

The integration tests catch issues that schema validation alone would
miss, like assertion failures in
`StackRunConfig.validate_server_stores()` or provider-specific runtime
logic.

Resolves #3311
Related to #3237

tests/backward_compat/test_run_config.py

@@ -0,0 +1,52 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the terms described in the LICENSE file in
+# the root directory of this source tree.
+
+"""
+Backward compatibility test for run.yaml files.
+
+This test ensures that changes to StackRunConfig don't break
+existing run.yaml files from previous versions.
+"""
+
+import os
+from pathlib import Path
+
+import pytest
+import yaml
+
+from llama_stack.core.datatypes import StackRunConfig
+
+
+def get_test_configs():
+    configs_dir = os.environ.get("COMPAT_TEST_CONFIGS_DIR")
+    if configs_dir:
+        # CI mode: test configs extracted from main/release
+        config_dir = Path(configs_dir)
+        if not config_dir.exists():
+            pytest.skip(f"Config directory not found: {configs_dir}")
+
+        config_files = sorted(config_dir.glob("*.yaml"))
+        if not config_files:
+            pytest.skip(f"No .yaml files found in {configs_dir}")
+
+        return config_files
+    else:
+        # Local mode: test current distribution configs
+        repo_root = Path(__file__).parent.parent.parent
+        config_files = sorted((repo_root / "src" / "llama_stack" / "distributions").glob("*/run.yaml"))
+
+        if not config_files:
+            pytest.skip("No run.yaml files found in distributions/")
+
+        return config_files
+
+
+@pytest.mark.parametrize("config_file", get_test_configs(), ids=lambda p: p.stem)
+def test_load_run_config(config_file):
+    with open(config_file) as f:
+        config_data = yaml.safe_load(f)
+
+    StackRunConfig.model_validate(config_data)