# MLflow Prompts Provider - Integration Tests

This directory contains integration tests for the MLflow Prompts Provider. These tests require a running MLflow server.

## Prerequisites

1. **MLflow installed**: `pip install 'mlflow>=3.4.0'` (or `uv pip install 'mlflow>=3.4.0'`)
2. **MLflow server running**: See setup instructions below
3. **Test dependencies**: `uv sync --group test`

## Quick Start

### 1. Start MLflow Server

```bash
# Start MLflow server on localhost:5555
mlflow server --host 127.0.0.1 --port 5555

# Keep this terminal open - server will continue running
```

### 2. Run Integration Tests

In a separate terminal:

```bash
# Set MLflow URI (optional - defaults to localhost:5555)
export MLFLOW_TRACKING_URI=http://localhost:5555

# Run all integration tests
uv run --group test pytest -sv tests/integration/providers/remote/prompts/mlflow/

# Run specific test
uv run --group test pytest -sv tests/integration/providers/remote/prompts/mlflow/test_end_to_end.py::TestMLflowPromptsEndToEnd::test_create_and_retrieve_prompt
```

### 3. Run Manual Test Script (Optional)

For quick validation without pytest:

```bash
# Run manual test script
uv run python scripts/test_mlflow_prompts_manual.py

# View output in MLflow UI
open http://localhost:5555
```

## Test Organization

### Integration Tests (`test_end_to_end.py`)

Comprehensive end-to-end tests covering:

- ✅ Create and retrieve prompts
- ✅ Update prompts (version management)
- ✅ List prompts (default versions only)
- ✅ List all versions of a prompt
- ✅ Set default version
- ✅ Variable auto-extraction
- ✅ Variable validation
- ✅ Error handling (not found, wrong version, etc.)
- ✅ Complex templates with multiple variables
- ✅ Edge cases (empty templates, no variables, etc.)

**Total**: 17 test scenarios

### Manual Test Script (`scripts/test_mlflow_prompts_manual.py`)

Interactive test script with verbose output for:

- Server connectivity check
- Provider initialization
- Basic CRUD operations
- Variable extraction
- Statistics retrieval

## Configuration

### MLflow Server Options

**Local (default)**:
```bash
mlflow server --host 127.0.0.1 --port 5555
```

**Remote server**:
```bash
export MLFLOW_TRACKING_URI=http://mlflow.example.com:5000
uv run --group test pytest -sv tests/integration/providers/remote/prompts/mlflow/
```

**Databricks**:
```bash
export MLFLOW_TRACKING_URI=databricks
export MLFLOW_REGISTRY_URI=databricks://profile
uv run --group test pytest -sv tests/integration/providers/remote/prompts/mlflow/
```

### Test Timeout

Tests have a default timeout of 30 seconds per MLflow operation. Adjust in `conftest.py`:

```python
MLflowPromptsConfig(
    mlflow_tracking_uri=mlflow_tracking_uri,
    timeout_seconds=60,  # Increase for slow connections
)
```

## Fixtures

### `mlflow_adapter`

Basic adapter for simple tests:

```python
async def test_something(mlflow_adapter):
    prompt = await mlflow_adapter.create_prompt(...)
    # Test continues...
```

### `mlflow_adapter_with_cleanup`

Adapter with automatic cleanup tracking:

```python
async def test_something(mlflow_adapter_with_cleanup):
    # Creates are tracked and attempted cleanup on teardown
    prompt = await mlflow_adapter_with_cleanup.create_prompt(...)
```

**Note**: MLflow doesn't support deletion, so cleanup is best-effort.

## Troubleshooting

### Server Not Available

**Symptom**:
```
SKIPPED [1] conftest.py:35: MLflow server not available at http://localhost:5555
```

**Solution**:
```bash
# Start MLflow server
mlflow server --host 127.0.0.1 --port 5555

# Verify it's running
curl http://localhost:5555/health
```

### Connection Timeout

**Symptom**:
```
requests.exceptions.Timeout: ...
```

**Solutions**:
1. Check MLflow server is responsive: `curl http://localhost:5555/health`
2. Increase timeout in `conftest.py`: `timeout_seconds=60`
3. Check firewall/network settings

### Import Errors

**Symptom**:
```
ModuleNotFoundError: No module named 'mlflow'
```

**Solution**:
```bash
uv pip install 'mlflow>=3.4.0'
```

### Permission Errors

**Symptom**:
```
PermissionError: [Errno 13] Permission denied: '...'
```

**Solution**:
- Ensure MLflow has write access to its storage directory
- Check file permissions on `mlruns/` directory

### Test Isolation Issues

**Issue**: Tests may interfere with each other if using same prompt IDs

**Solution**: Each test creates new prompts with unique IDs (generated by `Prompt.generate_prompt_id()`). If needed, use `mlflow_adapter_with_cleanup` fixture.

## Viewing Results

### MLflow UI

1. Start MLflow server (if not already running):
   ```bash
   mlflow server --host 127.0.0.1 --port 5555
   ```

2. Open in browser:
   ```
   http://localhost:5555
   ```

3. Navigate to experiment `test-llama-stack-prompts`

4. View registered prompts and their versions

### Test Output

Run with verbose output to see detailed test execution:

```bash
uv run --group test pytest -vv tests/integration/providers/remote/prompts/mlflow/
```

## CI/CD Integration

To run tests in CI/CD pipelines:

```yaml
# Example GitHub Actions workflow
- name: Start MLflow server
  run: |
    mlflow server --host 127.0.0.1 --port 5555 &
    sleep 5  # Wait for server to start

- name: Wait for MLflow
  run: |
    timeout 30 bash -c 'until curl -s http://localhost:5555/health; do sleep 1; done'

- name: Run integration tests
  env:
    MLFLOW_TRACKING_URI: http://localhost:5555
  run: |
    uv run --group test pytest -sv tests/integration/providers/remote/prompts/mlflow/
```

## Performance

### Expected Test Duration

- **Individual test**: ~1-5 seconds
- **Full suite** (17 tests): ~30-60 seconds
- **Manual script**: ~10-15 seconds

### Optimization Tips

1. Use local MLflow server (faster than remote)
2. Run tests in parallel (if safe):
   ```bash
   uv run --group test pytest -n auto tests/integration/providers/remote/prompts/mlflow/
   ```
3. Skip integration tests in development:
   ```bash
   uv run --group dev pytest -sv tests/unit/
   ```

## Coverage

Integration tests provide coverage for:

- ✅ Real MLflow API calls
- ✅ Network communication
- ✅ Serialization/deserialization
- ✅ MLflow server responses
- ✅ Version management
- ✅ Alias handling
- ✅ Tag storage and retrieval

Combined with unit tests, achieves **>95% code coverage**.