phoenix-oss/llama-stack-mirror
0
0
Fork 1
mirror of https://github.com/meta-llama/llama-stack.git synced 2025-08-12 13:00:39 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat: Enable DPO training with HuggingFace inline provider (#2825)
Some checks failed
Integration Tests / discover-tests (push) Has been skipped
Details
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 7s
Details
Integration Tests / record-tests (push) Has been skipped
Details
Integration Tests / run-tests (push) Has been skipped
Details
Vector IO Integration Tests / test-matrix (3.12, inline::milvus) (push) Failing after 22s
Details
Python Package Build Test / build (3.13) (push) Failing after 16s
Details
Test Llama Stack Build / generate-matrix (push) Successful in 19s
Details
Vector IO Integration Tests / test-matrix (3.13, inline::milvus) (push) Failing after 21s
Details
Vector IO Integration Tests / test-matrix (3.12, inline::sqlite-vec) (push) Failing after 31s
Details
Test Llama Stack Build / build-custom-container-distribution (push) Failing after 32s
Details
Test External API and Providers / test-external (venv) (push) Failing after 32s
Details
Vector IO Integration Tests / test-matrix (3.13, remote::chromadb) (push) Failing after 36s
Details
Vector IO Integration Tests / test-matrix (3.12, remote::chromadb) (push) Failing after 39s
Details
Update ReadTheDocs / update-readthedocs (push) Failing after 31s
Details
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 42s
Details
Test Llama Stack Build / build-single-provider (push) Failing after 37s
Details
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Failing after 35s
Details
Test Llama Stack Build / build-ubi9-container-distribution (push) Failing after 37s
Details
Vector IO Integration Tests / test-matrix (3.13, remote::pgvector) (push) Failing after 40s
Details
Vector IO Integration Tests / test-matrix (3.12, remote::pgvector) (push) Failing after 42s
Details
Unit Tests / unit-tests (3.12) (push) Failing after 36s
Details
Vector IO Integration Tests / test-matrix (3.13, inline::sqlite-vec) (push) Failing after 40s
Details
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 45s
Details
Test Llama Stack Build / build (push) Failing after 6s
Details
Python Package Build Test / build (3.12) (push) Failing after 1m1s
Details
Unit Tests / unit-tests (3.13) (push) Failing after 1m0s
Details
Vector IO Integration Tests / test-matrix (3.13, inline::faiss) (push) Failing after 1m6s
Details
Vector IO Integration Tests / test-matrix (3.12, inline::faiss) (push) Failing after 1m8s
Details
Pre-commit / pre-commit (push) Successful in 1m50s
Details

Browse source 
What does this PR do?

This PR adds support for Direct Preference Optimization (DPO) training
via the existing HuggingFace inline provider. It introduces a new DPO
training recipe, config schema updates, dataset integration, and
end-to-end testing to support preference-based fine-tuning with TRL.

Test Plan

Added integration test:

tests/integration/post_training/test_post_training.py::TestPostTraining::test_preference_optimize

Ran tests on both CPU and CUDA environments

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-43-83.ec2.internal>
Co-authored-by: Ashwin Bharambe <ashwin.bharambe@gmail.com>
This commit is contained in:
Nehanth Narendrula 2025-07-31 02:33:36 -04:00 • committed by GitHub
parent 2665f00102
commit cf73146132
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 913 additions and 215 deletions
Download patch file Download diff file Expand all files Collapse all files

4
docs/source/providers/post_training/inline_huggingface.md
View file

@ -24,6 +24,10 @@ HuggingFace-based post-training provider for fine-tuning models using the Huggin
| `weight_decay` | `<class 'float'>` | No | 0.01 | |
| `dataloader_num_workers` | `<class 'int'>` | No | 4 | |
| `dataloader_pin_memory` | `<class 'bool'>` | No | True | |
| `dpo_beta` | `<class 'float'>` | No | 0.1 | |
| `use_reference_model` | `<class 'bool'>` | No | True | |
| `dpo_loss_type` | `Literal['sigmoid', 'hinge', 'ipo', 'kto_pair'` | No | sigmoid | |
| `dpo_output_dir` | `<class 'str'>` | No | ./checkpoints/dpo | |
## Sample Configuration