llama-stack-mirror/llama_stack
Botao Chen c294a01c4b
[2/n][torchtune integration] implement job management and return training artifacts (#593)
### Context 
In this PR, we 
- Implement the post training job management and get training artifacts
apis
  - get_training_jobs
  - get_training_job_status
  - get_training_job_artifacts
- get_training_job_logstream is deleted since the trace can be directly
accessed by UI with Jaeger
https://llama-stack.readthedocs.io/en/latest/building_applications/telemetry.html#jaeger-to-visualize-traces
- Refactor the post training and training types definition to make them
more intuitive.
- Rewrite the checkpointer to make it compatible with llama-stack file
system and can be recognized during inference


### Test
Unit test
`pytest llama_stack/providers/tests/post_training/test_post_training.py
-m "torchtune_post_training_huggingface_datasetio" -v -s --tb=short
--disable-warnings`

<img width="1506" alt="Screenshot 2024-12-10 at 4 06 17 PM"
src="https://github.com/user-attachments/assets/16225029-bdb7-48c4-9d13-e580cc769c0a">


e2e test with client side call

<img width="888" alt="Screenshot 2024-12-10 at 4 09 44 PM"
src="https://github.com/user-attachments/assets/de375e4c-ef67-4dcc-a045-4037d9489191">
2024-12-13 15:00:04 -08:00
..
apis [2/n][torchtune integration] implement job management and return training artifacts (#593) 2024-12-13 15:00:04 -08:00
cli doc: llama-stack build --config help text references old directory (#596) 2024-12-10 17:42:02 -08:00
distribution add embedding model by default to distribution templates (#617) 2024-12-13 12:48:00 -08:00
providers [2/n][torchtune integration] implement job management and return training artifacts (#593) 2024-12-13 15:00:04 -08:00
scripts Integrate distro docs into the restructured docs 2024-11-20 23:20:05 -08:00
templates add embedding model by default to distribution templates (#617) 2024-12-13 12:48:00 -08:00
__init__.py export LibraryClient 2024-12-13 12:08:00 -08:00