llama-stack-mirror/llama_stack/providers/inline
Botao Chen c294a01c4b
[2/n][torchtune integration] implement job management and return training artifacts (#593)
### Context 
In this PR, we 
- Implement the post training job management and get training artifacts
apis
  - get_training_jobs
  - get_training_job_status
  - get_training_job_artifacts
- get_training_job_logstream is deleted since the trace can be directly
accessed by UI with Jaeger
https://llama-stack.readthedocs.io/en/latest/building_applications/telemetry.html#jaeger-to-visualize-traces
- Refactor the post training and training types definition to make them
more intuitive.
- Rewrite the checkpointer to make it compatible with llama-stack file
system and can be recognized during inference


### Test
Unit test
`pytest llama_stack/providers/tests/post_training/test_post_training.py
-m "torchtune_post_training_huggingface_datasetio" -v -s --tb=short
--disable-warnings`

<img width="1506" alt="Screenshot 2024-12-10 at 4 06 17 PM"
src="https://github.com/user-attachments/assets/16225029-bdb7-48c4-9d13-e580cc769c0a">


e2e test with client side call

<img width="888" alt="Screenshot 2024-12-10 at 4 09 44 PM"
src="https://github.com/user-attachments/assets/de375e4c-ef67-4dcc-a045-4037d9489191">
2024-12-13 15:00:04 -08:00
..
agents add tracing back to the lib cli (#595) 2024-12-11 08:44:20 -08:00
datasetio Telemetry API redesign (#525) 2024-12-04 11:22:45 -08:00
eval Add ability to query and export spans to dataset (#574) 2024-12-05 21:07:30 -08:00
inference add embedding model by default to distribution templates (#617) 2024-12-13 12:48:00 -08:00
ios/inference impls -> inline, adapters -> remote (#381) 2024-11-06 14:54:05 -08:00
memory Make embedding generation go through inference (#606) 2024-12-12 11:47:50 -08:00
meta_reference Telemetry API redesign (#525) 2024-12-04 11:22:45 -08:00
post_training/torchtune [2/n][torchtune integration] implement job management and return training artifacts (#593) 2024-12-13 15:00:04 -08:00
safety use logging instead of prints (#499) 2024-11-21 11:32:53 -08:00
scoring [/scoring] add ability to define aggregation functions for scoring functions & refactors (#597) 2024-12-11 10:03:42 -08:00
telemetry add tracing back to the lib cli (#595) 2024-12-11 08:44:20 -08:00
__init__.py impls -> inline, adapters -> remote (#381) 2024-11-06 14:54:05 -08:00