llama-stack/llama_stack/apis
Botao Chen c294a01c4b
[2/n][torchtune integration] implement job management and return training artifacts (#593)
### Context 
In this PR, we 
- Implement the post training job management and get training artifacts
apis
  - get_training_jobs
  - get_training_job_status
  - get_training_job_artifacts
- get_training_job_logstream is deleted since the trace can be directly
accessed by UI with Jaeger
https://llama-stack.readthedocs.io/en/latest/building_applications/telemetry.html#jaeger-to-visualize-traces
- Refactor the post training and training types definition to make them
more intuitive.
- Rewrite the checkpointer to make it compatible with llama-stack file
system and can be recognized during inference


### Test
Unit test
`pytest llama_stack/providers/tests/post_training/test_post_training.py
-m "torchtune_post_training_huggingface_datasetio" -v -s --tb=short
--disable-warnings`

<img width="1506" alt="Screenshot 2024-12-10 at 4 06 17 PM"
src="https://github.com/user-attachments/assets/16225029-bdb7-48c4-9d13-e580cc769c0a">


e2e test with client side call

<img width="888" alt="Screenshot 2024-12-10 at 4 09 44 PM"
src="https://github.com/user-attachments/assets/de375e4c-ef67-4dcc-a045-4037d9489191">
2024-12-13 15:00:04 -08:00
..
agents memory retrival to print only the bytes injected 2024-12-10 09:32:18 -08:00
batch_inference Add version to REST API url (#478) 2024-11-18 22:44:14 -08:00
common [2/n][torchtune integration] implement job management and return training artifacts (#593) 2024-12-13 15:00:04 -08:00
datasetio Telemetry API redesign (#525) 2024-12-04 11:22:45 -08:00
datasets unregister API for dataset (#507) 2024-12-03 21:18:30 -08:00
eval Add version to REST API url (#478) 2024-11-18 22:44:14 -08:00
eval_tasks Add version to REST API url (#478) 2024-11-18 22:44:14 -08:00
inference Console span processor improvements (#577) 2024-12-06 11:46:16 -08:00
inspect Remove "routing_table" and "routing_key" concepts for the user (#201) 2024-10-10 10:24:13 -07:00
memory Console span processor improvements (#577) 2024-12-06 11:46:16 -08:00
memory_banks Make embedding generation go through inference (#606) 2024-12-12 11:47:50 -08:00
models add embedding model by default to distribution templates (#617) 2024-12-13 12:48:00 -08:00
post_training [2/n][torchtune integration] implement job management and return training artifacts (#593) 2024-12-13 15:00:04 -08:00
safety Console span processor improvements (#577) 2024-12-06 11:46:16 -08:00
scoring Add version to REST API url (#478) 2024-11-18 22:44:14 -08:00
scoring_functions [/scoring] add ability to define aggregation functions for scoring functions & refactors (#597) 2024-12-11 10:03:42 -08:00
shields Console span processor improvements (#577) 2024-12-06 11:46:16 -08:00
synthetic_data_generation Add version to REST API url (#478) 2024-11-18 22:44:14 -08:00
telemetry Regenerate openapi 2024-12-08 20:46:22 -08:00
__init__.py API Updates (#73) 2024-09-17 19:51:35 -07:00
resource.py Enable sane naming of registered objects with defaults (#429) 2024-11-12 11:18:05 -08:00
version.py Fix the pyopenapi generator avoid potential circular imports 2024-11-18 23:37:52 -08:00