mirror of
https://github.com/meta-llama/llama-stack.git
synced 2025-06-27 18:50:41 +00:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Ihar Hrachyshka
|
a2f054607d
|
fix: cancel scheduler tasks on shutdown (#2130)
# What does this PR do?
Scheduler: cancel tasks on shutdown.
Otherwise the currently running tasks will never exit (before they
actually complete), which means the process can't be properly shut down
(only with SIGKILL).
Ideally, we let tasks know that they are about to shutdown and give them
some time to do so; but in the lack of the mechanism, it's better to
cancel than linger forever.
[//]: # (If resolving an issue, uncomment and update the line below)
[//]: # (Closes #[issue-number])
## Test Plan
Start a long running task (e.g. torchtune or external kfp-provider
training).
Ctr-C the process in TTY. Confirm it exits in reasonable time.
```
^CINFO: Shutting down
INFO: Waiting for application shutdown.
13:32:26.187 - INFO - Shutting down
13:32:26.187 - INFO - Shutting down DatasetsRoutingTable
13:32:26.187 - INFO - Shutting down DatasetIORouter
13:32:26.187 - INFO - Shutting down TorchtuneKFPPostTrainingImpl
Traceback (most recent call last):
File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/Users/ihrachys/src/llama-stack-provider-kfp-trainer/.venv/lib/python3.12/site-packages/kfp/dsl/executor_main.py", line 109, in <module>
executor_main()
File "/Users/ihrachys/src/llama-stack-provider-kfp-trainer/.venv/lib/python3.12/site-packages/kfp/dsl/executor_main.py", line 101, in executor_main
output_file = executor.execute()
^^^^^^^^^^^^^^^^^^
File "/Users/ihrachys/src/llama-stack-provider-kfp-trainer/.venv/lib/python3.12/site-packages/kfp/dsl/executor.py", line 361, in execute
result = self.func(**func_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/folders/45/1q1rx6cn7jbcn2ty852w0g_r0000gn/T/tmp.RKpPrvTWDD/ephemeral_component.py", line 118, in component
asyncio.run(recipe.setup())
File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/runners.py", line 194, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/runners.py", line 123, in run
raise KeyboardInterrupt()
KeyboardInterrupt
13:32:31.219 - ERROR - Task 'component' finished with status FAILURE
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
INFO 2025-05-09 13:32:31,221 llama_stack.providers.utils.scheduler:221 scheduler: Job
test-jobc3c2e1e4-859c-4852-a41d-ef29e55e3efa: Pipeline [1m[95m'test-jobc3c2e1e4-859c-4852-a41d-ef29e55e3efa'[1m[0m
finished with status [1m[91mFAILURE[1m[0m. Inner task failed: [1m[96m'component'[1m[0m.
ERROR 2025-05-09 13:32:31,223 llama_stack_provider_kfp_trainer.scheduler:54 scheduler: Job
test-jobc3c2e1e4-859c-4852-a41d-ef29e55e3efa failed.
╭───────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────╮
│ /Users/ihrachys/src/llama-stack-provider-kfp-trainer/src/llama_stack_provider_kfp_trainer/scheduler.py:45 │
│ in do │
│ │
│ 42 │ │ │ │
│ 43 │ │ │ job.status = JobStatus.running │
│ 44 │ │ │ try: │
│ ❱ 45 │ │ │ │ artifacts = self._to_artifacts(job.handler().output) │
│ 46 │ │ │ │ for artifact in artifacts: │
│ 47 │ │ │ │ │ on_artifact_collected_cb(artifact) │
│ 48 │
│ │
│ /Users/ihrachys/src/llama-stack-provider-kfp-trainer/.venv/lib/python3.12/site-packages/kfp/dsl/base_compon │
│ ent.py:101 in __call__ │
│ │
│ 98 │ │ │ │ f'{self.name}() missing {len(missing_arguments)} required ' │
│ 99 │ │ │ │ f'{argument_or_arguments}: {arguments}.') │
│ 100 │ │ │
│ ❱ 101 │ │ return pipeline_task.PipelineTask( │
│ 102 │ │ │ component_spec=self.component_spec, │
│ 103 │ │ │ args=task_inputs, │
│ 104 │ │ │ execute_locally=pipeline_context.Pipeline.get_default_pipeline() is │
│ │
│ /Users/ihrachys/src/llama-stack-provider-kfp-trainer/.venv/lib/python3.12/site-packages/kfp/dsl/pipeline_ta │
│ sk.py:187 in __init__ │
│ │
│ 184 │ │ ]) │
│ 185 │ │ │
│ 186 │ │ if execute_locally: │
│ ❱ 187 │ │ │ self._execute_locally(args=args) │
│ 188 │ │
│ 189 │ def _execute_locally(self, args: Dict[str, Any]) -> None: │
│ 190 │ │ """Execute the pipeline task locally. │
│ │
│ /Users/ihrachys/src/llama-stack-provider-kfp-trainer/.venv/lib/python3.12/site-packages/kfp/dsl/pipeline_ta │
│ sk.py:197 in _execute_locally │
│ │
│ 194 │ │ from kfp.local import task_dispatcher │
│ 195 │ │ │
│ 196 │ │ if self.pipeline_spec is not None: │
│ ❱ 197 │ │ │ self._outputs = pipeline_orchestrator.run_local_pipeline( │
│ 198 │ │ │ │ pipeline_spec=self.pipeline_spec, │
│ 199 │ │ │ │ arguments=args, │
│ 200 │ │ │ ) │
│ │
│ /Users/ihrachys/src/llama-stack-provider-kfp-trainer/.venv/lib/python3.12/site-packages/kfp/local/pipeline_ │
│ orchestrator.py:43 in run_local_pipeline │
│ │
│ 40 │ │
│ 41 │ # validate and access all global state in this function, not downstream │
│ 42 │ config.LocalExecutionConfig.validate() │
│ ❱ 43 │ return _run_local_pipeline_implementation( │
│ 44 │ │ pipeline_spec=pipeline_spec, │
│ 45 │ │ arguments=arguments, │
│ 46 │ │ raise_on_error=config.LocalExecutionConfig.instance.raise_on_error, │
│ │
│ /Users/ihrachys/src/llama-stack-provider-kfp-trainer/.venv/lib/python3.12/site-packages/kfp/local/pipeline_ │
│ orchestrator.py:108 in _run_local_pipeline_implementation │
│ │
│ 105 │ │ │ ) │
│ 106 │ │ return outputs │
│ 107 │ elif dag_status == status.Status.FAILURE: │
│ ❱ 108 │ │ log_and_maybe_raise_for_failure( │
│ 109 │ │ │ pipeline_name=pipeline_name, │
│ 110 │ │ │ fail_stack=fail_stack, │
│ 111 │ │ │ raise_on_error=raise_on_error, │
│ │
│ /Users/ihrachys/src/llama-stack-provider-kfp-trainer/.venv/lib/python3.12/site-packages/kfp/local/pipeline_ │
│ orchestrator.py:137 in log_and_maybe_raise_for_failure │
│ │
│ 134 │ │ logging_utils.format_task_name(task_name) for task_name in fail_stack) │
│ 135 │ msg = f'Pipeline {pipeline_name_with_color} finished with status │
│ {status_with_color}. Inner task failed: {task_chain_with_color}.' │
│ 136 │ if raise_on_error: │
│ ❱ 137 │ │ raise RuntimeError(msg) │
│ 138 │ with logging_utils.local_logger_context(): │
│ 139 │ │ logging.error(msg) │
│ 140 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Pipeline [1m[95m'test-jobc3c2e1e4-859c-4852-a41d-ef29e55e3efa'[1m[0m finished with status
[1m[91mFAILURE[1m[0m. Inner task failed: [1m[96m'component'[1m[0m.
INFO 2025-05-09 13:32:31,266 llama_stack.distribution.server.server:136 server: Shutting down
DistributionInspectImpl
INFO 2025-05-09 13:32:31,266 llama_stack.distribution.server.server:136 server: Shutting down ProviderImpl
INFO: Application shutdown complete.
INFO: Finished server process [26648]
```
[//]: # (## Documentation)
Signed-off-by: Ihar Hrachyshka <ihar.hrachyshka@gmail.com>
|
||
Charlie Doern
|
d12f195f56
|
feat: drop python 3.10 support (#2469)
# What does this PR do? dropped python3.10, updated pyproject and dependencies, and also removed some blocks of code with special handling for enum.StrEnum Closes #2458 Signed-off-by: Charlie Doern <cdoern@redhat.com> |
||
|
Ihar Hrachyshka
|
9e6561a1ec
|
chore: enable pyupgrade fixes (#1806)
# What does this PR do? The goal of this PR is code base modernization. Schema reflection code needed a minor adjustment to handle UnionTypes and collections.abc.AsyncIterator. (Both are preferred for latest Python releases.) Note to reviewers: almost all changes here are automatically generated by pyupgrade. Some additional unused imports were cleaned up. The only change worth of note can be found under `docs/openapi_generator` and `llama_stack/strong_typing/schema.py` where reflection code was updated to deal with "newer" types. Signed-off-by: Ihar Hrachyshka <ihar.hrachyshka@gmail.com> |
||
|
Ihar Hrachyshka
|
3ed4316ed5
|
feat: Implement async job execution for torchtune training (#1437)
# What does this PR do? Now a separate thread is started to execute training jobs. Training requests now return job ID before the job completes. (Which fixes API timeouts for any jobs that take longer than a minute.) Note: the scheduler code is meant to be spun out in the future into a common provider service that can be reused for different APIs and providers. It is also expected to back the /jobs API proposed here: https://github.com/meta-llama/llama-stack/discussions/1238 Hence its somewhat generalized form which is expected to simplify its adoption elsewhere in the future. Note: this patch doesn't attempt to implement missing APIs (e.g. cancel or job removal). This work will belong to follow-up PRs. [//]: # (If resolving an issue, uncomment and update the line below) [//]: # (Closes #[issue-number]) ## Test Plan [Describe the tests you ran to verify your changes with result summaries. *Provide clear instructions so the plan can be easily re-executed.*] Added unit tests for the scheduler module. For the API coverage, did manual testing and was able to run a training cycle on GPU. The initial call returned job ID before the training completed, as (now) expected. Artifacts are returned as expected. ``` JobArtifactsResponse(checkpoints=[{'identifier': 'meta-llama/Llama-3.2-3B-Instruct-sft-0', 'created_at': '2025-03-07T22:45:19.892714', 'epoch': 0, 'post_training_job_id': 'test-job2ee77104-2fd3-4a4e-84cf-f83f8b8f1f50', 'path': '/home/ec2-user/.llama/checkpoints/meta-llama/Llama-3.2-3B-Instruct-sft-0', 'training_metrics': None}], job_uuid='test-job2ee77104-2fd3-4a4e-84cf-f83f8b8f1f50') ``` The integration test is currently disabled for the provider. I will look into how it can be enabled in a different PR / issue context. [//]: # (## Documentation) Signed-off-by: Ihar Hrachyshka <ihar.hrachyshka@gmail.com> |