dragon.workflows.parsl_executor
Functions
|
Classes
A Dragon multiprocessing-based executor. |
- class DragonPoolExecutor
Bases:
ParslExecutor,RepresentationMixinA Dragon multiprocessing-based executor.
Parameters
- max_processesint
Number of processes. Default is 2.
- storage_accesslist of
Staging Specifications for accessing data this executor remotely.
- __init__(label: str = 'dragon', max_processes: int = 2, storage_access: List[Staging] | None = None, working_dir: str | None = None)
- start()
Start the executor.
Any spin-up operations (for example: starting thread pools) should be performed here.
- submit(func, resource_specification, *args, **kwargs)
Submits work to the process pool.
This method is simply pass through and behaves like a submit call as described here Python docs:
- scale_out(workers=1)
Scales out the number of active workers by 1.
This method is notImplemented and will raise the error if called.
- Raises:
NotImplemented exception
- scale_in(blocks)
Scale in the number of active blocks by specified amount.
This method is not implemented and will raise the error if called.
- Raises:
NotImplemented exception
- shutdown(block=True)
Shutdown the DragonPool. The underlying concurrent.futures process pool implementation will not terminate tasks that are being executed, because it does not provide a mechanism to do that. With block set to false, this will return immediately and it will appear as if the DFK is shut down, but the python process will not be able to exit until the process pool has emptied out by task completions. In either case, this can be a very long wait.
- Kwargs:
block (Bool): To block for confirmations or not
- monitor_resources()
Resource monitoring sometimes deadlocks, so this function returns false to disable it.
- main()