dragon.workflows.parsl_executor
Functions
|
Classes
A Dragon multiprocessing-based executor. |
- class DragonPoolExecutor
Bases:
ParslExecutor
,RepresentationMixin
A Dragon multiprocessing-based executor.
Example usage:
import dragon import multiprocessing as mp import parsl from parsl.config import Config from parsl import python_app @python_app def platform(): import platform import time time.sleep(2) return platform.uname() if __name__ == "__main__": mp.set_start_method("dragon") config = Config( executors=[ DragonPoolExecutor( max_processes=4, ), ], strategy=None, ) parsl.load(config) calls = [platform() for i in range(4)] print(calls) for c in calls: print("Got result: ", c.result())
- __init__(label: str = 'dragon', max_processes: int = 2, storage_access: List [Staging] | None = None, working_dir: str | None = None)
- start()
Start the executor.
Any spin-up operations (for example: starting thread pools) should be performed here.
- submit(func, resource_specification, *args, **kwargs)
Submits work to the process pool.
This method is simply pass through and behaves like a submit call as described here Python docs:
- scale_out(workers=1)
Scales out the number of active workers by 1.
This method is notImplemented and will raise the error if called.
- Raises:
NotImplemented exception
- scale_in(blocks)
Scale in the number of active blocks by specified amount.
This method is not implemented and will raise the error if called.
- Raises:
NotImplemented exception
- shutdown(block=True)
Shutdown the DragonPool. The underlying concurrent.futures process pool implementation will not terminate tasks that are being executed, because it does not provide a mechanism to do that. With block set to false, this will return immediately and it will appear as if the DFK is shut down, but the python process will not be able to exit until the process pool has emptied out by task completions. In either case, this can be a very long wait.
- Kwargs:
block (Bool): To block for confirmations or not
- monitor_resources()
Resource monitoring sometimes deadlocks, so this function returns false to disable it.
- main()