dragon.workflows.parsl_executor

Functions

main()

Classes

DragonPoolExecutor

A Dragon multiprocessing-based executor.

class DragonPoolExecutor

A Dragon multiprocessing-based executor.

Parameters

max_processesint

Number of processes. Default is 2.

storage_accesslist of Staging

Specifications for accessing data this executor remotely.

__init__(label: str = 'dragon', max_processes: int = 2, storage_access: Optional[List[Staging]] = None, working_dir: Optional[str] = None)

start()

Start the executor.

Any spin-up operations (for example: starting thread pools) should be performed here.

submit(func, resource_specification, *args, **kwargs)

Submits work to the process pool.

This method is simply pass through and behaves like a submit call as described here Python docs:

scale_out(workers=1)

Scales out the number of active workers by 1.

This method is notImplemented and will raise the error if called.

Raises:

NotImplemented exception

scale_in(blocks)

Scale in the number of active blocks by specified amount.

This method is not implemented and will raise the error if called.

Raises:

NotImplemented exception

shutdown(block=True)

Shutdown the DragonPool. The underlying concurrent.futures process pool implementation will not terminate tasks that are being executed, because it does not provide a mechanism to do that. With block set to false, this will return immediately and it will appear as if the DFK is shut down, but the python process will not be able to exit until the process pool has emptied out by task completions. In either case, this can be a very long wait.

Kwargs:

• block (Bool): To block for confirmations or not

monitor_resources()

Resource monitoring sometimes deadlocks, so this function returns false to disable it.