Workflows

The following reference provides details about the Dragon Workflows API. Tools for graph-based execution of tasks, data proessing, and integrations with other workflow systems are included.

Python Reference

Batch for Graph-based Execution

Run functions, executables, and parallel applications through a high-level, graph-based tasking API.

Batch	Graph-based distributed scheduling for functions, executables, and parallel applications
BatchTopology	Describes the placement of managers and worker pools across nodes.

Function
Job
MakeTask	Note on terminology: we use the terms "process" and "resolve" in a number of places in this class and in helper functions for this class.
SubmitAfterCloseError	Exception for submitting work to a batch instance after it's been closed.
Task

Data Processing Tools

Prototype API for creating dataflow processing pipelines.

data_mover

Classes for efficiently moving data between a host and devices

Proxy for Multi-System Workflow Execution

Create workflows that span multiple systems, with a proxy API that abstracts away the details of remote execution and data movement

Proxy	Object providing API to manage Dragon runtime connections for remote execution.
publish(name[, publish_dir])	Publish the current runtime descriptor to make it discoverable by other systems.
lookup(system, name[, timeout_in, publish_dir])	Look up and retrieve a published runtime descriptor from a remote system.
attach(sdesc_str[, oob_ssh_tunnel_override, ...])	Attach to a remote Dragon runtime using a serialized descriptor.

Other Workflow Tools

Dragon can integrate into other workflow tools and enhance their scalability and performance. These APIs are integrations into other tools. These are currently works-in-progress and are useful primarily for experimentation.

parsl_batch_executor	An expiremental executor for parsl that uses Dragon's multiprocessing pool to submit batches of tasks to workers.
parsl_executor
parsl_mpi_app	Dragon's experimental @mpi_app decorator and DragonMPIExecutor for parsl.