The API Stack

Before you start programming with Dragon, you need to decide which API you want to program to. The runtime provides a stack of interfaces abstracting resources of a distributed system, ranging from low-level shared memory to a distributed dictionary. It is composable, meaning the APIs are built on top of each other (see Fig. 6). We deliberately expose the whole stack of APIs so that you can choose if you want to interact with the complete runtime or want to use only parts of it. For an overview, see table 1 in the reference section.

../_images/dragon_api_stack.png

Fig. 6 The Dragon API stack

Lower level interfaces yield less convenient objects. Thus new and experienced users should consider programming to Dragon in two ways:

  1. In Python using Multiprocessing with Dragon, if they want to make an existing Python code scale to a distributed system quickly.

  2. In C, C++, Fortran and Python using the Dragon Native API, if they want to take advantage of all Dragon features or need to use languages other than Python.

The objects provided by these two APIs have the following properties:

  • interoperable: a named Python Multiprocessing Queue object can be used as a managed Dragon Native Queue in C with the same name.

  • transparent: objects can be used everywhere on a system of distributed or even federated nodes.

  • shareable: objects can be serialized and passed to other programs, processes or threads via stdin.

  • managed: objects can be looked up by name or uid to retrieve their serialized descriptor.

  • refcounted: object life-cycle is controlled by the Dragon runtime. It is automatically removed by the runtime if no program, process or thread is using it anymore.

  • thread-safe: users do not have to care about race conditions in objects when they are shared among programs, processes or threads.

  • performant: objects scale to distributed systems and efficiently support workflows across federated systems.

Only the Dragon Native API is polyglot, i.e. available in all supported programming languages.

In the future, experienced developers can further program to the

  1. Unmanaged Dragon Native API, if they want to use composite objects with improved performance. See Performance Costs.

  2. Dragon Client API or Dragon Infrastructure API, if they want to extend the functionality of the Dragon runtime by extending Dragon Services.

  3. Dragon Core API, to use core functionality in their own programs without starting the runtime. To use the Dragon core API on its own, see also Multiprocessing and Dragon without Patching

Architecture of a Dragon Program

../_images/api_use_python.svg

Fig. 7 Architecture of a user program using Dragon with Python Multiprocessing or Dragon Native. Internal Dragon APIs are not shown.

In Fig. 7 we show a component diagram of the architecture of a Dragon program using either the Python Multiprocessing with Dragon API, or the Dragon Native API.

  • User programs using managed Dragon native objects directly call into the Dragon stack. The Dragon native implementation uses the core and client components to implement its objects on top of the four primary objects. Dragon services manage the primary objects and communicate using the infrastructure message component on top of the core API.

  • Python Multiprocessing with Dragon programs only use the Multiprocessing API. Our MPBridge component translates the Multiprocessing objects into Dragon native objects by heavily modifying the object APIs. This way we achieve limited interoperability between both APIs.

Architecture of Advanced Use Cases

../_images/api_use_core.svg

Fig. 8 Architecture of advanced use cases for the Dragon runtime. Internal APIs are not shown.

In Fig. 8 we show a component diagram of the architecture of advanced use cases for Dragon. Note that these use cases are not supported yet.

  • User programs using unmanaged Dragon native objects directly call into Dragon, but do not require the infrastructure services to track names and uids of their objects. This reduces the load on infrastructure services, which only provide transparency across distributed or federated systems.

  • Users may choose to extend the Dragon native API with their own composite objects, using Dragons native, client and core APIs for maximum flexibility.

  • User may want to use only the Dragon core components to extend their own programs with its components. In that case the infrastructure components of Dragon do not need to be started, Dragon core components can be directly imported and used.