dragon.mpbridge.monkeypatching

A class to monkeypatch Dragon classes into Python Multiprocessing.

Monkeypatching fixes direct imports and is necessary, if the user asks for Multiprocessing classes and functions without the context. In particular, people may use direct imports before the start_method is actually set. Hence we have to do this in dragons __init__.py, i.e. before import Multiprocessing is executed by the user for the first time.

Consider:

import dragon
import multiprocessing as mp
from multiprocessing.connection import wait

if __name__ == "__main__":
    mp.set_start_method("dragon")
    ctx = mp.get_context()
    print(f"{wait.__module__=}")
    print(f"{ctx.wait.__module__=}")

without correct patching in __init__.py this would print:

wait.__module__='multiprocessing.connection'
ctx.wait.__module__='dragon.mpbridge.context'

Functions

patch_multiprocessing()

Add Dragon to the list of Multiprocessing start methods.

Classes

AugmentedDefaultContext

This class inserts the Dragon start method 'dragon' in Multiprocessing and enables context switching.

Monkeypatcher

Class managing the monkeypatching functionality.
class Monkeypatcher

Class managing the monkeypatching functionality.

We replace the Multiprocessing API at the top level and at the submodule level to facilitate direct imports. The difference is that the top level import returns a function, while the submodule import returns the class.

To understand this, consider:

import multiprocessing as mp
from multiprocessing.pool import Pool

p1 = mp.Pool # is a method
p2 = Pool # is a class
p3 = mp.Pool() # is the object
p4 = Pool() # is also an object

Note that multiprocessing is not designed in the same way everywhere. The standard pattern is that a function (ctx.Queue()) does import multiprocessing.queues.Queue and returns Queue(). However, some parts of the public API are not below the context, other parts have to be explicitly imported. A mechanism that works for the standard pattern, will leave other parts of the API unchanged.

We store the original classes and functions in private attributes for the brave.

If the switching is done before the user code is parsed, this replaces _all_ mentions of Multiprocessing objects with Dragon ones, including the inheritance tree.

__init__()

Store the original Multiprocessing classes

switch_out(ctx) → None

Replace the standard Multiprocessing classes and functions outside the context with Dragons versions.

To do so, we need the instantiated Dragon context class which is held next to the other context in multiprocessing.context._concrete_contexts.

Parameters

ctx (dragon.mpbridge.context.DragonContext) – The actual Dragon context.

switch_in() → None

Restore standard Multiprocessing classes and functions

class AugmentedDefaultContext

This class inserts the Dragon start method ‘dragon’ in Multiprocessing and enables context switching.

__init__(context, _actual_context)
set_start_method(method: str, *args, **kwargs) → None

Dragons replacement monkeypatches the class as well.

Parameters

method (str) – name of the start method

patch_multiprocessing()

Add Dragon to the list of Multiprocessing start methods.

This function is called when a Python program is started using the Launcher, i.e. Dragon is invoked with the dragon command.

  1. Insert Dragon’s MPBridge context into the list of Multiprocessing contexts.

  2. Replace Multiprocessings default context with our own version to swap Dragon objects in and out when the start methods is changed.

  3. Replace the complete exported API (multiprocessing.X) with Dragon versions.

  4. Replace multiprocessing.spawns start method function with Dragon versions.

  5. Switch out the class hierarchy so the resultion of the inheritance tree can find Dragon’s objects before the entrypoint is reached.