dragon.native.pool

The Dragon native pool manages a pool of child processes.

Functions

mapstar(args)
rebuild_exc(exc, tb)

Classes

ApplyResult	Returned by apply_async and has get method to retrieve result.
AsyncResult	alias of ApplyResult
ExceptionWithTraceback
MapResult
Pool	A Dragon native Pool relying on native Process Group and Queues

Exceptions

MaybeEncodingError	Wraps possible unpickleable errors, so they can be safely sent through the socket.
RemoteTraceback

exception RemoteTraceback

__init__(tb)

exception MaybeEncodingError

Wraps possible unpickleable errors, so they can be safely sent through the socket.

__init__(exc, value)

class ApplyResult

Returned by apply_async and has get method to retrieve result.

__init__(pool: Pool, callback: callable, error_callback: callable)

Initializes a result

Parameters

• pool (dragon.native.pool.Pool) – the pool where work has been submitted

• callback (callable) – function called on returned result

• error_callback (callable) – function called on returned result if an error was raised

ready() → bool

Checks whether the result is ready.

Returns

returns True if the result is ready or False if it isn’t

Return type

bool

successful() → bool

Checks if the result is ready and returns whether the function call was successful.

Raises

ValueError – raised if result is not ready

Returns

returns True if function call was successful

Return type

bool

wait(timeout: Optional[float] = None) → None

Waits on event for result to be ready

Parameters

timeout (float, optional) – timeout indicating how long to wait for result to be ready, defaults to None

get(timeout: Optional[float] = None) → Any

Retrieves returned values of work submitted to pool

Parameters

timeout (float, optional) – timeout for getting result, defaults to None

Raises

• TimeoutError – raised if result is not ready in specified timeout

• self._value – raises error if returned work was unsuccessful

Returns

value returned by func(*args, **kwargs)

Return type

Any

AsyncResult

alias of ApplyResult

class MapResult

__init__(pool: Pool, chunksize: int, length: int, callback: callable, error_callback: callable)

Initialization method

Parameters

• pool (dragon.native.pool.Pool) – the pool where work is submitted

• chunksize (int) – size that work is supposed to be broken up into

• length (int) – number of items in iterable

• callback (callable) – function to be called upon return of result

• error_callback (callable) – function to be called upon return of result if an error was raised.

class Pool

A Dragon native Pool relying on native Process Group and Queues

The interface resembles the Python Multiprocessing.Pool interface and focuses on the most generic functionality. Here we directly support the asynchronous API. The synchronous version of these calls are supported by calling get on the objects returned from the asynchronous functions. By using a Dragon native Process Group to coordinate the worker processes this implementation addresses scalability limitations with the patched base implementation of Multiprocessing.Pool.

At this time, both terminate and close send a signal.SIGTERM that the workers catch. Using close guarantees that all work submitted to the pool is finished before the signal is sent while terminate sends the signal immediately. The user is expected to call join following both of these calls. If join is not called, zombie processes may be left.

__init__(processes: Optional[int] = None, initializer: Optional[callable] = None, initargs: tuple = (), maxtasksperchild: Optional[int] = None, policy: Optional[Policy] = None, *args, **kwargs)

Init method

Parameters

• processes (int, optional) – number of worker processes, defaults to None

• initializer (callable, optional) – initializer function, defaults to None

• initargs (tuple, optional) – arguments for initializer function, defaults to ()

• maxtasksperchild (int, optional) – maximum tasks each worker will perform, defaults to None

• policy (dragon.infrastructure.policy.Policy) – determines the placement of the processes

Raises

ValueError – raised if number of worker processes is less than 1

terminate() → None

This sends the signal.SIGTERM and sets the threading event immediately. Calling this method before blocking on results may lead to some work being undone. If all work should be done before stopping the workers, close should be used. If join is not called after terminate zombie processes may be left over and interfere with future pool or process group use.

close() → None

This method starts a thread that waits for all submitted jobs to finish. This thread then sends signal.SIGTERM to the process group and sets the threading event to close the handle_results thread. Waiting and then sending the signals within the thread allows close to return without all work needing to be done.

Raises

ValueError – raised if close or terminate have been previously called

join() → None

Waits for all workers to return. The user must have called close or terminate prior to calling join.

Raises

ValueError – raised if close or terminate were not previously called

apply_async(func: callable, args: tuple = (), kwargs: dict = {}, callback: Optional[callable] = None, error_callback: Optional[callable] = None) → ApplyResult

Equivalent to calling func(*args, **kwargs) in a non-blocking way. A result is immediately returned and then updated when func(*args, **kwargs) has completed.

Parameters

• func (callable) – user function to be called by worker function

• args (tuple, optional) – input args to func, defaults to ()

• kwargs (dict, optional) – input kwargs to func, defaults to {}

• callback (callable, optional) – user provided callback function, defaults to None

• error_callback (callable, optional) – user provided error callback function, defaults to None

Raises

ValueError – raised if pool has already been closed or terminated

Returns

A result that has a get method to retrieve result of func(*args, **kwargs)

Return type

ApplyResult

map_async(func: callable, iterable: Iterable, chunksize: Optional[int] = None, callback: Optional[callable] = None, error_callback: Optional[callable] = None) → MapResult

Apply func to each element in iterable. The results are collected in a list that is returned immediately. It is equivalent to map if the get method is called immediately on the returned result.

Parameters

• func (callable) – user provided function to call on elements of iterable

• iterable (iterable) – input args to func

• chunksize (int, optional) – size of work elements to be submitted to input work queue, defaults to None

• callback (callable, optional) – user provided callback function, defaults to None

• error_callback (callable, optional) – user provided error callback function, defaults to None

Returns

A result that has a get method that returns an iterable of the output from applying func to each element of iterable.

Return type

MapResult