Serializables

Serializable is a wrapper class used for serializing and deserializing C++ objects that will be used in conjunction with the C++ DDict client or the C++ Queue. Additionally, several more specific classes may be used for both Queues and DDicts. The Serializable class definitions are provided here along with a few typedefs/using definitions for common instances of Serializables.

The DerivedSerializable documentation provides an example to be used when writing your own subclasses of SerializableBase or any of the other classes.

template<class Type>
class DerivedSerializable : public dragon::SerializableBase

This class provides the outline of what a subclass of Serializable should look like.

Use this documentation as an outline for writing your own subclasses of Serializable. DO NOT instantiate this class and expect it to do anything.

Public Functions

inline DerivedSerializable(Type obj)

Constructor for DerivedSerializable.

Write your own subclass of Serializable and a constructor for it. You may pass multiple arguments. The constructor is for your own program’s use and is not used by Dragon.

Parameters:

x – A Type of object to wrap.

inline virtual void serialize(dragonFLISendHandleDescr_t *sendh, uint64_t arg, const bool buffer, const timespec_t *timeout) const

Serialize a C++ object.

This method should be overridden in the implementing derived subclass. It should write the bytes of the serialized object to the FLI send handle using the FLI send_bytes interface defined in fli.h. The arg argument should simply be passed through from the serialize function call to the FLI API call for sending bytes. The buffer argument should typically just be passed through to FLI send_byte operations. It will be determined by the context in which serialize is called. For DDict keys, the writes are buffered. For DDict values, the writes are not. But in some cases you may wish to buffer serialized objects and specify true to cause the writes to be consilidated into one network communication.

Parameters:

• sendh – is an FLI send handle used for writing

• arg – is a provided hint. You may override this in some circumstances to create your own hint.

• buffer – is provided or you can override. A value of true on FLI sends will cause written data to be consolidated into one network transfer. The arg is written through to the receiver only when buffer is false.

• timeout – A value of nullptr will wait forever to serialize/transfer data. If value of {0,0} will try once. Otherwise, the timeout specifies how long to wait for the serialization/transfer to be completed.

inline Type val() const

Get the wrapped value for the object.

This method may be named whatever you like. It is not part of the SerializableBase class. And you may define more than one accessor method like this to retrieve parts of your object. You will need something like this to access your deserialized object in your program. The wrapped value (i.e. Type) may also be more than one value which would then be passed to the constructor and you would then have multiple accessor methods to get the various pieces out after deserialization.

Returns:

A value.

inline virtual int type_id() const

Return a unique type id for this type.

This method should return a unique integer to be used to identify this type. The SerializableType enum can provide these values. The return type is left as int to facilitate subclassing and returning your own type values.

Returns:

A unique type id.

Public Static Functions

static inline DerivedSerializable<Type> deserialize(dragonFLIRecvHandleDescr_t *recvh, uint64_t *arg, const timespec_t *timeout)

Deserialize a serialized C++ object.

This method should be written in the implementing derived subclass and should return the derived subtype of SerializableBase. It may throw a DragonError exception when a byte stream is not deserializable. It should throw a EmptyError when EOT is received while reading bytes as it deserializes a value. Assuming that the deserialization succeeds, the deserialize method should return a deserialized value of the derived type after it has read the serialized object’s bytes. The arg argument should be passed through to the FLI API calls for reading bytes or pool memory and will be set according to what was sent when it was serialized. This function relies on NRVO in C++17 and above. This optimization means that the object is initialized in the caller’s space so when the value is returned, it is already in-place. This means we can return a value without making an extra copy.

Parameters:

• recvh – An FLI receive handle. The receive handle is used to read the data of the object. You can read the data using any FLI recvh methods.

• arg – A pointer to a variable to hold the received arg value.

• timeout – A value of nullptr will wait forever. A value of {0,0} will try once. Otherwise, wait for the specified time to receive the object.

Returns:

A DerivedSerializable instance.

class Serializable : public dragon::SerializableBase

The Serializable class can be used to encompass any of the pre-defined Serializable types, providing a means to communicate any Serializable over a Dragon FLI, in particular Queues and DDicts.

The Serializable class wraps objects of other types allowing for them to be safely shared over and FLI connection and understood at the other end by the receiver when they are deserialized. All the pre-defined types are safely wrapped and unwrapped into/from the Serializable class as needed.

class SerializableString : public dragon::SerializableBase

A Serializable string class.

The class provides the Serializable interface for strings.

Public Functions

SerializableString()

Default Constructor for Serializable Strings.

Constructs an empty string.

SerializableString(std::string x)

Constructor for Serializable Strings.

This provides a wrapper class for string values that need to be serialized/deserialized in a Dragon program.

Parameters:

x – An string value to wrap.

virtual void serialize(dragonFLISendHandleDescr_t *sendh, uint64_t arg, const bool buffer, const timespec_t *timeout) const

See the DerivedSerializable serialize description.

std::string val() const

Get the wrapped value for the object.

Returns:

The wrapped value.

virtual int type_id() const

See the DerivedSerializable type_id description.

Public Static Functions

static SerializableString deserialize(dragonFLIRecvHandleDescr_t *recvh, uint64_t *arg, const timespec_t *timeout)

See the DerivedSerializable deserialize description.

template<class Type, int TVal>
class SerializableScalar : public dragon::SerializableBase

A SerializableScalar class.

The class provides the Serializable interface for all Scalar types in C++. There are two pre-defined types provided as instances of this template: SerializableInt and SerializableDouble. Users can define additional SerializableScalars by creating additional instances of this template.

Public Functions

inline SerializableScalar(Type x)

Constructor for SerializableScalar.

This provides a wrapper class for Type values that need to be serialized/deserialized in a Dragon program.

Parameters:

x – An double value to wrap.

inline SerializableScalar()

Constructor for SerializableScalar.

Default constructor.

inline virtual void serialize(dragonFLISendHandleDescr_t *sendh, uint64_t arg, const bool buffer, const timespec_t *timeout) const

See the DerivedSerializable serialize description.

inline Type val() const

Get the wrapped value for the object.

Returns:

The wrapped value.

inline virtual int type_id() const

See the DerivedSerializable type_id description.

Public Static Functions

static inline SerializableScalar deserialize(dragonFLIRecvHandleDescr_t *recvh, uint64_t *arg, const timespec_t *timeout)

See the DerivedSerializable deserialize description.

template<class Type, int TVal>
class SerializableVector : public dragon::SerializableBase

A Serializable Vector of Type.

The class provides the Serializable interface for all vector types in Dragon C++ code. There are two pre-defined types provided as instances of this template: SerializableIntVector and SerializableDoubleVector. Users can define additional SerializableVectors by creating additional instances of this template.

Public Functions

SerializableVector() = default

Default Constructor for Serializable Vector of Type.

Provides an empty vector.

inline SerializableVector(std::vector<Type> obj)

Constructor for Serializable Vector of Type.

This provides a wrapper class for a vector of Type values that need to be serialized/deserialized in a Dragon program.

Parameters:

vec – A Type vector value to wrap.

inline SerializableVector(size_t size)

Constructor for Serializable Vector of Type.

Contruct an empty serializable Type vector with size elements.

Parameters:

size – The number of elements for the empty vector.

inline virtual void serialize(dragonFLISendHandleDescr_t *sendh, uint64_t arg, const bool buffer, const timespec_t *timeout) const

See the DerivedSerializable serialize description.

inline std::vector<Type> val() const

Get the wrapped value for the object.

Returns:

The wrapped value.

inline virtual int type_id() const

See the DerivedSerializable type_id description.

Public Static Functions

static inline SerializableVector<Type, TVal> deserialize(dragonFLIRecvHandleDescr_t *recvh, uint64_t *arg, const timespec_t *timeout)

See the DerivedSerializable deserialize description.

template<class Type, int TVal>
class Serializable2DMatrix : public dragon::SerializableBase

A Serializable 2D Matrix of Type.

The class provides the Serializable interface for all Matrix types in Dragon C++ code. There are two pre-defined types provided as instances of this template: Serializable2DIntMatrix and Serializable2DDoubleMatrix. Users can define additional Serializable2DMatrices by creating additional instances of this template.

Public Functions

Serializable2DMatrix() = default

Default Constructor for Serializable 2D Matrix of Type.

Provides an empty matrix.

inline Serializable2DMatrix(std::vector<std::vector<Type>> obj)

Constructor for Serializable 2D Matrix of Type.

This provides a wrapper class for a matrix of Type values that need to be serialized/deserialized in a Dragon program.

Parameters:

vec – A Type vector value to wrap.

inline Serializable2DMatrix(size_t rows, size_t cols)

Constructor for Serializable 2D Matrix of Type.

Contruct an empty serializable Type Matrix with size elements.

Parameters:

• rows – The number of rows for the empty matrix.

• cols – The number of columns for the empty matrix.

inline virtual void serialize(dragonFLISendHandleDescr_t *sendh, uint64_t arg, const bool buffer, const timespec_t *timeout) const

See the DerivedSerializable serialize description.

inline std::vector<std::vector<Type>> val() const

Get the wrapped value for the object.

Returns:

The wrapped value.

inline virtual int type_id() const

See the DerivedSerializable type_id description.

Public Static Functions

static inline Serializable2DMatrix<Type, TVal> deserialize(dragonFLIRecvHandleDescr_t *recvh, uint64_t *arg, const timespec_t *timeout)

See the DerivedSerializable deserialize description.

template<class Type, int TVal>
class SerializableNDArray : public dragon::SerializableBase

A Serializable NDArray.

The class provides the Serializable interface for generalized matrices of any dimension and size. The Python counterpart is a numpy ndarray. This class makes sharing of data with ndarray objects in Python possible. The C++ version allows the creation of slices of the n-dimensional arrays. All slices within a process share the same cached data to be efficient in storage as possible.

The underlying implementation stores all data within a Dragon DDict and it is locally cached lazily when the array is first indexed. If the data is not indexed, then only metadata is communicated between processes to be as efficient in passing an NDArray between processes, through a Queue or DDict, as possible.

Since there is locally cached data when working with an NDArray, sync should be called when wishing to flush the cache back so all others can see it and refresh should be called when it is known the cache should be refreshed. Note that calling refresh immediately overwrites any locally cached data with the globally available version without regard to possible modifications of the cache. It is up to the program to provide synchronization around syncing and refreshing NDArray data.

Public Functions

inline SerializableNDArray(const std::vector<int> &dimensions, void *data, const char *ser_ddict, const timespec_t *timeout)

Constructor for Serializable NDArray.

Contruct a serializable ndarray with meta data of the ndarray encoded within and the data stored in the provided DDict. When serializing and passing around the ndarray, only the metadata is passed between processes while the ndarray data exists in the DDict and can be retrieved separately when needed.

This means you can pass SerializableNDArrays between processes without copying all the data they refer to until you are actually going to use it. Once referenced, all copies of an NDArray object in a process refer to the same cached data so that multiple copies are not kept within a process (or its threads). This makes working on data from multiple threads convenient. However, care must be taken so that multiple threads are not modifying the same cached data at the same time. When data has been updated in a process, sync should be called to copy that data back to the shared DDict so it can be found by other processes. If another process has updated the NDArray, then refresh can be called to load the latest data into cached data for this process.

Parameters:

• dimensions – A vector of length matching the dimensions of the ndarray. Each value in dimensions is the size of the ndarray in that dimension.

• data – A pointer to the ndarray data. It will be copied into the provided DDict.

• ddict_ser – A serialized Distributed Dictionary in which to store the ndarray data.

• timeout – A pointer to a timeout structure or NULL. The timeout is used for all interactions with the provided DDict attached from the ddict_ser value.

inline void destroy()

Destroy an ndarray by removing its data from the DDict backing store.

An NDArray stores its data in a DDict so only meta data is transferred between processes. This method cleans up that DDict backed data when the ndarray is no longer needed.

inline void ddict_detach()

Detach from an NDArray DDict.

Only call detach if you are completely done with the ndarray and no longer need access to the underlying DDict. Call this after calling destroy on the ndarray.

inline SerializableNDArray(const SerializableNDArray<Type, TVal> &other)

Copy Constructor for Serializable NDArray.

Construct a copy of an existing SerializableNDArray. The copy shares the same underlying DDict entry and key, meaning both ndarrays refer to the same data.

inline virtual int type_id() const override

Please see the documentation for the DerivedSerializable class.

The DerivedSerializable documentation provides a description of what you must write to subclass Serializable and use it in your program.

inline virtual void serialize(dragonFLISendHandleDescr_t *sendh, uint64_t arg, const bool buffer, const timespec_t *timeout) const override

Please see the documentation for the DerivedSerializable class.

The DerivedSerializable documentation provides a description of what you must write to subclass SerializableBase and use it in your program.

inline SerializableNDArray<Type, TVal> operator[](int idx) const

Index operator.

The user can index into a serializable ndarray up to the dimensions of the data. Once all dimensions have been specified, the user can access individual elements of the ndarray. Elements must be of element_size as specified when the ndarray was created. The user can also get the address of any element, including rows within the ndarray. It is assumed that data within the ndarray is stored in row major form abstracted to higher dimensions. The cached data is read lazily when it is actually going to be referenced. This behavior is triggered by calling the index of operator.

Parameters:

idx – The index into the current dimension.

inline void *operator&() const

Address Of.

Provide the address of a particular row or element within the ndarray. This address is only valid within the current process and should not be shared.

inline void refresh() const

Refresh cached data.

If accessing the data of the ndarray, data is cached locally. This method will retrieve the shared data from the DDict source and throw away any currently cached data.

inline void sync()

Update NDArray with Cached data.

Calling this rewrites the ndarray data with cached data.

inline Type val()

Access an element of an NDArray.

When indices have fully specified the location of an element, that element will be yielded by calling this method.

inline int size() const

Return the size of a dimension in the ndarray.

Calling this will return the size of a vector in the ndarray. The vector does not really exist. What is returned is the size of the next unspecified dimension of the ndarray.

inline operator Type()

Conversion operator.

Convert the SerializableNDArray to its template Type. This is useful when all indices have been provided and the user want to access an element at the specified set of indices.

inline SerializableNDArray<Type, TVal> &operator=(Type value)

Assignment operator.

Store a value at the indexed location. The region size for the remaining un-indexed dimensions must match the size of a double.

inline SerializableNDArray<Type, TVal> &operator=(const SerializableNDArray<Type, TVal> &other)

Assignment operator for another NDArray.

Copy data from another ndarray to this ndarray. Both ndarrays must have the same region size.

inline size_t region_size() const

Get the size of the specified region.

Compute and return the size in bytes of the region specified by the current indices. This is the size of the un-indexed sub-region in bytes.

inline SerializableNDArray<Type, TVal> regionAsNDArray() const

Create a new ndarray from the current region.

Constructs a new ndarray representing the current indexed region with the remaining un-indexed dimensions. The new ndarray has empty indices and can be indexed further or accessed directly.

Calling regionAsNDArray makes a copy of the region. It does not share data with the current NDArray. If you want to share data, you can index into the current array and get the address of any specific region.

class SerializableBarrier : public dragon::SerializableBase

A Serializable Dragon Barrier.

The class provides the Serializable interface for Dragon Barriers so that a Barrier may be sent through a Queue or stored in a DDict. Only the base64 encoded descriptor of the Barrier travels between processes, exactly as it does when a Barrier is serialized in Python and attached to in C++. The lifetime of the Barrier is still managed by whoever created it.

Public Functions

SerializableBarrier() = default

Default constructor providing an empty descriptor.

inline SerializableBarrier(const std::string &serialized)

Construct from a base64 encoded serialized Barrier descriptor.

inline SerializableBarrier(const char *serialized)

Construct from a base64 encoded serialized Barrier descriptor.

inline SerializableBarrier(Barrier &barrier)

Construct from an attached Barrier.

This is defined in barrier.hpp because it requires the complete Barrier class.

inline virtual void serialize(dragonFLISendHandleDescr_t *sendh, uint64_t arg, const bool buffer, const timespec_t *timeout) const override

See the DerivedSerializable serialize description.

inline std::string val() const

Get the base64 encoded descriptor of the Barrier.

Assigning a received Serializable to a Barrier attaches to it directly, but the descriptor is available here when an action is needed on the attach. For instance

Barrier barrier(serializable_barrier.val().c_str(), my_action);

Returns:

The serialized descriptor of the Barrier.

inline virtual int type_id() const override

See the DerivedSerializable type_id description.

Public Static Functions

static inline SerializableBarrier deserialize(dragonFLIRecvHandleDescr_t *recvh, uint64_t *arg, const timespec_t *timeout)

See the DerivedSerializable deserialize description.

template<class Serializable>
class SerializableQueue : public dragon::SerializableBase

A Serializable Dragon Queue.

The class provides the Serializable interface for Dragon Queues so that a Queue may itself be sent through a Queue or stored in a DDict. Only the base64 encoded descriptor of the Queue travels between processes, exactly as it does when a Queue is serialized in Python and attached to in C++. The lifetime of the Queue is still managed by whoever created it.

The template argument mirrors the Queue template argument. It is the type of the values carried by the Queue that this descriptor refers to.

Public Functions

SerializableQueue() = default

Default constructor providing an empty descriptor.

inline SerializableQueue(const std::string &serialized)

Construct from a base64 encoded serialized Queue descriptor.

inline SerializableQueue(const char *serialized)

Construct from a base64 encoded serialized Queue descriptor.

SerializableQueue(Queue<Serializable> &queue)

Construct from an attached Queue.

This is defined in queue.hpp because it requires the complete Queue template.

inline virtual void serialize(dragonFLISendHandleDescr_t *sendh, uint64_t arg, const bool buffer, const timespec_t *timeout) const override

See the DerivedSerializable serialize description.

inline std::string val() const

Get the base64 encoded descriptor of the Queue.

Pass this to the Queue attach constructor to interact with the Queue. For instance

Queue<Serializable> queue(serializable_queue.val().c_str(), nullptr);

Returns:

The serialized descriptor of the Queue.

inline virtual int type_id() const override

See the DerivedSerializable type_id description.

Public Static Functions

static inline SerializableQueue<Serializable> deserialize(dragonFLIRecvHandleDescr_t *recvh, uint64_t *arg, const timespec_t *timeout)

See the DerivedSerializable deserialize description.

class SerializableSemaphore : public dragon::SerializableBase

A Serializable Dragon Semaphore.

The class provides the Serializable interface for Dragon Semaphores so that a Semaphore may be sent through a Queue or stored in a DDict. Only the base64 encoded descriptor of the Semaphore travels between processes, exactly as it does when a Semaphore is serialized in Python and attached to in C++. The lifetime of the Semaphore is still managed by whoever created it.

Public Functions

SerializableSemaphore() = default

Default constructor providing an empty descriptor.

inline SerializableSemaphore(const std::string &serialized)

Construct from a base64 encoded serialized Semaphore descriptor.

inline SerializableSemaphore(const char *serialized)

Construct from a base64 encoded serialized Semaphore descriptor.

inline SerializableSemaphore(Semaphore &semaphore)

Construct from an attached Semaphore.

This is defined in semaphore.hpp because it requires the complete Semaphore class.

inline virtual void serialize(dragonFLISendHandleDescr_t *sendh, uint64_t arg, const bool buffer, const timespec_t *timeout) const override

See the DerivedSerializable serialize description.

inline std::string val() const

Get the base64 encoded descriptor of the Semaphore.

Returns:

The serialized descriptor of the Semaphore.

inline virtual int type_id() const override

See the DerivedSerializable type_id description.

Public Static Functions

static inline SerializableSemaphore deserialize(dragonFLIRecvHandleDescr_t *recvh, uint64_t *arg, const timespec_t *timeout)

See the DerivedSerializable deserialize description.