dragon.ai.inference.config.HardwareConfig

class HardwareConfig

Bases: object

Hardware allocation and resource configuration.

Controls which Dragon allocation nodes and GPUs the inference service uses, and how many GPU inference workers are grouped under each CPU-head worker.

Parameters:

• num_nodes (int ) – Number of nodes to use from the current Dragon allocation. -1 means use all available nodes.

• num_gpus (int ) – Number of GPUs to use per selected node. -1 means use all visible GPUs on each selected node.

• num_inf_workers_per_cpu (int ) – Maximum number of tensor-parallel inference workers assigned to one CPU-head worker. -1 means auto-calculate the value as num_gpus // tp_size (with a minimum of one) once the GPU count is resolved.

• node_offset (int ) – Starting node index within the allocation. Use this to give multiple inference services disjoint node slices.

• inf_wrkr_queue_maxsize (int ) – Maximum size of the per-CPU-head inference worker input queue. -1 means auto-calculate the value as num_inf_workers_per_cpu * 2.

__init__(num_nodes: int = -1, num_gpus: int = -1, num_inf_workers_per_cpu: int = -1, node_offset: int = 0, inf_wrkr_queue_maxsize: int = -1) → None

Methods

__init__([num_nodes, num_gpus, ...])
calculate_inf_workers_per_cpu(num_gpus, tp_size)	Calculate recommended inference workers per CPU when set to auto (-1).
validate(all_nodes)	Validate hardware configuration against available resources.

Attributes

inf_wrkr_queue_maxsize
node_offset
num_gpus
num_inf_workers_per_cpu
num_nodes

num_nodes: int = -1

num_gpus: int = -1

num_inf_workers_per_cpu: int = -1

node_offset: int = 0

inf_wrkr_queue_maxsize: int = -1

validate(all_nodes: dict ) → None

Validate hardware configuration against available resources.

Parameters:

all_nodes (dict ) – Dictionary of all available nodes in the cluster. Keys are hostnames, values are dragon.native.machine.Node objects.

Raises:

ValueError – If any configuration parameter is invalid or exceeds available resources.

calculate_inf_workers_per_cpu(num_gpus: int , tp_size: int ) → int

Calculate recommended inference workers per CPU when set to auto (-1).

Formula: num_gpus // tp_size

Rationale: - Each inference worker requires tp_size GPUs - num_gpus // tp_size gives the maximum model instances per node - All model instances are assigned to a single CPU worker per node - Ensures at least 1 worker per CPU

Parameters:

• num_gpus (int ) – Number of GPUs per node.

• tp_size (int ) – Tensor parallelism size (GPUs per model instance).

Returns:

Recommended number of inference workers per CPU worker.

Return type:

int

__init__(num_nodes: int = -1, num_gpus: int = -1, num_inf_workers_per_cpu: int = -1, node_offset: int = 0, inf_wrkr_queue_maxsize: int = -1) → None