dragon.ai.inference.config.BatchingConfig

class BatchingConfig

Bases: object

Request batching configuration.

Controls whether requests are sent to vLLM one at a time, collected by the service over a short time window, or supplied as caller-created batches.

Parameters:

• enabled (bool ) – Enable batching. If False, requests are processed with a batch size of one.

• batch_type (str ) – Batching mode. "dynamic" lets the service assemble batches; "pre-batch" expects callers to submit prompt lists.

• batch_wait_seconds (float ) – Maximum time to hold a dynamic batch open before flushing it to the LLM process.

• max_batch_size (int ) – Maximum number of requests in one vLLM generation call. Also used as vLLM max_num_seqs.

__init__(enabled: bool = True, batch_type: str = 'dynamic', batch_wait_seconds: float = 0.1, max_batch_size: int = 60) → None

Methods

__init__([enabled, batch_type, ...])
validate()	Validate batching configuration.

Attributes

batch_type
batch_wait_seconds
enabled
max_batch_size

enabled: bool = True

batch_type: str = 'dynamic'

batch_wait_seconds: float = 0.1

max_batch_size: int = 60

validate() → None

Validate batching configuration.

__init__(enabled: bool = True, batch_type: str = 'dynamic', batch_wait_seconds: float = 0.1, max_batch_size: int = 60) → None