dragon.ai.inference.batching.Batch

class Batch

Bases: object

A collection of items to be processed together.

__init__(items: List[BatchItem], batch_id: int, created_at: float)

Initialize a Batch instance.

Parameters:

• items (list[BatchItem]) – List of BatchItem instances.

• batch_id (int) – Unique identifier for the batch.

• created_at (float) – Timestamp when the batch was created.

Methods

__init__(items, batch_id, created_at)

Initialize a Batch instance.

Attributes

continue_final_message_list	Extract chat-template continuation flags from the batch.
formatted_prompts	Extract formatted prompts from batch items.
json_schema_list	Extract per-request JSON schema overrides from the batch.
latency_metrics	Extract latency metrics from batch items.
response_queues	Extract response queues from batch items.
size	Get the batch size.
tools_list	Extract per-request tool definitions from the batch.
user_prompts	Extract user prompts from batch items.

__init__(items: List[BatchItem], batch_id: int, created_at: float)

Initialize a Batch instance.

Parameters:

• items (list[BatchItem]) – List of BatchItem instances.

• batch_id (int) – Unique identifier for the batch.

• created_at (float) – Timestamp when the batch was created.

property size: int

Get the batch size.

Returns:

Number of items in the batch.

Return type:

int

property user_prompts: List[str]

Extract user prompts from batch items.

Returns:

List of user prompts.

Return type:

list[str]

property formatted_prompts: List[str]

Extract formatted prompts from batch items.

Returns:

List of formatted prompts.

Return type:

list[str]

property response_queues: List[Queue]

Extract response queues from batch items.

Returns:

List of response queues.

Return type:

list[dragon.native.Queue]

property latency_metrics: List[Tuple[float, float, float]]

Extract latency metrics from batch items.

Returns:

List of latency metrics tuples.

Return type:

list[tuple[float, float, float]]

property tools_list: List[List[Dict[str, Any]] | None]

Extract per-request tool definitions from the batch.

Chat requests may carry OpenAI-style tool schemas. The batch preserves one entry per request so the LLM process can apply each request’s tool definitions when formatting chat messages.

Returns:

Tool definition lists, or None for requests without tools.

Return type:

list[list[dict] | None]

property json_schema_list: List[dict | None]

Extract per-request JSON schema overrides from the batch.

A schema entry enables vLLM guided or structured decoding for that request. None means the request should use free-form generation.

Returns:

JSON schema dictionaries, or None for free-form requests.

Return type:

list[dict | None]

property continue_final_message_list: List[bool]

Extract chat-template continuation flags from the batch.

When True, the LLM process asks the tokenizer to continue the final assistant message instead of adding a new generation prompt.

Returns:

One continuation flag per batch item.

Return type:

list[bool]