dragon.ai.inference.batching.BatchItem

class BatchItem[source]

Bases: object

A single item to be batched.

Handles both plain-text prompts and chat-format conversations. Chat items carry optional tools, json_schema_override, and continue_final_message; plain-text items leave them at their defaults.

__init__(user_prompt: str | List [Dict [str , Any ]], formatted_prompt: str | List [Dict [str , Any ]], response_queue: Queue, latency_metrics: Tuple [float , float , float ], tools: List [Dict [str , Any ]] | None = None, json_schema_override: dict | None = None, continue_final_message: bool = False)[source]

Initialize a BatchItem instance.

Parameters:

  • user_prompt (str ) – Raw user input.

  • formatted_prompt (str ) – Formatted prompt with system instructions.

  • response_queue (dragon.native.Queue) – dragon.native.Queue used to send the response.

  • latency_metrics (tuple [float , float , float ]) – Tuple of (entry_time, cpu_latency, guard_latency).

  • tools (list [dict [str , Any]] | None) – Optional tool definitions for chat requests.

  • json_schema_override (dict | None) – Per-request JSON schema for guided decoding.

  • continue_final_message (bool ) – Whether to continue the final assistant message instead of adding a generation prompt.

Methods

__init__(user_prompt, formatted_prompt, ...)

Initialize a BatchItem instance.
__init__(user_prompt: str | List [Dict [str , Any ]], formatted_prompt: str | List [Dict [str , Any ]], response_queue: Queue, latency_metrics: Tuple [float , float , float ], tools: List [Dict [str , Any ]] | None = None, json_schema_override: dict | None = None, continue_final_message: bool = False)[source]

Initialize a BatchItem instance.

Parameters:

  • user_prompt (str ) – Raw user input.

  • formatted_prompt (str ) – Formatted prompt with system instructions.

  • response_queue (dragon.native.Queue) – dragon.native.Queue used to send the response.

  • latency_metrics (tuple [float , float , float ]) – Tuple of (entry_time, cpu_latency, guard_latency).

  • tools (list [dict [str , Any]] | None) – Optional tool definitions for chat requests.

  • json_schema_override (dict | None) – Per-request JSON schema for guided decoding.

  • continue_final_message (bool ) – Whether to continue the final assistant message instead of adding a generation prompt.