[go: up one dir, main page]
Skip to content

vllm.v1.kv_hints

Modules:

Classes:

  • KvHintAction –

    A versioned orchestrator-provided KV hint action.

  • KvHintsEnvelope –

    Orchestrator-provided KV hints optionally attached to one inference request.

KvHintAction

Bases: Struct

A versioned orchestrator-provided KV hint action.

Source code in vllm/v1/kv_hints/protocol.py 
 9
10
11
12
13
14
15
16
17
class KvHintAction(msgspec.Struct, frozen=True):  # type: ignore[call-arg]
    """A versioned orchestrator-provided KV hint action."""

    action_id: str
    action_type: str
    action_version: str
    # `payload` must contain JSON-compatible values. Action-specific schemas are
    # validated by the component implementing the action.
    payload: dict[str, Any]

KvHintsEnvelope

Bases: Struct

Orchestrator-provided KV hints optionally attached to one inference request. A KV hint envelope is a collection of KV hint action(s).

Source code in vllm/v1/kv_hints/protocol.py 
20
21
22
23
24
25
26
class KvHintsEnvelope(msgspec.Struct, frozen=True):  # type: ignore[call-arg]
    """Orchestrator-provided KV hints optionally attached to one inference request.
    A KV hint envelope is a collection of KV hint action(s)."""

    protocol_version: str
    message_id: str
    actions: list[KvHintAction]