Generate Text (vLLM)

How it fits

Typical backends

Caveats

• Hard constraintA colocated vLLM service serving model_name is required; it is declared as a dependency and deployed alongside, with its address read from the environment at startup. An API key is sent only when one is configured in the environment, so the service must accept whichever form the client uses.
• I/O contractThe two inputs are the user message and the system message for one chat turn; generated text leaves on the first output and an end-of-turn marker on the second. With streaming on the reply arrives as incremental chunks, otherwise as one chunk at completion, and emit_stream_end is what produces the StreamEnd.
• Parameter interactionhistory_messages set to zero runs stateless, negative keeps the full conversation, and positive keeps the last N turns; max_parallel only takes effect in the stateless case, since carrying history forces sequential turns.
• Parameter interactionpreempt_on_new_input cancels the in-flight stream when a fresh prompt arrives, so the abandoned turn surfaces downstream as a truncated stream.
• Fallbackjson_mode requests a JSON object from the service; service builds that do not honour it return free-form text instead, and the output is not re-validated here.
• Latencytimeout_seconds caps each request, and max_tokens caps per-turn length; long generations need a higher timeout or they fail with a network error.
• State lifetimeConversation history lives in the process and resets on restart, and every knob binds once at startup, so changes require a redeploy.