Generate Text (SGLang)

How it fits

Typical backends

Caveats

• Hard constraintA colocated SGLang service serving model_name is required; it is declared as a dependency and deployed alongside, with its address read from the environment at startup and its OpenAI-style route called without authentication, so the service must accept unauthenticated requests.
• I/O contractThe two inputs are the user message and the system message for one chat turn; generated text leaves on the first output and an end-of-turn marker on the second. With streaming on the reply arrives as incremental chunks, otherwise as one chunk at completion, and emit_stream_end is what produces the StreamEnd.
• Parameter interactionhistory_messages set to zero runs stateless, negative keeps the full conversation, and positive keeps the last N turns; max_parallel only takes effect in the stateless case, since carrying history forces sequential turns.
• Parameter interactionpreempt_on_new_input cancels the in-flight stream when a fresh prompt arrives, so the abandoned turn surfaces downstream as a truncated stream.
• Accuracyjson_mode asks the service for a JSON object but the output is not re-validated here, so a non-conforming response propagates unchanged.
• Hard constraintmax_tokens caps per-turn generation; exceeding the service's context length surfaces as a backend error rather than a local truncation.
• LatencyEach request runs under a fixed HTTP timeout; very long generations need the timeout raised at deploy time.
• State lifetimeConversation history lives in the process and resets on restart, and every knob binds once at startup, so changes require a redeploy.