Docs

File schema

A component does not embed the large artifacts it needs. Instead it declares them, and the platform provisions each File on the node before the component starts. The declaration is the contract a backend binds against: it names the slots a component fills, the type each slot accepts, and the config parameter that carries the resolved file into the component at runtime.

Declaring file dependencies

Input files are declared under worker.file_schema. Each entry is a named slot:

worker:  file_schema:    model:      file_type: "weights"      config_key: "model_path"      is_optional: false      tag: "detector"

The slot key (model here) is what a backend binds against — ppl backend add-file --key model. file_type gates which uploaded File is allowed in the slot: it is one value, or a |-separated list when the component genuinely reads several formats. The full vocabulary is in File types.

config_key is how the chosen File reaches the running component. The platform synthesises this parameter from the binding and writes it at deploy time, so it must not also appear under config_schema — declaring it twice is a duplicate-parameter error. What the value contains depends on the optional component target: with no target, the File loads into this component and the value is a component-local path like /files/model; with a target set (triton, torchserve, vllm, sglang), the File loads into that service and the value is the File's display name.

The remaining fields tune the slot: is_optional allows the slot to stay unbound, tag surfaces a slot id at binding time and narrows model-artifact candidates, and mutable allows the binding to change while the backend stays deployed.

Declaring files a component produces

A component that writes a File declares it under worker.generated_file_schema:

worker:  generated_file_schema:    report:      name: "Evidence report"      file_type: "pdf"      config_key: "report_path"

Output declarations are simpler than inputs: name, a single file_type, and a config_key. The | list form is not accepted here — an output is exactly one kind of File.

Committing what a component produces

Writing a file is not the same as producing one. At deploy the platform sets the slot's config_key to a writable directory the component writes the slot's files into; files written there are not saved until they are committed. A file becomes a workspace File only when the component commits it, and only committed files are ever saved — by a lease collect, by a teardown with --save, or for the operator.

For the commit, the platform injects two more string parameters next to config_key: <config_key>_commit_url, the URL a finished file of the slot is committed to, and <config_key>_commit_token, the slot's commit token. Neither is declared in config_schema. For the report slot above they are report_path_commit_url and report_path_commit_token.

A commit is one HTTP POST of the file's raw bytes to the commit URL with two headers: X-Pipelogic-Ingest-Token carrying the token, and X-Pipelogic-File-Name carrying the name the stored File takes — one plain path element, with no separators, not . or .., no control characters, and at most 255 bytes. The request must carry a Content-Length; a chunked body is refused with 422. A 202 Accepted means the platform accepted the file and will save it. A 403 means the token header is missing or is not this slot's <config_key>_commit_token, and a 422 means the name or length is invalid; neither changes on retry. Transport errors and 5xx answers are worth retrying with backoff.

When to commit follows from what the file is:

  • Commit an artifact once it is complete — never a temporary or half-written file.
  • A file the component rewrites in place, such as a snapshot or an index, is committed again after each rewrite. Every commit becomes its own stored File, oldest first.
  • An empty (0-byte) file is a valid artifact.
  • Every commit must have returned before the worker exits; a commit still in flight at exit is lost.
import os, time, urllib.error, urllib.requestfrom pipelogic.worker import configdef commit(key, path):    """Commit the finished file at path to the slot whose config_key is key."""    with open(path, "rb") as f:        body = f.read()    req = urllib.request.Request(str(getattr(config, key + "_commit_url")), data=body, method="POST", headers={        "X-Pipelogic-Ingest-Token": str(getattr(config, key + "_commit_token")),        "X-Pipelogic-File-Name": os.path.basename(path),        "Content-Type": "application/octet-stream",    })    delay = 0.25    for attempt in range(5):        try:            with urllib.request.urlopen(req, timeout=60):                return        except urllib.error.HTTPError as e:            if e.code < 500:                raise RuntimeError(f"commit of {path} refused: HTTP {e.code}") from e        except urllib.error.URLError:            pass        time.sleep(delay)        delay *= 2    raise RuntimeError(f"commit of {path} failed after retries")

In C++ the released Output JSONL File component commits with libcurl; the file egress pattern shows a C++ sink that commits every file it writes.

Related

Was this page helpful?