Calibrates a camera online from each Image: finds orthogonal vanishing points from scene line structure to recover the focal length into a running CameraIntrinsics centred on the frame, and, given [BoundingBox] references and a MetricDepthMap, measures the depth scale that turns the depth model's units into metres.
Typical backends
Pure vanishing-point calibration from a single video stream of a structured scene, publishing the recovered intrinsics over HTTP. No detector or depth model is wired; the focal length comes entirely from the orthogonal line structure.
Metric scene calibration: detect people and estimate metric depth from the same frame, then publish the intrinsics together with the depth scale. Set references to the person class with its standing height and width 0, for example [(0, 1.78, 0.0)]; multiplying the depth map by depth_scale then gives distances in metres.
Caveats
- Hard constraintThe vanishing-point path requires a Manhattan scene — visible families of parallel and mutually orthogonal straight lines such as building edges, floor tiles, doorways or road markings. On a frame with no vanishing-point focal, reference detections at their ground-contact depth stand in, else the estimate stays as it was.
- LimitationFocal samples outside 300 to 5000 pixels are rejected as implausible, so very low-resolution frames never calibrate. The intrinsics are Nothing until the first frame with a focal estimate, and stay Nothing on a scene with neither vanishing points nor reference objects at a depth map.
- I/O contractThe output resolves to the arm the connected consumer accepts: a bare Maybe<CameraIntrinsics>, a record of the same intrinsics plus depth_scale, or that record plus floor. The consumer must accept the Maybe: a stage that takes a plain
CameraIntrinsics needs the Nothing case handled before it. Inside Just intrinsics the distortion field is the measured [k1, 0]; k1 is 0 when the scene shows no measurable bow. - I/O contractMultiply a
MetricDepthMap value by depth_scale to get metres. depth_scale and floor both need a focal estimate first and a wired depth map, and are Nothing until a frame has measured them. floor is the ground plane in the camera coordinates (x right, y down, z forward) of the depth map multiplied by depth_scale, or of the depth map as estimated while depth_scale is Nothing: normal is a unit vector pointing up (negative y) and offset is the camera's height above the floor in those units (metres once depth_scale is known), so normal . X + offset is a point X's height above the floor. floor needs no reference objects. - I/O contractThe variadic input tail accepts zero, one, or both of a
[BoundingBox] detection stream and a MetricDepthMap; absent tail inputs disable the known-size and depth-scale paths and never degrade the vanishing-point estimate. - Hard constraintA frame that breaks the input contract stops the worker with an error naming the violation and emits no output; the worker's exit fails the whole deployment, so one malformed frame from an upstream stage stops the graph: an empty
Image, a MetricDepthMap whose shape is not (H, W) or (H, W, 1) or whose data length is not H times W, or a BoundingBox with a non-finite confidence or corner. Zero-area boxes measure nothing and are skipped. Startup fails when the linked OpenCV has no working line segment detector or when references lists a class id twice. - Accuracydepth_scale needs a visible floor in the depth map and reference objects standing on it with their feet in view; boxes cut off by the image bottom, hidden behind a lower overlapping box, or inside a box of another reference class are skipped. A wrong reference height biases the scale by the same factor.
references lists the classes most trusted first: the scale comes from the first-listed class seen so far, an earlier-listed class replaces it when it first appears, so the scale jumps once at that frame, and later-listed classes then no longer move it. Listing people first and a vehicle class after them gives a scene with no people a scale too. - AccuracyThe model is a square-pixel pinhole, so the recovered fx and fy are equal and skew is always 0. The radial distortion is the division-model [k1, k2] vector (k2 is 0) from a plumb-line search over long edge ridges.
- LatencyEvery frame runs line segment detection, a vanishing-point RANSAC, and on frames with a focal estimate an edge-ridge distortion search over the full image; a wired depth map adds a floor-plane fit per frame. Large frames cost proportionally more CPU time per output.
- State lifetimeFocal length, distortion, depth_scale and floor are running values: each frame's sample is weighted by
decay raised to its age and the estimate is the weighted sum over the sum of weights, so the first frame weighs no more than later ones. Each stays as it was on a frame that measures none and resets only on restart. Every frame that meets the input contract emits exactly one output.
