Convert Depth to Point Cloud

@bleenco1 VersionTransform

Back-projects a MetricDepthMap through a pinhole camera into 3D, emitting a dense per-pixel PointMap3D or a filtered PointCloud. Uses a wired CameraIntrinsics or a focal-length fallback, and can colour the cloud from an optional Image.

How it fits

Typical backends

Estimate metric depth from images, lift it to 3D points, and project them onto a ground plane.

Calibrate the camera from reference detections, then lift the same metric depth into 3D with the recovered intrinsics for accurate geometry.

Estimate depth with a HuggingFace model and lift it into a coloured point cloud using the source image for per-point RGB.

Caveats

I/O contractThe output arm is fixed by the downstream connection: a PointMap3D keeps one 3D point per pixel in image grid order (invalid pixels become NaN so the grid shape is preserved), while a PointCloud flattens only the finite, positive-depth points into an unordered `[N, 3]` set.
I/O contractThe required input is the MetricDepthMap; an optional CameraIntrinsics supplies a real fx/fy/cx/cy, and an optional Image colours the PointCloud (its RGB is sampled per kept point into `colors`). Either optional input may be omitted.
AccuracyWith no CameraIntrinsics wired, the back-projection assumes a pinhole camera with focal_length_x / focal_length_y and a principal point at the image centre; wiring a calibrated CameraIntrinsics (for example from calibrate_camera_depth) removes that approximation.
Hard constraintBoth fallback focal lengths must be positive; the component rejects a zero focal_length_x or focal_length_y at startup rather than producing degenerate points.
LimitationGeometry is only as good as the upstream depth: an estimator that produces relative rather than metric depth yields points at the wrong scale.

Versions

8b493707latestdefault
Rebuild against the current pipeml
7/29/2026