mirror of
https://github.com/harry7557558/spirula-studio.git
synced 2026-10-20 05:49:54 +08:00
168 lines
8.7 KiB
Markdown
168 lines
8.7 KiB
Markdown
# Pose normalization: orientation, centering, scale, and train_frame
|
|
|
|
`orientation_method`, `center_method` and `auto_scale_poses` are accepted by
|
|
both trainers but only *partially* implemented natively. This note records
|
|
what the C++ does, what it does not, and where the reference implementation
|
|
is, so the port does not have to be reverse-engineered from a diff.
|
|
|
|
## Status
|
|
|
|
| config | native behaviour |
|
|
|---|---|
|
|
| `orientation_method` | always `up`; anything else warns and is approximated |
|
|
| `center_method` | always `poses`; ditto |
|
|
| `auto_scale_poses` | effectively always on — the scale is folded into `train_frame_scale` |
|
|
| `train_frame` | only `points`; `normalized` / `camera` are rejected by `check_config()` |
|
|
|
|
**Where the frame now comes from.** `spirula sfm` applies this same
|
|
similarity to the model before writing it (`src/sfm/map/Orient.h`,
|
|
`--no-orient` to decline), so for a dataset this repository reconstructed, the
|
|
"raw SfM frame" below already *is* the normalized frame: `train_frame_scale`
|
|
comes out 1 and `train_to_normalized` the identity. Everything in this note
|
|
still describes what the trainer does — it is simply being handed a scene that
|
|
has nothing left to normalize. Datasets from COLMAP, Metashape or nerfstudio
|
|
are unchanged and still arrive in whatever gauge their producer chose.
|
|
|
|
Because `train_frame="points"` keeps splats in the raw SfM frame, the
|
|
orientation/centering choice does **not** move the training coordinates. It
|
|
only changes `train_frame_scale` (and hence the LR / regularizer rescaling
|
|
that hangs off it) and `train_to_normalized`, the similarity the web viewer
|
|
uses to place its default camera. That is why the native trainer can warn and
|
|
carry on rather than refuse: the approximation is real but bounded.
|
|
|
|
`src/data/parsers/DatasetCommon.cpp::compute_normalized_transform` is the C++
|
|
side. It implements the `up`/`poses` path only.
|
|
|
|
## Reference implementation
|
|
|
|
- `reference/python/camera_utils.py` — the algorithms
|
|
(`auto_orient_and_center_poses` for pca/up/vertical/none x poses/focus/none,
|
|
`orient_and_center_poses_gsplat` for the `gsplat` value of both). That module
|
|
is on no code path; it exists for this.
|
|
- The call site below — how those outputs become `transform_matrix`,
|
|
`scale_factor`, `train_frame_scale` and `train_to_normalized_transform`.
|
|
This half is at least as fiddly as the algorithms, and it is the part a
|
|
reimplementation is most likely to get subtly wrong.
|
|
|
|
## The original call site
|
|
|
|
Verbatim from `spirulae_splat/modules/dataparser.py::_parse_nerfstudio_data`
|
|
at commit `af96993`, the last commit before the Python parser was deleted
|
|
(`git show af96993:spirulae_splat/modules/dataparser.py` for the full file).
|
|
Reproduced rather than paraphrased so it cannot drift from what was actually
|
|
verified against the native parser.
|
|
|
|
```python
|
|
poses = torch.from_numpy(np.array(poses).astype(np.float32))
|
|
if self.config.center_method == "gsplat" or self.config.orientation_method == "gsplat":
|
|
poses, transform_matrix, scene_scale = camera_utils.orient_and_center_poses_gsplat(
|
|
poses.numpy(), metadata['points3D_xyz'].numpy()
|
|
)
|
|
poses = torch.from_numpy(poses).float()
|
|
transform_matrix = torch.from_numpy(transform_matrix).float()[:3, :]
|
|
scale_factor = 1.0 / scene_scale # TODO: this messes up scale_reg
|
|
else:
|
|
poses, transform_matrix = camera_utils.auto_orient_and_center_poses(
|
|
poses,
|
|
method=self.config.orientation_method,
|
|
center_method=self.config.center_method,
|
|
)
|
|
scale_factor = 1.0
|
|
if self.config.auto_scale_poses:
|
|
scale_factor /= float(torch.max(torch.abs(poses[:, :3, 3])))
|
|
|
|
transform_matrix_scale = np.abs(np.cbrt(np.linalg.det(transform_matrix[:3, :3])))
|
|
transform_matrix /= transform_matrix_scale
|
|
poses[:, :3, 3] /= transform_matrix_scale
|
|
scale_factor *= transform_matrix_scale
|
|
|
|
# Normalize all poses to (N, 4, 4) for the train-frame branching below.
|
|
# auto_orient_and_center_poses returns (N, 3, 4); gsplat returns (N, 4, 4).
|
|
if poses.shape[-2] == 3:
|
|
bottom = torch.tensor([[[0, 0, 0, 1]]], dtype=poses.dtype).expand(len(poses), 1, 4)
|
|
poses = torch.cat([poses, bottom], dim=1)
|
|
|
|
# Apply transformation to 3D points (orient/center only; scale handled per train_frame)
|
|
metadata['points3D_xyz'] = (
|
|
torch.cat(
|
|
(
|
|
metadata['points3D_xyz'],
|
|
torch.ones_like(metadata['points3D_xyz'][..., :1]),
|
|
),
|
|
-1,
|
|
)
|
|
@ transform_matrix.T
|
|
)
|
|
|
|
# Build the similarity that maps a point in the would-be camera-frame
|
|
# into the normalized frame: p_n = scale_factor * (R_t @ p_c + t_t).
|
|
transform_4x4 = torch.eye(4, dtype=transform_matrix.dtype)
|
|
transform_4x4[:3, :] = transform_matrix
|
|
T_n_from_camera = torch.eye(4, dtype=transform_matrix.dtype)
|
|
T_n_from_camera[:3, :3] = transform_matrix[:3, :3] * scale_factor
|
|
T_n_from_camera[:3, 3] = transform_matrix[:3, 3] * scale_factor
|
|
|
|
applied_transform_4x4 = None
|
|
if "applied_transform" in meta:
|
|
at = torch.tensor(meta["applied_transform"], dtype=transform_matrix.dtype)
|
|
applied_transform_4x4 = torch.eye(4, dtype=transform_matrix.dtype)
|
|
applied_transform_4x4[:3, :] = at
|
|
|
|
train_frame = self.config.train_frame
|
|
if train_frame == "normalized":
|
|
# Current behavior: poses and points get the scale_factor applied.
|
|
poses[:, :3, 3] *= scale_factor
|
|
metadata['points3D_xyz'] *= scale_factor
|
|
train_frame_scale = 1.0
|
|
train_to_normalized_transform = torch.eye(4, dtype=transform_matrix.dtype)
|
|
elif train_frame in ("camera", "points"):
|
|
# Undo the rigid orient+center on poses and points (already applied
|
|
# by auto_orient/gsplat above) so they're back in the camera frame.
|
|
transform_4x4_inv = torch.linalg.inv(transform_4x4)
|
|
poses = transform_4x4_inv.unsqueeze(0) @ poses
|
|
homog = torch.cat(
|
|
(metadata['points3D_xyz'],
|
|
torch.ones_like(metadata['points3D_xyz'][..., :1])), -1)
|
|
metadata['points3D_xyz'] = (homog @ transform_4x4_inv.T)[..., :3]
|
|
T_train_from_camera = torch.eye(4, dtype=transform_matrix.dtype)
|
|
if train_frame == "points" and applied_transform_4x4 is not None:
|
|
applied_inv = torch.linalg.inv(applied_transform_4x4)
|
|
poses = applied_inv.unsqueeze(0) @ poses
|
|
homog = torch.cat(
|
|
(metadata['points3D_xyz'],
|
|
torch.ones_like(metadata['points3D_xyz'][..., :1])), -1)
|
|
metadata['points3D_xyz'] = (homog @ applied_inv.T)[..., :3]
|
|
T_train_from_camera = applied_inv
|
|
# scale_factor is NOT applied to poses/points in this branch.
|
|
train_frame_scale = 1.0 / scale_factor if scale_factor != 0 else 1.0
|
|
# c2w_train = (T_n)^-1 @ c2w_normalized for camera mode;
|
|
# c2w_train = (T_n @ applied)^-1 @ c2w_normalized for points mode.
|
|
T_n_from_train = T_n_from_camera @ torch.linalg.inv(T_train_from_camera)
|
|
train_to_normalized_transform = torch.linalg.inv(T_n_from_train)
|
|
else:
|
|
raise ValueError(f"Unknown train_frame: {train_frame}")
|
|
|
|
all_dataparser_outputs = []
|
|
for split_id, indices in enumerate([i_train, i_eval]):
|
|
|
|
# Choose image_filenames and poses based on split, but after auto orient and scaling the poses.
|
|
image_filenames_split = [image_filenames[i] for i in indices]
|
|
```
|
|
|
|
## Notes for whoever ports this
|
|
|
|
- `transform_matrix_scale` (the cube root of the rotation block's determinant)
|
|
is divided out of the transform and folded into `scale_factor`. The gsplat
|
|
path returns a non-unit-determinant matrix; the up/poses path does not, so
|
|
this is a no-op there. Getting it wrong scales the whole scene.
|
|
- The `gsplat` branch sets `scale_factor = 1 / scene_scale` and carries a
|
|
`# TODO: this messes up scale_reg` comment from before `train_frame_scale`
|
|
existed. Worth re-checking rather than porting the comment.
|
|
- `train_frame="points"` additionally undoes nerfstudio's `applied_transform`
|
|
so poses and points land back in the original SfM frame. The native parser
|
|
does implement this (`NerfstudioParser.cpp`); it is only the
|
|
orientation/centering *choice* that is missing.
|
|
- The `train_frame_scale = 1 / scale_factor` relationship is what the engine
|
|
uses to rescale `means_lr`, `scale_reg`, `max_world_size` and `noise_lr`.
|
|
A port that changes `scale_factor` changes effective learning rates.
|