Methodology

What the leaderboard numbers mean and how they are produced. The metric definitions are shared with Evaluation & metrics, so a local run and a published entry measure the same thing.

Sensor configurations

Submissions are grouped into seven mutually-exclusive categories. Ranking happens within a category — a single-camera intrinsics calibration and a four-IMU rig calibration are not the same problem, and a combined ranking would imply otherwise.

Category	Shorthand	Example methods
Camera only — mono	0 IMU · 1 cam	COLMAP intrinsic calibration
Camera only — stereo	0 IMU · 2 cams	OpenCV stereo calibration
Camera only — multi	0 IMU · ≥ 3 cams	Multi-cam factor graphs
1 IMU + 1 cam	1 IMU · 1 cam	Kalibr, OpenVINS, Basalt
1 IMU + multi cams	1 IMU · ≥ 2 cams	Kalibr multi-cam VI
Multi IMU + 1 cam	≥ 2 IMUs · 1 cam	MVIS (partial)
Multi IMU + multi cams	≥ 2 IMUs · ≥ 2 cams	MVIS

Algorithm families

Reported so that a difference in results can be attributed to the approach rather than left unexplained. The discrete- versus continuous-time distinction in particular is discussed in Continuous-time refinement.

Family	Description
Factor graph (discrete-time)	GTSAM / Ceres batch MAP optimisation. MVIS
Continuous-time B-spline	SE(3) B-spline trajectory parameterisation. Kalibr, Basalt
Filter (EKF)	Online filtering with calibration in the state. OpenVINS
Learning-based	Networks regressing calibration parameters
Other / hybrid	Anything else — flagged in the submission

Metrics

Metric	Units	Notes
Reprojection RMSE	px	Mean over all cameras and sequences
Extrinsic rotation error	deg	vs. consensus baseline
Extrinsic translation error	cm	vs. consensus baseline
Time-offset error	ms	vs. consensus baseline
Runtime	s	Wall-clock, single machine, hardware reported
Convergence rate	%	Successful runs / total runs

Each is reported as value ± stddev (n runs) where the submitter ran it more than once. Convergence rate is the one to read first: a method with excellent errors and a 60% convergence rate is not better than one that always converges.

The consensus baseline

The datasets used here have no metrological ground truth — nobody measured these rigs on a coordinate-measuring machine. Error metrics are therefore computed against a consensus baseline: the agreed reference calibration for each dataset, derived from independent runs across toolboxes.

A baseline is not ground truth

Where methods systematically agree and are systematically wrong, the baseline inherits that error and no entry’s score reveals it. Reported differences are meaningful relative to each other; absolute accuracy claims need external validation, such as a dataset that publishes its own calibration (TUM VI, Monado).

Ranking

Sorted by the selected metric — ascending for errors, descending for convergence rate. There is no aggregate score across metrics: weighting rotation error against runtime would encode a judgement about what matters that belongs to the reader, not the benchmark. This follows KITTI’s lesson against composite rankings.

Missing values

— means the submitter did not report the metric. Missing entries are never imputed, and a method is never penalised into a default value — that would let an aggregate ranking be gamed by omission.

Verification

Submissions are schema-checked in CI. Maintainer re-runs, where possible, earn a verified badge — the distinction between reported and reproduced is kept visible. The submission flow is in Submit results; governance and review are in Governance.

Versioning

The leaderboard is snapshotted so that a comparison can be cited stably: “Open MVIS leaderboard vX.Y, accessed <date>”. Later submissions never silently change a published snapshot.