Evaluation & metrics

How to tell whether a calibration is good. The same metrics back the leaderboard, so a local run and a published entry are comparable.

Read the report first

Every run writes a self-contained HTML report. The section to read before the error numbers is calibration data quality: motion-excitation metrics with letter grades, rotation-axis spread, angular-velocity timeline, and a feature-coverage heatmap with convex hull.

Reprojection error is not accuracy

It measures how well the solution explains the data it was fitted to. A parameter the motion never excited settles near its initial value without raising the residual. A low error on a poorly-excited sequence means very little.

Metrics

Metric	Units	Against	Notes
Reprojection median / P95	px	own observations	Robust statistics, not mean — P95 is where model mismatch shows
Extrinsic rotation error	deg	a reference	Geodesic angle
Extrinsic translation error	cm	a reference	Includes the camera–IMU lever arm
Stereo baseline	mm	a reference	The most stable cross-check on a rig
Time-offset error	ms	a reference	Per camera and auxiliary IMU
Runtime	s	—	Wall-clock, hardware reported
Convergence rate	%	repeated runs	Successful / total

Read convergence first: excellent errors at a 60% convergence rate is not a better toolbox than one that always converges.

Ground truth, where it exists

Per-dataset ground-truth YAMLs live under examples/regression/gt/. The dashboard annotates each parameter with its deviation — Δ +0.27% GT=190.97 — colour-coded by accuracy band. Version 3 results match the published TUM-VI and Intel T265 calibrations to under 0.6% on focals.

Round-trip: the strongest check without ground truth

mvis roundtrip -c cam_imu.yaml

PASS requires IMU intrinsics to collapse to identity on the corrected bag and extrinsics and stereo baseline to be preserved. Two conditions pulling in opposite directions is what makes it hard to satisfy by accident. See Round-trip validation.

Repeatability across sequences

The most practical field check. Run a batch sweep and compare: parameters that move between sequences were not constrained by the motion in them. The regression harness reports this directly as mean ± stddev of intrinsics, distortion, and baseline across a collection.

Regression harness

MVIS validates itself with a three-tier regression suite: every row re-solves a real recording and compares the result against a committed per-collection baseline with per-metric tolerances (focal, distortion, extrinsics, time offsets, readout, reprojection). Tiers are supersets — a deeper tier re-runs everything above it:

Tier	Command	Rows	Runtime	Role
fast	--level fast	9	~30 min	the gate — every change passes it
mid	--level mid	27	~1.5–2.5 h	routine deep check (3 recordings per family)
full	--level full	102	~6–10 h	every collection of every suite dataset

python scripts/run_regression.py --mode all --level fast
python scripts/run_regression.py --only mvis_elp/imu_bag1      # one row
python scripts/run_regression.py --only <row> --update-baseline # re-bless

The 9 seed rows span the calibration space on four datasets: global- and rolling-shutter, mono/stereo/3-camera, single-, two- and mixed-IMU configurations. The other 93 row configs are generated from the seeds (only the recording changes) and drift-guarded by a test that regenerates them, so solver settings cannot silently diverge across a family. The design story is on the regression testing page.

Baselines

One committed JSON per (dataset, collection) under examples/regression/baselines/ holds the blessed metrics. A gate run re-solves and compares; --update-baseline re-blesses after an intentional change, and the diff of the baseline file is the review artifact — every numeric shift a change causes is visible in the PR.