Methodology
Use this page to read the leaderboard: what its numbers mean and how we make them. The metric definitions are shared with Evaluation & metrics. So a local run and a published entry measure the same thing.
What the board shows
Section titled “What the board shows”- The Leaderboard shows every dataset and family on one screen; each dataset eval page (for example VI-Rig 4 IMUs + 3 cams) charts every parameter and holds the full tables: one table per regression family (one test in the MVIS regression suite), then per sensor.
- Three tools: MVIS, Kalibr and Basalt each calibrate the recordings of a rig.
- Each cell: a parameter as median ± sd (n), across the recordings that tool solved. sd is the standard deviation; n is the number of recordings.
- No scores or ranks: a single number would encode a judgement about which parameters matter.
Differences
Section titled “Differences”Each difference column subtracts the medians of two tools. A ◆ marks a difference larger than the larger of the two spreads.
We show no difference where it would mislead:
- values a tool takes as input
- values shown only
- missing values
- rows where the tools’ models differ
Reprojection error isn’t compared, because it measures fit.
Values that aren’t estimates
Section titled “Values that aren’t estimates”| Label | Meaning |
|---|---|
fixed | Kalibr’s camera + IMU runs take this value as input, for example intrinsics |
shown | The tool estimated it with a different model, so there’s no difference against it |
no run or not run | The tool has no result here, and the family’s notes give the reason |
(3/15) | The tool solved only 3 of the family’s 15 recordings |
Known limits
Section titled “Known limits”- Dates: MVIS rows and the Kalibr and Basalt runs were not all made on the same date. Data settings differ on T6, T7 and T8.
- Recording counts: these differ per tool on some families, so a difference can mix recording sets. These cells are marked.
- IMU model: the MVIS column uses Kalibr’s IMU model on camera + IMU families, so the IMU rows share one model. The MVIS default fills fewer rows and is in a closed fold under each family.
Configs
Section titled “Configs”Each family on the leaderboard has a closed Run config section per tool. It holds:
- the command for one recording
- the settings
- what differs on the other recordings
Kalibr logs no command line. So we rebuild its camera-only commands from the settings the run passed.
The Basalt runs use our own headless build.
The files are kept with the records in
calibench-vi/results.
Versioning
Section titled “Versioning”Each batch is dated and kept in the history of
calibench-vi/results.
So a comparison can be cited as “OpenMVIS leaderboard, batch 2026-11-10a, accessed <date>”.