Skip to content

Methodology

Use this page to read the leaderboard: what its numbers mean and how we make them. The metric definitions are shared with Evaluation & metrics. So a local run and a published entry measure the same thing.

  • The Leaderboard shows every dataset and family on one screen; each dataset eval page (for example VI-Rig 4 IMUs + 3 cams) charts every parameter and holds the full tables: one table per regression family (one test in the MVIS regression suite), then per sensor.
  • Three tools: MVIS, Kalibr and Basalt each calibrate the recordings of a rig.
  • Each cell: a parameter as median ± sd (n), across the recordings that tool solved. sd is the standard deviation; n is the number of recordings.
  • No scores or ranks: a single number would encode a judgement about which parameters matter.

Each difference column subtracts the medians of two tools. A ◆ marks a difference larger than the larger of the two spreads.

We show no difference where it would mislead:

  • values a tool takes as input
  • values shown only
  • missing values
  • rows where the tools’ models differ

Reprojection error isn’t compared, because it measures fit.

LabelMeaning
fixedKalibr’s camera + IMU runs take this value as input, for example intrinsics
shownThe tool estimated it with a different model, so there’s no difference against it
no run or not runThe tool has no result here, and the family’s notes give the reason
(3/15)The tool solved only 3 of the family’s 15 recordings
  • Dates: MVIS rows and the Kalibr and Basalt runs were not all made on the same date. Data settings differ on T6, T7 and T8.
  • Recording counts: these differ per tool on some families, so a difference can mix recording sets. These cells are marked.
  • IMU model: the MVIS column uses Kalibr’s IMU model on camera + IMU families, so the IMU rows share one model. The MVIS default fills fewer rows and is in a closed fold under each family.

Each family on the leaderboard has a closed Run config section per tool. It holds:

  • the command for one recording
  • the settings
  • what differs on the other recordings

Kalibr logs no command line. So we rebuild its camera-only commands from the settings the run passed. The Basalt runs use our own headless build. The files are kept with the records in calibench-vi/results.

Each batch is dated and kept in the history of calibench-vi/results. So a comparison can be cited as “OpenMVIS leaderboard, batch 2026-11-10a, accessed <date>”.