Conformal Calibration for Multi-modal Regression with Missing Modalities

Abstract

Prediction intervals for multi-modal regression with tabular variables, text, images, or other input sources are difficult to calibrate when those sources disagree or one is missing. A single global quantile averages these regimes together instead of calibrating to the modality pattern observed at test time. We address this through a modality-aware conformal calibration layer. The layer trains or reuses one predictor per modality, computes a disagreement score from the modality-specific predictions, and uses that score in split conformal calibration under a strict split protocol. We use the score in two complementary ways. First, a continuous weighted version reallocates interval width across examples while preserving the usual marginal split-conformal guarantee. Second, a Mondrian version calibrates within fixed disagreement or modality-availability groups, giving group guarantees when the regime rule is fixed before final calibration and calibration/test examples are exchangeable within group. Across four multi-modal datasets and 60 paired calibration runs, the weighted layer reduces mean interval width in 39 runs and interval CRPS in 46 runs. Counting exact ties as matched performance, it matches or improves the marginal conformal baseline in 52 of 60 width comparisons and 59 of 60 interval CRPS comparisons while keeping empirical coverage near the 95% target. In seven reported missing-modality stress tests, regime-specific Mondrian calibration recovers up to 20.2 percentage points of coverage and brings the hardest masked regimes back near the target, sometimes at substantial width cost. The result is a simple, model-agnostic reliability layer for multi-modal regression systems.

Preprint under review, and available at request