This paper introduces MultiSEMF, a Multi-modal Supervised Expectation-Maximization Framework that extends SEMF to integrate heterogeneous data modalities within a unified probabilistic model. MultiSEMF allows modality-specific architectures, such as convolutional networks for images, transformers for text, and gradient boosting models for tabular data, to jointly learn a shared latent representation. Copulas are incorporated to model dependencies between modalities, improving the understanding of inter-modal relationships. To handle complex, non-Gaussian latent structures, MultiSEMF borrows ideas from normalizing flows and KL-divergence regularization. Empirical evaluations on real-world and curated datasets show that MultiSEMF achieves strong predictive performance and reliable uncertainty estimation while maintaining a flexible and interpretable multi-modal learning approach.
Accurate uncertainty quantification is critical for reliable predictive modeling. Existing methods typically address either aleatoric uncertainty due to measurement noise or epistemic uncertainty resulting from limited data, but not both in a balanced manner. We propose CLEAR, a calibration method with two distinct parameters, γ1 and γ2, to combine the two uncertainty components and improve the conditional coverage of predictive intervals for regression tasks. CLEAR is compatible with any pair of aleatoric and epistemic estimators; we show how it can be used with (i) quantile regression for aleatoric uncertainty and (ii) ensembles drawn from the Predictability–Computability–Stability (PCS) framework for epistemic uncertainty. Across 17 diverse real-world datasets, CLEAR achieves an average improvement of 28.2% and 17.4% in the interval width compared to the two individually calibrated baselines while maintaining nominal coverage. Similar improvements are observed when applying CLEAR to Deep Ensembles (epistemic) and Simultaneous Quantile Regression (aleatoric). The benefits are especially evident in scenarios dominated by high aleatoric or epistemic uncertainty.
Accurate tropical cyclone (TC) intensity prediction is crucial for disaster preparedness and risk mitigation. While existing approaches primarily rely on single modalities such as satellite imagery or atmospheric measurements, we present MultiCyclone, the first multi-modal deep learning framework that integrates tabular meteorological data, reanalysis wind fields, and textual weather discussions for TC intensity prediction. Our approach leverages 197 tropical cyclones from the Atlantic basin (2005-2020) comprising 3,871 observations to make several key contributions: (1) the first use of textual weather reports for TC intensity prediction through fine-tuned transformer models, (2) multi-modal fusion strategies combining structured and unstructured data, and (3) uncertainty quantification for multi-modal predictions.
This work introduces the Supervised Expectation-Maximization Framework (SEMF), a versatile and model-agnostic approach for generating prediction intervals with any ML model. SEMF extends the Expectation-Maximization algorithm, traditionally used in unsupervised learning, to a supervised context, leveraging latent variable modeling for uncertainty estimation. Through extensive empirical evaluation of diverse simulated distributions and 11 real-world tabular datasets, SEMF consistently produces narrower prediction intervals while maintaining the desired coverage probability, outperforming traditional quantile regression methods. Furthermore, without using the quantile (pinball) loss, SEMF allows point predictors, including gradient-boosted trees and neural networks, to be calibrated with conformal quantile regression. The results indicate that SEMF enhances uncertainty quantification under diverse data distributions and is particularly effective for models that otherwise struggle with inherent uncertainty representation.
Multi-modal data is widely available for online real estate listings. Announcements can contain various forms of data, including visual data and unstructured textual descriptions. Nonetheless, many traditional real estate pricing models rely solely on well-structured tabular features. This work investigates whether it is possible to improve the performance of the pricing model using additional unstructured data, namely images of the property and satellite images. We compare four models based on the type of input data they use: (1) tabular data only, (2) tabular data and property images, (3) tabular data and satellite images, and (4) tabular data and combination of property and satellite images. In a supervised context, branches of dedicated neural networks for each data type are fused (concatenated) to predict log rental prices. The novel dataset devised for the study (SRED) consists of 11,105 flat rentals advertised over the internet in Switzerland. The results reveal that using all three sources of data generally outperforms machine learning models built on only the tabular information. The findings pave the way for further research on integrating other non-structured inputs, for instance, the textual descriptions of properties.