SEMF: Supervised Expectation-Maximization Framework for Predicting Intervals

Abstract

This work introduces the Supervised Expectation-Maximization Framework (SEMF), a versatile and model-agnostic framework that generates prediction intervals for datasets with complete or missing data. SEMF extends the Expectation-Maximization (EM) algorithm, traditionally used in unsupervised learning, to a supervised context, enabling it to extract latent representations for uncertainty estimation. The framework demonstrates robustness through extensive empirical evaluation across 11 tabular datasets, achieving—in some cases—narrower normalized prediction intervals and higher coverage than traditional quantile regression methods. Furthermore, SEMF integrates seamlessly with existing machine learning algorithms, such as gradient-boosted trees and neural networks, exemplifying its usefulness for real-world applications. The experimental results highlight SEMF’s potential to advance state-of-the-art techniques in uncertainty quantification.

Preprint

Code

BibTeX

@article{azizi2024semf,
  title={SEMF: Supervised Expectation-Maximization Framework for Predicting Intervals},
  author={Azizi, Ilia and Boldi, Marc-Olivier and Chavez-Demoulin, Val{\'e}rie},
  journal={arXiv preprint arXiv:2405.18176},
  year={2024}
}