Deep Dive: Fit
Friday, November 6, 2026
Fit is a profound mathematical innovation that enables one to determine the optimal combination of observations and predictive variables to be used in a prediction. Additionally, it provides a principled way to treat missing data, and it gives advance guidance about a prediction’s reliability. In this issue’s Deep Dive, we will explain fit intuitively, describe how fit is calculated, show its connection to an important statistical concept, and discuss how it is used in relevance-based prediction (RBP).
Intuition: To grasp fit, it is necessary first to understand the concept of relevance. Relevance is a statistic that gives a precise measure of an observation’s importance to a prediction. It is composed of similarity and informativeness which are both measured as Mahalanobis distances. (See Volume 1 Number 1 for a Deep Dive into relevance.) Fit measures the average alignment of relevance and outcomes within all pairs of observations that go into a prediction task. Suppose, for example, that you ask two highly regarded baseball scouts whom you judge to have equal expertise about a minor league player’s chance of being brought up to the major leagues within a specified period and they both give the same answer. You would have confidence in the answer because their expertise aligns with their responses. It forms a pattern. Now suppose you ask the same two scouts about the player, and they give opposite answers. You wouldn’t trust either answer because they contradict each other. There is no pattern. Fit gives a formal mathematical assessment of the extent to which the data contains useful patterns for making a prediction. It detects patterns between relevance, which can be thought of as expertise, and outcomes. Predictions that are formed from data with many useful patterns are more reliable than predictions formed from data with few useful patterns. Moreover, these patterns are detectable before the prediction is rendered.
Calculation: Consider a pair of observations that is used to form a prediction. If both observations have high relevance and high outcomes, they are aligned. If both observations have low relevance and low outcomes, they are aligned. If one observation has high relevance and a low outcome and the other has high relevance and a low outcome they are aligned. However, if one observation has high relevance and a low outcome and the other has high relevance and a high outcome, they are not aligned. Fit is calculated as the average alignment of relevance weights and outcomes, both expressed in standardized units, of all pairs of observations that go into a prediction task. It can also be viewed as the squared correlation between relevance weights and outcomes.
Connection to important statistical concepts: Fit is not arbitrary. An informativeness-weighted average of fit across all prediction tasks converges precisely to R-squared. (See Volume 1 Number 1 for a description of informativeness.) This convergence highlights an important insight. R-squared measures reliability only on average across all predictions from a model. It does not necessarily give a good indication of a specific prediction’s reliability. Fit, however, pertains to specific predictions. It is, therefore, much more nuanced and powerful than R-squared.
How fit is used in RBP: Fit is used in three ways in RBP. First, it is used to provide a principled way to determine how to combine predictive variables and observations to form a prediction. RBP employs grid prediction to address the co-dependence of observations and predictive variables. The different columns of the grid represent different combinations of predictive variables used to form a prediction, while the rows of the grid represent different subsamples of observations used to form a prediction. Therefore, each cell gives a prediction for a unique combination of predictive variables and observations. And each prediction has an associated fit. (These fit values are further adjusted for asymmetry and the number of predictive variables.) RBP forms a composite prediction as a weighted average of all the cells’ predictions in which the weights are the relative adjusted fits. Fit effectively adjusts the composite prediction to account for the strength of the patterns in the different calibrations. Second, fit enables RBP to treat missing observations in a principled way because the fit of cells with important missing observations, all else equal, will have lower fit than cells with less important missing observations. Third, each composite prediction has an associated composite fit. This measure of composite fit enables users to distinguish more reliable predictions from less reliable predictions and to adjust how they act on the different predictions.
