Information gain by better modelling

Following Kent (1983),^[1] we use the Fraser information (Fraser 1965)^[2]

${\displaystyle F(\theta )=\int {\textrm {d}}r\,g(r)\,\ln f(r;\theta )}$

where ${\displaystyle g(r)}$ is the probability density of a random variable ${\displaystyle R\,}$, and ${\displaystyle f(r;\theta )\,}$ with ${\displaystyle \theta \in \Theta _{i}}$ (${\displaystyle i=0,1\,}$) are two families of parametric models. Model family 0 is the simpler one, with a restricted parameter space ${\displaystyle \Theta _{0}\subset \Theta _{1}}$.

Parameters are determined by maximum likelihood estimation,

${\displaystyle \theta _{i}=\operatorname {argmax} _{\theta \in \Theta _{i}}F(\theta ).}$

The information gain of model 1 over model 0 is written as

${\displaystyle \Gamma (\theta _{1}:\theta _{0})=2[F(\theta _{1})-F(\theta _{0})]\,}$

where a factor of 2 is included for convenience. Γ is always nonnegative; it measures the extent to which the best model of family 1 is better than the best model of family 0 in explaining g(r).

Information gain by a conditional model

Assume a two-dimensional random variable ${\displaystyle R=(X,Y)}$ where X shall be considered as an explanatory variable, and Y as a dependent variable. Models of family 1 "explain" Y in terms of X,

${\displaystyle f(y\mid x;\theta )}$,

whereas in family 0, X and Y are assumed to be independent. We define the randomness of Y by ${\displaystyle D(Y)=\exp[-2F(\theta _{0})]}$, and the randomness of Y, given X, by ${\displaystyle D(Y\mid X)=\exp[-2F(\theta _{1})]}$. Then,

${\displaystyle \rho _{C}^{2}=1-D(Y\mid X)/D(Y)}$

can be interpreted as proportion of the data dispersion which is "explained" by X.

Linear regression

The fraction of variance unexplained is an established concept in the context of linear regression. The usual definition of the coefficient of determination is based on the fundamental concept of explained variance.

Correlation coefficient as measure of explained variance

Let X be a random vector, and Y a random variable that is modeled by a normal distribution with centre ${\displaystyle \mu =\Psi ^{\textrm {T}}X}$. In this case, the above-derived proportion of explained variation ${\displaystyle \rho _{C}^{2}}$ equals the squared correlation coefficient ${\displaystyle R^{2}}$.

Note the strong model assumptions: the centre of the Y distribution must be a linear function of X, and for any given x, the Y distribution must be normal. In other situations, it is generally not justified to interpret ${\displaystyle R^{2}}$ as proportion of explained variance.

In principal component analysis

Explained variance is routinely used in principal component analysis. The relation to the Fraser–Kent information gain remains to be clarified.