Notes on correlation
Pearson correlation coefficient, coefficient of determination
Pearson correlation coefficient For a population
For a sample
coefficient of determination ()
Definition: the proportion of the variance in the dependent variable that is predictable from the independent variable(s).
When we test the models performance, measures the size of the residuals from the model compared to the size of the residuals from a null model (mean value). On the training data, as the model cannot be worse than the null model, will always be positive. It may not be the case for the testing data as the model can do worse than the null model.
Things above are some basics that we all know. However, recently colleagues of mine proposes that our traditional approach of evaluating our prediction model’s performance is problematic.
Previously, when we want to evaluate our model, we will first use cross-validation to generate predicted values and then calculate its Pearson correlation () between predicted vs observed values. However, when we do this, reports the degree to which the predictions are correlated with observations, this is not the same as where it represents the accuracy of the model.
In the training set, when we regress observations () on the predicted values (), the model will simply be as the regression model already assures to minimize residual sum of squares (SSR) and is already the linear combination of predictors. So there is no additional fitting when we evaluate the model on training data. will just be the square root of .
However, in the testing data, when we calculate , we are actually fitting a secondary model , and this will result in a higher than we can get from directly apply the definition equation of . Some test set observations have gone to the secondary model which inflates the result. So reporting this secondary may not give you what you think you want.
All of this is tricky and if we want to say that is the explained variance of the model, we have to make sure that “the model” is the one we trained, not the nontrivial secondary model.