How do I adjust SSE (sum of squared errors) or RMSE (root-mean-square errors) for the number of free parameters in the model?
Is there an "adjusted" RMSD metric similar to the adjusted r-squared metric?
4
-
$\begingroup$ please don't assume that we know what RMSD stands for. The more effort you invest in carefully phrasing your question, the more motivated people will feel to answer it. $\endgroup$– Artem KaznatcheevCommented Feb 8, 2013 at 5:04
-
4$\begingroup$ This sounds more like a question better suited for stats.stackexchange.com . Is there an aspect of this that relates particularly to cognitive modelling? $\endgroup$– Jeromy AnglimCommented Feb 8, 2013 at 5:13
-
$\begingroup$ Not sure to understand what you mean by "adjusted" here ? Do you mean adjusted as for "goodness of fit" or as correction in regards to some parameters ? $\endgroup$– Cheatboy2Commented Feb 8, 2013 at 14:30
-
$\begingroup$ @Cheatboy2 I believe he is referring to a correction in the number of parameters, as in adjusted $r^2$ $\endgroup$– JeffCommented Feb 8, 2013 at 23:17
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
To my knowledge, there is no adjusted RMSD.
RMSD, unlike $R^2$, isn't typically used to compare models across the literature. $R^2$ represents the proportion of variance explained by the model, a construct which translates well across different experimental designs. Adjusted $R^2$ distorts this by accounting for the number of parameters in your model, but is a better estimate of the proportion of variance in the population explained by your model.
RMSD is in whatever units you happen to be using. It should be interpreted against some benchmark for what you think a good model should be, or whatever has been deemed an "acceptable error rate". It's not clear to me that adjusting for the number of parameters in RMSD would provide a number that is inherently meaningful, but I'm not sure if this is true.
If you would like to select among several model candidates, you may want to compute AIC (Akaike Information Criterion) or BIC (Bayesian Information Criterion), both of which are attempts to maximize the likelihood of your model while minimizing the number of parameters. Both metrics are commonly used in cognitive modeling, and I think this may be what you are looking for.
While I think this question is on-topic here, you will likely get better answers on stats.stackexchange.com (as Jeromy points out) because your question is not really specific to cognitive modeling.
$\begingroup$
$\endgroup$
There are at least 3 ways to discount SSE (or RMSE) by the number of free params:
$$ \text{adjusted RMSE} = \sqrt{\frac{SSE}{n - k}} $$
$$ AIC = n \times ln\left(\frac{SSE}{n}\right) - k \times ln(n) $$
$$ BIC = n \times ln\left(\frac{SSE}{n}\right) - 2 \times k $$
or in computer code style:
k = number of free params
n = number of DV's
SSE = sum of squared errors
adjusted RMSE = sqrt ( SSE / n - k )
AIC = n * ln(SSE/n) - k * ln(n)
BIC = n * ln(SSE/n) - 2 * k
