|
|
|
|
Occam's razor
|
(Definition)
|
|
|
The following example in statistics illustrates the principle of Occam's razor. See Article on Occam's Razor on Wikipedia for a thorough discussion, mathematical or not, of this principle, which is also known as the principle of parsimony.
Example in statistics:
Given the following ten pairs of hypothetical observations of continuous response variable $Y$ and continuous explanatory variable $X$ :
| $X$ |
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
| $Y$ |
11 |
13 |
15 |
15 |
16 |
18 |
21 |
22 |
22 |
24 |
be fitted using a linear regression model given by $$Y=\alpha + \beta X.$$ Using the method of least squares, the regression coefficients are found to be $\alpha=9.867$ and $\beta=1.424$ , so that the regression line is $$Y=9.867+1.424X.$$ The p-values for both regression coefficients are found to be less than 0.0001 and the p-value for the overall fit of the model is also less than
0.0001. The p-value analysis can be summarized by the following table:
| |
$\alpha$ |
$\beta$ |
overall fit of model |
| p-value |
$<0.0001$ |
$<0.0001$ |
$<0.0001$ |
This indicates that the linear regression equation fits the data very well.
Next, fit the data using a 2nd order polynomial regression model, given by $$Y=\alpha + \beta X + \gamma X^2.$$ Least square estimation shows that the regression equation is given by $$Y=9.783 + 1.466X - 0.004X^2.$$ The following table shows the result of the p-value analysis of the 2nd order polynomial regression model:
| |
$\alpha$ |
$\beta$ |
$\gamma$ |
overall fit of model |
| p-value |
$<0.0001$ |
$0.0085$ |
0.9188 |
$<0.0001$ |
The p-value for the fit of the overall model suggests that the polynomial regression equation also fits the data well. The same can be said about the estimates of the regression coefficients $\alpha$ and $\beta$ . However, the coefficient $\gamma=-0.004$ shows that the additional term does not contribute too much to the overall fit of the model to the observations. Furthermore, its p-value is very high, indicating that the $X^2$ term is not significant.
In light of the above analysis, we prefer the simpler model $Y=\alpha + \beta X$ over the more complicated one $Y=\alpha + \beta X + \gamma X^2$ . This example shows that, in statistical modeling, when a simpler, easier to interpret model exists in the presence of more complicated ones, the simpler one should be chosen. This is Occam's razor at work!
|
"Occam's razor" is owned by CWoo. [ full author list (4) | owner history (1) ]
|
|
(view preamble | get metadata)
| Other names: |
law of parsimony |
|
|
Cross-references: term, coefficient, estimates, polynomial, polynomial regression model, order, equation, analysis, p-values, line, regression coefficients, least squares, linear regression model, explanatory variable, response variable, continuous, observations, statistics
There are 6 references to this entry.
This is version 6 of Occam's razor, born on 2004-10-14, modified 2007-12-18.
Object id is 6371, canonical name is OccamsRazor.
Accessed 8122 times total.
Classification:
| AMS MSC: | 00A20 (General :: General and miscellaneous specific topics :: Dictionaries and other general reference works) | | | 62-07 (Statistics :: Data analysis) | | | 62A01 (Statistics :: Foundational and philosophical topics) |
|
|
|
|
|
|
Pending Errata and Addenda
|
|
|
|
|
|
|
|
|
|
|
