## You are here

Homebias

## Primary tabs

# bias

Background. In estimating a parameter from a statistical model, one is interested in how the estimates deviate from the true value of the parameter. The deviations generally come from two sources. One source is known as the *noise*, or *random error*, which has to do with the random nature of observations or measurements in general. For example, when a fair coin is tossed 100 times and the number of heads is counted. One might get 51 even though the true parameter is 50. The difference of 1 is due to the random nature of coin tossing.

The other source of deviation is known as the *bias*, or *systematic error*, which has to do with how the observations are made, how the instruments are set up to make the measurements, and most of all, how these observations or
measurements are tallied and summarized to come up with an estimate of the true parameter. For example, a rating scheme is proposed for an online collaborative encyclopedia on entries contributed by individuals who are members of the online website hosting the encyclopedia. The purpose of this rating scheme is to give the readers, members or non-members inclusive, a better idea on the quality of the entries by their corresponding numerical values. Suppose that members are asked to rate an entry from a scale of 1 to
10. For simplicity, members who are intimately familiar with the concept in the entry rate it with a perfect 10. Next, members who are not that familiar with the entry give it a 5. Finally, the remaining members choose to not participate and the rating scale from them default to a 0. A simple arithmetic average is computed and a rating of 2.5 is produced. Would this 2.5 be a good indicator
of the overall quality of the entry? Maybe not. Here, biases are introduced. First, the participants of the rating scheme do not include non-members, who, collectively, may very well represent a different level of understanding of the rated entry than members. Secondly, even among the members, there is a considerable amount of differences in terms of levels of understanding of the entry, etc… To some, the entry may be accurately and perfectly written, not
everyone will rate it the same way in the end. Finally, there are the non-raters. We have no idea as to how they would rate the entries. Their votes should certainly count if they decide to rate in the last minute. The final rate, however, would most likely be different.

The difference between the bias and the noise is that one can be reduced while the other can not. Mathematically, we have the following:

Definition. If $\theta$ is a parameter in a statistical model, the bias of an estimator $\hat{\theta}$ of $\theta$, is the difference between expectation of $\hat{\theta}$ and the value of $\theta$, which, by abuse of notation, is also denoted $\theta$:

$\operatorname{Bias}(\hat{\theta}):=\operatorname{E}[\hat{\theta}]-\theta.$ |

An estimator is called an *unbiased estimator* if its bias is
zero at *all* values of $\theta$. Otherwise, it is a
*biased estimator*.

Note that the random error does not appear in the above definition because its expectation is zero.

Examples.

1. If observations $X_{1},\ldots,X_{n}$ are iid from a normal distribution with mean $\mu$ and variance $\sigma^{2}$, then the sample mean estimator $\overline{X}$ is an unbiased estimator for $\mu$. To see this, recall the definition of a sample mean

$\overline{X}=\frac{1}{n}(X_{1}+\cdots+X_{n})$ so that

$\operatorname{E}[\overline{X}]=\frac{1}{n}(\operatorname{E}[X_{1}]+\cdots+% \operatorname{E}[X_{n}]).$ But $\mu=\operatorname{E}[X_{1}]=\cdots=\operatorname{E}[X_{n}]$, the above expression reduces to

$\operatorname{E}[\overline{X}]=\frac{1}{n}(n\mu)=\mu,$ showing that the bias of $\overline{X}$ is zero. Note that even though $\overline{X}$ depends on the size of the sample $n$, its expectation, however, does not, and is identically $\mu$, for all values of $\mu$.

2. Here is another example of an unbiased estimator. Again, assume observations $X_{i}$, $i=1,\ldots,n$ are iid as normal distribution $N(\mu,\sigma^{2})$. The sample variance estimator $s^{2}$ is defined by

$s^{2}=\frac{1}{n-1}\sum_{{i=1}}^{{n}}(X_{i}-\overline{X})^{2}.$ Expressing $s^{2}$ explicitly in terms of the random variables $X_{i}$, we have

$\displaystyle(n-1)s^{2}$ $\displaystyle=$ $\displaystyle\sum_{{i=1}}^{{n}}X_{i}^{2}-\frac{1}{n}(\sum_{{i=1}}^{{n}}X_{i})^% {2}$ (1) $\displaystyle=$ $\displaystyle\frac{1}{n}[(n-1)\sum_{{i=1}}^{{n}}X_{i}^{2}-2\sum_{{i<j}}X_{i}X_% {j}]$ (2) Now, for $i\neq j$, $X_{i}$ and $X_{j}$ are independent so that

$\operatorname{E}[X_{i}X_{j}]=\operatorname{E}[X_{i}]\operatorname{E}[X_{j}]=% \mu^{2}=\operatorname{E}[X_{i}]^{2},$ for all $i=1,\ldots,n$. Hence

$\displaystyle(n-1)\operatorname{E}[s^{2}]$ $\displaystyle=$ $\displaystyle\frac{1}{n}\{(n-1)\sum_{{i=1}}^{{n}}\operatorname{E}[X_{i}^{2}]-2% \sum_{{i<j}}\operatorname{E}[X_{i}X_{j}]\}$ (3) $\displaystyle=$ $\displaystyle\frac{1}{n}\{(n-1)\sum_{{i=1}}^{{n}}\operatorname{E}[X_{i}^{2}]-2% \sum_{{i<j}}\mu^{2}\}$ (4) $\displaystyle=$ $\displaystyle\frac{1}{n}\{(n-1)\sum_{{i=1}}^{{n}}\operatorname{E}[X_{i}^{2}]-2% \cdot\frac{n(n-1)}{2}\mu^{2}\}$ (5) $\displaystyle=$ $\displaystyle\frac{n-1}{n}\sum_{{i=1}}^{{n}}\{\operatorname{E}[X_{i}^{2}]-\mu^% {2}\}$ (6) $\displaystyle=$ $\displaystyle\frac{n-1}{n}\sum_{{i=1}}^{{n}}\{\operatorname{E}[X_{i}^{2}]-% \operatorname{E}[X_{i}]^{2}\}$ (7) $\displaystyle=$ $\displaystyle\frac{n-1}{n}\sum_{{i=1}}^{{n}}\operatorname{Var}[X_{i}]=(n-1)% \sigma^{2}.$ (8) This shows that $s^{2}$ is an unbiased estimator for $\sigma^{2}$.

3. However, $s^{2}$ would be biased if we were to define it by

$s^{2}=\frac{1}{n}\sum_{{i=1}}^{{n}}(X_{i}-\overline{X})^{2},$ since

$\operatorname{E}[s^{2}]=\frac{n-1}{n}\sigma^{2}$ would depend on the sample size $n$ and would not equal to $\sigma^{2}$ at any $n$.

Remark. In practice, unbiased estimators are rare. There is another, larger class of estimators that are biased with smaller samples, but the bias gets smaller and tends to 0 as the sample size gets larger. Such an estimator is called an *asymptotically unbiased estimator*. For example, if we were to define $s^{2}$ as in Example 3 above, $s^{2}$ would be an asymptotically unbiased estimator for $\sigma^{2}$.

## Mathematics Subject Classification

62A01*no label found*

- Forums
- Planetary Bugs
- HS/Secondary
- University/Tertiary
- Graduate/Advanced
- Industry/Practice
- Research Topics
- LaTeX help
- Math Comptetitions
- Math History
- Math Humor
- PlanetMath Comments
- PlanetMath System Updates and News
- PlanetMath help
- PlanetMath.ORG
- Strategic Communications Development
- The Math Pub
- Testing messages (ignore)

- Other useful stuff

## Recent Activity

new question: Lorenz system by David Bankom

Oct 19

new correction: examples and OEIS sequences by fizzie

Oct 13

new correction: Define Galois correspondence by porton

Oct 7

new correction: Closure properties on languages: DCFL not closed under reversal by babou

new correction: DCFLs are not closed under reversal by petey

Oct 2

new correction: Many corrections by Smarandache

Sep 28

new question: how to contest an entry? by zorba

new question: simple question by parag

Sep 26

new question: Latent variable by adam_reith